Keeping the Distribution Point (DP) servers Lights On
Point-of-View: from a ConfigMgr Administrator to his Manager, discussing how he/she spends their day managing a ConfigMgr environment.
I had a real difficult time the other day trying to explain to my manager what I spent the entire day working on. Well… one of our ConfigMgr Distribution Point servers had a heart-attack. This came out of the blue, it wasn’t like it had been on life support or anything like that. In fact, it was only built up almost a year ago and had been operating perfectly fine since then. Typically when something breaks on a server, it’s because something recent has changed, and is usually (falsely) blamed on security or application updates – a reboot usually remedies the problem. Well, I tried that, and it didn’t work. Grrrrr…
Just like a Microsoft Knowledge Base article, let’s describe the problem first…
A new application request came into the team which was packaged up and imported into ConfigMgr. It was distributed to our three staging DP’s in order to complete the required Support and User Acceptance Testing. Fast forward almost four weeks, and we now had sign off from the business to deploy ApplicationA. I assigned and distributed the content to all production DP’s. The next day I was reminded by Outlook that I needed to check the Distribution Status reports and doing so I had discovered that the content had only successfully distributed to 28 out of 36 servers. This isn’t peculiar – in fact this is more common than you may know!
Reviewing the messages within each of the reports I was able to discover the following:
Six of the DP’s were just taking a long time to receive the content. I had to check the log files on the Primary Site server, as well as the log files on the DP servers. I could see that the Content Distribution Manager component had tried many times to copy files to the remote servers and generally timing out “network path not found” etc. This is mainly due to poor WAN links to these sites within North America and South East Asia. We have to leave these DP’s to keep on trying. But if they fail completely, then we’ll need to manually copy the content to the site and pre-stage it (it's already failing so why would this work?), or send it via a hard disk & courier (yeah, right!).
One DP hadn’t even received the policy, why not!? I restarted the ConfigMgr services, and after a few hours of processing existing back-logged files, it eventually started to download the content.
The last DP on the list is the one that had the heart-attack. As I said earlier, this DP is one of our newer DP’s and this is the first problem we’ve encountered with it, so we tried a simple restart of the ConfigMgr services (no luck) and then resorted to rebooting the server. After an hour or two of monitoring the Distribution Manager and Despooler log files, it was evident this server wasn’t going to ever receive the content. It was behaving as if it was blind to the ConfigMgr Over-Lord (aka the Primary Site server).
I’ve seen lots of seemingly random issues with “healthy” DP’s. The resolutions range from restarting the Content Distribution Manager component, having to delete transferred temp files, editing the database, removing files from SCCMContentLib, to simply restarting the ConfigMgr services, or worst case scenario completely restarting the server! And this all takes time, my time! And then there are problems that re-occur, for example, IIS Content Filtering mis-configuration. Set it once and forget about it, right? If IIS is ever re-installed, you may need to re-configure IIS Content Filtering, again.
This was just one package, ONE package! And all up it took one day to monitor and resolve the distribution problem of ONE package. We distribute lots of content every day so the possibility of this happening time and time again is very high, like several times a month! Keeping ConfigMgr Distribution Points healthy requires some effort. I’ve only got 140 DP servers in this ConfigMgr environment, and that’s already a headache. Imagine if I had 400, or even 4,000 DP’s! There must be another way…
This is part one of a story I experienced in a past life as a ConfigMgr Administrator. The second part of this story is what we actually did to reduce this DP server management nightmare.
Software Licence Audits – The Ultimate Defence Kit.
Software vendors are getting more predatory about license audits. If you're passive about it, you're the prey. If you're active, you'll save millions and fly through your audits (they hate that). Download our four weapons and fight back.
