What happens when failing to run wsusutil configureSSL, and how to troubleshoot the symptoms

To quickly answer the question, so that you do not have to read the entire blog:

• The RED sections of both logfile screen-captures at the bottom, demonstrates what you will see in the respective logfiles when you fail to run WSUSUTIL.EXE command-line.
• The GREEN sections of both logfile screen-captures at the bottom, demonstrates what you will see in the respective logfiles after running the WSUSUTIL.EXE command-line.

…however, the rest of the blog entry is pretty interesting, so you might want to read on!!
When configuring a ConfigMgr site to use an Active SUP, it does not arbitrarily select which hostname and ports it's going to use to connect to the site system server running WSUS. Under Site Settings -> Component Configuration -> Software Update Point Component , you must specify the correct port numbers for HTTP & HTTPS connections. You are required to use the port number(s) configured for the website WSUS was installed under. Most admins choose either the Default Web Site (e.g. generally ports TCP/80 & 443) or have the WSUS installer create a custom website called WSUS Administration (e.g. generally TCP 8530 & 8531), when installing WSUS. Nothing new here. Life is good…Right??
After configuring the IIS website with the desired ports, and say you are simply using HTTP to connect to the Active SUP running WSUS (e.g. generally using ports TCP/80 or TCP/8530), you really shouldn't have many problems doing so (if any). To confirm this, check the WCM.log and WSUSCtrL.log, as they will most likely be clear of any connection-related errors. You will most often have this kind of experience when sites are running in Mixed Mode.
However, when the IIS website on the Active SUP server is configured to use an SSL certificate and the SUP component configuration in the ConfigMgr site hierarchy settings is configured to use the same SSL port (e.g. generally TCP/443 or TCP/8531) to connect to the Active SUP server, there is yet an additional step to execute. The additional step is required to be run on the Active SUP: wsusutil configureSSL <FQDN of Active SUP>. What this accomplishes is to write the ServerCertificateName and PortNumber values to the WSUS registry-key (e.g. HKLM\Software\Microsoft\Update Services\Server\Setup).
 

 
The ConfigMgr Documentation Library covers this under How to Configure the WSUS Web Site to Use SSL and Carol Bailey did a write-up on it as well.
In Native Mode sites where Internet-Based Client Management (IBCM) will be used, many have had a difficult time determining which FQDN to use on the command-line (e.g. wsusutil configureSSL <inTRAnet OR InTERnet FQDN of Active SUP??>) . Carol Bailey has provided another great blog to help in this decision-making process.
I just wanted to give a little background before getting to why I'm writing this blog in the first place…I hope you didn't mind that long intro'?
Ok, so what I want to cover is not so much about what the ConfigMgr Documentation Library or Carol have already covered, but how to use the WSUSCtrl.log and IIS logfiles together to help you determine whether you may have "forgotten" to execute the command: wsusutil configureSSL <inTRAnet OR InTERnet FQDN of Active SUP??>. Said differently, "I would like to remind you to run the command".
Before I get started, let me provide you some background on how I implemented IBCM within a Native Mode environment:
I am using the exact configurations you see documented in Scenario 3 with SQL Server Replica (in DMZ), which is arguably the most secure method to implementing Internet-Based Client Management (IBCM) because no connections are required to be initiated by site system server(s) in the DMZ to servers on the inTRAnet.
 

Addition configuration info':

• Certificate Revocation List (CRL) checking is enabled on the ConfigMgr site
• All site server (in inTRAnet) to site system server (in DMZ) communication is initiated by the site server
• Site database (in inTRAnet) is using transactional push replication to the site database replica (in DMZ)
• Software Update Point (in inTRAnet) is configured to only accept connections from InTRAnet-clients
• Software Update Point (in DMZ) is configured to only accept connections from InTERnet-clients
• CRL distribution points are located in DMZ (e.g. used by Internet clients)
• The WSUS metadata on the Active remote Internet SUP (in DMZ) is synchronized manually using the "WSUS import" command.

So while implementing IBCM, I ran into a problem configuring the Active remote Internet SUP in the DMZ and here's how I went about troubleshooting it.
If you look at the RED section of the WSUSCtrl.log screen-capture below, you can see the WSUS Admin Console is confirmed to be the correct installed version on the SUP, but then fails to connect to the Active SUP server with an error of The request failed with HTTP status 403 and Failed to set WSUS Local Configuration. WSUS is installed under an IIS website which is actually configured with a Web Server Certificate (e.g. Server Authentication) and is also configured to listen for incoming SSL connections on TCP/8531. Along with that, the SSL settings of the WSUS vdirs (e.g. ApiRemoting30, ClientWebService, DssAuthWebService, ServerSyncWebService, SimpleAuthWebService) are configured to "Require SSL". OK…ya' got that?
So I'm reading the WSUSCtrl.log, feeling like I've configured everything properly…So why am I having this problem? Surely this must be a bug, right?? (famous last words!!). C'mon…give me some love…you've done it too!!
Why is it trying to connect using "HTTP" over TCP/8530? This is definitely a problem.
The WSUSCtrl.log gave me some things to go on, but not enough to figure out what the problem is…or maybe I'm just slow?? Hhhmm??
So I decided to check out the IIS Logfile…
In the RED section of the IIS Logfile screen-capture below, you'll notice that the site system server is trying to POST to the ApiRemoting30 vdir on the WSUS server using HTTP (e.g. TCP/8530) and is failing with IIS error-code 403-Access Forbidden (see it in the third to last column of each log-entry?). What else is strange, is that I can still open the WSUS Admin Console and connect to the WSUS server using "SSL" over TCP/8531. Go figure!?!?
So I started reviewing my steps, realizing I had not run the WSUSUTIL.EXE command on the WSUS server (e.g. wsusutil configureSSL <InTERnet FQDN of Active remote Internet SUP??>). Hhmm??
Note: Since the Active remote Internet SUP in the DMZ will be used to only accept connections from Internet clients, then in the wsusutil command above, I used the InTERnet FQDN and not the InTRAnet FQDN.
So I did that and the problem was fixed!! That's all it was…
Looking at the GREEN section in each of the logfile screen-captures below, you can see that once I ran the WSUSUTIL.EXE command, the site system server was able to successfully connect to the Active remote Internet SUP. Reflecting on my statement earlier in the blog about what happens when the command is executed: It's not until ServerCertificateName and PortNumber values are written to the Active SUP's registry-key (e.g. HKLM\Software\Microsoft\Update Services\Server\Setup) , will the site system server running the SUP role be able to connect to the Active SUP, which is server running WSUS. Keep in mind, this is the case even if the IIS website WSUS is installed under is configured with the correct ports and the Web Server Certificate used by the website is configured properly.
Software Update Management (SUM) has proven to be kind of difficult for people to troubleshoot, because there are a number of "moving parts" involved. When you introduce Native Mode and Internet-Based Client Management, it can get even more complicated to troubleshoot. Native Mode and Internet-Based Client Management are not for the faint of heart. You really have to plan it out wisely and test, test, test…and then test a little more!!
I have more topics like this that I will be blogging about.
I hope you found this one helpful!!
WSUSCtrl.log (on Active remote Internet SUP in DMZ)
 

 
IIS Logfile (on Active remote Internet SUP in DMZ)