Yet Another ADFS Looping Issue

I recently applied the Dynamics 365 Update to a CRM 2016 Service Pack 1 environment that was setup for IFD and ran into some unexpected behavior. This environment had about 15 organization setup in it. When trying to log on using the external URLs (e.g. orgname.domain.com) access worked without issue but using the internal URL for any organization (e.g. crm.domain.com/orgname) CRM and ADFS would run in a continuous loop. Sometimes it would error out, sometimes it would just keep looping forever.

Started off by checking event viewers on both the CRM and ADFS server. I noticed that on the times it did error out after a few loops instead of looping nonstop, it resulted in an event log similar to one I saw before caused by a bug with the 0.1 Update for CRM 2016 (http://blog.gagepennisi.com/2016/04/crm-2016-update-01-bug-with-adfs-for.html):

MSIS7042: The same client browser session has made '6' requests in the last '17' seconds. Contact your administrator for details.

So immediately I thought “Great, here we go again.” There was also nothing in the CRM event viewer.

Hours of trying different different troubleshooting tactics for similar situations (including this one with the same looping behavior but with the external URL - http://blog.gagepennisi.com/2016/01/adfs-logon-page-loop-issue.html) yielded nothing. Rebuilt the Relying Party Trusts, reconfigured IFD, created a self-signed certificate to verify it wasn’t a certificate issue, etc…

Instead of keeping on with the guess and check, I realized I needed to get more information around the problem so I started with a Fiddler trace – great web debugging tool, if you haven’t heard of it (http://www.telerik.com/fiddler). Unfortunately, in this case all it did was confirm the behavior I was seeing in the browser which was a constant redirect between D365 and ADFS.

After that, I decided to run a platform trace on the CRM server to see if CRM would give me any insight but nothing really stood out to me. Almost at my wit’s end, I decided to rope in my colleague Dan Francis to get another set of eyes on it. After some review of the platform trace he noticed one little line that seemed a bit odd:

>Multi-org sharable cache loading system and non-system metadata with build number 7.0.0.3543 and language 1033

CRM 2016 is version 8.0 and D365 is 8.2 so the fact that we were seeing any refernce to 7.0.X.XXXX (CRM 2015) was not inline with what was to be expected. We checked the Deployment Manger and realized there was an old, disabled organization from CRM 2015 that was not upgraded (because it was disabled at the time of the original CRM 2016 upgrade). After some deliberation, we decided to remove that organization from the deployment and boom! The internal URL began working as expected!

Basically what we were able to surmise from this is that when the internal URL is being used, all organizations – whether enabled or disabled – are checked for versioning and the lowest build number is used to load this “multi-org sharable cache” and apparently D365 doesn’t play nicely with older versions hanging around. Moral of the story – update/upgrade all of your orgs or just remove them.

Big thanks to Dan the Man!

Calling LoadLibraryEx on ISAPI filter DefaultAddonFilter.dll failed

I recently had an interesting issue with a client’s CRM 2011 to 2016 go-live. Following all of my upgrade work and initial testing, I typically like to reboot the CRM and SQL servers and then test again. In this particular instance, when the servers came back online I tested CRM again only to receive a lovely error message from Internet Explorer:

HTTP Error 500.0 – Internal Server Error

Calling LoadLibraryEx on ISAPI filter C:\Program Files\Microsoft Dynamics CRM\Server\bin\DefaultAddonFilter.dll failed.

After a fair amount of digging I noticed that the Visual C++ 2013 Redistributable (64-bit) was not present in the add/remove programs list on the CRM server. It seemingly uninstalled itself during the reboot because otherwise I would have had this issue pre-reboot. So, I went out and grabbed the installer package for Visual C++ 2013 Redistributable (64-bit) from MS (http://go.microsoft.com/fwlink/p/?LinkId=402059) and reinstalled it. After that and an IISReset, CRM began functioning as normal and everything retested everything successfully.

Unfortunately, I am at a loss as to why this happened – perhaps something to do with server patching. Also of note is that the redistributable did not remove itself from the SQL server.

Host Header Configured on Selected Site Does Not Resolve to an IP Address that is Assigned to the Local System.

I was recently performing an in-place upgrade from CRM 2015 to CRM 2016 when during the system checks I received the following warning:

Host Header "notCRM.domain.com" configured on selected site does not resolve to an IP address that is assigned to the local system. This conflict must be reconciled for CRM Server to function properly. 

The fact that it was giving me a host header that did not pertain to the CRM website was odd so I took a look at IIS. What I discovered was that there was another website setup in IIS on the CRM Server. And that site had the site ID of 1.

As I was the original installer of CRM for this deployment, I knew that this site wasn’t here when CRM was installed. CRM originally had the site ID of 1, not this other site. Somebody had been in here messing around and changed the site IDs manually. After asking enough questions, I learned who did this and why. To make a long story short, their site needed to have the ID of 1. CRM, on the other hand, does not care so I rolled with the punches and decided I would work around this rather than cause a fuss.

From what I could deduce about the error message, the CRM installer was being told to look at the IIS website that had an ID of 1 to perform the upgrade on. I decided to switch the CRM site back to the ID of 1. To do this, you must first change the other site’s ID to a number not in use (e.g. 3, in my case). To do this, go into the site’s advanced settings.

After the other site was changed, I reverted the CRM site back to 1 and reset IIS.

The upgrade wizard will need to be restarted for it to pick up the change but once you make it back to the system checks, the warning should be gone.

The upgrade succeeded without issue but we weren’t done just yet. Since the other site needed to have the ID of 1, I needed to revert changes to the site IDs which in itself isn’t complicated. What was not taken into account was a registry key in HKLM\Software\Microsoft\MSCRM called “website”.

This registry key indicates which website directory in IIS CRM is using. Since CRM is using the ID of 2, this key needed to be switched to “LM/W3SVC/2”. In case you have been wondering what the ID controls, it is just an identifier used in the directory name for trace files and log files. It’s nothing critical that will not let CRM operate as normal.

By now you are probably thinking “Why didn’t you change this key rather than switching the site IDs to resolve the warning message during the upgrade?!?!” Well... it didn’t occur to me until afterward I completed the upgrade. Hindsight is 20/20!