ADFS Cache Issues

From time-to-time, employees may leave a company but end up coming back. With all that comes user management, accounts, access, permissions, etc…. While that is all easy enough from an Active Directory Standpoint, this can cause some complications with access into CRM/Dynamics 365 when using ADFS for claims-based authentication. Particularly if the user’s original AD account was deleted and then recreated with the same username.

After the account is recreated, you may learn that there isn’t any issue re-enabling the user in CRM or Dynamics 365 but that the user cannot login via ADFS. There is also a similar scenario when companies migrate domains and keep the same user names, only changing the domain. Some behavior you may notice are 404 errors, wrong username showing in event viewer logs, and general ADFS errors. While I haven’t been able to pinpoint why this happens sometimes and not others, what I do know is the problem lies within ADFS.

Basically what seems to be going on is the ADFS server has a cache of information already built up about the users, including GUIDs, SIDs, UPNs, etc… So when an authentication request comes through it checks for the username in its cache along with any associated records and matches what it can rather than pulling from Active Directory. When it does this it can pull the information from the old, previously-deleted account and then error out.

Anyways, without further ado – here are two methods that I’ve used successfully to get through these situations. Sometimes one works and sometimes the other works – just some more mystery around this situation…

Method 1: Disable the LSA Cache
This method involves disabling the LSA cache by setting it’s maximum size to 0 – essentially making it nonexistent.

1. Log on to the ADFS Server.
2. Open Registry Editor.
3. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa.
4. Create a new DWORD value in Lsa named LsaLookupCacheMaxSize and set the value to 0.
5. Close Registry Editor.
6. Have problem user try to log in again.

Whether or not this works, go back in after testing and delete the value you just created. ADFS may suffer from performance issues if it is left in place for a long period of time.

Method 2: Use PSTools to update SID
Going this route requires the download of PsTools – a command-line tool that helps with server administration. In our scenario, it is the PsGetSid command that will be useful as it will force update the SID of the account held in cache from AD.

1. Log on to the ADFS server.
2. Download PSTools from https://docs.microsoft.com/en-us/sysinternals/downloads/psgetsid.
3. Extract out the contents of the zip file into an accessible directory.
4. Run PSGetSid from command prompt using the following syntax: C:\{Directory}\PsGetsid.exe domain\crmuser
5. Have problem user try to log in again.

If neither of those methods works, there may be something else wrong. I would start by verifying that all other users can access CRM/D365 without issue and go from there. My apologies for not having exact error messages or screenshots on this post – it was just an issue I’ve run into in the past a few times and thought it may help someone out regardless as I’ve realized there isn’t much information out there about it.

Connecting Dynamics 365 On-Premise to Power BI

If you're looking to setup the integration between Dynamics 365 On-Premise (using IFD) and Power BI, you may stumble upon this lovely TechNet: https://technet.microsoft.com/en-us/library/dn708055.aspx#PBI_op

All seems great, right? Well, that would be the case if all the steps were accurate...

To start, the PowerShell commands in step 1 to enable OAuth on the Dynamics 365 Server just do not work. The first three will work fine but upon execution of the 4th to actually set the change, it will throw an error. Instead, use this syntax to accomplish the same thing:

Add-PSSnapin Microsoft.Crm.PowerShell
$ClaimsSettings = Get-CrmSetting -SettingType OAuthClaimsSettings
$ClaimsSettings.Enabled = $true
Set-CrmSetting -Setting $ClaimsSettings

Afterwards, run Get-CrmSetting -SettingType OAuthClaimsSettings and you should see enabled set to "True" as shown below.

Moving onward to step 3 in the process where we register the Power BI Desktop OAuth 2.0 client with ADFS. You will notice in the instructions that it states "open a Windows PowerShell window and run the following PowerShell command on the PC where you are running Power BI Desktop". This is simply wrong. The "Add-AdfsClient" command is for... ADFS. Shocking, I know. So ignore the statement about the PC running Power BI and run the command provided on your ADFS server instead. Luckily, this one is correct and works. If you run Get-AdfsClient afterwards, you will see the addition of Power BI.

One last helpful tip - When copying commands from websites (even here), I always recommend pasting into Notepad before bringing them where they need to be executed (e.g. SSMS, PowerShell, etc...). This will get rid of any problematic formatting.

That's really it from a server perspective. Best of luck!

Issue Installing CRM 2016 SRS Data Connector with SQL 2016

In my last post, I mentioned that I had received some Windows Server 2016 boxes to test setup of CRM 2016/Dynamics 365. In addition to them being the 2016 OS, they also came with SQL Server 2016. The installation of CRM went off without a hitch (after working through the Search service problem) and the databases created in SQL as expected.

However, when it came to installation of the SRS Data Connector, things got a little hairy. The installation began but quickly threw this error message:

Error| System.Exception: Action Microsoft.Crm.Setup.SrsDataConnector.AddBindingRedirectForRdlHelper failed. ---> System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Program Files\Microsoft SQL Server\MSRS13.MSSQLSERVER\Reporting Services\ReportManager\web.config'.

To make a long story short, SQL 2016 no longer creates/uses the path specified in the error message and the RTM Data Connector installer does not check the version of SSRS being used so it will just keep trying to use that directory.

Now, I had some previous experience installing the Data Connector with SQL 2016 involved. From that, I recalled that the installer, when asked to check for updates on the first screen, actually went out and got some new install bits. This time around I was not so lucky – it just told me there were no updates available (I even verified that the Windows Firewall was off and that the box was allowing Windows Updates)… it had internet access but if you’re reading this and your SQL server does not have access to the internet, you should keep reading. Regardless, I needed a solution and the internet was not full of suggestions with this being pretty new technology.

I decided to check the Microsoft Update Catalog to see if I could download and manually patch the update in. To my delight, I came across a listing for “Setup Update for Microsoft Dynamics CRM 2016 Reporting Extensions” and this ultimately helped solve my issue. Here is a step-by-step process of what to do if you are facing the same situation:

1. If your initial install failed, go into the server’s control panel and uninstall the data connector.

2. Go to http://www.catalog.update.microsoft.com/

3. Search for “3129794” and you should get back one listing. Click the download button.

4. A new screen will pop up with a ton of .cab files. Find the one named “srs_kb3129794_amd64_1033_b585b78e247dc7611e1e406a6c132cdee112690c.cab” (just look for 1033) and download that. I will admit that I am not sure of the purpose of all these files but the 1033 one did the trick for me. Later on, you will notice that another file we work with is also “1033” so that probably has something to do with it.

5. Once you have the .cab file downloaded to the server, open it up to reveal the patch file. Right-click on it and choose “Extract”.

6. After the extraction, you will have the .msp file available for use. Go to the directory of the SRS Data Connector’s installation files and locate a folder named “Update”. In here, you will see another .msp file named “Srs_KB3121695_Amd64_1033”.

7. Cut and paste that file out of the “Update” folder – just put it anywhere else on the server.

8. Copy and paste the .msp you just downloaded into the now blank “Update” folder - essentially replacing it.

You should go from this:

To this:

9. Re-run the installer as normal and it should complete without issue.


Just to note a few things:

• I initially played a bunch of trial and error with getting the update patched in before I discovered the ability to replace the .msp in the Update folder. I first went down the path of editing the config.xml to include the patch and then called the installer from a command line specifying it to use the config file. This is how I have always patched in updates for the server installer but it does not seem to work the same way for the Data Connector. I also tried loading the patch into the temp files that the installer looks to… also didn’t work.

• If your base installer’s update file has a different number than “1033” at the tail end of it, you may need to match it up with the correct .cab file from the Update Catalog. I did not encounter this scenario so I cannot test or prove out this theory but I think logic dictates… 

• There is another blog out there about applying SP1/2 to the failed installation and then manually editing file structure. I will not discredit that being a possible workaround but personally, I do not like the idea of relying on an update to fix a broken install.

Good luck!

Issue Installing CRM 2016 (Dynamics 365) on Windows Server 2016

I recently got my hands on some Windows Server 2016 boxes and wanted to test out the CRM 2016/Dynamics 365 installation process on them. Preparation began as usual and I kicked off the wizard. Some time after that I was presented with the following error message:

Action.Microsoft.Crm.Setup.Common.InstallWindowsSearchAction failed. The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (Exception from HRESULT: 0x80070422)

What I quickly learned was that the Windows Search service, while coming pre-installed on Windows Server 2016, is set to disabled by default.

I enabled the service by setting it to automatic startup, started it, and then hit retry on the error diagloue from the CRM installation wizard. The installation continued and finished with no further issues.

I love the easy fixes!

URL Redirect Based on IP Address

As you are most likely aware, CRM/D365 has two URLs to access it when setup for IFD. The first URL, referred to as the “internal URL”, will allow authentication to be passed through and not display the ADFS login page for credentials. As the name suggests, this URL should only be available from within the network and not exposed out over the internet. The second URL, (not surprisingly) referred to as the “external URL”, will not pass through authentication and will display the ADFS login form for users to enter credentials. This URL is the one exposed out over the internet but is also available from within the network.

Customers often like the convenience of not having to enter credentials using the internal URL but do not like having two URLs to use. As a result, we are tasked with providing a solution to this dilemma. Back in the days of ADFS 2.0, we could easily handle this with a much simpler tweak in IIS of the ADFS server. Unfortunately, with the newer versions of ADFS no longer using IIS as its backbone this is no longer an option and a URL Rewrite rule now must be implemented on the CRM/D365 server. Here’s how:

1. Open IIS on the CRM/D365 Front End (web) Server and go to the URL Rewrite module of the CRM Website.

2. Click the link to “Add Rule(s)” and just select “Blank Rule” under Inbound rules.

3. Give the rule an applicable name and in the Match URL section, leave both the ”Requested URL” and “Using” fields as their defaults (shown in screen capture). In the “Pattern” field type in (.*).

4. This is where it gets fun. In the Conditions section, you will add three rules in the order shown below.

     a. For the first rule, change “crmorgname” to the actual name of your organization.
     b. For the second rule, change the pattern to match that of your internal IP range.
     c. Just enter the third rule as shown – no changes needed – ([^\.]*)\.(.*)

5. In the Action section, change the “Action type” to “Redirect” and for the Redirect URL enter your internal CRM/D365 URL followed by the syntax shown in the screenshot. Leave “Append query string” checked and set the Redirect type to “Permanent (301)”.

6. Apply the URL Rewrite rule and test it by going to the external URL from a machine that has an IP address matching that of the ranges specified on your network. If everything was setup correctly, the external URL will redirect to the internal URL and pass-through authentication will occur.

Yet Another ADFS Looping Issue

I recently applied the Dynamics 365 Update to a CRM 2016 Service Pack 1 environment that was setup for IFD and ran into some unexpected behavior. This environment had about 15 organization setup in it. When trying to log on using the external URLs (e.g. orgname.domain.com) access worked without issue but using the internal URL for any organization (e.g. crm.domain.com/orgname) CRM and ADFS would run in a continuous loop. Sometimes it would error out, sometimes it would just keep looping forever.

Started off by checking event viewers on both the CRM and ADFS server. I noticed that on the times it did error out after a few loops instead of looping nonstop, it resulted in an event log similar to one I saw before caused by a bug with the 0.1 Update for CRM 2016 (http://blog.gagepennisi.com/2016/04/crm-2016-update-01-bug-with-adfs-for.html):

MSIS7042: The same client browser session has made '6' requests in the last '17' seconds. Contact your administrator for details.

So immediately I thought “Great, here we go again.” There was also nothing in the CRM event viewer.

Hours of trying different different troubleshooting tactics for similar situations (including this one with the same looping behavior but with the external URL - http://blog.gagepennisi.com/2016/01/adfs-logon-page-loop-issue.html) yielded nothing. Rebuilt the Relying Party Trusts, reconfigured IFD, created a self-signed certificate to verify it wasn’t a certificate issue, etc…

Instead of keeping on with the guess and check, I realized I needed to get more information around the problem so I started with a Fiddler trace – great web debugging tool, if you haven’t heard of it (http://www.telerik.com/fiddler). Unfortunately, in this case all it did was confirm the behavior I was seeing in the browser which was a constant redirect between D365 and ADFS.

After that, I decided to run a platform trace on the CRM server to see if CRM would give me any insight but nothing really stood out to me. Almost at my wit’s end, I decided to rope in my colleague Dan Francis to get another set of eyes on it. After some review of the platform trace he noticed one little line that seemed a bit odd:

>Multi-org sharable cache loading system and non-system metadata with build number 7.0.0.3543 and language 1033

CRM 2016 is version 8.0 and D365 is 8.2 so the fact that we were seeing any refernce to 7.0.X.XXXX (CRM 2015) was not inline with what was to be expected. We checked the Deployment Manger and realized there was an old, disabled organization from CRM 2015 that was not upgraded (because it was disabled at the time of the original CRM 2016 upgrade). After some deliberation, we decided to remove that organization from the deployment and boom! The internal URL began working as expected!

Basically what we were able to surmise from this is that when the internal URL is being used, all organizations – whether enabled or disabled – are checked for versioning and the lowest build number is used to load this “multi-org sharable cache” and apparently D365 doesn’t play nicely with older versions hanging around. Moral of the story – update/upgrade all of your orgs or just remove them.

Big thanks to Dan the Man!

Calling LoadLibraryEx on ISAPI filter DefaultAddonFilter.dll failed

I recently had an interesting issue with a client’s CRM 2011 to 2016 go-live. Following all of my upgrade work and initial testing, I typically like to reboot the CRM and SQL servers and then test again. In this particular instance, when the servers came back online I tested CRM again only to receive a lovely error message from Internet Explorer:

HTTP Error 500.0 – Internal Server Error

Calling LoadLibraryEx on ISAPI filter C:\Program Files\Microsoft Dynamics CRM\Server\bin\DefaultAddonFilter.dll failed.

After a fair amount of digging I noticed that the Visual C++ 2013 Redistributable (64-bit) was not present in the add/remove programs list on the CRM server. It seemingly uninstalled itself during the reboot because otherwise I would have had this issue pre-reboot. So, I went out and grabbed the installer package for Visual C++ 2013 Redistributable (64-bit) from MS (http://go.microsoft.com/fwlink/p/?LinkId=402059) and reinstalled it. After that and an IISReset, CRM began functioning as normal and everything retested everything successfully.

Unfortunately, I am at a loss as to why this happened – perhaps something to do with server patching. Also of note is that the redistributable did not remove itself from the SQL server.

CRM 2016 Update 0.1 Bug with ADFS for Server 2012 R2

Following a successful upgrade to CRM 2016 and installation of the 0.1 Update, users could no longer authenticate against ADFS using the “internal” URL.

On the ADFS server I was seeing Event ID 364 in the Event Viewer:

Exception details:
Microsoft.IdentityServer.Web.InvalidRequestException: MSIS7042: The same client browser session has made '6' requests in the last '0' seconds. Contact your administrator for details.

In efforts to get this working, I tried completely re-configuring CRM for claims and IFD and even recreated the Relying Party Trusts. All with no luck but apparently, I was not alone on this issue and Microsoft has acknowledged it as a bug with Update 0.1. The following forum thread has more info:

https://social.msdn.microsoft.com/Forums/windowsserver/en-US/9d4040eb-81fa-4144-ae4b-85ca4610aa1d/crm-2016-problem-with-claimsbased-authentication?forum=crm

These are the only real details on this issue currently available:

There were major code changes in Ara UR1 for authentication. The affected code is in Microsoft.Crm.Core.Security.Identity.IdentityExtensions.GetUserPrincipalName(). We are unable to cast from type ClaimsIdentity to a new type CrmIdentity. Therefore, the variable is null, and we cannot retrieve the information.

Please note that only the ‘internal’ URL is affected when IFD is setup. The ‘external’ URL (e.g. https://orgname.domain.com) that uses forms authentication works fine still. I will update this blog post as more information becomes available.

UPDATE: Service Pack 1 for CRM 2016 resolves this issue.

Next Button Grayed Out in Internet-Facing Deployment Configuration Wizard

While recently assisting a colleague with setting up IFD, we stumbled upon an interesting bug with CRM 2016 in which we could not advance in the IFD wizard because the “Next” button was grayed out and disabled. No matter what we typed in the field, it would not enable the button. Tried an IISReset, restarting CRM services, and even a reboot but they all failed to help.

Realizing this had to be an issue within MSCRM_CONFIG database, we took a look at the tables that hold the IFD properties that we input in these wizards. What we found was that somehow the NVarCharColumn value of ExternalRelyingPartyPassiveIdentifier in the FederationProviderProperties table was set to “uri:ifdMicrosoftCRM”. Typically we would see the Auth URL in this field so we updated it manually to “https://auth.domain.com” in SQL. Make sure to include the “https://” - even though we do not enter this in the field during the configuration wizard, it gets appended automatically and set in the database. Issue an IISRESET on the CRM server and then re-launch the CRM Deployment Manager.

Please note that editing the database manually via SQL is not supported by Microsoft and you are doing so at your own risk. Always take backups and use caution.

CRM 2016 New Form Rendering Engine

With the release of CRM 2016 (2015 Update 1 for CRM Online), comes a new form rendering engine built to provide better performance of form loads. The two main changes are focused around loading process of the form and the handling of the cache.

However, while the new rendering engine was built to help with performance you may actually notice the opposite taking place. In heavily customized environment, you might experience long stalls during form loads with messages reading “requesting data from CRM” and/or “loading business logic”.

If you are currently being plagued by these messages, it may be a good idea to turn the new rendering engine off. To do this, simply go to: Administration -> System Setting, scroll all the way down and you will see the “Use legacy form rendering” option. Turning it to “yes” will disable the new engine.

While this will help in the short term, it is advised to figure out what on your form is causing conflict with the new engine so that it can be fixed and the new engine can be turned back on for even better performance.