ADFS Cache Issues

From time-to-time, employees may leave a company but end up coming back. With all that comes user management, accounts, access, permissions, etc…. While that is all easy enough from an Active Directory Standpoint, this can cause some complications with access into CRM/Dynamics 365 when using ADFS for claims-based authentication. Particularly if the user’s original AD account was deleted and then recreated with the same username.

After the account is recreated, you may learn that there isn’t any issue re-enabling the user in CRM or Dynamics 365 but that the user cannot login via ADFS. There is also a similar scenario when companies migrate domains and keep the same user names, only changing the domain. Some behavior you may notice are 404 errors, wrong username showing in event viewer logs, and general ADFS errors. While I haven’t been able to pinpoint why this happens sometimes and not others, what I do know is the problem lies within ADFS.

Basically what seems to be going on is the ADFS server has a cache of information already built up about the users, including GUIDs, SIDs, UPNs, etc… So when an authentication request comes through it checks for the username in its cache along with any associated records and matches what it can rather than pulling from Active Directory. When it does this it can pull the information from the old, previously-deleted account and then error out.

Anyways, without further ado – here are two methods that I’ve used successfully to get through these situations. Sometimes one works and sometimes the other works – just some more mystery around this situation…

Method 1: Disable the LSA Cache
This method involves disabling the LSA cache by setting it’s maximum size to 0 – essentially making it nonexistent.

1. Log on to the ADFS Server.
2. Open Registry Editor.
3. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa.
4. Create a new DWORD value in Lsa named LsaLookupCacheMaxSize and set the value to 0.
5. Close Registry Editor.
6. Have problem user try to log in again.

Whether or not this works, go back in after testing and delete the value you just created. ADFS may suffer from performance issues if it is left in place for a long period of time.

Method 2: Use PSTools to update SID
Going this route requires the download of PsTools – a command-line tool that helps with server administration. In our scenario, it is the PsGetSid command that will be useful as it will force update the SID of the account held in cache from AD.

1. Log on to the ADFS server.
2. Download PSTools from https://docs.microsoft.com/en-us/sysinternals/downloads/psgetsid.
3. Extract out the contents of the zip file into an accessible directory.
4. Run PSGetSid from command prompt using the following syntax: C:\{Directory}\PsGetsid.exe domain\crmuser
5. Have problem user try to log in again.

If neither of those methods works, there may be something else wrong. I would start by verifying that all other users can access CRM/D365 without issue and go from there. My apologies for not having exact error messages or screenshots on this post – it was just an issue I’ve run into in the past a few times and thought it may help someone out regardless as I’ve realized there isn’t much information out there about it.

Connecting Dynamics 365 On-Premise to Power BI

If you're looking to setup the integration between Dynamics 365 On-Premise (using IFD) and Power BI, you may stumble upon this lovely TechNet: https://technet.microsoft.com/en-us/library/dn708055.aspx#PBI_op

All seems great, right? Well, that would be the case if all the steps were accurate...

To start, the PowerShell commands in step 1 to enable OAuth on the Dynamics 365 Server just do not work. The first three will work fine but upon execution of the 4th to actually set the change, it will throw an error. Instead, use this syntax to accomplish the same thing:

Add-PSSnapin Microsoft.Crm.PowerShell
$ClaimsSettings = Get-CrmSetting -SettingType OAuthClaimsSettings
$ClaimsSettings.Enabled = $true
Set-CrmSetting -Setting $ClaimsSettings

Afterwards, run Get-CrmSetting -SettingType OAuthClaimsSettings and you should see enabled set to "True" as shown below.

Moving onward to step 3 in the process where we register the Power BI Desktop OAuth 2.0 client with ADFS. You will notice in the instructions that it states "open a Windows PowerShell window and run the following PowerShell command on the PC where you are running Power BI Desktop". This is simply wrong. The "Add-AdfsClient" command is for... ADFS. Shocking, I know. So ignore the statement about the PC running Power BI and run the command provided on your ADFS server instead. Luckily, this one is correct and works. If you run Get-AdfsClient afterwards, you will see the addition of Power BI.

One last helpful tip - When copying commands from websites (even here), I always recommend pasting into Notepad before bringing them where they need to be executed (e.g. SSMS, PowerShell, etc...). This will get rid of any problematic formatting.

That's really it from a server perspective. Best of luck!

Issue Installing CRM 2016 SRS Data Connector with SQL 2016

In my last post, I mentioned that I had received some Windows Server 2016 boxes to test setup of CRM 2016/Dynamics 365. In addition to them being the 2016 OS, they also came with SQL Server 2016. The installation of CRM went off without a hitch (after working through the Search service problem) and the databases created in SQL as expected.

However, when it came to installation of the SRS Data Connector, things got a little hairy. The installation began but quickly threw this error message:

Error| System.Exception: Action Microsoft.Crm.Setup.SrsDataConnector.AddBindingRedirectForRdlHelper failed. ---> System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Program Files\Microsoft SQL Server\MSRS13.MSSQLSERVER\Reporting Services\ReportManager\web.config'.

To make a long story short, SQL 2016 no longer creates/uses the path specified in the error message and the RTM Data Connector installer does not check the version of SSRS being used so it will just keep trying to use that directory.

Now, I had some previous experience installing the Data Connector with SQL 2016 involved. From that, I recalled that the installer, when asked to check for updates on the first screen, actually went out and got some new install bits. This time around I was not so lucky – it just told me there were no updates available (I even verified that the Windows Firewall was off and that the box was allowing Windows Updates)… it had internet access but if you’re reading this and your SQL server does not have access to the internet, you should keep reading. Regardless, I needed a solution and the internet was not full of suggestions with this being pretty new technology.

I decided to check the Microsoft Update Catalog to see if I could download and manually patch the update in. To my delight, I came across a listing for “Setup Update for Microsoft Dynamics CRM 2016 Reporting Extensions” and this ultimately helped solve my issue. Here is a step-by-step process of what to do if you are facing the same situation:

1. If your initial install failed, go into the server’s control panel and uninstall the data connector.

2. Go to http://www.catalog.update.microsoft.com/

3. Search for “3129794” and you should get back one listing. Click the download button.

4. A new screen will pop up with a ton of .cab files. Find the one named “srs_kb3129794_amd64_1033_b585b78e247dc7611e1e406a6c132cdee112690c.cab” (just look for 1033) and download that. I will admit that I am not sure of the purpose of all these files but the 1033 one did the trick for me. Later on, you will notice that another file we work with is also “1033” so that probably has something to do with it.

5. Once you have the .cab file downloaded to the server, open it up to reveal the patch file. Right-click on it and choose “Extract”.

6. After the extraction, you will have the .msp file available for use. Go to the directory of the SRS Data Connector’s installation files and locate a folder named “Update”. In here, you will see another .msp file named “Srs_KB3121695_Amd64_1033”.

7. Cut and paste that file out of the “Update” folder – just put it anywhere else on the server.

8. Copy and paste the .msp you just downloaded into the now blank “Update” folder - essentially replacing it.

You should go from this:

To this:

9. Re-run the installer as normal and it should complete without issue.


Just to note a few things:

• I initially played a bunch of trial and error with getting the update patched in before I discovered the ability to replace the .msp in the Update folder. I first went down the path of editing the config.xml to include the patch and then called the installer from a command line specifying it to use the config file. This is how I have always patched in updates for the server installer but it does not seem to work the same way for the Data Connector. I also tried loading the patch into the temp files that the installer looks to… also didn’t work.

• If your base installer’s update file has a different number than “1033” at the tail end of it, you may need to match it up with the correct .cab file from the Update Catalog. I did not encounter this scenario so I cannot test or prove out this theory but I think logic dictates… 

• There is another blog out there about applying SP1/2 to the failed installation and then manually editing file structure. I will not discredit that being a possible workaround but personally, I do not like the idea of relying on an update to fix a broken install.

Good luck!

Issue Installing CRM 2016 (Dynamics 365) on Windows Server 2016

I recently got my hands on some Windows Server 2016 boxes and wanted to test out the CRM 2016/Dynamics 365 installation process on them. Preparation began as usual and I kicked off the wizard. Some time after that I was presented with the following error message:

Action.Microsoft.Crm.Setup.Common.InstallWindowsSearchAction failed. The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. (Exception from HRESULT: 0x80070422)

What I quickly learned was that the Windows Search service, while coming pre-installed on Windows Server 2016, is set to disabled by default.

I enabled the service by setting it to automatic startup, started it, and then hit retry on the error diagloue from the CRM installation wizard. The installation continued and finished with no further issues.

I love the easy fixes!

High Availability and Disaster Recovery Options for Dynamics 365 (CRM)

Planning the infrastructure for your Dynamics 365 environment is a critical step in the deployment process, especially when the software is mission critical to business needs and day-to-day operations. In scenarios like these, it is imperative to have high availability and even disaster recovery plans in place. So the question is: What options do you have when it comes to Dynamics 365 on premise? In this article, we will break it down from both an application (Dynamics 365) and database (SQL Server) perspective.

High Availability – Dynamics 365
For most companies, having high availability for their Dynamics 365 deployment is a must and the setup for this is similar to that of other web-based applications. Keep in mind that through this section we are strictly referring to Dynamcis 365 servers with the “Front End” or “Full” role – in other words, the servers that host the website in IIS. Having multiple “Back End”-only servers (those with the Async and Sandbox services) is also a good idea but those will natively load balance themselves if they are in the same deployment.

• Network Load Balancing – Probably the most obvious method for high availability is running multiple Dynamics 365 servers in a load-balancing configuration. This will not only help with carrying the day-to-day load of traffic but also serve as the foundation for a good high availability option. If a server were to fail, just pop it out of the NLB pool. Load balancing comes in a few different flavors because of all the applications out there but mainly it comes down to two options – hardware and software.

• Hardware Load Balancing involves the use of a third-party appliance such as F5 or NetScaler that sits in front of the Dynamics 365 servers to handle which server receives the user requests. While this is the preferred option as it does not create any additional processes for the servers to handle and usually provides a wider array of configuration options (e.g. monitoring, security, filtering, etc…), it can be more difficult and much more costly to implement.

• Software Load Balancing on the other hand is relatively easy to setup and in the case of Windows NLB, is free, as you have already paid for your Windows Server licenses. Critics of software load balancing will contend that it is not “true” high-availability due to the lack of many key features found in hardware load balancing. The fact stands that Windows NLB is just Round Robin DNS so if a server were to go down, requests would still be sent to the faulted server until it is told otherwise.

• As one can probably infer, this is simply a cost vs. ROI decision – which is the case with most IT situations. If the hardware option is being considered, make sure to reach out to your appliance vendor to ensure there are no known issues with it and Dynamics 365. It is also a good idea to inquire about documentation for setup. For more information about load balancing Dynamics 365 in general and the required configurations, check out this TechNet: https://technet.microsoft.com/en-us/library/hh699803.aspx. 

• Backup Server(s) – While this method may be a little more rigid, it is still effective and again, has a couple options. The meat and potatoes of this route is having a server to fall back to in the event that your primary Dynamics 365 server(s) fails and NLB has not been implemented.

• VM Snapshots – If you are reading this, you most likely understand the concept of VM snapshots so we will not dive too heavily into this. The idea is to have automatic snapshots taken automatically on a set interval that the Dynamics 365 server can be reverted to or new server built from in the event of failure. Reverting the existing server is the easier option but also the more risky of the two. Spinning up a new server is safer but more involved – and requires the shutdown of the existing server as you will need to use the same host name and IP address.

• Laying in wait – We have had some customers opt for this route as the return to uptime is a little quicker and/or VMs were not being used. Dynamics 365 deployments can obviously be comprised of multiple servers – but not all need to be active or enabled. The concept here is to build a secondary Dynamics 365 server as part of the deployment but leave it disabled until it is needed. If the primary server were to fail, all that would be required is enabling of the server via the Deployment Manager and changes to DNS (and firewall rules, if applicable).

High Availability – SQL Server

Now that we have covered high availability from an application perspective, what about the database? Don’t worry! Microsoft has that covered too!

• SQL Server Failover Cluster – There is nothing new about this concept. Two SQL Servers are setup to use a shared disk, one server goes down and the other picks up the load. When installing Dynamics 365 just make sure to point connection strings to the cluster name and not a single node and you should not have any issues but be sure to test failover prior to going live. Just to note - although you can install Dynamics 365 to a SQL Server cluster configured for either active-active or active-passive clustering, the cluster will function in an active-passive manner. While clustering is tried and true, Microsoft decided to go a step further with...

• SQL Server Always On – New to SQL Server 2012 (Enterprise Edition), Always On introduced the concept of Availability Groups. An Availability Group is simply a container for a set of databases that fail over together. For purposes of Dynamics 365 databases, this is important considering there are always at least two databases per deployment. With Always On, the databases in the Availability Groups are synchronized amongst all the configured replicas. We won’t get into the details of setting up Always On (there are tons of articles out there already) but be sure to reference this TechNet for how to configure Dynamics 365 with SQL Always On: https://technet.microsoft.com/en-us/library/jj822357.aspx 

Disaster Recovery – Dynamics 365 and SQL Server

While having a single server fail is far more likely than seeing your entire data center swallowed into a pit only to be burnt up in the Earth’s core, it is a situation that should also be planned for! Unlike the sections regarding high availability, we need to think of a DR scenario in terms of the entire infrastructure – both Dynamics 365 and SQL Server – because if there is a disaster, chances are it is taking both so these plans go hand-in-hand. 

Over the years, we have implemented CRM/Dynamics 365 disaster recovery plans for numerous customers and as a result, have learned (what we consider) the best option – which is what we will be covering. This is not to say there are no other viable methods (such as VM snapshot migrations or SQL Log Shipping), but we will keep focus on our preferred option. In a nutshell, we are going to have a separate, self-contained Dynamics 365 environment in the DR data center that has data synced from the production SQL servers in the primary site.

SQL Server

Let us start from a database perspective since it is a little more complex and setting up Dynamics 365 for DR will require this information. As previously touched on, SQL Always On will come in handy yet again and recall the concept of Availability Groups as this will play heavy into this strategy.

For DR scenarios with Dynamics 365, we will need two Availability Groups in SQL AlwaysOn – one for the MSCRM_CONFIG database to be synchronized between the two primary data center nodes and another to synchronize the Organization_MSCRM database between not only the two primary data center SQL Servers but also a third SQL Server in your DR data center. The reason for this will become clear as you keep reading.

Dynamics 365

Going into the Dynamics 365 server installation(s), the third SQL node should already be built and ready with SQL Always On. Unlike the installation of the Dynamics 365 servers in the primary data center, you do NOT want to set the SQL connection to use the Availability Group but rather just the SQL Server located in the DR data center. The reasoning behind this is the MSCRM_CONFIG database – this database is specific per deployment of Dynamics 365 and two cannot exist in the same SQL server/instance (nor can the name be changed). Remember that this Dynamics 365 DR deployment is to be isolated from the primary.

At this point, you may be asking yourself how this all comes together. When the primary data center goes down (mainly both primary SQL nodes), the organization database will be failed over to the third node in DR. Once that happens, launch the deployment manager on the Dynamics 365 DR Server and import the organization database into the deployment. That should take about 5 minutes and then Dynamics 365 will be back up and running.

Great! We have the servers all setup in DR and are ready for that giant pit to open up beneath the data center! Now what? RUN THROUGH A TEST DISASTER RECOVERY SCENARIO! This cannot be stressed enough – you do not want to wait for a real disaster to find some small configuration issue messed up the entire plan.

URL Redirect Based on IP Address

As you are most likely aware, CRM/D365 has two URLs to access it when setup for IFD. The first URL, referred to as the “internal URL”, will allow authentication to be passed through and not display the ADFS login page for credentials. As the name suggests, this URL should only be available from within the network and not exposed out over the internet. The second URL, (not surprisingly) referred to as the “external URL”, will not pass through authentication and will display the ADFS login form for users to enter credentials. This URL is the one exposed out over the internet but is also available from within the network.

Customers often like the convenience of not having to enter credentials using the internal URL but do not like having two URLs to use. As a result, we are tasked with providing a solution to this dilemma. Back in the days of ADFS 2.0, we could easily handle this with a much simpler tweak in IIS of the ADFS server. Unfortunately, with the newer versions of ADFS no longer using IIS as its backbone this is no longer an option and a URL Rewrite rule now must be implemented on the CRM/D365 server. Here’s how:

1. Open IIS on the CRM/D365 Front End (web) Server and go to the URL Rewrite module of the CRM Website.

2. Click the link to “Add Rule(s)” and just select “Blank Rule” under Inbound rules.

3. Give the rule an applicable name and in the Match URL section, leave both the ”Requested URL” and “Using” fields as their defaults (shown in screen capture). In the “Pattern” field type in (.*).

4. This is where it gets fun. In the Conditions section, you will add three rules in the order shown below.

     a. For the first rule, change “crmorgname” to the actual name of your organization.
     b. For the second rule, change the pattern to match that of your internal IP range.
     c. Just enter the third rule as shown – no changes needed – ([^\.]*)\.(.*)

5. In the Action section, change the “Action type” to “Redirect” and for the Redirect URL enter your internal CRM/D365 URL followed by the syntax shown in the screenshot. Leave “Append query string” checked and set the Redirect type to “Permanent (301)”.

6. Apply the URL Rewrite rule and test it by going to the external URL from a machine that has an IP address matching that of the ranges specified on your network. If everything was setup correctly, the external URL will redirect to the internal URL and pass-through authentication will occur.

Tracing ADFS Logon Failures - Enabling ADFS Auditing

If you are ever faced with a situation where you are seeing a ton of logon failures in your ADFS logs and you’re not sure where they are coming from, you will soon learn that the basic logs do not provide any insight into their origins. But fear not! Some additional auditing can be enabled to help track down your problem child.

You will likely start with “Event 342 – The user name or password is incorrect” in the ADFS Admin logs – this is about as useless as it gets.

In my scenario, I was seeing about 1,500 of those events per minute. Needless to say, something was wrong but we had no idea what could possibly be trying to authenticate that much. Luckily, ADFS has some built-in auditing that can be of more use in situations like this. Start out by opening the ADFS Management Console and choose the option “Edit Federation Service Properties…” (it’s in the column on the right). Once in the properties screen, click on the “Events” tab. Here you should see 5 checkboxes – 2 of which are unchecked. Check those boxes (Success audits and Failure audits) and click OK.

Now that ADFS is setup for auditing, you need to tell the server to allow it. Head on over to the Local Security Policy of the ADFS server – this can be found from the Server Manager. Once here, drop down “Advanced Audit Policy Configuration” and then “System Audit Policies – Local Group Policy Object”. Locate the “Object Access” policy and then open the “Audit Application Generated” subcategory. Check the three boxes on this screen and then click OK.

You should now be all set to revisit your Event Viewer. Look into the Security events under the Windows Logs and you should now see events with ID 411 for “Classic Audit Failure” with the source as “AD FS Auditing”.

Go ahead and open one of those bad boys up…. Ahhhh finally some useful information! A quick look through will reveal the Client IP, which should be the machine sending the invalid authentication requests to ADFS.

The process from here is self-explanatory but head on over to the machine with the offending client IP address and find out what the problem is. In my case, there was a long-forgotten email router still setup which kept trying to access CRM. After figuring out the problem, it is recommended to disable auditing.