Thursday, October 01, 2009

SharePoint Farm using Kerberos on IIS7

Kerberos is recommended as the authentication mode in SharePoint due to security, performance and not least delegation (impersonation, double-hop) reasons. I won't go into details on the benefits here, but rather point to some good resources you will need when configuring Kerberos for SharePoint. Configuration of Kerberos is not part of the SharePoint installer, except for popup boxes warning you that your system administrator must perform some extra configuration for this to work. If you're lucky, that system admin is not you.

I recommend you follow these guides to ensure a working Kerberos environment:
Follow the guidelines meticulously, taking great care when adding SPNs. Use SETSPN -S rather than -A to avoid duplicates (which breaks Kerberos and fallback to NTLM), and use SETSPN -X to check for duplicates. These are new SETSPN switches on Windows Server 2008.

After setting up all the SPNs, a strange problem affected our load balanced web-applications. We got 401 Unauthorized "the requested resource requires user authentication" and "access is denied due to invalid credentials" when browsing sites on the WFE servers.

Our servers run Windows Server 2008 and hosts the sites in IIS7. IIS7 use a new "kernel mode" by default for integrated Windows Authentication, and the computer account (NETWORK SERVICE) is used rather than the application pool identity by Kerberos to do SPN ticket decryption. This affects farm sites, as a domain account must be used rather than a local account to successfully decrypt SPN tickets across the farm. See Service Principal Name (SPN) checklist for Kerberos authentication with IIS 7.0 for further details.

So you have two options to make this work on IIS7: either disable kernel mode authentication, or use the undocumented useAppPoolCredentials in combination with useKernelMode in ApplicationHost.config (C:\Windows\System32\inetsrv\config\applicationHost.config). Add the attribute to the specific site's <windowsAuthentication> element so it looks like this:

<windowsAuthentication enabled="true" useAppPoolCredentials="true" useKernelMode="true">

Do not use this setting across the board, only use useAppPoolCredentials on the sites on those servers that give a Kerberos 401 not authorized error. And always make a backup copy of the file before editing it.

The last step in the configure Kerberos guide is to turn on "negotiate" authentication for the SSP:

STSADM –o setsharedwebserviceauthn –negotiate

Click on "Office SharePoint Server Search" in Central Admin > Operations > Services on Server on the Query role, after selecting a WFE server, to confirm that the SSP work correctly after enabling Kerberos. If the settings page opens all went well. If you get this message, it sure didn't:

An unhandled exception occurred in the user interface. Exception Information: Could not connect to server MOSS-WFE01. This error might occur if the firewall or proxy configuration is preventing the server from being contacted.

The above test checks the root-level Office Server Web Services used for MOSS server-to-server communication.

In addition, trying to open the SSP search settings page gave this error:

The search service is currently offline. Visit the Services on Server page in SharePoint Central Administration to verify whether the service is enabled. This might also be because an indexer move is in progress.

The above test checks the Office Server Web Services used for SSP services such as search administration.

Don't panic. Diagnosing this is quite easy. SharePoint uses the Office Server Web Services when assigning roles to servers and to communicate between the servers. Browse the MOSS-WFE01 root-level services from the MOSS-ADMIN server and vice versa:
If you cannot browse these root-level Office Server Web Services (OSWS) on WFE from the Cental Admin server, then you have a problem. If you get a 404 "not found" message, the check the firewalls on the servers and try pinging them. If you get a response, but things still do not work it is typically 401 or 403 errors. Never mind the untrusted certificate message, the real problem is any 401 Unauthorized messages you might get.

First, ensure that all your SPNs are correctly configured for the server and app-pool accounts, including the HOST, HTTP and MSSP service classes. HOST or HTTP is used for the root-level OSWS, when the app-pool idenity is a machine account or domain account repectively. MSSP is used for the SSP services.

Then ensure that any SPN account involved in Kerberos is trusted for delegation in AD. This also applies to computer accounts used to run sites, typically as the "network service" local account. Also ensure that DCOM is correctly configured on all the servers running SharePoint in the farm.

If all else fails, change the application pool used to run the Office Server Web Services web-application. By default, this is OfficeServerApplicationPool running as "network service". But you can't really change that application pool's identity. You can, but SharePoint will automatically set it back to "network service". As you learnt above, using a local account for a farm service in IIS7 is not recommended.

So, reuse the SSP service application pool or create a new IIS "OfficeServerKerberosApplicationPool" using the domain account of the SSP service application pool and make the Office Server Web Services run using the new app-pool. This must be done on all WFE servers and on the indexing application server. Provided that the selected domain account has a valid HTTP SPN for the OSWS machine NETBOIS name, internal MOSS server-to-server calls should now work. Test and confirm that both the root-level and SSP search server settings pages now opens.

Finally, check that the security event log contains audit success entries for logon of type Kerberos on the servers. Kerberos ticket decryption failures are shown in the client's system event log with event id 4 and error code KRB_AP_ERR_MODIFIED. Remember that an app-pool account cannot decrypt the Kerberos ticket if the SPN is registered on another account.

Note that if you're following the least privilege approach and have different accounts for the SSP service application host and the SSP services, then you might not be able to directly browse the SSP web-services such as SearchAdmin.asmx, because the HTTP SPN is assigned to a different account than the account used to run the SSP service. This is not a problem because the SSP services internally use the MSSP SPN to get and validate Kerberos tickets.

Still having problems? Read Jesper M. Christensen's excellent series Troubleshooting Kerberos in a SharePoint environment Part 1 / Part 2 / Part3.

End note: Turn off UAC on Windows Server 2008 if you get "access denied" errors when running STSADM commands.


liquidpooled said...

UAC is in place for a reason. Rather than a blanket disabling of the functionality, you can run your command prompts as an administration (right click, "Run As Administrator"). Doing this will allow you to execute stsadm and psconfig command.

Matt Adkins said...

If all else fails...reuse the SSP application pool! Great advice! Appreciate the article.