SharePoint Farm using Kerberos on IIS7

Kerberos is recommended as the authentication mode in SharePoint due to security, performance and not least delegation (impersonation, double-hop) reasons. I won't go into details on the benefits here, but rather point to some good resources you will need when configuring Kerberos for SharePoint. Configuration of Kerberos is not part of the SharePoint installer, except for popup boxes warning you that your system administrator must perform some extra configuration for this to work. If you're lucky, that system admin is not you.

I recommend you follow these guides to ensure a working Kerberos environment:

• Configure Kerberos Authentication MOSS 2007 (really long article)
• How to configure SharePoint for Kerberos authentication (much shorter)

Follow the guidelines meticulously, taking great care when adding SPNs. Use SETSPN -S rather than -A to avoid duplicates (which breaks Kerberos and fallback to NTLM), and use SETSPN -X to check for duplicates. These are new SETSPN switches on Windows Server 2008.

After setting up all the SPNs, a strange problem affected our load balanced web-applications. We got 401 Unauthorized "the requested resource requires user authentication" and "access is denied due to invalid credentials" when browsing sites on the WFE servers.

Our servers run Windows Server 2008 and hosts the sites in IIS7. IIS7 use a new "kernel mode" by default for integrated Windows Authentication, and the computer account (NETWORK SERVICE) is used rather than the application pool identity by Kerberos to do SPN ticket decryption. This affects farm sites, as a domain account must be used rather than a local account to successfully decrypt SPN tickets across the farm. See Service Principal Name (SPN) checklist for Kerberos authentication with IIS 7.0 for further details.

So you have two options to make this work on IIS7: either disable kernel mode authentication, or use the undocumented useAppPoolCredentials in combination with useKernelMode in ApplicationHost.config (C:\Windows\System32\inetsrv\config\applicationHost.config). Add the attribute to the specific site's <windowsAuthentication> element so it looks like this:

<windowsAuthentication enabled="true" useAppPoolCredentials="true" useKernelMode="true">
...
</windowsAuthentication>

Do not use this setting across the board, only use useAppPoolCredentials on the sites on those servers that give a Kerberos 401 not authorized error. And always make a backup copy of the file before editing it.

The last step in the configure Kerberos guide is to turn on "negotiate" authentication for the SSP:

STSADM –o setsharedwebserviceauthn –negotiate

Click on "Office SharePoint Server Search" in Central Admin > Operations > Services on Server on the Query role, after selecting a WFE server, to confirm that the SSP work correctly after enabling Kerberos. If the settings page opens all went well. If you get this message, it sure didn't:

An unhandled exception occurred in the user interface. Exception Information: Could not connect to server MOSS-WFE01. This error might occur if the firewall or proxy configuration is preventing the server from being contacted.

The above test checks the root-level Office Server Web Services used for MOSS server-to-server communication.

In addition, trying to open the SSP search settings page gave this error:

The search service is currently offline. Visit the Services on Server page in SharePoint Central Administration to verify whether the service is enabled. This might also be because an indexer move is in progress.

The above test checks the Office Server Web Services used for SSP services such as search administration.

Don't panic. Diagnosing this is quite easy. SharePoint uses the Office Server Web Services when assigning roles to servers and to communicate between the servers. Browse the MOSS-WFE01 root-level services from the MOSS-ADMIN server and vice versa:

• Browse https://moss-wfe01:56738/searchadmin.asmx from MOSS-ADMIN
• Browse https://moss-admin:56738/searchadmin.asmx from MOSS-WFE01

If you cannot browse these root-level Office Server Web Services (OSWS) on WFE from the Cental Admin server, then you have a problem. If you get a 404 "not found" message, the check the firewalls on the servers and try pinging them. If you get a response, but things still do not work it is typically 401 or 403 errors. Never mind the untrusted certificate message, the real problem is any 401 Unauthorized messages you might get.

First, ensure that all your SPNs are correctly configured for the server and app-pool accounts, including the HOST, HTTP and MSSP service classes. HOST or HTTP is used for the root-level OSWS, when the app-pool idenity is a machine account or domain account repectively. MSSP is used for the SSP services.

Then ensure that any SPN account involved in Kerberos is trusted for delegation in AD. This also applies to computer accounts used to run sites, typically as the "network service" local account. Also ensure that DCOM is correctly configured on all the servers running SharePoint in the farm.

If all else fails, change the application pool used to run the Office Server Web Services web-application. By default, this is OfficeServerApplicationPool running as "network service". But you can't really change that application pool's identity. You can, but SharePoint will automatically set it back to "network service". As you learnt above, using a local account for a farm service in IIS7 is not recommended.

So, reuse the SSP service application pool or create a new IIS "OfficeServerKerberosApplicationPool" using the domain account of the SSP service application pool and make the Office Server Web Services run using the new app-pool. This must be done on all WFE servers and on the indexing application server. Provided that the selected domain account has a valid HTTP SPN for the OSWS machine NETBOIS name, internal MOSS server-to-server calls should now work. Test and confirm that both the root-level and SSP search server settings pages now opens.

Finally, check that the security event log contains audit success entries for logon of type Kerberos on the servers. Kerberos ticket decryption failures are shown in the client's system event log with event id 4 and error code KRB_AP_ERR_MODIFIED. Remember that an app-pool account cannot decrypt the Kerberos ticket if the SPN is registered on another account.

Note that if you're following the least privilege approach and have different accounts for the SSP service application host and the SSP services, then you might not be able to directly browse the SSP web-services such as SearchAdmin.asmx, because the HTTP SPN is assigned to a different account than the account used to run the SSP service. This is not a problem because the SSP services internally use the MSSP SPN to get and validate Kerberos tickets.

Still having problems? Read Jesper M. Christensen's excellent series Troubleshooting Kerberos in a SharePoint environment Part 1 / Part 2 / Part3.

End note: Turn off UAC on Windows Server 2008 if you get "access denied" errors when running STSADM commands.

WCF: WS-Security using Kerberos over SSL

WCF support typical scenarios with the system provided bindings such as basicHttpBinding and wsHttpBinding. These bindings work very well in WCF-only environments and for interop scenarios where all parties are completely WS* standard compliant.

However, interop always seems to need some tweaking, and the ootb bindings allow you to change many aspects of the binding configuration. But there will always be scenarios where these bindings just cannot be configured to fit your needs. E.g. if you need a binding that supports custom wsHttp over SOAP1.1 using a non-negotiated/direct Kerberos WS-Security token secured by TLS. In this case you need a custom binding, because e.g. even if you can turn of spnego for wsHttpBinding, you cannot tweak it into using SOAP1.1

WCF <customBinding> to the rescue:
<customBinding>
<binding name="wsBindingCustom">
<security authenticationMode="KerberosOverTransport"
requireDerivedKeys="false"
includeTimestamp="true">
</security>
<textMessageEncoding messageVersion="Soap11" />
<httpsTransport />
</binding>
</customBinding>

This custom binding combines HTTPS transport with direct Kerberos. The timestamp is required for security reasons when using authentication mode Kerberos over SSL. Note that using the UPN/SPN type <identity> element is only supported NTLM or negotiated security, and cannot be used with direct Kerberos.

WCF by default requires that the response always contain a timestamp when it is in the request. This is a typical interop issue with e.g. WSS4J, Spring-WS and WebSphere DataPower.

The response with the signed security header timestamp should be like this:
<soapenv:Header>
<wsse:Security soapenv:mustUnderstand="1" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
<wsu:Timestamp wsu:Id="uuid-c9f9cf30-2685-4090-911b-785d59718267" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
<wsu:Created>2008-09-29T12:02:59Z</wsu:Created>
<wsu:Expires>2008-09-29T12:12:59Z</wsu:Expires>
</wsu:Timestamp>


Note the casing of the wsse: and wsu: namespace elements and attributes. This replay attack detection timestamp can be turned off using the includeTimestamp attribute, or it can be configured using the <localClientSettings> and <localServiceSetting> security elements.

Refer to the <security> element of the <customBinding> documentation for more details. Also review the settings for initiating a WS-SecureConversation if a session is needed.

WCF Security: wsHttpBinding with brokered Kerberos

Now that I have implemented a separate ASMX endpoint for interoperability with Flex 2, it was time to switch from basicHttpBinding to wsHttpBinding for better security (authentication, encryption, signing). Using this binding will provide message level security, preventing e.g. HTTP spies on the client from snooping the data exchange with our services.

Editing the WCF client and server configuration files is a quite daunting task, but luckily the new version of WSSF offers nice wizards for configuring security for your services. The 'WCF security' menu allows you to easily add support for the most common providers: client X.509 certificates, Kerberos, ADAM, SQL Server, ActiveDirectory, server certificate.


Among the available providers, the Kerberos provider is the simplest to use if you don't want to use a certificate nor HTTPS/SSL, or you want/has to use Cassini (the VS2005 developer web-server), as shown in the figure (click to enlarge):

The wizard is straightforward, just remember to include metadata (MEX) in your service if you want to expose a WSDL file. Adding a list of authorized clients is not mandatory for this security mode. I recommend that you delete all content in the <system.serviceModel> element in the service config file before using the 'WCF Security' menu. This ensures a clean configuration settings section afterwards.

I then updated the service reference of our SmartClient application to get the correct settings for the client config file. This step is important, as the common configuration settings must be equal on both client and server for WCF communication to work.

With all the configuration finished, it was time for some testing. I got this error when invoking a WCF operation:

System.ServiceModel.Security.MessageSecurityException: The token provider cannot get tokens for target 'http://localhost.:1508/eApproval.Host/ProjectDocumentServices.svc'. ---> System.IdentityModel.Tokens.SecurityTokenValidationException: The NetworkCredentials provided were unable to create a Kerberos credential, see inner execption for details. ---> System.IdentityModel.Tokens.SecurityTokenException: InitializeSecurityContent failed. Ensure the service principal name is correct. ---> System.ComponentModel.Win32Exception: No credentials are available in the security package
--- End of inner exception stack trace ---

My client config file contained these security settings, which to me looked fine and dandy:

<security mode="Message">
<transport clientCredentialType="Windows" proxyCredentialType="None" realm="" />
<message clientCredentialType="Windows" negotiateServiceCredential="false" algorithmSuite="Default" establishSecurityContext="false" />
</security>
My suspects were the extra <message> element attributes not covered in Keith Brown's "Security in WCF" article: negotiateServiceCredential and establishSecurityContext. As usual with new Microsoft technology, the online help is rather lame, so some research was required.

Googling lead me to this excellent article by Michèle Leroux Bustamante: "Fundamentals of WCF Security" (6 pages). The 'Service Credentials and Negotiation' section contained the explanation for my error: "When negotiation is disabled for Windows client credentials, a Kerberos domain must exist". So by turning service identity negotiation on in both the client and the server config files, my client can now communicate securely with our WCF service. I also recommend turning on the 'secure session' context for better performance, unless your service is rarely used by the client.

A confusing part of the client side authentication settings for the 'Windows' security mode is the <endpoint/identity> element, which will typically contain your "user principal name" (UPN) in the <userPrincipalName> element when hosting with Cassini or self-hosting. The identity setting is not for specifying who the client user is, but for identityfying and authenticating the service itself. If you use IIS as the host, the identity must contain the "service principal name" (SPN). Note that if your IIS app pool identity is not NETWORK SERVICE, you must typically create the SPN explicitly.
The <identity> element is only used with NTLM or negotiated security. If you use non-negotiated security such as direct Kerberos, this element is not supported.

The security config now looks like this (server side settings):

<bindings><wsHttpBinding>
<binding name="BrokeredAuthenticationKerberosBinding">
<security mode="Message"> <message clientCredentialType="Windows" negotiateServiceCredential="true" establishSecurityContext="true" />
</security>
</binding>
</wsHttpBinding>
</bindings>

Checking the on-the-wire format with Fiddler shows that the request and response messages now are encrypted; without using HTTPS or SSL transport level security:

The system provided bindings and the WSSF wizards do a good job in guiding you in the WCF settings jungle; but when your automated guide gets lost, you'd better have some survival skills. Knowing the inner workings of WCF is a must for professional service developers, and I strongly recommend reading Michèle's article. The same goes for her blogs and the WCF stuff at IDesign.

Fiddler is highly recommended and can be downloaded from http://www.fiddlertool.com/fiddler/.