How to Debug SharePoint Solutions in a Multi-Server Farm

Some tips on deploying and debugging code in a multi-server SharePoint 2010 farm:

When debugging on a multi-server SharePoint farm, with Visual Studio on the app-server, then add AAM mapping for the app-server to the web-application to debug, otherwise VS can't attach to the local w3wp process.

For example, to debug the web-app hosted on

• http://azure-sp2010web:8383/ 

you must add the app-server URL first to AAM and then use it as Site URL when debugging

• http://azure-sp2010app:8383/

Make sure to browse the site using the added app-server URL to load the code in a local w3wp process.

If the breakpoints have yellow warning triangles, then VS could not load the correct code to the w3wp processes attached to the debugger. Solve by rebuild and deploy to get the latest bits into the [14] hive and GAC. Note that you can’t activate WSPs on deploy in a multi-server farm, set the "Active Deployment Configuration" to “no activation” in Visual Studio project properties. If VS still can’t deploy the WSP, then use powershell to first Add-SPSolution and then Install-SPSolution across the farm.

Validate the WSP deployment status in “Manage farm solutions” in Central Admin first, and make sure that your feature is activated in the site or subsite you try to debug.

Happy debugging :)

SharePoint Farm using Kerberos on IIS7

Kerberos is recommended as the authentication mode in SharePoint due to security, performance and not least delegation (impersonation, double-hop) reasons. I won't go into details on the benefits here, but rather point to some good resources you will need when configuring Kerberos for SharePoint. Configuration of Kerberos is not part of the SharePoint installer, except for popup boxes warning you that your system administrator must perform some extra configuration for this to work. If you're lucky, that system admin is not you.

I recommend you follow these guides to ensure a working Kerberos environment:

• Configure Kerberos Authentication MOSS 2007 (really long article)
• How to configure SharePoint for Kerberos authentication (much shorter)

Follow the guidelines meticulously, taking great care when adding SPNs. Use SETSPN -S rather than -A to avoid duplicates (which breaks Kerberos and fallback to NTLM), and use SETSPN -X to check for duplicates. These are new SETSPN switches on Windows Server 2008.

After setting up all the SPNs, a strange problem affected our load balanced web-applications. We got 401 Unauthorized "the requested resource requires user authentication" and "access is denied due to invalid credentials" when browsing sites on the WFE servers.

Our servers run Windows Server 2008 and hosts the sites in IIS7. IIS7 use a new "kernel mode" by default for integrated Windows Authentication, and the computer account (NETWORK SERVICE) is used rather than the application pool identity by Kerberos to do SPN ticket decryption. This affects farm sites, as a domain account must be used rather than a local account to successfully decrypt SPN tickets across the farm. See Service Principal Name (SPN) checklist for Kerberos authentication with IIS 7.0 for further details.

So you have two options to make this work on IIS7: either disable kernel mode authentication, or use the undocumented useAppPoolCredentials in combination with useKernelMode in ApplicationHost.config (C:\Windows\System32\inetsrv\config\applicationHost.config). Add the attribute to the specific site's <windowsAuthentication> element so it looks like this:

<windowsAuthentication enabled="true" useAppPoolCredentials="true" useKernelMode="true">
...
</windowsAuthentication>

Do not use this setting across the board, only use useAppPoolCredentials on the sites on those servers that give a Kerberos 401 not authorized error. And always make a backup copy of the file before editing it.

The last step in the configure Kerberos guide is to turn on "negotiate" authentication for the SSP:

STSADM –o setsharedwebserviceauthn –negotiate

Click on "Office SharePoint Server Search" in Central Admin > Operations > Services on Server on the Query role, after selecting a WFE server, to confirm that the SSP work correctly after enabling Kerberos. If the settings page opens all went well. If you get this message, it sure didn't:

An unhandled exception occurred in the user interface. Exception Information: Could not connect to server MOSS-WFE01. This error might occur if the firewall or proxy configuration is preventing the server from being contacted.

The above test checks the root-level Office Server Web Services used for MOSS server-to-server communication.

In addition, trying to open the SSP search settings page gave this error:

The search service is currently offline. Visit the Services on Server page in SharePoint Central Administration to verify whether the service is enabled. This might also be because an indexer move is in progress.

The above test checks the Office Server Web Services used for SSP services such as search administration.

Don't panic. Diagnosing this is quite easy. SharePoint uses the Office Server Web Services when assigning roles to servers and to communicate between the servers. Browse the MOSS-WFE01 root-level services from the MOSS-ADMIN server and vice versa:

• Browse https://moss-wfe01:56738/searchadmin.asmx from MOSS-ADMIN
• Browse https://moss-admin:56738/searchadmin.asmx from MOSS-WFE01

If you cannot browse these root-level Office Server Web Services (OSWS) on WFE from the Cental Admin server, then you have a problem. If you get a 404 "not found" message, the check the firewalls on the servers and try pinging them. If you get a response, but things still do not work it is typically 401 or 403 errors. Never mind the untrusted certificate message, the real problem is any 401 Unauthorized messages you might get.

First, ensure that all your SPNs are correctly configured for the server and app-pool accounts, including the HOST, HTTP and MSSP service classes. HOST or HTTP is used for the root-level OSWS, when the app-pool idenity is a machine account or domain account repectively. MSSP is used for the SSP services.

Then ensure that any SPN account involved in Kerberos is trusted for delegation in AD. This also applies to computer accounts used to run sites, typically as the "network service" local account. Also ensure that DCOM is correctly configured on all the servers running SharePoint in the farm.

If all else fails, change the application pool used to run the Office Server Web Services web-application. By default, this is OfficeServerApplicationPool running as "network service". But you can't really change that application pool's identity. You can, but SharePoint will automatically set it back to "network service". As you learnt above, using a local account for a farm service in IIS7 is not recommended.

So, reuse the SSP service application pool or create a new IIS "OfficeServerKerberosApplicationPool" using the domain account of the SSP service application pool and make the Office Server Web Services run using the new app-pool. This must be done on all WFE servers and on the indexing application server. Provided that the selected domain account has a valid HTTP SPN for the OSWS machine NETBOIS name, internal MOSS server-to-server calls should now work. Test and confirm that both the root-level and SSP search server settings pages now opens.

Finally, check that the security event log contains audit success entries for logon of type Kerberos on the servers. Kerberos ticket decryption failures are shown in the client's system event log with event id 4 and error code KRB_AP_ERR_MODIFIED. Remember that an app-pool account cannot decrypt the Kerberos ticket if the SPN is registered on another account.

Note that if you're following the least privilege approach and have different accounts for the SSP service application host and the SSP services, then you might not be able to directly browse the SSP web-services such as SearchAdmin.asmx, because the HTTP SPN is assigned to a different account than the account used to run the SSP service. This is not a problem because the SSP services internally use the MSSP SPN to get and validate Kerberos tickets.

Still having problems? Read Jesper M. Christensen's excellent series Troubleshooting Kerberos in a SharePoint environment Part 1 / Part 2 / Part3.

End note: Turn off UAC on Windows Server 2008 if you get "access denied" errors when running STSADM commands.

IIS6: App-Domain could not be created

Each time when I have to deploy a ASP.NET/WCF application to a new Windows Server 2003 IIS web-site, I run into a "Server Application Unavailable" error and this logged in the application event log:

Failed to execute request because the App-Domain could not be created. Error: 0x80070005 Access is denied.

Exception: System.IO.FileLoadException
Message: Could not load file or assembly 'System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. Access is denied.

As always, I'm a bit puzzled by the access denied error as I have granted NTFS read+execute rights to the ASPNET and IUSR accounts as applicable to the application's disk folder and to the .NET 2.0 and 3.0 folders. Time for Microsoft System Internals Filemon to log file access failures to diagnose the problem.

Note to self: The cause of the problem is that the identity (e.g. NETWORK SERVICE) of the IIS application pool used to run the web application, needs read access to the application's web.config file to be able to launch the ASP.NET worker process.

Read the excellent MSDN article 'Extend Your WCF Services Beyond HTTP With WAS' to learn more about IIS/ASP.NET/WAS application pools, process models and app-domains.

WCF Streaming: Upload files over HTTP

We have a web application that requires a robust and reliable document upload mechanism for very large files, and are currently using Microsoft BITS technology to achieve this. With the introduction of WCF in the solution, I set out to implement an upload service prototype using WCF and a HTTP binding.

You have two main options for uploading files in WCF: streamed or buffered/chunked. The latter option is the one that provides reliable data transfer as you can use wsHttpBinding with WS-ReliableMessaging turned on. Reliable messaging cannot be used with streaming as the WS-RM mechanism requires processing the data as a unity to apply checksums, etc. Processing a large file this way would require a huge buffer and thus a lot of available memory on both client and server; denial-of-service comes to mind. The workaround for this is chunking: splitting the file into e.g. 64KB fragments and using reliable, ordered messaging to transfer the data.

If you do not need the robustness of reliable messaging, streaming lets you transfer large amount of data using small message buffers without the overhead of implementing chuncking. Streaming over HTTP requires you to use basicHttpBinding; thus you will need SSL to encrypt the transferred data. Buffered transfer, on the other hand, can use wsHttpBinding, which by default provides integrity and confidentiality for your messages; thus there is no need for SSL then.

Read more about 'Large Data and Streaming' at MSDN.

So I implemented my upload prototype following the 'How to: Enable Streaming' guidelines, using a VS2005 web application (Cassini) as the service host. I made my operation a OneWay=true void operation with some SOAP headers to provide meta-data:

[OperationContract(Action = "UploadFile", IsOneWay = true)]

void UploadFile(ServiceContracts.FileUploadMessage request);

[MessageContract]

public class FileUploadMessage

{

[MessageHeader(MustUnderstand = true)]

public DataContracts.DnvsDnvxSession DnvxSession

[MessageHeader(MustUnderstand = true)]

public DataContracts.EApprovalContext Context

[MessageHeader(MustUnderstand = true)]

public DataContracts.FileMetaData FileMetaData

[MessageBodyMember(Order = 1)]

public System.IO.Stream FileByteStream

}

The reason for using message headers for meta-data is that WCF requires that the stream object is the only item in the message body for a streamed operation. Headers are the recommended way for sending meta-data when streaming. Note that headers are always sent before the body, thus you can depend on receving the header data before processing the streamed data.

Note that you should provide a separate endpoint (address, binding, contract) for your streamed services. The main reason for this is that configuration such as transferMode = "Streamed" applies to all operations in the endpoint. The same goes for transferMode, maxBufferSize, maxReceivedMessageSize, receiveTimeout, etc.

I use MTOM as the encoding format for the streamed data and set the max file size to 64MB. I also set the buffer size to 64KB even if I read the input stream in 4KB chucks, this to avoid receive buffer underruns. My binding config looks like this:

<basicHttpBinding>

<!-- buffer: 64KB; max size: 64MB -->

<binding name="FileTransferServicesBinding"

closeTimeout="00:01:00" openTimeout="00:01:00" 

receiveTimeout="00:10:00" sendTimeout="00:01:00"

transferMode="Streamed" messageEncoding="Mtom"

maxBufferSize="65536" maxReceivedMessageSize="67108864">

<security mode="None">

<transport clientCredentialType="None"/>

</security>

</binding>

</basicHttpBinding>

I have set the transport security to none as I am testing the prototype without using SSL. I have not modified the service behavior; the default authentication (credentials) and authorization behaviors of the binding is used.

Note that the setting for transferMode does not propagate to clients when using a HTTP binding. You must manually edit the client config file to set transferMode = "Streamed" after using 'Add service reference' or running SVCUTIL.EXE. If you forget to do this, the transfer mode will be "Buffered" and you will get an error like this:

System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://localhost:1508/Host/FileTransferService.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.


After ensuring that the client basicHttpBinding config was correct, I ran the unit test once more. Now I got this error from VS2005 Cassini:

System.ServiceModel.ProtocolException: The remote server returned an unexpected response: (400) Bad Request.
System.Net.WebException: The remote server returned an error: (400) Bad Request.


I have inspected the HTTP traffic using Fiddler and can see nothing wrong with the MTOM encoded request. Googling lead me to a lot of frustrated "streamed" entries at the forums.microsoft.com "Indigo" group, but no solution to my problem. So I made a self-hosted service using a console application, using the code provided in the WCF samples (see below), and it worked like a charm. I expect that it is just VS2005 Cassini that does not support streamed transfers/MTOM encoding.

I then installed IIS to check whether HTTP streaming works better with IIS. The first error I ran into when uploading my 1KB test file was this:

System.ServiceModel.CommunicationException: The underlying connection was closed: The connection was closed unexpectedly.
System.Net.WebException: The underlying connection was closed: The connection was closed unexpectedly.


Inspecting the traffic with Fiddler, I found the HTTP request to be fine, but no HTTP response. Knowing that this is a sure sign of a server-side exception, I debugged the service operation. The error was caused by the IIS application pool identity (ASPNET worker process) lacking NTFS rights to store the uploaded file on disk. Note that WCF by default will not impersonate the caller, thus the identity used to run your service will need to have sufficient permissions on all resources that your service needs to access. This includes the temp folder, which is used to provide the WSDL file for HTTP GET.

As the UploadFile operation has [OperationContract(IsOneWay = true)], it cannot have a response and neither a fault contract. Thus, there will be no HTTP response when a server-side exception occurs, just the famous "The underlying connection was closed" message.

I assigned the correct NTFS rights to my upload folder and re-ran the unit test: streamed upload to a WCF service hosted by IIS worked. The next unit test used a 43MB file to check if uploading data larger than the WCF buffer size works. The test gave me this error:


System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:59.8590000'.System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host


I was a bit puzzled as the file was less than the WCF maxReceivedMessageSize limit set to 64MB, and the stream read buffer was only 4KB as compared to the WCF maxBufferSize being 64KB. Knowing that IIS/ASP.NET has a system enforced limit on the size of a HTTP request to prevent denial-of-service attacks, which seems to be unrelated to WCF as I do not use the AspNetCompatibilityRequirements service setting, I decided to set the maxRequestLength limit to 64MB also (default: 4096KB). This time the "stream large file upload" test passed!

So for WCF streaming to IIS to work properly, you need this <system.web> config in addition to the <system.serviceModel> config in the IIS web.config file:

<!-- maxRequestLength (in KB) max size: 2048MB -->

<httpRuntime 

maxRequestLength="65536"

/>

Make the maximum HTTP request size equal to the WCF maximum request size.

I have to recommend the 'Service Trace Viewer' (SvcTraceViewer.EXE) and the 'Service Configuration Editor' (SvcConfigEditor.EXE) provided in the WCF toolbox (download .NET3 SDK). These tools make it really simple to add tracing and logging to your service for debugging and diagnosing errors. Start the WCF config editor, open your WCF config file, select the 'Diagnostics' node and just click "Enable tracing" in the "Tracing" section to turn on tracing. The figure shows some typical settings (click to enlarge):



Save the config file and run the unit test to invoke the service. This will generate the web_tracelog.svclog file to use as input for the trace viewer. Open the trace file in the trace viewer and look for the errors (red text). Click an error to see the exception details. The figure shows some of the details available for the "Maximum request length exceeded" error (click to enlarge):


The WCF tools installed by the SDK is located here:
C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin\

The code to process the incoming stream is rather simple, storing the uploaded files to a single folder on disk and overwriting any existing files:

public void UploadFile(FileUploadMessage request)

{

FileStream targetStream = null;

Stream sourceStream = request.FileByteStream;

string uploadFolder = @"C:\work\temp\";

string filename = request.FileMetaData.Filename;

string filePath = Path.Combine(uploadFolder, filename);

using (targetStream = new FileStream(filePath, FileMode.Create, 

                          FileAccess.Write, FileShare.None))

{

//read from the input stream in 4K chunks

//and save to output stream

const int bufferLen = 4096;

byte[] buffer = new byte[bufferLen];

int count = 0;

while ((count = sourceStream.Read(buffer, 0, bufferLen)) > 0)

{

targetStream.Write(buffer, 0, count);

}

targetStream.Close();

sourceStream.Close();

}

}

The 'Stream Sample' available on MSDN contains all the code you need to upload a file as a stream to a self-hosted WCF service and then save it to disk on the server by reading the stream in 4KB chunks. The download contains about 150 solutions with more than 4800 files, so there is a lot of stuff in it. Absolutely worth a look. The streaming sample is located here:
<unzip folder> \TechnologySamples\Basic\Contract\Service\Stream

Please let me know if you have been able to get HTTP streaming with MTOM encoding to work with Microsoft VS2005 Cassini.
Finally, a useful tip for streaming download: How to release streams and files using Disponse() on the the message contract.