Wednesday, February 07, 2007

WCF Streaming: Upload files over HTTP

We have a web application that requires a robust and reliable document upload mechanism for very large files, and are currently using Microsoft BITS technology to achieve this. With the introduction of WCF in the solution, I set out to implement an upload service prototype using WCF and a HTTP binding.

You have two main options for uploading files in WCF: streamed or buffered/chunked. The latter option is the one that provides reliable data transfer as you can use wsHttpBinding with WS-ReliableMessaging turned on. Reliable messaging cannot be used with streaming as the WS-RM mechanism requires processing the data as a unity to apply checksums, etc. Processing a large file this way would require a huge buffer and thus a lot of available memory on both client and server; denial-of-service comes to mind. The workaround for this is chunking: splitting the file into e.g. 64KB fragments and using reliable, ordered messaging to transfer the data.

If you do not need the robustness of reliable messaging, streaming lets you transfer large amount of data using small message buffers without the overhead of implementing chuncking. Streaming over HTTP requires you to use basicHttpBinding; thus you will need SSL to encrypt the transferred data. Buffered transfer, on the other hand, can use wsHttpBinding, which by default provides integrity and confidentiality for your messages; thus there is no need for SSL then.

Read more about 'Large Data and Streaming' at MSDN.

So I implemented my upload prototype following the 'How to: Enable Streaming' guidelines, using a VS2005 web application (Cassini) as the service host. I made my operation a OneWay=true void operation with some SOAP headers to provide meta-data:

[OperationContract(Action = "UploadFile", IsOneWay = true)]
void UploadFile(ServiceContracts.FileUploadMessage request);
[MessageContract]
public class FileUploadMessage
{
[MessageHeader(MustUnderstand = true)]
public DataContracts.DnvsDnvxSession DnvxSession
[MessageHeader(MustUnderstand = true)]
public DataContracts.EApprovalContext Context
[MessageHeader(MustUnderstand = true)]
public DataContracts.FileMetaData FileMetaData
[MessageBodyMember(Order = 1)]
public System.IO.Stream FileByteStream
}

The reason for using message headers for meta-data is that WCF requires that the stream object is the only item in the message body for a streamed operation. Headers are the recommended way for sending meta-data when streaming. Note that headers are always sent before the body, thus you can depend on receving the header data before processing the streamed data.

Note that you should provide a separate endpoint (address, binding, contract) for your streamed services. The main reason for this is that configuration such as transferMode = "Streamed" applies to all operations in the endpoint. The same goes for transferMode, maxBufferSize, maxReceivedMessageSize, receiveTimeout, etc.

I use MTOM as the encoding format for the streamed data and set the max file size to 64MB. I also set the buffer size to 64KB even if I read the input stream in 4KB chucks, this to avoid receive buffer underruns. My binding config looks like this:

<basicHttpBinding>
<!-- buffer: 64KB; max size: 64MB -->
<binding name="FileTransferServicesBinding"
closeTimeout="00:01:00" openTimeout="00:01:00" 
receiveTimeout="00:10:00" sendTimeout="00:01:00"
transferMode="Streamed" messageEncoding="Mtom"
maxBufferSize="65536" maxReceivedMessageSize="67108864">
<security mode="None">
<transport clientCredentialType="None"/>
</security>
</binding>
</basicHttpBinding>

I have set the transport security to none as I am testing the prototype without using SSL. I have not modified the service behavior; the default authentication (credentials) and authorization behaviors of the binding is used.

Note that the setting for transferMode does not propagate to clients when using a HTTP binding. You must manually edit the client config file to set transferMode = "Streamed" after using 'Add service reference' or running SVCUTIL.EXE. If you forget to do this, the transfer mode will be "Buffered" and you will get an error like this:

System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://localhost:1508/Host/FileTransferService.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.



After ensuring that the client basicHttpBinding config was correct, I ran the unit test once more. Now I got this error from VS2005 Cassini:

System.ServiceModel.ProtocolException: The remote server returned an unexpected response: (400) Bad Request.
System.Net.WebException: The remote server returned an error: (400) Bad Request.



I have inspected the HTTP traffic using Fiddler and can see nothing wrong with the MTOM encoded request. Googling lead me to a lot of frustrated "streamed" entries at the forums.microsoft.com "Indigo" group, but no solution to my problem. So I made a self-hosted service using a console application, using the code provided in the WCF samples (see below), and it worked like a charm. I expect that it is just VS2005 Cassini that does not support streamed transfers/MTOM encoding.

I then installed IIS to check whether HTTP streaming works better with IIS. The first error I ran into when uploading my 1KB test file was this:

System.ServiceModel.CommunicationException: The underlying connection was closed: The connection was closed unexpectedly.
System.Net.WebException: The underlying connection was closed: The connection was closed unexpectedly.



Inspecting the traffic with Fiddler, I found the HTTP request to be fine, but no HTTP response. Knowing that this is a sure sign of a server-side exception, I debugged the service operation. The error was caused by the IIS application pool identity (ASPNET worker process) lacking NTFS rights to store the uploaded file on disk. Note that WCF by default will not impersonate the caller, thus the identity used to run your service will need to have sufficient permissions on all resources that your service needs to access. This includes the temp folder, which is used to provide the WSDL file for HTTP GET.

As the UploadFile operation has [OperationContract(IsOneWay = true)], it cannot have a response and neither a fault contract. Thus, there will be no HTTP response when a server-side exception occurs, just the famous "
The underlying connection was closed" message.

I assigned the correct NTFS rights to my upload folder and re-ran the unit test: streamed upload to a WCF service hosted by IIS worked. The next unit test used a 43MB file to check if uploading data larger than the WCF buffer size works. The test gave me this error:



System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:59.8590000'.System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host



I was a bit puzzled as the file was less than the WCF maxReceivedMessageSize limit set to 64MB, and the stream read buffer was only 4KB as compared to the WCF maxBufferSize being 64KB. Knowing that IIS/ASP.NET has a system enforced limit on the size of a HTTP request to prevent denial-of-service attacks, which seems to be unrelated to WCF as I do not use the AspNetCompatibilityRequirements service setting, I decided to set the maxRequestLength limit to 64MB also (default: 4096KB). This time the "stream large file upload" test passed!

So for WCF streaming to IIS to work properly, you need this <system.web> config in addition to the <system.serviceModel> config in the IIS web.config file:


<!-- maxRequestLength (in KB) max size: 2048MB -->
<httpRuntime 
maxRequestLength="65536"
/>

Make the maximum HTTP request size equal to the WCF maximum request size.

I have to recommend the 'Service Trace Viewer' (SvcTraceViewer.EXE) and the 'Service Configuration Editor' (SvcConfigEditor.EXE) provided in the WCF toolbox (download .NET3 SDK). These tools make it really simple to add tracing and logging to your service for debugging and diagnosing errors. Start the WCF config editor, open your WCF config file, select the 'Diagnostics' node and just click "Enable tracing" in the "Tracing" section to turn on tracing. The figure shows some typical settings (click to enlarge):




Save the config file and run the unit test to invoke the service. This will generate the web_tracelog.svclog file to use as input for the trace viewer. Open the trace file in the trace viewer and look for the errors (red text). Click an error to see the exception details. The figure shows some of the details available for the "Maximum request length exceeded" error (click to enlarge):


The WCF tools installed by the SDK is located here:
C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin\

The code to process the incoming stream is rather simple, storing the uploaded files to a single folder on disk and overwriting any existing files:

public void UploadFile(FileUploadMessage request)
{
FileStream targetStream = null;
Stream sourceStream = request.FileByteStream;
 
string uploadFolder = @"C:\work\temp\";
string filename = request.FileMetaData.Filename;
string filePath = Path.Combine(uploadFolder, filename);
 
using (targetStream = new FileStream(filePath, FileMode.Create, 
                          FileAccess.Write, FileShare.None))
{
//read from the input stream in 4K chunks
//and save to output stream
const int bufferLen = 4096;
byte[] buffer = new byte[bufferLen];
int count = 0;
while ((count = sourceStream.Read(buffer, 0, bufferLen)) > 0)
{
targetStream.Write(buffer, 0, count);
}
targetStream.Close();
sourceStream.Close();
}
}

The 'Stream Sample' available on MSDN contains all the code you need to upload a file as a stream to a self-hosted WCF service and then save it to disk on the server by reading the stream in 4KB chunks. The download contains about 150 solutions with more than 4800 files, so there is a lot of stuff in it. Absolutely worth a look. The streaming sample is located here:
<unzip folder> \TechnologySamples\Basic\Contract\Service\Stream


Please let me know if you have been able to get HTTP streaming with MTOM encoding to work with Microsoft VS2005 Cassini.
Finally, a useful tip for streaming download: How to release streams and files using Disponse() on the the message contract.

37 comments:

Christian / info@critech.dk said...

Still no luck on getting it to work with cassini?

Kjell-Sverre Jerijærvi said...

I haven't really tried after making it work with IIS. Still no solution over at indigo@forums.microsoft.com either.

Markus Strobl said...

Hi!

Thanks for the information provided!

I'm currently messing around with a streamed WCF upload/download-service as well but ran into another problem:

after downloading a file the serverside file seems to stay locked by the client coz consecutive delete-operations on the serverfile fail!

I'm closing and disposing all streams in the client-side download-method.

Do you have any suggestions what i've missed?

Best wishes Markus

Kjell-Sverre Jerijærvi said...

Have you tried to close the client-side proxy to ensure that any WCF session is closed and released? Also, as this is download, have you closed and disposed of all stream/file objects on the server side?

Markus Strobl said...

Hi!

Thank you very much for the quick reply. Closing the client proxy didn't help eather because i found out that actually the server process locked the downloaded file because the FileStream passed to the client still stays open.

It's a little weird in my oppinion because i can't close it in my download-method as at this point the client hasn't received all of the data.

So what i did now to work around this behaviour is that i keep a list of Streams on the server-side and added a CloseDownload()-Method to my services which actually closes the server-side Stream.

If there might be a better way to solve this problem i'd love to hear any comments but it seems again that after scratching on the surface of microsoft's gifts there are always some funky inconsistencies showing up :)

Anyway ... Best wishes Markus

Anonymous said...

Thanks for the sample.
I try to accomplish similar thing for uploading files. However I found out that stream request does not support http authentication. I got error:
HTTP request streaming cannot be used in conjunction with HTTP authentication.
Either disable request streaming or specify anonymous HTTP authentication.
This almost kills my scenario. Do you have any suggestion on this issue?

Thanks!
Shirley

Kjell-Sverre Jerijærvi said...

Use HTTPS/SSL for transport security and send username+password (or a STS ticket) using message headers as shown in the post.

yannick said...

If you send username or password to auth a user before processing message using MessageContract and MessageBody Attributes, willn't you add to each SOAP message an extra info with this information, thus growing total size of the global message ?

Kjell-Sverre Jerijærvi said...

Well, you only add it to the FileStreamMessage and the lenght of the user name and password will normally be dwarfed by the amount of data in the file.

In addition, as you need a separate endpoint for streaming, you would use this type of authN and encryption only for streaming.

Anonymous said...

Excellent post. It helped me a lot.
Thank you!

Ksenia said...

Hi!

I'm just working on this and seem to be stuck... When hosting service in IIS I continue to get error:
The HTTP request is unauthorized with client authentication scheme 'Anonymous'. The authentication header received from the server was 'Negotiate,NTLM'.
As it is streaming scenario, I can't set ClientCredentialType to Ntlm. It seems that the problem is in Directory Security settings for virtual directory in IIS (Integrated Windows authentication is set, Anonymous access is enabled). Could you, please, help resolving the problem? Thanks in advance,
Ksenia

Kjell-Sverre Jerijærvi said...

Yes, the problem you have stems from the combination of IIS security and the WCF security not being compatible. Try turning off 'integrated windows authN' and use only anonymous. You'll find several helpful posts at the WCF (Indigo) forum.

Anonymous said...

Markus,

Have your [MessageContract]-decorated class implement IDisposable and close/dispose the stream in the Dispose method. The WCF libraries dispose your message object for you when it's done if it is IDisposable.

Kjell-Sverre Jerijærvi said...

No, I have not implemented IDisposable. Our project ended up not using WCF for uploads due to our Flex client, so I cannot give any guarantees for how the code behaves under heavy load. Check the MSDN samples to see if they have been updated.

Anonymous said...

Waht is the deal with manually edit the client config file to set transferMode = "Streamed"? Why is this necessary?

Ragnar Engnes said...

I have been struggeling with transferMode = "Streamed" on IIS7. The call returned whith a SendTimeout when I switched from buffered to streamed.
However, when I changed the Application Pool from DefaultAppPool to Classic .NET AppPool - it worked.

Kjell-Sverre Jerijærvi said...

See this post by Matevz Gacnik's blog on hosting under IIS7 and the new HTTP pipeline integration modes: http://www.request-response.com/blog/PermaLink,guid,3a99c75b-d108-4fb7-8d1d-8e18c32e659b.aspx

PRAVINTH G said...

Hi,
Thanks for the sample. i am trying to upload file from client to server.At client side i opened the file like File.OpenRead(..) so i got a FileStream and assigned it to the datacontract memeber of type stream. when i run the app. i am getting the error FileStreamnot serializable. What to do? Please give me your valuable suggestions.

Kjell-Sverre Jerijærvi said...

Strange, this should work fine:
using (System.IO.FileStream stream = new System.IO.FileStream(filename, System.IO.FileMode.Open, System.IO.FileAccess.Read))
{...}

roy.zhu said...

I was using vs2008 and .NET 3.5 to construct my service/client apps. The service is hosted in a non-UI thread, the client is console app. The service provides the only method, UploadData, with the only one input Stream type param. However, every time the client app complains, 'Specified method is not supported', no matter how to modify the config files.

It seems I have to use the buffered/chunk mode to transfer large data file.

Anonymous said...

How does this compare to BITS in terms of performance and efficency?

Kjell-Sverre Jerijærvi said...

At least the non-Vista BITS versions is very asymmetric with max one upload to server, as it is focus on downloading. In addition, to my knowledge, BITS should only use free bandwith to trickle the files over the network in the background. So the speed will depend on the amount of leftover capasity. The new peer-to-peer mechanism might alleviate these issues.

WCF streaming might be sensitive to config such as throttling, max connections, etc.

sekarsaravanan said...

Excellent post . Really helpful . The solution you suggested was pretty good and no unnecessary hacks and workaround.

Holt said...

This project has been a great help. I'm using it over netTCP with slightly better performance than HTTP.

mark said...

Thanks for the useful post, I used it to get straming uploads on IIS over http and https.

http://mark-csharp.blogspot.com/2009/01/wcf-file-transfer-streaming-chunking.html

Anonymous said...

Thanks for the sample!

I can't seem to stream the content of the file to the service on IIS 6.0. The message headers like length, filename, etc. can be sent from the client to the IIS server which has permission to create a new file, but the message body (file content) doesn't get sent. I verified this by checking the hash value of the content. I checked the service log, but there are no errors. I set the transferMode="Streamed" messageEncoding="Mtom"
maxReceivedMessageSize="67108864" on both client and server but still no luck. Do you have any suggestions?

Thanks!
-h

Kjell-Sverre Jerijærvi said...

Check with Fiddler that the content actually isn't sent. If it is, turn on WCF tracing and logging as shown in the article, as that is you only "debugging" option when no messages are dispatched to your operation.

ShinyHat said...

this made my week. you are awesome, thanks.

danang said...

Hi,
Thanks for the info.

I've created an upload app using streamed. it worked, the file successfully uploaded to the server.
But the problem is when your service is hosted in IIS, i don't think the stream is working.

I've tried to solve it based on this arcticle.
http://weblogs.asp.net/jclarknet/archive/2008/02/14/wcf-streaming-issue-under-iis.aspx
but somehow it's still not working.

Do you have an idea for this?

Kjell-Sverre Jerijærvi said...

I got it working on IIS6. For IIS7 you might need to use classic mode, see one of the comments above. Try the WCF forum to get support from Microsoft.

wizardnet said...

Man you saved my night, after I spent last one, I finally found your post , and hint to use the .Net SDK tools to trace the problem. My issue was that I have defined the binding correctly with all parameters but forget to link it with endpoint using bindingConfiguration attribute of the endpoint. The ws still used the default configuration for basicHttpBinding...

So thank you again for the hint with the tools, they are just great.

Anonymous said...

Excellent. This article helped me lot. Thank you.
- Raj.

Horacio said...

I know this post is kinda old but I'm completely frustrated and desperate to find a solution to my upload streaming WCF IIS hosted web service not working.
This is the first error I get:
Content Type multipart/related; type="application/xop+xml";start="";boundary="uuid:2a2af39d-17ab-46f9-a061-6fb42fe10c29+id=1";start-info="text/xml" was not supported by service http://localhost/streamWCF/streamTransferService.svc. The client and service bindings may be mismatched.

The second is:
The remote server returned an error: (415) Cannot process the message because the content type 'multipart/related; type="application/xop+xml";start="";boundary="uuid:2a2af39d-17ab-46f9-a061-6fb42fe10c29+id=1";start-info="text/xml"' was not the expected type 'text/xml; charset=utf-8'..

My bindings are the same in server and client, I'm using mtom and my trasnfermode is streamed.
At this point I could cry or laugh.
Any help would be highly appreciated.

Anonymous said...

Is it possible to stream more than 2 Gb files without going for chunking and hosting in IIS 7?

The httpruntime's maxRequestLength property is limiting the size upto 2Gb.

Kjell-Sverre Jerijærvi said...

I doubt that the request can be more than 2GB, so you must do streaming in smaller, throttled chuncks. Look into IIS7 media services: http://www.iis.net/overview/IntegratedMediaPlatform

Sathis said...

I have created a WCF service in My application, which Invoked by Silverlight App and Windows Application – ActiveX.
I am seeing different performance (response/request time) in Silverlight app and Windows App, the Silverlight app can sending a request (with image bytes array) below 4 sec, but the windows app is taking 8 to 12 sec to send request (with image bytes array).

What is reason for the delay in windows application and Is there any technique to resolve this issue, we will discuss in the call detail.

tugberk said...

I implemented this but get bad request error. I did everything by the book but still, bad request.