Managed File Transfer using BITS, dark horse is RoboCopy

My current project is considering using Microsoft BITS (Background Intelligent Transfer Service) for managed file transfer between several document management file servers around the world. BITS is already in use in this solution to allow customers to upload very large files in a robust and reliable manner.

As BITS is "designed for C and C++ developers", you will need to create some interop asseblies to be able to use BITS from a .NET application or service. I have not been able to find any official PIA provided by Microsoft. Luckily, the community already provides some very useful stuff:

• Managed BITS wrapper for v1, 2 and 3 by RodgerB: adjusts to which version that's installed and makes event handling really simple
• Basic BITS C# wrapper by Eddie Tse: includes support for the SetCredentials method needed for authentication

Most of the articles you'll find about BITS are about downloading files from the remote IIS file server, in addition to being a few years old. BITS is, however, also capable of uploading files to a remote IIS server that have BITS installed and configured. Read Using BITS to Upload Files with .NET by Phil Wilson to learn more.

I wish the new peer file transfer mechanism of v3 was an option for us, but it is limited to Vista and Windows Server "Longhorn". As our use of BITS is for internal managed file transfer between file servers, we have to stick with what's supported on WS2000 and WS2003.

Alas, RoboCopy might be sufficient for getting robust and reliable file transfer, so I'm looking into it's capabilities for bandwidth throttling, background transfers, and for instrumentation and monitoring. The robustness when faced with server restarts, and not just dropped LAN/WAN connections, is on top of my list of issues.

The main reason for considering RoboCopy as an alternative to BITS is that this is internal LAN/WAN managed file transfer and the need for installing IIS at the remote file servers when using BITS. As BITS operates on static files only and have some upload limitations (single file per job, job scalability, target IIS virtual directory, etc), RoboCopy seems to be a good alternative - especially for mass uploading of folders and files to remote locations. Download the Windows Resource Kit Tools to get RoboCopy, including documentation of the multitude of options. TechNet provides a RoboCopy GUI.

Speaking of operations and monitoring, you will need the BitsAdmin tool included in the WinXP Support Tools download. It's just a simple command line tool, but it lets you list and manage BITS jobs.

EntLib3 April 2007: still some VAB issues

It's been quiet around here the last month, as I've been "busy" finishing the March sprint of our current project - after skiing in Avoriaz, France for one week first. Today I'm back after a one week Easter vacation, skiing in Trysil, Norway.

The production version of EntLib 3 was released last week, and I have downloaded it and tested the validation application block with our WCF services. The first thing I noticed when recompiling was that a breaking change has been introduced in EntLib3 April 2007: the attribute parameters for specifying a custom validation message are gone from the attribute constructors. You now need to use named parameters to specify your message in a VAB attribute, e.g. MessageTemplate = "message" as shown here:

[RegexValidator(@"^.*$", RegexOptions.IgnoreCase, MessageTemplate = "Invalid ticket")]

When my code again compiled, I checked if some of the issues I've reported in the February CTP had been fixed. The most serious issue for my current project is validating recursive object collections, e.g. a structure of folders that can contain files and sub-folders.

The [ObjectCollectionValidator] issue has not been fixed, the NUnit tests still silently fails - caused by a stack overflow due to an infinite loop when building the validator tree for the recursive object structure.

The second issue is that the [IgnoreNulls] attribute should get a sibling [IgnoreNullOrEmpty] attribute for strings. This is especially important for validating WCF messages and data contracts; if an optional string element is not specified (null) in the message on-the-wire, the DataContractSerializer will deserialize the missing string element by applying the default value for strings: String.Empty. So, now the null value sent by the client is actually String.Empty when applying the validation rules on the server, and thus the [IgnoreNulls] rule is not sufficient for easily validating WCF messages.

Specifying a set of validation attributes to build a ruleset for covering "ignore if null or empty, otherwise string must be 5-10 chars long" using the [ValidatorComposition] attribute is not simple, thus I have logged the need for [IgnoreNullOrEmpty] as an issue.