Managed File Transfer, RoboCopy, Sheduled Tasks

This spring I blogged about our research for a simple and inexpensive Managed File Transfer (MFT) solution. After reviewing the pros and cons of BITS for two-way robust high volume file transfers for a WAN scenario, RoboCopy was chosen due to its simplicity, configurability and logging capabilities.

Our MFT needs are covered by setting up a set of source-target pairs of UNC shares on the WAN and using the continuous monitoring mode of RoboCopy to move any files dropped in the source folders in a robust and reliable manner to the target folders. Use e.g. these RoboCopy options to do reliable transfers every five minutes when there is at least one new file:
ROBOCOPY.EXE . . . /Z /S /MOV /MON:1 /MOT:5
Refer to the documentation installed with RoboCopy to learn more about the different options.

I recommend making a command file (.CMD) that contains the RoboCopy jobs for each source-target folder pair. Note that when running in continuous monitoring mode RoboCopy will run "forever" and will not exit. Thus, you cannot just put multiple RoboCopy jobs into the command file, as the first job will block the command file, hindering all but the first process to start. Luckily, you can use the 'start' command to spawn/fork new processes:

start robocopy.exe \\mftserver001\uncshare1 \\dmserver001\uncshare1
start robocopy.exe \\dmserver001\uncshare2 \\mftserver001\uncshare2

We have used Scheduled Tasks to launch the RoboCopy processes instead of implementing a Windows Service. This makes configuring and running the jobs more accessible to the operations department as all they need to know is standard Scheduled Tasks (and saved me from implementing another Windows Service and installer). A few configuration tips:

• Use 'At system startup' to launch the command file only once and when the server (re)starts; i.e. like a service that has automatic startup
  
• Use 'Run As' to set the identity used to run the processes, i.e. the account that has the applicable NTFS and UNC share permissions
  
• Clear 'Run only if logged on' to ensure that the task is run like a service
  
• Clear the 'Stop the task if it runs for' option as the task should run "forever" in continuous monitoring mode
  

I strongly advice against running the command file using an account with admin rights. We use a limited account for the RoboCopy tasks, this identity has only the permissions needed to perform the managed file transfer.

Using a minimum permission set did of course cause an error when testing the RoboCopy task on a Windows Server 2003 machine: just running the command file worked fine, but running the scheduled task item got the status "Could not start'. To see the actual error of a task, use 'Advanced-View log' in the Scheduled Tasks main window and look for errors. You will typically find this error:

0x80070005: Access is denied.

I had granted the MFT identity read+execute rights on RoboCopy.exe and on the command file, so I was a bit puzzled. To make a long story short, Googling led me to "Access is denied" error message when you run a batch job on a Windows Server 2003-based computer. The problem was missing rights on the one part of the task that you do not think about: CMD.EXE itself.

Now everything was fine and dandy; and a restart of the server to simulate a server crash, followed by reviewing the processes using Task Manager showed the correct number of ROBOCOPY.EXE processes running in the background.

This MFT solution is truly based on the "Make everything as simple as possible, but not simpler" principle.

Managed File Transfer using BITS, dark horse is RoboCopy

My current project is considering using Microsoft BITS (Background Intelligent Transfer Service) for managed file transfer between several document management file servers around the world. BITS is already in use in this solution to allow customers to upload very large files in a robust and reliable manner.

As BITS is "designed for C and C++ developers", you will need to create some interop asseblies to be able to use BITS from a .NET application or service. I have not been able to find any official PIA provided by Microsoft. Luckily, the community already provides some very useful stuff:

• Managed BITS wrapper for v1, 2 and 3 by RodgerB: adjusts to which version that's installed and makes event handling really simple
• Basic BITS C# wrapper by Eddie Tse: includes support for the SetCredentials method needed for authentication

Most of the articles you'll find about BITS are about downloading files from the remote IIS file server, in addition to being a few years old. BITS is, however, also capable of uploading files to a remote IIS server that have BITS installed and configured. Read Using BITS to Upload Files with .NET by Phil Wilson to learn more.

I wish the new peer file transfer mechanism of v3 was an option for us, but it is limited to Vista and Windows Server "Longhorn". As our use of BITS is for internal managed file transfer between file servers, we have to stick with what's supported on WS2000 and WS2003.

Alas, RoboCopy might be sufficient for getting robust and reliable file transfer, so I'm looking into it's capabilities for bandwidth throttling, background transfers, and for instrumentation and monitoring. The robustness when faced with server restarts, and not just dropped LAN/WAN connections, is on top of my list of issues.

The main reason for considering RoboCopy as an alternative to BITS is that this is internal LAN/WAN managed file transfer and the need for installing IIS at the remote file servers when using BITS. As BITS operates on static files only and have some upload limitations (single file per job, job scalability, target IIS virtual directory, etc), RoboCopy seems to be a good alternative - especially for mass uploading of folders and files to remote locations. Download the Windows Resource Kit Tools to get RoboCopy, including documentation of the multitude of options. TechNet provides a RoboCopy GUI.

Speaking of operations and monitoring, you will need the BitsAdmin tool included in the WinXP Support Tools download. It's just a simple command line tool, but it lets you list and manage BITS jobs.