Managed File Transfer, RoboCopy, Sheduled Tasks

This spring I blogged about our research for a simple and inexpensive Managed File Transfer (MFT) solution. After reviewing the pros and cons of BITS for two-way robust high volume file transfers for a WAN scenario, RoboCopy was chosen due to its simplicity, configurability and logging capabilities.

Our MFT needs are covered by setting up a set of source-target pairs of UNC shares on the WAN and using the continuous monitoring mode of RoboCopy to move any files dropped in the source folders in a robust and reliable manner to the target folders. Use e.g. these RoboCopy options to do reliable transfers every five minutes when there is at least one new file:
ROBOCOPY.EXE . . . /Z /S /MOV /MON:1 /MOT:5
Refer to the documentation installed with RoboCopy to learn more about the different options.

I recommend making a command file (.CMD) that contains the RoboCopy jobs for each source-target folder pair. Note that when running in continuous monitoring mode RoboCopy will run "forever" and will not exit. Thus, you cannot just put multiple RoboCopy jobs into the command file, as the first job will block the command file, hindering all but the first process to start. Luckily, you can use the 'start' command to spawn/fork new processes:

start robocopy.exe \\mftserver001\uncshare1 \\dmserver001\uncshare1
start robocopy.exe \\dmserver001\uncshare2 \\mftserver001\uncshare2

We have used Scheduled Tasks to launch the RoboCopy processes instead of implementing a Windows Service. This makes configuring and running the jobs more accessible to the operations department as all they need to know is standard Scheduled Tasks (and saved me from implementing another Windows Service and installer). A few configuration tips:

• Use 'At system startup' to launch the command file only once and when the server (re)starts; i.e. like a service that has automatic startup
  
• Use 'Run As' to set the identity used to run the processes, i.e. the account that has the applicable NTFS and UNC share permissions
  
• Clear 'Run only if logged on' to ensure that the task is run like a service
  
• Clear the 'Stop the task if it runs for' option as the task should run "forever" in continuous monitoring mode
  

I strongly advice against running the command file using an account with admin rights. We use a limited account for the RoboCopy tasks, this identity has only the permissions needed to perform the managed file transfer.

Using a minimum permission set did of course cause an error when testing the RoboCopy task on a Windows Server 2003 machine: just running the command file worked fine, but running the scheduled task item got the status "Could not start'. To see the actual error of a task, use 'Advanced-View log' in the Scheduled Tasks main window and look for errors. You will typically find this error:

0x80070005: Access is denied.

I had granted the MFT identity read+execute rights on RoboCopy.exe and on the command file, so I was a bit puzzled. To make a long story short, Googling led me to "Access is denied" error message when you run a batch job on a Windows Server 2003-based computer. The problem was missing rights on the one part of the task that you do not think about: CMD.EXE itself.

Now everything was fine and dandy; and a restart of the server to simulate a server crash, followed by reviewing the processes using Task Manager showed the correct number of ROBOCOPY.EXE processes running in the background.

This MFT solution is truly based on the "Make everything as simple as possible, but not simpler" principle.