Today I was asked to look at a puzzling problem with TableAdapters at another project at my current customer. The problem they had was that they could not make the TableAdapter read back the generated SQL Server identity column values when saving a dataset (ta.Update) to insert new rows.
The first thing I checked was the 'Advanced options' of the TableAdapter to see if the 'Refresh the data table' setting was on. It wasn't. "But, we checked that option when we added the adapter". I checked the option and finished the wizard, then checked the setting again. It wasn't set...
I had noticed that details in the wizard summary page was rather short, so I did the wizard steps and inspected the summary again: "Generated SELECT statement" and "Generated INSERT statement", but no "Generated UPDATE statement" and no "Generated DELETE statement". I suspected that this was the cause of the problem, but what could cause the missing SQL commands ? The UpdateCommand and DeleteCommand of the TableAdapter's property window was empty. According to the online help, these statements "will be generated if there is enough information", but no hints to which information is needed. Really helpful.
Looking at the table definition in SQL Server, I then noticed that the table had no defined primary key. I selected the identity column and applied the "Set Primary Key" action and saved the table definition in SQL Server. After reconfiguring the TableAdapter in Visual Studio, the data table now has a defined primary key and - lo and behold - update/delete commands and refresh data on save.
Problem solved.
Wednesday, May 02, 2007
Monday, April 16, 2007
Managed File Transfer using BITS, dark horse is RoboCopy
My current project is considering using Microsoft BITS (Background Intelligent Transfer Service) for managed file transfer between several document management file servers around the world. BITS is already in use in this solution to allow customers to upload very large files in a robust and reliable manner.
As BITS is "designed for C and C++ developers", you will need to create some interop asseblies to be able to use BITS from a .NET application or service. I have not been able to find any official PIA provided by Microsoft. Luckily, the community already provides some very useful stuff:
I wish the new peer file transfer mechanism of v3 was an option for us, but it is limited to Vista and Windows Server "Longhorn". As our use of BITS is for internal managed file transfer between file servers, we have to stick with what's supported on WS2000 and WS2003.
Alas, RoboCopy might be sufficient for getting robust and reliable file transfer, so I'm looking into it's capabilities for bandwidth throttling, background transfers, and for instrumentation and monitoring. The robustness when faced with server restarts, and not just dropped LAN/WAN connections, is on top of my list of issues.
The main reason for considering RoboCopy as an alternative to BITS is that this is internal LAN/WAN managed file transfer and the need for installing IIS at the remote file servers when using BITS. As BITS operates on static files only and have some upload limitations (single file per job, job scalability, target IIS virtual directory, etc), RoboCopy seems to be a good alternative - especially for mass uploading of folders and files to remote locations. Download the Windows Resource Kit Tools to get RoboCopy, including documentation of the multitude of options. TechNet provides a RoboCopy GUI.
Speaking of operations and monitoring, you will need the BitsAdmin tool included in the WinXP Support Tools download. It's just a simple command line tool, but it lets you list and manage BITS jobs.
As BITS is "designed for C and C++ developers", you will need to create some interop asseblies to be able to use BITS from a .NET application or service. I have not been able to find any official PIA provided by Microsoft. Luckily, the community already provides some very useful stuff:
- Managed BITS wrapper for v1, 2 and 3 by RodgerB: adjusts to which version that's installed and makes event handling really simple
- Basic BITS C# wrapper by Eddie Tse: includes support for the SetCredentials method needed for authentication
I wish the new peer file transfer mechanism of v3 was an option for us, but it is limited to Vista and Windows Server "Longhorn". As our use of BITS is for internal managed file transfer between file servers, we have to stick with what's supported on WS2000 and WS2003.
Alas, RoboCopy might be sufficient for getting robust and reliable file transfer, so I'm looking into it's capabilities for bandwidth throttling, background transfers, and for instrumentation and monitoring. The robustness when faced with server restarts, and not just dropped LAN/WAN connections, is on top of my list of issues.
The main reason for considering RoboCopy as an alternative to BITS is that this is internal LAN/WAN managed file transfer and the need for installing IIS at the remote file servers when using BITS. As BITS operates on static files only and have some upload limitations (single file per job, job scalability, target IIS virtual directory, etc), RoboCopy seems to be a good alternative - especially for mass uploading of folders and files to remote locations. Download the Windows Resource Kit Tools to get RoboCopy, including documentation of the multitude of options. TechNet provides a RoboCopy GUI.
Speaking of operations and monitoring, you will need the BitsAdmin tool included in the WinXP Support Tools download. It's just a simple command line tool, but it lets you list and manage BITS jobs.
Tuesday, April 10, 2007
EntLib3 April 2007: still some VAB issues
It's been quiet around here the last month, as I've been "busy" finishing the March sprint of our current project - after skiing in Avoriaz, France for one week first. Today I'm back after a one week Easter vacation, skiing in Trysil, Norway.
The production version of EntLib 3 was released last week, and I have downloaded it and tested the validation application block with our WCF services. The first thing I noticed when recompiling was that a breaking change has been introduced in EntLib3 April 2007: the attribute parameters for specifying a custom validation message are gone from the attribute constructors. You now need to use named parameters to specify your message in a VAB attribute, e.g. MessageTemplate = "message" as shown here:
[RegexValidator(@"^.*$", RegexOptions.IgnoreCase, MessageTemplate = "Invalid ticket")]
When my code again compiled, I checked if some of the issues I've reported in the February CTP had been fixed. The most serious issue for my current project is validating recursive object collections, e.g. a structure of folders that can contain files and sub-folders.
The [ObjectCollectionValidator] issue has not been fixed, the NUnit tests still silently fails - caused by a stack overflow due to an infinite loop when building the validator tree for the recursive object structure.
The second issue is that the [IgnoreNulls] attribute should get a sibling [IgnoreNullOrEmpty] attribute for strings. This is especially important for validating WCF messages and data contracts; if an optional string element is not specified (null) in the message on-the-wire, the DataContractSerializer will deserialize the missing string element by applying the default value for strings: String.Empty. So, now the null value sent by the client is actually String.Empty when applying the validation rules on the server, and thus the [IgnoreNulls] rule is not sufficient for easily validating WCF messages.
Specifying a set of validation attributes to build a ruleset for covering "ignore if null or empty, otherwise string must be 5-10 chars long" using the [ValidatorComposition] attribute is not simple, thus I have logged the need for [IgnoreNullOrEmpty] as an issue.
The production version of EntLib 3 was released last week, and I have downloaded it and tested the validation application block with our WCF services. The first thing I noticed when recompiling was that a breaking change has been introduced in EntLib3 April 2007: the attribute parameters for specifying a custom validation message are gone from the attribute constructors. You now need to use named parameters to specify your message in a VAB attribute, e.g. MessageTemplate = "message" as shown here:
[RegexValidator(@"^.*$", RegexOptions.IgnoreCase, MessageTemplate = "Invalid ticket")]
When my code again compiled, I checked if some of the issues I've reported in the February CTP had been fixed. The most serious issue for my current project is validating recursive object collections, e.g. a structure of folders that can contain files and sub-folders.
The [ObjectCollectionValidator] issue has not been fixed, the NUnit tests still silently fails - caused by a stack overflow due to an infinite loop when building the validator tree for the recursive object structure.
The second issue is that the [IgnoreNulls] attribute should get a sibling [IgnoreNullOrEmpty] attribute for strings. This is especially important for validating WCF messages and data contracts; if an optional string element is not specified (null) in the message on-the-wire, the DataContractSerializer will deserialize the missing string element by applying the default value for strings: String.Empty. So, now the null value sent by the client is actually String.Empty when applying the validation rules on the server, and thus the [IgnoreNulls] rule is not sufficient for easily validating WCF messages.
Specifying a set of validation attributes to build a ruleset for covering "ignore if null or empty, otherwise string must be 5-10 chars long" using the [ValidatorComposition] attribute is not simple, thus I have logged the need for [IgnoreNullOrEmpty] as an issue.
Thursday, March 01, 2007
WCF: Validating Message and Data Contracts using VAB
Last year I wrote about how we used a 'broken rules' engine and the Noogen validation component for doing validation across all layers including the business logic and the WinForms client. Using a validation engine allows for better design and separation of concerns in your components, and makes the code easier to understand as the validation logic is clearly separated from your domain logic.
These days I have started to use the Validation application block (VAB) from the upcoming Enterprise Library 3 provided by the patterns & practices team at Microsoft. VAB works just like our custom 'broken rules' engine, it allows you to declaratively add validation rules as [attributes] to your objects instead of implementing the rules as code; and then passing an object instance to the Validation.Validate() method to interpret the declared rules and provide a list of those rules that are broken, i.e. the rules that the object instance does not comply with.
The nice feature provided by VAB is that you can actually completely separate the validation rules from your domain logic code by allowing you to declare the rules in an external XML config file instead of adding [attributes] on your domain objects. It is really nice as there is a VS2005 editor tool that allows you to create and edit the rules instead of hand-coding XML. The figure shows how easy the editor makes it to e.g. choose which objects and properties to validate (click to enlarge):
Remember to set the "default rule set" property for each type/class that you add. Forgetting to set it will give no warning, but neither will your rules be applied unless you actually specify a rule set name in the call to Validation.Validate() (more on this problem below).
By having the validation rules in config, you can actually modify the rules without having to recompile and redeploy your solution to keep up with the ever changing business requirements. Read more about this and other VAB features at David Hayden's blog. Also read his four part VAB introduction series to see some code.
I use VAB to validate the incoming message and data contracts of our WCF service. I have decorated all the message and data contracts with VAB attributes in addition to the WCF attributes. This allows for validating just only a single data contract or for validating the complete structure of a message including contained data contracts. The capability of validating an object graph in a single operation is what I love most about VAB. These are the object graph validators:
All messages derive from a common base class that ensures that all messages contain the same set of [MessageHeader] info required by our service. This allows us to validate all messages using a single method:
public static void ValidateMessage<T>(T message)
where T : DefaultMessage, new()
{
ValidationResults results = Validation.Validate(message);
//NOTE: DO NOT EXPOSE IMPLEMENTATION DETAILS
if(results.IsValid==false)
{
FaultContracts.DefaultFaultContract fault = new FaultContracts.DefaultFaultContract();
fault.ErrorId = (int)EApprovalErrorCode.InvalidMessage;
fault.ErrorMessage = Utilities.MakeDetailedErrorMessage(results);
throw new FaultException<DefaultFaultContract> (fault, new FaultReason(InvalidMessageReason));
}
}
Note that as VAB uses generics in the Validation.Validate(<T>) method, the code must ensure that type fidelity is not lost before calling the method. I.e. you must use a generic <T> type input and not the base class DefaultMessage in your message validator method input. The problem by using a base class type input is that even if you pass a derived class at run-time, the compile-time type will be used to deduce the validation rules - thus only the base class will get validated.
Using VAB gives us better control over the validity of the input than the DataContractSerializer is able to provide. E.g. it allows for checking string lengths and controlling that string elements are actually not just empty values; and validating e-mail addresses, guids, etc; rules that cannot be expressed using WCF attributes. These are some of the field level validators:
Validators can be composed into Boolean expressions using [AndCompositeValidator] and [OrCompositeValidator]. In addition, there are plans to provide cross-property validation in the final release of the validation application block. This will allow for comparing class members against each other instead of just literals, values sets and type/conversions. Sometimes, however, this is not sufficient for complex rules.
When your rules are too complex to be supported by the standard validators, VAB allows you to do self validation by providing [SelfValidation] methods:
[HasSelfValidation]
[MessageContract]
public class ImportProjectDocumentsMessage : DefaultMessage
{
[SelfValidation]
public void ValidateFileXorData(ValidationResults results)
{
//XOR operation
if ( (_fieldRootNode == null ^ String.IsNullOrEmpty(_fieldImportXmlFilename) ) == false)
{
results.AddResult(new ValidationResult("One of the parameters ProjectDocuments or ImportXmlFilename must be specified", this, "ImportProjectDocumentsMessage", null, null));
}
}
. . .
}
All classes containing self-validation must be marked with the [HasSelfValidation] attribute. VAB allows you to mix standard validators with self-validation. You just got to love this kind of flexibility.
Note that as there will be no exceptions if your validation rules are not interpreted by VAB for some reason (such as the above lost type fidelity problem, missing "default rule", etc), I recommend hard-coding one of the rules in your logic to ensure that such mishaps are detected during development and testing. This fail-safe can be enclosed in #if DEBUG so that it is not included in your final solution. Regardless, you should always have a unit-test expected to fail when validating; if the test does not fail, you know that the rules are not being interpreted. Use [ExpectedException] or Assert.IsFalse() to verify that invalid input actually causes the validation to fail.
Download the EntLib3 Feb2007 CTP beta including VAB from CodePlex. Read the documentation and check the 'quick start' samples to get started.
[UPDATE] Watch the 'Taking Advantage of the Enterprise Library in Your Site' web-cast for an introduction to EntLib3.
[UPDATE] Watch the 'New Capabilities in Enterprise Library 3.0' web-cast for even more details about the new features in EntLib3.
Another part of enterprise library 3 to look forward to, is the Policy Injection block. Read more about PIAB at Tom Hollander's blog. This looks good for adding cross-cutting concerns such as logging and exception handling to the domain logic without actually modifying the logic to add non-domain stuff. Alas, this is in fact just 'aspect oriented programming' (AOP) under a new name - read Patrik Löwendahl's comments and the response from the p&p team at his blog.
[UPDATE] Also check out the new WCF stuff in the Orcas March CTP: integration of WF and WCF with new activity types and a new WorkFlowServiceHost; plus JSON endcoding and webHttpBinding for better REST + POX/JSON support.
These days I have started to use the Validation application block (VAB) from the upcoming Enterprise Library 3 provided by the patterns & practices team at Microsoft. VAB works just like our custom 'broken rules' engine, it allows you to declaratively add validation rules as [attributes] to your objects instead of implementing the rules as code; and then passing an object instance to the Validation.Validate() method to interpret the declared rules and provide a list of those rules that are broken, i.e. the rules that the object instance does not comply with.
The nice feature provided by VAB is that you can actually completely separate the validation rules from your domain logic code by allowing you to declare the rules in an external XML config file instead of adding [attributes] on your domain objects. It is really nice as there is a VS2005 editor tool that allows you to create and edit the rules instead of hand-coding XML. The figure shows how easy the editor makes it to e.g. choose which objects and properties to validate (click to enlarge):
Remember to set the "default rule set" property for each type/class that you add. Forgetting to set it will give no warning, but neither will your rules be applied unless you actually specify a rule set name in the call to Validation.Validate() (more on this problem below).
By having the validation rules in config, you can actually modify the rules without having to recompile and redeploy your solution to keep up with the ever changing business requirements. Read more about this and other VAB features at David Hayden's blog. Also read his four part VAB introduction series to see some code.
I use VAB to validate the incoming message and data contracts of our WCF service. I have decorated all the message and data contracts with VAB attributes in addition to the WCF attributes. This allows for validating just only a single data contract or for validating the complete structure of a message including contained data contracts. The capability of validating an object graph in a single operation is what I love most about VAB. These are the object graph validators:
- [ObjectValidator] is for validating a composite child object of the current object
- [ObjectCollectionValidator(typeof(...))] is for validating a collection of child objects
All messages derive from a common base class that ensures that all messages contain the same set of [MessageHeader] info required by our service. This allows us to validate all messages using a single method:
public static void ValidateMessage<T>(T message)
where T : DefaultMessage, new()
{
ValidationResults results = Validation.Validate(message);
//NOTE: DO NOT EXPOSE IMPLEMENTATION DETAILS
if(results.IsValid==false)
{
FaultContracts.DefaultFaultContract fault = new FaultContracts.DefaultFaultContract();
fault.ErrorId = (int)EApprovalErrorCode.InvalidMessage;
fault.ErrorMessage = Utilities.MakeDetailedErrorMessage(results);
throw new FaultException<DefaultFaultContract> (fault, new FaultReason(InvalidMessageReason));
}
}
Note that as VAB uses generics in the Validation.Validate(<T>) method, the code must ensure that type fidelity is not lost before calling the method. I.e. you must use a generic <T> type input and not the base class DefaultMessage in your message validator method input. The problem by using a base class type input is that even if you pass a derived class at run-time, the compile-time type will be used to deduce the validation rules - thus only the base class will get validated.
Using VAB gives us better control over the validity of the input than the DataContractSerializer is able to provide. E.g. it allows for checking string lengths and controlling that string elements are actually not just empty values; and validating e-mail addresses, guids, etc; rules that cannot be expressed using WCF attributes. These are some of the field level validators:
- [NotNullValidator] ensures that required fields are not null; just got to love this, no more String.IsNullOrEmpty() checks in my domain logic
- [StringLengthValidator(min, max)] controls the string length; must be combined with [IgnoreNulls] for non-required strings; no more 'field would be truncated' exceptions
- [RegexValidator] validates the field against a regular expression; useful for validating e-mail addresses, phone numbers, guids, etc.
- A guid validator example, with optional { and }: [RegexValidator(@ "^(\{){0,1}
[0-9A-F]{8}\-[0-9A-F]{4}\-[0-9A-F]{4}\-[0-9A-F]{4}\-[0-9A-F]{12}
(\}){0,1}$", RegexOptions.IgnoreCase)] - E-mail validator for 99% of all the possible formats: [RegexValidator(@ "^[A-Z0-9._%-]+@(?:[A-Z0-9-]+\.)+[A-Z]{2,4}$", RegexOptions.IgnoreCase)]
Validators can be composed into Boolean expressions using [AndCompositeValidator] and [OrCompositeValidator]. In addition, there are plans to provide cross-property validation in the final release of the validation application block. This will allow for comparing class members against each other instead of just literals, values sets and type/conversions. Sometimes, however, this is not sufficient for complex rules.
When your rules are too complex to be supported by the standard validators, VAB allows you to do self validation by providing [SelfValidation] methods:
[HasSelfValidation]
[MessageContract]
public class ImportProjectDocumentsMessage : DefaultMessage
{
[SelfValidation]
public void ValidateFileXorData(ValidationResults results)
{
//XOR operation
if ( (_fieldRootNode == null ^ String.IsNullOrEmpty(_fieldImportXmlFilename) ) == false)
{
results.AddResult(new ValidationResult("One of the parameters ProjectDocuments or ImportXmlFilename must be specified", this, "ImportProjectDocumentsMessage", null, null));
}
}
. . .
}
All classes containing self-validation must be marked with the [HasSelfValidation] attribute. VAB allows you to mix standard validators with self-validation. You just got to love this kind of flexibility.
Note that as there will be no exceptions if your validation rules are not interpreted by VAB for some reason (such as the above lost type fidelity problem, missing "default rule", etc), I recommend hard-coding one of the rules in your logic to ensure that such mishaps are detected during development and testing. This fail-safe can be enclosed in #if DEBUG so that it is not included in your final solution. Regardless, you should always have a unit-test expected to fail when validating; if the test does not fail, you know that the rules are not being interpreted. Use [ExpectedException] or Assert.IsFalse() to verify that invalid input actually causes the validation to fail.
Download the EntLib3 Feb2007 CTP beta including VAB from CodePlex. Read the documentation and check the 'quick start' samples to get started.
[UPDATE] Watch the 'Taking Advantage of the Enterprise Library in Your Site' web-cast for an introduction to EntLib3.
[UPDATE] Watch the 'New Capabilities in Enterprise Library 3.0' web-cast for even more details about the new features in EntLib3.
Another part of enterprise library 3 to look forward to, is the Policy Injection block. Read more about PIAB at Tom Hollander's blog. This looks good for adding cross-cutting concerns such as logging and exception handling to the domain logic without actually modifying the logic to add non-domain stuff. Alas, this is in fact just 'aspect oriented programming' (AOP) under a new name - read Patrik Löwendahl's comments and the response from the p&p team at his blog.
[UPDATE] Also check out the new WCF stuff in the Orcas March CTP: integration of WF and WCF with new activity types and a new WorkFlowServiceHost; plus JSON endcoding and webHttpBinding for better REST + POX/JSON support.
Subscribe to:
Posts (Atom)