WCF Queued Dual Router Updated

I've updated the source for the WCF Queued Dual HTTP Request Response Router to now support sending out-of-band fault and notification messages back to the consumer.

The added unified fault handling support for router and services is using ChannelFactory<T> were T is IOutputChannel to support generic one-way channel shapes. The same generic WCF channel shape is used for sending notification messages.

The messages sent over the IOutputChannel are created using Message CreateMessage(...) to keep things generic in the router and on the service end. The fault and notification messages on the consumer end are of course strongly typed.

Faults and notification messages are sent from service providers using a ThreadStart delegate to send messages on separate threads to avoid being blocked by ongoing processing. This code is in the RoutedResponseScope class.

The sample code also shows how to integration "unit" test async dual WCF request-response MEPs with static EventHandler<T> events using anonymous delegates and ManualResetEvent thread synchronization.

Download the router example code here. The code is provided 'as-is' with no warranties and confers no rights. This example requires Windows Process Activation Service (Vista/WinServer2008), MSMQ 4.0 and .NET 3.0 WCF Non-HTTP Activation to be installed.

WCF Queued Dual HTTP Request Response Router

There are a lot of examples available for how to make WCF-based publish/subscribe messaging solutions, but not very many thay provides a simple queued dual HTTP request-response router. IDesign provides the queued Response Service from Juval Löwy's book Programming WCF Services, which provides a feature rich set of WCF goodies. Recommended. I've used it in combination with Sasha Goldshtein's Generic Forwarding Router to create a dual channel queued router.

The router container uses HTTP endpoints externally to interact with consumers, and MSMQ queues internally to talk to the service providers. This gives fewer queues to manage and monitor, and the consumers need not know anything about MSMQ.

The message forwarding ChannelFactory is cached for best performance. Read more about Channel and ChannelFactory caching in Wenlong Dong's post Performance Improvement for WCF Client Proxy Creation in .NET 3.5 and Best Practices.

Some useful resources related to making this router work:

• MSDN mag articles "Building a WCF router" part 1 and part 2
• Article series "MSMQ, WCF and IIS: Getting them to play nice" part 1, part 2 and part 3
• MSMQ activation of WAS
• Poison message handling in MSMQ 4.0 (and in MSMQ 3.0)
• NetMsmqBinding dead letter queues
• Correctly disposing WCF channels: Dispose extension method
• Really disabling the Visual Studio WCF Service Host, look for the <FlavorProperties/AutoStart> element

Note the poorly documented requirement that the WAS activated service endpoint path must be reflected in the MSMQ queue name:

<add key="targetQueueName" value=".\private$\ServiceModelSamples/service.svc" />

<endpoint address=
"net.msmq://localhost/private/servicemodelsamples/service.svc"

Failure to adhere to this convention will prevent messages added to the queue from automatically activating the WAS-hosted WCF service. Also ensure that the WAS app-pool identity has access rights on the queues.

Download the router example code here. The code is provided 'as-is' with no warranties and confers no rights. This example requires Windows Process Activation Service (Vista/WinServer2008), MSMQ 4.0 and .NET 3.0 WCF Non-HTTP Activation to be installed.

WCF: Message Headers and XmlSerializerFormat

Sometimes you need to use the classic XmlSerializer due to interoperability or when doing schema first based on XSD contracts that contains e.g. XML attributes in complexTypes. I've used the [XmlSerializerFormat] switch on services many times without any problems, but recently I had to make use of a custom header - and that took me quite some time to get working.

This is the wire format of the header:

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" >
<s:Header>
<h:CorrelationContext xmlns:h="urn:QueuedPubSub.CorrelationContext" xmlns="urn:QueuedPubSub.CorrelationContext" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" >
<CorrelationId>c4e03aae-9501-46e9-bbb8-a9ddf6c4fe15</CorrelationId>
<FaultAddress>feil</FaultAddress>
<Priority>0</Priority>
<ResponseAddress>svar</ResponseAddress>
</h:CorrelationContext>
. . .

The WCF OperationContext provides access to the message headers:

CorrelationContext context = OperationContext.Current. IncomingMessageHeaders.GetHeader<CorrelationContext> (Constants.CorrelationContextHeaderName, Constants.CorrelationContextNamespace);

I had used [XmlElement] attributes to control the name and namespace of the [Serializable] class members, only to get this error:

'EndElement' 'CorrelationContext' from namespace 'urn:QueuedPubSub.CorrelationContext' is not expected. Expecting element 'CorrelationId'.

That really puzzled me. To make a long story short, the MessageHeaders class and GetHeader<T> method only supports serializers derived from XmlObjectSerializer, such as the DataContractSerializer - but not XmlSerializer. To make this work, your header class must be implemented as a [DataContract] class even for [XmlSerializerFormat] services.

My working message header contracts looks like this:

[DataContract(Name = Constants.CorrelationContextHeaderName, Namespace = Constants.CorrelationContextNamespace)]
public class CorrelationContext
{
[DataMember]
public Guid CorrelationId { get; set; }
[DataMember]
public string FaultAddress { get; set; }
[DataMember]
public int Priority { get; set; }
[DataMember]
public string ResponseAddress { get; set; }
}


[MessageContract(IsWrapped = true)]
[Serializable]
public abstract class MessageWithCorrelationHeaderBase
{
[MessageHeader(MustUnderstand = true, Name = Constants.CorrelationContextHeaderName, Namespace = Constants.CorrelationContextNamespace)]
[XmlElement(ElementName = Constants.CorrelationContextHeaderName, Namespace = Constants.CorrelationContextNamespace)]
public CorrelationContext CorrelationContext { get; set; }
}

This code was made and tested on .NET 3.5 SP1.

Service Compatibility - A Primer

In a comment on the InfoQ article Contract Versioning, Compatibility and Composability about my definition of service forwards and backwards compatibility, the problem of talking about compatibility of services compared to the definition of schema compatibility is acknowledged.

The "problem" is that a service version that is compatible with the specifications of older versions of the service, can be achieved using both backwards and forwards compatible schemas. That is correct, but doesn't preclude having a definition of forwards and backwards compatibility for service providers, a.k.a "services". Service compatibility is based on ability to validate a message, it is not based on using wildcards in the schema definition. For a definition of the three types of forward compatibility, see my post on schema, service and routing compatibility.

Seen only from the service provider perspective, how it handles incoming messages is what defines if a service is forwards or backwards compatible (or both). How consumers handle messages sent by the service is really not of any concern for the provider - wait, read on.

Thinking about this within SOAP 1.x constraints, where a WSDL operation has a request message and a response message with fixed schema definitions/version (unilateral contracts), will lead to the conclusion that operations cannot be classified as forwards or backwards compatible, only the message schemas. This is a limitation of SOAP, but not of messaging in general.

In the following examples, the v1.2 service provider is backwards compatible and interacts with a v1.1 consumer. However, the schemas are not designed to be forwards compatible - they do not support XML extensibility (schema wildcards). In this scenario, the consumer can do either XSD validation of response messages against the v1.1 schema, or do 'validation by projection' of response messages - i.e. do "ignore unkown" validation. Doing 'validation by projection' is a recommended practice for compatibility and really simplifies building SOA solutions - this is also how WCF validates messages. So how to handle consumers that only do XSD validation, without relying on schema wildcards?

In REST, the consumer can put an "accept formats" header in the v1.1 request message, and the service provider can then respond with a v1.1 schema even if the service version is v1.2. The service provider adheres to it's obligations by being backwards compatible, and the consumer is allowed to express it's version expectation - the service has bilateral contracts.

Service Virtualization is a mechanism that can help with service versioning. The task of such an abstract endpoint intermediary is to handle versioning through both service compatibility and schema compatibility. A virtual service supports multiple versions of the service on the same endpoint, and must be capable of processing older requests. The virtual endpoint must have a mechanism that allows for the latest major v1.2 provider to handle v1.1 consumers. The intermediary mediates between the schema versions by transforming or projecting/enriching the messages as needed.

Back to the example, the service provider v1.2 response message can be stripped down to a v1.1 message by the intermediary as it is sent back to the consumer. The net effect is that the service has virtual bilateral contracts.

In messaging in general, by definition there are no duplex channels, only one-way channels (see Enterprise Integration Patterns by Hohpe/Woolf). On top of this, you can have a logical two-way channel for message exchange patterns such as request-response, specified using a "reply-to" address and a "reply-format" (bilateral contracts). The message compatibility is defined by the schema constructs, but just as in REST, the version of the incoming message does not dictate the version of the response message. It is the implementation of the endpoint that processes the messages that defines the compatibility policy of the endpoint, not the schemas.

So, service compatibility do not require using forwards compatible schemas in addition to backwards compatible schemas. The message validation policy is what defines service compatibility.

It is of course much simpler to just have a service compatibility policy based on that the schemas used in the services must be both forwards and backwards compatible - as shown in the "SOAP-style unilateral contracts" service compatibility figures.



Click figures to enlarge.

This way, the service provider or consumer platform need not handle "request-format" and "reply-format" that have different versions. In such a unilateral schema compatibility policy world, services are just intrinsically compatible through schema compatibility.

Service+Schema Versioning: Flexible/Strict Strategy

In a few weeks time I will be giving a session at NNUG on SOA service and schema versioning strategies and practices. A central topic will be schema compatibility rules, where I will recommend creating a policy based on the "Strict", "Flexible" and "Loose" versioning strategies described in chapter 20.4 in the latest Thomas Erl series book Web Service Contract Design and Versioning for SOA. I guess David Orchard is the author/editor of part III in the book.

I recommend using a “Flexible/Strict” compatibility policy:

• Flexible: Safe changes to schemas are backwards compatible and cause just a point version


• Strict: All unsafe schema changes must cause a new schema version and thus a new service version


• Do not require forwards compatible schemas (Loose, wildcard schemas) - schemas should be designed for extensibility, not to avoid versioning


• Service interfaces should also have a Flexible/Strict policy

Safe changes is typically adding to schemas, while unsafe changes are typically modifying or removing schema components.

Note that forwards compatible schemas is not required to have forwards compatible services, as service compatibility is defined by the ability to validate messages. WCF uses a variant of 'validation by projection' (ignore unknown) for forwards compatibility, but also supports schema wildcards.


As Nicolai M. Josuttis shows in the book SOA in Practice (chapter 12.2.1 Trivial Domain-Driven Versioning), even simple backwards compatible changes might cause unpredicted side effects such as response times breaking SLAs and causing problems for consumers. It is much safer to provide a new service version with the new schema version, as if there is a problem, only the upgraded consumers that required the change will be affected.

Note that even adding backwards compatible schema components can be risky, but adding is typically safe. Josuttis recommends using "Strict" as it is a very simple and explicit policy, but I prefer "Flexible/Strict" as this gives more flexibility and less service versions to govern.

Avoid trying to implement some smart automagical mechanism for handling schema version issues in the service logic. Rather use backwards compatibility, explicit schema versions and support multiple active service versions. In addition, consider applying a service virtualization mechanism.

Enterprise-Level Business Concepts

Dan North has expanded on Arjen Poutsma's SOA 'holiday request form' analogy in his post 'A Low-Tech Approach to Understanding SOA'. Applying the paper form analogy to design the interactions and messages of your business processes is an approach that I really recommend.

What got my attention was the recommendation at the end of the post in the 'Avoid a universal domain model' section: Do not introduce an “enterprise information architecture” / “universal data dictionary” trying to force everyone to use the same domain model. Instead, introduce business concepts. This is effectively the higher-level, ubiquitous language that ties together all of the finer-grained domain models behind each service. The services use the enterprise-level business concepts when interacting, which decouples the service consumer from the service provider and allows them to evolve independently.

This is spot on with my advice for making a business process business process information model (BPIM) and avoid enforcing a "enterprise data model" or "canonical schemas" across your services. Not only does BPIM allow for loosely-coupled, evolvable services; it also allows for the services to be semantic covenants which is important for service composition.

Semantic Covenant: The Service is Always Right

This post is about service semantics and how to be “liberal in what you accept” in your service-oriented architecture implementation. Let me start by defining that in this post the term "service" is used for a specific business capability that can be composed and reused in different business processes. I.e. the idiom “service” used here is representing a building block business capability. This is not to say that a service equals a single business process step, it just makes it easier to talk about the topic of this post (which is not BPM): semantic covenants.

A service has a defined contract, but also some implied semantic. The semantic is hard to express through the contract, and the business compositions that utilizes (consumes) the operation can only assume that the service is a covenant, i.e. that the service will do B when provided with input A. Having semantic coupling is just another type of coupling that prevents services to be truly loosely-coupled and that will cause a breaking ripple effect through all consumers of the service when the inevitable thing happens: change.

There is some thinking in the SOA community that to avoid this ripple effect caused by a service contract or semantic change, it is up to the service to do the right thing independent of the information provided by the sender of the message. That is, that the service should behave as if the consumer is always right, and adapt its semantic to suit the sender.

A service can do no such thing, as it is an actual representation of a specific business capability. The capability does what it does, i.e. a credit check is just that – a credit check. It must be up to the business composition of the services to interpret the real-life events that have occurred and their context, and then invoke the right set of operations. Note that the service should still be liberal in what it accepts (message duck-typing), as it is the business event that is important, not the format of the event message.

It is easier to recognize that the service is always right when you think of messages as representing business events and apply EDA thinking such as publish/subscribe instead of a habitual SOA command and control pattern. A service will then subscribe to business events that must trigger the business capability, but the service need not know who the publisher is. Neither need the publisher know who is listening for the business process state changed events (event topics). Thinking in service “consumers” is very connected to having a request/reply, command and control style SOA. 

To simplify the “service is always right” notion, let us call the publisher a “consumer”. The consumer can send any message (business process event, context and state; mail-order analogy) it likes, as it is the publisher. However, as the consumer does not know who the subscribers are, it cannot know or depend on the services having specific semantic. The service cannot depend on who the publisher is, it just accepts that a business event happened that it must process. A business event message never specifies any handling operations, the logic to decide how to handle the event is part of the mechanism that routes messages to services. This ensures that the autonomous and loose coupling tenets of SOA are fulfilled.

When you apply EDA style thinking to service orientation, you’ll see that the service is always right. There cannot be any traditional “the consumer is always right” in relation to service semantic, as the services have no request/reply consumers.

Alas, still the consumer can still be right; it is just the responsibility of the process composition mechanism to accept what ever messages (events) the consumer (publisher) throws at it and invoke the applicable service operations. It is the business process composition mechanism that has to be the semantic covenant.

PS! Note that I have deliberately used the term "composition" here and not even mentioned using an ESB or BPM to implement this. The semantic covenant composition mechanism can simply be implemented as a mashup (composite application) instead of a composite service or an orchestration/saga.

WCF: Exception Handling with Logging

In my 'WCF Exception Shielding using Generics' post there was a "//TODO: add logging as applicable" comment, which I now have implemented using the EntLib3 Logging Application Block.

As part of the error information I wanted to write to the Windows Application Event Log was the actual request message causing the service operation exception. Thus, I googled for a solution for how to get the XML of the incoming message, so that I could log it along with the stack trace. There are a lot of options for getting at the message in WCF:

• Write a custom message interceptor at the channel level
• Use the Message class directly in the service operations definitions
• Use CreateMessage internally to get at the Message class methods to reproduce the behind-the-scenes stuff of WCF

The problem of the first option is that I only want to get at the message XML when handling exceptions. The second option is not an option (duh!) as I want my operations to be strongly typed using my data contracts. The third option was more like what I wanted, but it seemed to be unneccessary complicated code.

The code to get at the message XML from within an operation is quite simple:

private static string GetMessageXml(DefaultMessage request)
{
return OperationContext.Current. RequestContext.RequestMessage.ToString();
}

This approach does not work for operations with no RequestContext, such as one-way operations.

The logging code is added after creating the fault details, this makes it possible to use fault details data in the log. This is how my WCF exception shielding with logging looks like:

public static FaultException<T> NewFaultException<T>(Exception ex)
where T : FaultContracts.DefaultFaultContract, new()
{
return NewFaultException<T>(ex, null);
}

public static FaultException<T> NewFaultException<T>(Exception ex, DefaultMessage request)
where T : FaultContracts.DefaultFaultContract, new()
{
return CreateNewFaultException<T>(ex, request);
}

private static FaultException<T> CreateNewFaultException<T>(Exception ex, DefaultMessage request)
where T : FaultContracts.DefaultFaultContract, new()
{
StackFrame stackFrame = new StackFrame(2);
string methodName = stackFrame.GetMethod().Name;

T fault = new T();
Type type = fault.GetType();
type.InvokeMember("SetFaultDetails", BindingFlags.Public BindingFlags.Instance BindingFlags.InvokeMethod, null, fault, new object[] { ex });

string msg = String.Format("An unexpected error occurred in {0}.", methodName);
Dictionary<string, object> props = new Dictionary<string, object>();
props.Add("stackTrace", ex.ToString());
if(request!=null)
{
string xml = GetMessageXml(request);
props.Add("requestMessage", xml);
}
Logger.Write(msg, Category.General, Priority.Highest, (int)EventLogId.Exception, TraceEventType.Error, ex.Message, props);

string reasonText = methodName + " failed, check detailed fault information.";

#if DEBUG
reasonText = methodName + " error: " + ex.ToString();
#endif

return new FaultException<T>(fault, new FaultReason(reasonText));
}

Note that I now use an inner private method to create the new fault to be thrown and thus must skip two stack frames to get the details of the original method called. The DEBUG section is there to make it simple to see full exception details in NUnit or with Fiddler when debugging the service. This DEBUG code must never make it into release builds, as this is not exactly exception shielding.
I have not looked into the WCF app-blocks integration mechanisms added in the april release of EntLib3, that is next on my task list for quieter days. There is not much info about this to be found, but Guy Burstein provides some samples. Btw, you should check out the Patterns and Practices Guidance site, it contains useful stuff and tutorials about EntLib3.

WCF: Validating Message and Data Contracts using VAB

Last year I wrote about how we used a 'broken rules' engine and the Noogen validation component for doing validation across all layers including the business logic and the WinForms client. Using a validation engine allows for better design and separation of concerns in your components, and makes the code easier to understand as the validation logic is clearly separated from your domain logic.

These days I have started to use the Validation application block (VAB) from the upcoming Enterprise Library 3 provided by the patterns & practices team at Microsoft. VAB works just like our custom 'broken rules' engine, it allows you to declaratively add validation rules as [attributes] to your objects instead of implementing the rules as code; and then passing an object instance to the Validation.Validate() method to interpret the declared rules and provide a list of those rules that are broken, i.e. the rules that the object instance does not comply with.

The nice feature provided by VAB is that you can actually completely separate the validation rules from your domain logic code by allowing you to declare the rules in an external XML config file instead of adding [attributes] on your domain objects. It is really nice as there is a VS2005 editor tool that allows you to create and edit the rules instead of hand-coding XML. The figure shows how easy the editor makes it to e.g. choose which objects and properties to validate (click to enlarge):


Remember to set the "default rule set" property for each type/class that you add. Forgetting to set it will give no warning, but neither will your rules be applied unless you actually specify a rule set name in the call to Validation.Validate() (more on this problem below).

By having the validation rules in config, you can actually modify the rules without having to recompile and redeploy your solution to keep up with the ever changing business requirements. Read more about this and other VAB features at David Hayden's blog. Also read his four part VAB introduction series to see some code.

I use VAB to validate the incoming message and data contracts of our WCF service. I have decorated all the message and data contracts with VAB attributes in addition to the WCF attributes. This allows for validating just only a single data contract or for validating the complete structure of a message including contained data contracts. The capability of validating an object graph in a single operation is what I love most about VAB. These are the object graph validators:

• [ObjectValidator] is for validating a composite child object of the current object
  
• [ObjectCollectionValidator(typeof(...))] is for validating a collection of child objects
  

Combine the object validators with the [NotNullValidator] to make the composite objects required. Note that there seems to be a bug in the current VAB beta when using the collection validator on recursive structures (e.g. folders containing folders), your app-domain will just silently die. [UPDATE] More about this bug here.

All messages derive from a common base class that ensures that all messages contain the same set of [MessageHeader] info required by our service. This allows us to validate all messages using a single method:

public static void ValidateMessage<T>(T message)
where T : DefaultMessage, new()
{
ValidationResults results = Validation.Validate(message);

//NOTE: DO NOT EXPOSE IMPLEMENTATION DETAILS
if(results.IsValid==false)
{
FaultContracts.DefaultFaultContract fault = new FaultContracts.DefaultFaultContract();

fault.ErrorId = (int)EApprovalErrorCode.InvalidMessage;

fault.ErrorMessage = Utilities.MakeDetailedErrorMessage(results);

throw new FaultException<DefaultFaultContract> (fault, new FaultReason(InvalidMessageReason));
}
}

Note that as VAB uses generics in the Validation.Validate(<T>) method, the code must ensure that type fidelity is not lost before calling the method. I.e. you must use a generic <T> type input and not the base class DefaultMessage in your message validator method input. The problem by using a base class type input is that even if you pass a derived class at run-time, the compile-time type will be used to deduce the validation rules - thus only the base class will get validated.

Using VAB gives us better control over the validity of the input than the DataContractSerializer is able to provide. E.g. it allows for checking string lengths and controlling that string elements are actually not just empty values; and validating e-mail addresses, guids, etc; rules that cannot be expressed using WCF attributes. These are some of the field level validators:

• [NotNullValidator] ensures that required fields are not null; just got to love this, no more String.IsNullOrEmpty() checks in my domain logic
  
• [StringLengthValidator(min, max)] controls the string length; must be combined with [IgnoreNulls] for non-required strings; no more 'field would be truncated' exceptions
  
• [RegexValidator] validates the field against a regular expression; useful for validating e-mail addresses, phone numbers, guids, etc.
• A guid validator example, with optional { and }: [RegexValidator(@ "^(\{){0,1}
  [0-9A-F]{8}\-[0-9A-F]{4}\-[0-9A-F]{4}\-[0-9A-F]{4}\-[0-9A-F]{12}
  (\}){0,1}$", RegexOptions.IgnoreCase)]
• E-mail validator for 99% of all the possible formats: [RegexValidator(@ "^[A-Z0-9._%-]+@(?:[A-Z0-9-]+\.)+[A-Z]{2,4}$", RegexOptions.IgnoreCase)]
  

Note that you must apply the validators to public class members only; validators on private class members will not be interpreted, and neither will they cause an exception.

Validators can be composed into Boolean expressions using [AndCompositeValidator] and [OrCompositeValidator]. In addition, there are plans to provide cross-property validation in the final release of the validation application block. This will allow for comparing class members against each other instead of just literals, values sets and type/conversions. Sometimes, however, this is not sufficient for complex rules.

When your rules are too complex to be supported by the standard validators, VAB allows you to do self validation by providing [SelfValidation] methods:

[HasSelfValidation]
[MessageContract]
public class ImportProjectDocumentsMessage : DefaultMessage
{
[SelfValidation]
public void ValidateFileXorData(ValidationResults results)
{
//XOR operation
if ( (_fieldRootNode == null ^ String.IsNullOrEmpty(_fieldImportXmlFilename) ) == false)
{
results.AddResult(new ValidationResult("One of the parameters ProjectDocuments or ImportXmlFilename must be specified", this, "ImportProjectDocumentsMessage", null, null));
}
}
. . .
}

All classes containing self-validation must be marked with the [HasSelfValidation] attribute. VAB allows you to mix standard validators with self-validation. You just got to love this kind of flexibility.

Note that as there will be no exceptions if your validation rules are not interpreted by VAB for some reason (such as the above lost type fidelity problem, missing "default rule", etc), I recommend hard-coding one of the rules in your logic to ensure that such mishaps are detected during development and testing. This fail-safe can be enclosed in #if DEBUG so that it is not included in your final solution. Regardless, you should always have a unit-test expected to fail when validating; if the test does not fail, you know that the rules are not being interpreted. Use [ExpectedException] or Assert.IsFalse() to verify that invalid input actually causes the validation to fail.

Download the EntLib3 Feb2007 CTP beta including VAB from CodePlex. Read the documentation and check the 'quick start' samples to get started.

[UPDATE] Watch the 'Taking Advantage of the Enterprise Library in Your Site' web-cast for an introduction to EntLib3.

[UPDATE] Watch the 'New Capabilities in Enterprise Library 3.0' web-cast for even more details about the new features in EntLib3.

Another part of enterprise library 3 to look forward to, is the Policy Injection block. Read more about PIAB at Tom Hollander's blog. This looks good for adding cross-cutting concerns such as logging and exception handling to the domain logic without actually modifying the logic to add non-domain stuff. Alas, this is in fact just 'aspect oriented programming' (AOP) under a new name - read Patrik Löwendahl's comments and the response from the p&p team at his blog.

[UPDATE] Also check out the new WCF stuff in the Orcas March CTP: integration of WF and WCF with new activity types and a new WorkFlowServiceHost; plus JSON endcoding and webHttpBinding for better REST + POX/JSON support.

WCF: Importing InfoPath forms to Data Contracts

The solution that we are implementing includes a WCF service that allows a set of document metadata to be imported into our system. The operation is ImportProjectDocuments( ImportProjectDocumentsMessage request) and the message contains the ProjectDocuments data contract specifying the document structure. The data contract of course adhers to my 'separate structure from data' rule. The import operation is intended for application-to-application layer usage by our e-biz partners, for connecting our systems in a service-oriented distributed architecture.

Then came the need for allowing users to manually enter the document structure in a disconnected manner and submitting it to our service later on; i.e. a human-to-application layer service.

Enter InfoPath 2003 as the occasionally connected client (OCC), a perfect match for our existing data contract (see closing note) and a nice data entry application. Note that InfoPath is not used to submit the entered data directly to the WCF service endpoint, just to fill out a form and submitting it to our server as XML files - by e-mail or to a SharePoint form library when connected. You may think of the form library as the human friendly "message queue".

By the way - having a OCC service consumer is a rather good assessment of whether your services adheres to SOA best practices such as explicit boundaries, message based, share contract, and self-contained business event operations based on the paper form metaphor.

The submitted InfoPath forms must then be processed, typically by a workflow system such as K2.NET or even WWF, monitoring the form library. Each XML form must taken off the "message queue", deserialized using the DataContractSerializer and passed to our service, invoking the correct operation for the submitted XML data.

This is all quite trivial, but even if the XML generated by InfoPath adheres to the XSD schema, InfoPath adds some processing instructions that the deserializer chokes on:

System.Runtime.Serialization.SerializationException:
There was an error deserializing the object of type DNVS.DNVX.eApproval.DataContracts.ProjectDocuments.
Processing instructions (other than the XML declaration) and DTDs are not supported.

You must remove the InfoPath processing instructions before deserialization, and the easiest way to do this is by using a standard XmlTextReader instead of the WCF XmlDictionaryReader used in the MSDN documentation:

[Test]
public void DeserializeDataContractFromInfoPathXmlFileTest()
{
string fileName = @".\form1.xml";
ProjectDocuments documents = null;

DataContractSerializer xlizer = new DataContractSerializer(typeof(ProjectDocuments));

FileStream fs = new FileStream(fileName, FileMode.Open);
//NOTE: must get rid of InfoPath processing instructions
//XmlDictionaryReader reader = XmlDictionaryReader.CreateTextReader(fs, new XmlDictionaryReaderQuotas());

XmlTextReader reader = new XmlTextReader(fs);

documents = (ProjectDocuments) xlizer.ReadObject(reader, true);

reader.Close();
fs.Close();

Assert.IsNotNull(documents, "Could not deserialize the project documents XML");
}


The ReadObject() method is what actually converts the XML into a WCF data contract instance, ready for processing by the service layer just as if it had arrived through the WCF endpoint.

Note that the duration data type defined in the XSD schemas generated by WCF will cause an InfoPath 2003 parse error. You need to remove it before you start to design a new InfoPath form based on the data contract XSD schemas.

WCF Security: wsHttpBinding with brokered Kerberos

Now that I have implemented a separate ASMX endpoint for interoperability with Flex 2, it was time to switch from basicHttpBinding to wsHttpBinding for better security (authentication, encryption, signing). Using this binding will provide message level security, preventing e.g. HTTP spies on the client from snooping the data exchange with our services.

Editing the WCF client and server configuration files is a quite daunting task, but luckily the new version of WSSF offers nice wizards for configuring security for your services. The 'WCF security' menu allows you to easily add support for the most common providers: client X.509 certificates, Kerberos, ADAM, SQL Server, ActiveDirectory, server certificate.


Among the available providers, the Kerberos provider is the simplest to use if you don't want to use a certificate nor HTTPS/SSL, or you want/has to use Cassini (the VS2005 developer web-server), as shown in the figure (click to enlarge):

The wizard is straightforward, just remember to include metadata (MEX) in your service if you want to expose a WSDL file. Adding a list of authorized clients is not mandatory for this security mode. I recommend that you delete all content in the <system.serviceModel> element in the service config file before using the 'WCF Security' menu. This ensures a clean configuration settings section afterwards.

I then updated the service reference of our SmartClient application to get the correct settings for the client config file. This step is important, as the common configuration settings must be equal on both client and server for WCF communication to work.

With all the configuration finished, it was time for some testing. I got this error when invoking a WCF operation:

System.ServiceModel.Security.MessageSecurityException: The token provider cannot get tokens for target 'http://localhost.:1508/eApproval.Host/ProjectDocumentServices.svc'. ---> System.IdentityModel.Tokens.SecurityTokenValidationException: The NetworkCredentials provided were unable to create a Kerberos credential, see inner execption for details. ---> System.IdentityModel.Tokens.SecurityTokenException: InitializeSecurityContent failed. Ensure the service principal name is correct. ---> System.ComponentModel.Win32Exception: No credentials are available in the security package
--- End of inner exception stack trace ---

My client config file contained these security settings, which to me looked fine and dandy:

<security mode="Message">
<transport clientCredentialType="Windows" proxyCredentialType="None" realm="" />
<message clientCredentialType="Windows" negotiateServiceCredential="false" algorithmSuite="Default" establishSecurityContext="false" />
</security>
My suspects were the extra <message> element attributes not covered in Keith Brown's "Security in WCF" article: negotiateServiceCredential and establishSecurityContext. As usual with new Microsoft technology, the online help is rather lame, so some research was required.

Googling lead me to this excellent article by Michèle Leroux Bustamante: "Fundamentals of WCF Security" (6 pages). The 'Service Credentials and Negotiation' section contained the explanation for my error: "When negotiation is disabled for Windows client credentials, a Kerberos domain must exist". So by turning service identity negotiation on in both the client and the server config files, my client can now communicate securely with our WCF service. I also recommend turning on the 'secure session' context for better performance, unless your service is rarely used by the client.

A confusing part of the client side authentication settings for the 'Windows' security mode is the <endpoint/identity> element, which will typically contain your "user principal name" (UPN) in the <userPrincipalName> element when hosting with Cassini or self-hosting. The identity setting is not for specifying who the client user is, but for identityfying and authenticating the service itself. If you use IIS as the host, the identity must contain the "service principal name" (SPN). Note that if your IIS app pool identity is not NETWORK SERVICE, you must typically create the SPN explicitly.
The <identity> element is only used with NTLM or negotiated security. If you use non-negotiated security such as direct Kerberos, this element is not supported.

The security config now looks like this (server side settings):

<bindings><wsHttpBinding>
<binding name="BrokeredAuthenticationKerberosBinding">
<security mode="Message"> <message clientCredentialType="Windows" negotiateServiceCredential="true" establishSecurityContext="true" />
</security>
</binding>
</wsHttpBinding>
</bindings>

Checking the on-the-wire format with Fiddler shows that the request and response messages now are encrypted; without using HTTPS or SSL transport level security:

The system provided bindings and the WSSF wizards do a good job in guiding you in the WCF settings jungle; but when your automated guide gets lost, you'd better have some survival skills. Knowing the inner workings of WCF is a must for professional service developers, and I strongly recommend reading Michèle's article. The same goes for her blogs and the WCF stuff at IDesign.

Fiddler is highly recommended and can be downloaded from http://www.fiddlertool.com/fiddler/.

WCF: Core categories of data contracts

One of the famous SOA tenets is "services share contract, not class/implementation", meaning that it is the schema of your contract that is the main conveyor of how to consume the operations provided by your service. This has a huge impact on how you should design your contracts to provide for clear, understandable and comprehensive semantics, and also to minimize ambiguity in how to use your service. Contracts that have subtle or vague semantics are just more difficult to use and are thus more error prone. The same applies to contracts that are too flexible.

This post is about how to design data contracts that a simple to use, rather than easy to implement (simple vs easy); and at the same time keeping the number of data contracts to a minimum. The latter is important both for the consumers of your service and for the maintainability of your service. It is also important wrt SOA governance, the less stuff you have to govern, the better. Less schemas, less semantics, less maintenance, less governance.

Data contracts belong to one of these two groupings: altering state and querying information. Generally speaking, operations that modifies your system need to comply with stricter requirements and rules than operations that reads data from your system. This is because operations that can leave your system in an invalid state have greater technical impact on your business than operations that just returns information. Of course, if you disclose the incorrect information, your business could be in serious legal trouble.

The two data contract groupings can be further refined into several categories based on the different needs for expressing contract semantics and for being unambiguous. These five data contract core categories have manifested themselves through several more or less service-oriented solutions that I have implemented:

• Insert/update contracts: Typically one contract per domain object. Optional contained data contracts must be avoided or specifically handled.
• Delete contracts: Typically one contract per domain object.
• Specification/criteria contracts: Typically one contract per result contract, but it is not uncommon that a single specification can relate to multiple result contracts. Optional members are perfectly standard; the same applies to nullable criteria. Composite specifications are normal.
• Read/query result contracts: One or more contracts per domain object. Optional contained data contracts are allowed for flexibility and this is a key mechanism for keeping the number of result contracts to a minimum. Composite contracts are also allowed for the same reasons.
• Batch update/import contracts: Typically one contract per domain object batch operation type. Composite contracts are normal. Optional composite or contained contracts must be specifically handled.

These are core data contract categories for entity/core services. You will need to have more than just these core data contracts to provide good, event-driven, specialized business process services (EDA) in different contexts (sales, support, accounting, logistics, partners, suppliers, customers, etc).

The term ‘domain object’ also comprises complex objects (aggregate root objects) such as an order or a document card. The term ‘contained’ is used for complex objects. The term ‘composite’ is used for contracts that consists of several domain objects. The term ‘batch update’ includes insert, update and delete actions or a combination of these actions.

Note that I use CRUDy terms in the categories for simplicity (easier for me), to cover any real-life event that affects the state of a domain object. E.g. the “customer has moved” event falls into the “update” category.

A result contract will typically contain a composite structure of domain objects, defined by exactly the same unambiguous data contracts used for insert/update actions. The main reason for defining data contracts in the first place is to promote standardization and reuse across services and operations. To be able to support both the rigid insert/update data contact requirements and the flexible result contract requirements; it becomes a must to separate structure from data, isolating the structure/composition to the result set data contracts. Structural elements in a data contracts implicitly impose subtle semantics: how will the service handle the omission of composite/contained domain objects.

Insert/Update Contracts

It is important that insert/update contracts have little room for ambiguity, especially for complex domain objects. E.g. if the customer data contract contains a collection of addresses, what will happen if a customer update action is performed and no addresses are provided: does it mean that the customer no longer have any addresses or does it just mean that your can update a customers phone number without having to specify the addresses?

Such contained objects must be either A) required or B) specifically handled and by default optional/ignored. Controlled optional elements can be handled the way that the .NET 1.x XmlSerializer handled optional elements: using an extra property to indicate the state of the optional element. The XmlSerializer uses a Boolean XxxSpecified property for each optional element, e.g. OrderShippedDateSpecified.

Rather than using just a Boolean for the contained optional object, I recommend using an enumeration that contains Ignore (default value) and then some other applicable actions; much like cascading actions in SQL Server. The customer data contract should contain both an AddressList collection and an AddressListAction enumeration with e.g. the values Ignore, Replace, Alter, Purge. The point is that the user has to explicitly assign an action on the contained collection, rather than the service just assuming that an empty collection means deletion of the existing children. Assumptions are semantic coupling, and that is something you should strive to avoid.

Note that these 'insert/update' contract recommendations apply to entity/core services, which are not the services you want to expose publicly. Your public services need to reflect the events of your service-oriented business processes, and these "published" services belongs to the 'application to application services' category. By layering your services according to the four service categories, you will be able to expose more specialized operations with smaller contracts. Large contracts imply stronger coupling to the service, and as large contracts are more likely to change, your service will be more subject to breaking changes. Small contracts are simpler contracts, and simple contracts are important for the reusability, reliability, quality and robustness of your service (more about this in "Patterns for High-Integrity Data Consumption and Composition" by Dion Hinchcliffe).

You can still provide a very specific business operation that builds on the core service. E.g. the "customer has moved" event can be supported by an composite operation that takes only the customer key and the new postal address; which internally gets the complete customer, alters the address, and then stores the customer, in a single transaction using the core services. Services at the A2AS layer allows you to be "liberal in what you accept" as they shield the consumers from the details of the core services.

Read/Query Result Contracts

Result data contracts should be able to fit multiple needs and support several views of domain objects and composite result sets. At the same time, a consumer should be able to control how much information that gets returned from the service. E.g. one consumer might not be interested in address information when fetching customer data. Thus, a result data contract will most likely comprise optional elements, and consumers will not fail if some data is not present in the result set.

An empty collection does not normally imply the same ambiguity for reads as it does for insert/update contracts. A consumer will typically assume that if a fetched complex object contains no elements for a contained data contract, then the object does not have any such children; e.g. that a customer has no addresses if the customer AddressList collection is empty. An extra metadata property could be added to the result data contract as an indication of whether an optional element actually contains data even if not returned due to the processed query specification.

Note that ‘not present’ in the result set is not the same as ‘missing’ from the result set, which is clearly an error and should have caused a service fault.

Batch Update/Import Contracts

Batch update contracts are typically used to alter the state of a set of (related) domain objects. E.g. to update the TaskList collection of a project by sending a message that contains the tasks to add, modify and remove as one batch. Batch operations are a good way to avoid having to expose transactions outside your service; package all domain objects that must be altered in a transaction into a single message and perform the update using a single transacted operation.

Note that each data contract must still follow the rules described for ‘insert/update contracts’ even when used as part of a batch contract.

To be ideal objects for batch operations, domain objects should expose a “row-state” property; if they don’t, you need something like ‘Service Data Objects’ to make your batch contracts really simple to use. A message with one collection per action should be the last alternative.

Consume WCF BasicHttpBinding Services from ASMX Clients

The interoperability of WCF services has been touted quite a bit: just expose an endpoint using BasicHttpBinding and you will have a WS-I Basic Profile adherent web-service. This is of course backed by some simple examples of math services, all RPC-style and none of them messaging-style services.

We have for some time been consuming our BasicHttpBinding WCF services from a Flex 2 client and a WinForms smart client using a WCF generated client proxy (svcutil.exe). Just to be certain that our partners would have no problem setting up an application-to-application integration; I decided to test our services ASMX-style using "Add web reference" in Visual Studio (wsdl.exe). This to ensure that the services are consumable from non-WCF systems, i.e. systems without the .NET 3 run-time installed.

Well, surprise, surprise; it wasn't as straightforward as shown in the examples. There are several details that you need to be aware of, and some bugs in the service/message/data contract serializing mechanism.

I started by creating a new WinForms application and just used "Add web reference" to reference my "projectdocumentservices.svc?WSDL" file and WSDL.EXE generated an ASMX-style proxy for me. So far this looks good. I then added code to call my HelloWorld method on the proxy, which gave me this run-time error:

System.InvalidOperationException: Method ProjectDocumentServices.GetDocumentRequirement can not be reflected. ---> System.InvalidOperationException: The XML element 'KeyCriteriaMessage' from namespace 'http://DNVS.DNVX.eApproval.ServiceContracts/2006/11' references a method and a type. Change the method's message name using WebMethodAttribute or change the type's root element using the XmlRootAttribute.

Note that the Flex 2 client has no such problems with the BasicHttpBinding web-service, thus it must be related to how the generated proxy interprets the WSDL.

My service uses the specified message in several operations:

[System.ServiceModel.OperationContractAttribute(Action = "GetDocumentCard")]DocumentCardResponse GetDocumentCard(KeyCriteriaMessage request);

[System.ServiceModel.OperationContractAttribute(Action = "GetDocumentCategory")]DocumentCategoryResponse GetDocumentCategory(KeyCriteriaMessage request);

[System.ServiceModel.OperationContractAttribute(Action = "GetDocumentRequirement")]DocumentRequirementResponse GetDocumentRequirement(KeyCriteriaMessage request);

I turned to MSDN for more info and found the article 'ASMX Client with a WCF Service' (note the RPC-style math services) which lead me to 'Using the XmlSerializer'. So, accordingly, I added the [XmlSerializerFormat] attribute to the [ServiceContract] attribute in my service interface. Then I compiled and inspected the service, and got a series of different errors when trying to view the generated WSDL:

[SEHException (0x80004005): External component has thrown an exception.]

[CustomAttributeFormatException: Binary format of the specified custom attribute was invalid.]

[InvalidOperationException: There was an error reflecting type 'DNVS.DNVX.eApproval.DataContracts.DocumentCard'.]

Applying the well-known XML serialization attributes [Serializable] and [XmlIgnore] at relevant places in the data and message contracts helped me to isolate the problem to the use of collections and the WCF [CollectionDataContract] attribute.


To make a long story short, there is a bug in the WCF framework that affects .asmx but not .svc when using the [CollectionDataContract(Name="...")] attribute parameter:

"This happens only with Asmx pages but not for Svc due to a known Whidbey issue. Removing the "Name" parameter setting would work. If it is possible, you could also have your client instead use svc only."

Removing the Name parameter from all my [CollectionDataContract] attributes made all the WSDL reflection errors disappear, and I was again able to view the generated WSDL file. This time for a service that used the [XmlSerializerFormat] attribute.

Full of hope, I updated the web reference in my test client and ran it. Calling the test method lead me straight back to my original problem: the message contract used in multiple operations. Note that removing all but one of the operations makes the problem go away.

According to 'Message WSDL Considerations' WCF should have done this when generating the WSDL:

"When using the same message contract in multiple operations, multiple message types are generated in the WSDL document. The names are made unique by adding the numbers "2", "3", and so on for subsequent uses. When importing back the WSDL, multiple message contract types are created and are identical except for their names."

Inspecting the WSDL showed me that multiple XSD schemas are not generated. Certainly another WCF issue/bug. The problem can be solved by deriving the message class into new classes with unique names and using the derived classes in the operations. But when you have a message that is used ubiquitously, e.g. DefaultOperationResponse, this is not a viable solution.

As of now, our BasicHttpBinding WCF service cannot be consumed by ASMX clients. So much for interoperability...

[UPDATE] Read my new post about how to expose a WCF service also as an ASMX web-service.

JSR235: Service Data Objects

In my last post, I wrote about how the SOA 'boundaries are explicit' and 'services are autonomous' tenets should make you design service operations that work on an entity as a unity rather than on the bits and pieces making up the unity. This is the wellknown 'paper document' message metaphor.

The most common reason for developers breaking the "unity" rule is for optimizing transport performance, i.e. to make the amount of data passed across the wire as small as possible. The rationale behind some of the design decisions of e.g. the Amazon Web Services shopping cart follows this thinking: "Amazon has to be prepared for the one customer in 60 million with 20,000 items in their shopping cart" (Werner Vogels, vice president, world-wide architecture and CTO at Amazon.com). How would you implement an operation for changing a few of the 20,000 cart items ?

In my opinion, this kind of transport optimizations should not surface in the service operations; it should be handled by the service framework. Just as e.g. encoding and encryption should be handled by the framework.

The Java community is working on the 'Service Data Objects' specification (JSR 235) to create a better framework for handling entities in service-oriented architectures. The goals of SDO are to help simplify and unify data access across different data source types. Data transport is one aspect of SDO. Another aspect is the change tracking mechanism, which is closely related to data transport: unchanged values of the original tree are omitted when serializing the data graph.

Change tracking applies to any kind of data object, including what is called 'complex data objects' in the SDO spec - i.e. aggregate root objects/entities such as orders. This is a feature that anyone who has implemented an order service would wish for: having automagical support for knowing which changes have been made to the list of order items, simplifying e.g. altering the order in the database.

Service Data Objects has several interesting aspects, and I recommend that you read this IBM article and some of the resources referenced in it.

While you're at it, also read this Architecture Journal article 'Autonomous Services and Enterprise Entity Aggregation'. It discusses how the explicit/autonomous tenets influence the use of core system entities in a service-oriented event-driven design.

Service Architecture: JBOWS, SOA or WOA

A few days ago, I got into a discussion of whether the Amazon Web Services are true SOA or just a programming model exposed using web-services. The discussion started when I said that CRUD style operations are not according to SOA best practices; and that operating on parts of a domain object of type aggregate root should be avoided, favoring actions on the complete entity to be exposed as service operations.

Before making my SOA-vs-JBOWS case, a short introduction to WOA: as adhering to all the SOA service design principles and the different WS-* technologies can be quite daunting and complex, a lot of service-oriented solutions have emerged that take a simpler approach (REST, SOAP, POX, etc). Gartner has coined the acronym WOA (Web-Oriented Architecture) for these kinds of solutions that implement and expose services using more pragmatic techniques following the WOA tenets.

First the "what's in an operation name" issue:

It is not best practice to use CRUDy style operation names that conveys only that the domain object state is going to change in the repository (e.g. UpdateCustomerAddress). You should rather use operation names that reveals the event in the business process that caused the action (e.g. CustomerHasMoved). The point is that your service will most likely perform some business logic on the domain object in conjunction with storing it in the database. The operation name should convey the fact that the state of the real-world entity represented by the domain object has changed.

Note that you need not change the operation names very much, sometimes renaming UpdateXxx to ChangeXxx can be sufficient to convey the correct semantics. Focus on creating business process driven services rather than data-driven services.

The discussion I had was about if names such as AddXxx, ModifyXxx and DeleteXxx are CRUDy style names or not. I think they are to closely named after what the code is going to do with the domain object repository rather than reflecting changes to the real-life entity, i.e. it is more a programming model (WOA) than a SOA operation.

Note that CRUDy operations and names can be quite ok, especially for information services, which will still be needed to manage data in your repositories. There will still be a need for creating new customers, even in SOA.

Then on to the operations on different types of domain objects issue:

A common metaphor for designing SOA operation and data contracts is to think of paper forms being passed around to clerks to fulfil a business process. Another metaphor is mail orders, the point is that the 'document' contains all data needed to perform the business process.

Documents can be simple domain objects or aggregate root objects. An order is an example of an aggregate root object - it contains general order data and a set of order items. The order items belong to the order, i.e. it is an identifying relationship and not just a relation. As a general rule, an operation should process a whole document and never mess directly with 'identified items' within the document. Such operations would be very CRUDy style, and breaks the "boundaries are explicit" tenet.


Note that it is perfectly legal to operate on normal relations, such as adding orders to a customer. It is the nature of the domain and its business processes that decide if a relation is identifying or not, thus there is no hard rule to help you decide whether incremental operations are ok or not.

Using the 'document' metaphor, you would implement a 'RegisterOrder' operation to add new orders. You should, however, not implement operations for changing a subset of the order items; you must rather implement a 'ChangeOrder' or a 'CancelOrder' operation.

In my opinion, the Amazon shopping cart is analogous to the above order domain object. Thus, according to SOA best practices, they should not expose operations such as ChartModify that allows you to do incremental changes to the items in the chart. I understand that they do this for simplicity and performance reasons, and that the shopping cart is really not registered until it is submitted; but some developers use the AWS as general best practices for SOA and apply the same programming model blindly.

I don't propose that Amazon Web Services are not properly designed; they are just WOA rather than SOA.
A closing note: there is no "one or the other" between SOA and WOA, in fact they are both central in web 2.0. Read more about it in this report from the Web 2.0 Summit.