Service Compatibility - A Primer

In a comment on the InfoQ article Contract Versioning, Compatibility and Composability about my definition of service forwards and backwards compatibility, the problem of talking about compatibility of services compared to the definition of schema compatibility is acknowledged.

The "problem" is that a service version that is compatible with the specifications of older versions of the service, can be achieved using both backwards and forwards compatible schemas. That is correct, but doesn't preclude having a definition of forwards and backwards compatibility for service providers, a.k.a "services". Service compatibility is based on ability to validate a message, it is not based on using wildcards in the schema definition. For a definition of the three types of forward compatibility, see my post on schema, service and routing compatibility.

Seen only from the service provider perspective, how it handles incoming messages is what defines if a service is forwards or backwards compatible (or both). How consumers handle messages sent by the service is really not of any concern for the provider - wait, read on.

Thinking about this within SOAP 1.x constraints, where a WSDL operation has a request message and a response message with fixed schema definitions/version (unilateral contracts), will lead to the conclusion that operations cannot be classified as forwards or backwards compatible, only the message schemas. This is a limitation of SOAP, but not of messaging in general.

In the following examples, the v1.2 service provider is backwards compatible and interacts with a v1.1 consumer. However, the schemas are not designed to be forwards compatible - they do not support XML extensibility (schema wildcards). In this scenario, the consumer can do either XSD validation of response messages against the v1.1 schema, or do 'validation by projection' of response messages - i.e. do "ignore unkown" validation. Doing 'validation by projection' is a recommended practice for compatibility and really simplifies building SOA solutions - this is also how WCF validates messages. So how to handle consumers that only do XSD validation, without relying on schema wildcards?

In REST, the consumer can put an "accept formats" header in the v1.1 request message, and the service provider can then respond with a v1.1 schema even if the service version is v1.2. The service provider adheres to it's obligations by being backwards compatible, and the consumer is allowed to express it's version expectation - the service has bilateral contracts.

Service Virtualization is a mechanism that can help with service versioning. The task of such an abstract endpoint intermediary is to handle versioning through both service compatibility and schema compatibility. A virtual service supports multiple versions of the service on the same endpoint, and must be capable of processing older requests. The virtual endpoint must have a mechanism that allows for the latest major v1.2 provider to handle v1.1 consumers. The intermediary mediates between the schema versions by transforming or projecting/enriching the messages as needed.

Back to the example, the service provider v1.2 response message can be stripped down to a v1.1 message by the intermediary as it is sent back to the consumer. The net effect is that the service has virtual bilateral contracts.

In messaging in general, by definition there are no duplex channels, only one-way channels (see Enterprise Integration Patterns by Hohpe/Woolf). On top of this, you can have a logical two-way channel for message exchange patterns such as request-response, specified using a "reply-to" address and a "reply-format" (bilateral contracts). The message compatibility is defined by the schema constructs, but just as in REST, the version of the incoming message does not dictate the version of the response message. It is the implementation of the endpoint that processes the messages that defines the compatibility policy of the endpoint, not the schemas.

So, service compatibility do not require using forwards compatible schemas in addition to backwards compatible schemas. The message validation policy is what defines service compatibility.

It is of course much simpler to just have a service compatibility policy based on that the schemas used in the services must be both forwards and backwards compatible - as shown in the "SOAP-style unilateral contracts" service compatibility figures.



Click figures to enlarge.

This way, the service provider or consumer platform need not handle "request-format" and "reply-format" that have different versions. In such a unilateral schema compatibility policy world, services are just intrinsically compatible through schema compatibility.

WCF, SVCUTIL Proxy Problems

We've had some rather strange proxy problems with our WCF client code lately, with some PCs not being able to connect to the services using DNS names ("There was no endpoint listening at ... that could accept the message"), but working fine using the service's IP address. This happened even if all PCs had the same MSIE "bypass proxy server for local addresses" and the same exception list in "do not use proxy server for addresses beginning with".

To make a long story short, there is a "sneaky gotcha" in the proxy exceptions logic: port numbers are not considered to be a sub-domain. So an exception like this "*.blogspot.com" might not include e.g. "*.blogspot.com:12001" depending on your system setup. Alas, you can add your own <bypasslist> directly in the <system.net/defaultProxy> element. Note that the bypass list addresses must be valid regex expressions. Read more about proxies and <system.net> at Matt Ellis' blog.

<system.net>
<defaultProxy useDefaultCredentials="true" >
<proxy usesystemdefault="True" bypassonlocal="True"/>
<bypasslist>
<add address=".+\.blogspot\.com:\d{1,5}" />
</bypasslist>
</defaultProxy>
</system.net>

For more info about WCF and proxies and how to do this from code, see Setting Credentials for your HTTP Proxy by Kenny Wolf.

The <system.net> proxy configuration can also be applied to SVCUTIL.EXE when e.g. proxy authentication is required to get the WSDL from the MEX endpoint.

Create a file named svcutil.exe.config in the directory where SVCUTIL.EXE is located to apply the proxy configuration to the tool. To pass user name and password use "username:password@" in front of the URL in the proxyaddress attribute. Do not store passwords in clear text like this.

PS! note how the casing of the attributes and values in the <proxy> element differs from what's normal.

WCF: Using Shared Types

In WCF you can use shared types from existing assemblies when generating a service proxy. Use the advanced options in VS2008, otherwise use SVCUTIL.EXE with the /reference /r option. Note that you can only use /reference when your WSDL operations, types and messages adheres to the DataContractSerializer rules that I described this spring.

I'm currently trying to research why we are not able to use shared types for our wsdl:fault message parts.

<wsdl:message name="GetCustomerRequest">
<wsdl:part element="tns:GetCustomerRequest" name="parameters" />
</wsdl:message>
<wsdl:message name="GetCustomerResponse">
<wsdl:part element="tns:GetCustomerResponse" name="parameters" />
</wsdl:message>
<wsdl:message name="FaultDetailList">
<wsdl:part element="fault:FaultDetailList" name="detail" />
</wsdl:message>

<wsdl:portType name="CustomerService">
<wsdl:operation name="GetCustomer">
<wsdl:input message="tns:GetCustomerRequest" name="GetCustomerRequest" />
<wsdl:output message="tns:GetCustomerResponse" name="GetCustomerResponse" />
<wsdl:fault message="tns:FaultDetailList" name="FaultDetailList" />
</wsdl:operation>
</wsdl:portType>

<wsdl:binding name="CustomerServiceSoap11" type="tns:CustomerService">
<soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http" />
<wsdl:operation name="GetCustomer">
<soap:operation soapAction="" />
<wsdl:input name="GetCustomerRequest">
<soap:body use="literal" />
</wsdl:input>
<wsdl:output name="GetCustomerResponse">
<soap:body use="literal" />
</wsdl:output>
<wsdl:fault name="FaultDetailList">
<soap:fault name="FaultDetailList" use="literal" />
</wsdl:fault>
</wsdl:operation>
</wsdl:binding>

Even if the fault details contains only xs:string elements, we get this error when generating the proxy:

Cannot import wsdl:portTypeDetail: An exception was thrown while running a WSDL import extension: System.ServiceModel.Description.DataContractSerializerMessageContractImporterError: Referenced type 'SharedDataContracts.FaultDetailType, MyService, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' with data contract name 'FaultDetailType' in namespace 'urn:kjellsj.blogspot.com/FaultDetail/1.0' cannot be used since it does not match imported DataContract. Need to exclude this type from referenced types.

I thought that writing a unit-test to import the WSDL using the underlying SVCUTIL importer directly should help me deduce the cause of the error. You will find all the code that you need to import WSDL using DataContractSerializerMessageContractImporter here at the "nV Framework for Enterprise Integration" project as CodePlex.

Add these lines to enforce the use of /namespace, /reference and /ct from code (note that these switches can be used multiple times):

xsdDataContractImporter.Options.Namespaces. Add(new KeyValuePair<string, string>("*", "MyService"));
xsdDataContractImporter.Options.ReferencedCollectionTypes. Add(typeof(List<>));
xsdDataContractImporter.Options.ReferencedTypes. Add(typeof(SharedDataContracts.FaultDetailList));
xsdDataContractImporter.Options.ReferencedTypes. Add(typeof(SharedDataContracts.FaultDetailType));

Sure enough, I get the exact same exception in my unit-test. Inspecting the imported contract shows that the operation has a fault with a "null" DetailType XML Schema type (watch: contracts[0].Operations[0].Faults[0].DetailType). I suspect that it is the "null" DetailType in the FaultDescription of the imported ServiceDescription for the WSDL that causes the error for the referenced shared types.

As it turns out this is a bug that has been fixed in .NET 3.5 SP1. The workaround for earlier versions of SVCUTIL is to add nillable="true" to all elements that are reference types (strings and complexTypes) in your wsdl:types XSD schemas. Or just drop the /reference switch and cut'n'paste the generated code to use your shared types.

Configure MSE UDDI Integration

The Managed Services Engine repository can be synchronized with UDDI. All service endpoints hosted on an MSE runtime server can be exported to make your virtualized services discoverable from e.g. VisualStudio. View this web-cast (slide deck) by Raul Camacho to learn about SOA lifecycle management and MSE service virtualization.

Enable and configure the UDDI sync using the <serviceCatalogUddi> element in the config file of the MSE catalog service executable on the MSE server (Microsoft.MSE.Catalog.ServiceHost.exe.config).

NOTE: In UDDI Services, data published for a service, provider, or tModel can be modified only by the current owner. Make sure that the identity used to publish MSE data to UDDI is the owner of the configured UDDI provider (BusinessKey).

The UDDI sync configuration must adhere to these rules:

• Make sure that you use the "uddipublic" URLs for "UDDI" authN
• Make sure that you use the "uddi" URLs for "WINDOWS" authN
• The identity used to run the MSE catalog service must have access to the UDDI web-services
• The identity specified for the sync must have publish rights in UDDI and own the BusinessKey

The UDDI Services Console MMC snap-in must be used to configure domain groups for the different UDDI roles, including "publisher" rights.

Using "WINDOWS" authN is strongly recommended. You can specify "UDDI" as the <UddiAuthenticationScheme> and put a username + password of an identity that has access to UDDI in the config file to test the UDDI sync. Do not do this for production systems.

<!-- DO NOT CHANGE THE ORDER OF THE ELEMENTS -->
<serviceCatalogUddi>
<UddiIntegrationMode>true</UddiIntegrationMode>
<UddiPublishUrl>http://***/uddipublic/publish.asmx</UddiPublishUrl>
<UddiInquireUrl>http://***/uddipublic/inquire.asmx</UddiInquireUrl> <BusinessKey> UDDI PROVIDER GUID HERE </BusinessKey>
<UddiAuthenticationScheme>UDDI</UddiAuthenticationScheme>
<UddiUserName>***\***</UddiUserName>
<UddiPassword>*******</UddiPassword>
</serviceCatalogUddi>

Check the UDDI config if you have problems when publishing services and also check the Application Event Log. Use the MMC UDDI Services Console to adjust the logging level to diagnose errors. Note that you must set the level to info / verbose to see SQL exceptions, e.g. when you get an error like this:

Failed to Publish EndPoint service to UDDI due to error [Exception Information Type[FaultException] Message[A UDDI specific error occurred while publishing Endpoint data to UDDI] ].

Using "WINDOWS" against "uddipublic" will give you this error in the event log: UDDI_ERROR_AUTHTOKENREQUIRED_NOTOKENPUBLISHATTEMPT

Note that the UDDI web-portal requires ASP.NET 1.x. Check that the IIS web-sites use ASP.NET 1.x if the UDDI web-pages have errors (e.g. the treeview fails).

EntLib3 Replace Handler, ExceptionHandlingException

Using the Enterprise Library 3 exception replace handler can help you handle e.g. WCF FaultException<T> on the client and replace it with your own exception derived from ApplicationException. This will shield your client app from the different types of FaultException<T> where T: FaultContract types defined in the WSDL as wsdl:fault schemas and provide a normalized application exception type. This client-side exception shielding mechanism ensures that the proxy fault types need not be exposed outside your service agent.

There is one exception best practice to remember to implement for the @replaceExceptionType configuration to work: your derived exception must implement the three recommended constructors default, message, message and inner exception. If you forget, the entlib HandleException call will throw an "unable to handle exception" ExceptionHandlingException error.

Note the [[fault contract type]] syntax to be able to specify handlers for generic exceptions:

<add type="System.ServiceModel.FaultException`1[[MyServiceAgent.Proxy.GetCustomerListFault, MyServiceAgent.Proxy, Version=1.0.0.0]], System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
postHandlingAction="ThrowNewException" name="MyApplicationException">

You have to edit the .config file manually as the EntLib3 editor does not support generics

WCF: DataContractSerializer Schema Rules

Anyone who have tried to do contract/schema first design of data contracts or tried to generate proxies using SVCUTIL.EXE in an interop scenario with e.g. Spring-WS, will have run into how WCF magically decides to use the old XmlSerializer rather than the new DataContractsSerializer (DCS). And deducing which parts of a schema that cause this is not easy with real-life contracts.

The XML schema rules/limitations for DCS is well documented, but the proxy generator will not tell you which of the rules your XSD/WSDL does not adhere to. Even worse, if you specify /serializer:DataContractSerializer and the rules are broken; the proxy will get generated without any error, but also without any messages and data transfer objects.

Note that the SVCUTIL switches /collectionType /ct and /reference /r only work with the DataContractSerializer, not the XmlSerializer. Thus, if the proxy generator falls back to the XmlSerializer, you will not get collections (List / BindingList / etc) for arrays nor any reuse of existing shared object types.

The only way I've found that will tell you why DCS is not used, is by using the /DCONLY option to extract the data contracts from the WSDL. Using this switch will disclose data contract (XSD) errors one at a time, but it is still better than not knowing why. Note that it will not disclose any errors related to the messages not being document/literal wrapped (WSDL).

The most common XSD schema lapsus is not wrapping arrays in a two-level complex type, one for the list of items and one for the items themselves. This is not very well documented in the DCS rules. The array XSD schema must follow this construct:

<xs:element name="ItemList" type="tns:ItemListType" />

<xs:complexType name="ItemListType">
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" name="Item" type="tns:ItemType" />
</xs:sequence>
</xs:complexType>

<xs:complexType name="ItemType">
<xs:sequence>
<xs:element name="Id" type="xs:long" />
<xs:element name="Value" nillable="true" type="xs:string" />
</xs:sequence>
</xs:complexType>

Note that there is no need to use the prefix ArrayOf on the list element name.

The reason for wrapping the array might seem quite silly when the message contains elements of only one type. However, look at this XML segment that contains elements of multiple types:
<root><a/><b/><b/><b/></root>

The DCS does not like a XSD schema where the children of an element varies in numbers and are not of the same type; it requires that all child elements that varies in numbers must have the same type, thus the wrapper element:
<root><a/><itemList><b/><b/><b/></itemList></root>

This way the <root> element contains a fixed number of different elements, the <a> element and the <itemList> element; while the <itemList> element contains a varying number of <b> elements only. In addition, an empty/null array can still be represented in the XML.

The other typical WSDL lapsus wrt DCS is about the message part definitions and the related operation input and output elements plus the related binding section. The important rule here is to remember to use only one part in each message (WS-I Basic Profile) and to set the message part attribute @name = "parameters" (by convention) to make the messages document/literal wrapped. Other @name conventions are "header" for message headers and "detail" for fault message parts to be used as wsdl:fault elements in operations.

This is an example that will allow you to use the DataContractSerializer:

<wsdl:message name="GetItemRequest">
<wsdl:part element="tns:GetItemRequestType" name="parameters" />
</wsdl:message>
<wsdl:message name="GetItemResponse">
<wsdl:part element="tns:GetItemResponseType" name="parameters" />
</wsdl:message>
<wsdl:message name="FaultDetailList">
<wsdl:part element="fault:FaultDetailList" name="detail" />
</wsdl:message>

<wsdl:portType name="InspectValues">
<wsdl:operation name="GetItem">
<wsdl:input message="tns:GetItemRequest" name="GetItemRequest" />
<wsdl:output message="tns:GetItemResponse" name="GetItemResponse" />
<wsdl:fault message="tns:FaultDetailList" name="FaultDetailList" />
</wsdl:operation>
</wsdl:portType>

<wsdl:binding name="InspectValuesSoap11" type="tns:InspectValues">
<soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http" />
<wsdl:operation name="GetItem">
<soap:operation soapAction="" />
<wsdl:input name="GetItemRequest">
<soap:body use="literal" /></wsdl:input>
<wsdl:output name="GetItemResponse">
<soap:body use="literal" /></wsdl:output>
<wsdl:fault name="FaultDetailList">
<soap:fault name="FaultDetailList" use="literal"/></wsdl:fault>
</wsdl:operation>
</wsdl:binding>

Note that if your message part element is a wrapped array, the referenced element must be marked as nillable. This is a SVCUTIL WSDL requriement, and applies even if using /DCONLY generates the contracts just fine.

If the generated DCS-based proxy interface contains operations that has no input or output messages, then your WSDL does not adhere to the above rules. The same is true if you get any "cannot import" errors or warnings related to wsdl:binding, wsdl:portType or wsdl:port. Note also that the element referenced as a message part must be a DCS allowable <xs:complexType> with an <xs:sequence> child (can also be empty). Read more about message schema rules in Handcrafting WCF friendly WSDLs.

There are some known issues with SVCUTIL /reference regarding the mismatches between some XSD schema types and .NET types (e.g. xs:date, xs:positiveinteger), which will make /reference fail as the WSDL types don't correspond with the referenced .NET data contract types: .NET has no Date only data type, just DateTime - thus xs:date becomes string. Also watch out for enumerations being represented as pure strings without any defined enum items. These issues will cause /reference to fail.

Note also that your custom shared collection types must also be referenced using /ct even if already added using /reference. Forgetting this will give your duplicate classes for all your [CollectionDataContractAttribute] lists defined in your data contracts.

As you might have noticed, using nillable elements might be required for /reference to work properly. Always try setting nillable="true" if you can't get any code generated when using the DataContractSerializer.

Pre .NET 3.5 SP1: All xs:string elements are required to be nillable. The same applies to your custom collection types.

I wish the DCS rules documentation had included more XSD schema examples, both for allowable and invalid schema constructs.

The /reference option is very important when using an architecture like CAB/SCSF where the service implementation (proxy) is separated from the service definition (interface). Then, it is important that the data contracts are included in the service interface assembly for reuse across business modules; i.e. the DTO objects can not be inside the proxy implementation assembly. The proxy itself is most likely wrapped in a service agent and should never be exposed in the DDD service or repository pattern interface.

If you can't get the /dconly plus /reference combo to work, then you're stuck with generating the contracts and proxy into classes in a single file using a .Service.Interface namespace and then extracting the proxy code into a separate file in a .Service namespace using e.g. a RegEx script, followed by some namespace using/imports refactoring for the extracted proxy code file so that it references the data contracts in the interface namespace.