Service Compatibility - A Primer

In a comment on the InfoQ article Contract Versioning, Compatibility and Composability about my definition of service forwards and backwards compatibility, the problem of talking about compatibility of services compared to the definition of schema compatibility is acknowledged.

The "problem" is that a service version that is compatible with the specifications of older versions of the service, can be achieved using both backwards and forwards compatible schemas. That is correct, but doesn't preclude having a definition of forwards and backwards compatibility for service providers, a.k.a "services". Service compatibility is based on ability to validate a message, it is not based on using wildcards in the schema definition. For a definition of the three types of forward compatibility, see my post on schema, service and routing compatibility.

Seen only from the service provider perspective, how it handles incoming messages is what defines if a service is forwards or backwards compatible (or both). How consumers handle messages sent by the service is really not of any concern for the provider - wait, read on.

Thinking about this within SOAP 1.x constraints, where a WSDL operation has a request message and a response message with fixed schema definitions/version (unilateral contracts), will lead to the conclusion that operations cannot be classified as forwards or backwards compatible, only the message schemas. This is a limitation of SOAP, but not of messaging in general.

In the following examples, the v1.2 service provider is backwards compatible and interacts with a v1.1 consumer. However, the schemas are not designed to be forwards compatible - they do not support XML extensibility (schema wildcards). In this scenario, the consumer can do either XSD validation of response messages against the v1.1 schema, or do 'validation by projection' of response messages - i.e. do "ignore unkown" validation. Doing 'validation by projection' is a recommended practice for compatibility and really simplifies building SOA solutions - this is also how WCF validates messages. So how to handle consumers that only do XSD validation, without relying on schema wildcards?

In REST, the consumer can put an "accept formats" header in the v1.1 request message, and the service provider can then respond with a v1.1 schema even if the service version is v1.2. The service provider adheres to it's obligations by being backwards compatible, and the consumer is allowed to express it's version expectation - the service has bilateral contracts.

Service Virtualization is a mechanism that can help with service versioning. The task of such an abstract endpoint intermediary is to handle versioning through both service compatibility and schema compatibility. A virtual service supports multiple versions of the service on the same endpoint, and must be capable of processing older requests. The virtual endpoint must have a mechanism that allows for the latest major v1.2 provider to handle v1.1 consumers. The intermediary mediates between the schema versions by transforming or projecting/enriching the messages as needed.

Back to the example, the service provider v1.2 response message can be stripped down to a v1.1 message by the intermediary as it is sent back to the consumer. The net effect is that the service has virtual bilateral contracts.

In messaging in general, by definition there are no duplex channels, only one-way channels (see Enterprise Integration Patterns by Hohpe/Woolf). On top of this, you can have a logical two-way channel for message exchange patterns such as request-response, specified using a "reply-to" address and a "reply-format" (bilateral contracts). The message compatibility is defined by the schema constructs, but just as in REST, the version of the incoming message does not dictate the version of the response message. It is the implementation of the endpoint that processes the messages that defines the compatibility policy of the endpoint, not the schemas.

So, service compatibility do not require using forwards compatible schemas in addition to backwards compatible schemas. The message validation policy is what defines service compatibility.

It is of course much simpler to just have a service compatibility policy based on that the schemas used in the services must be both forwards and backwards compatible - as shown in the "SOAP-style unilateral contracts" service compatibility figures.



Click figures to enlarge.

This way, the service provider or consumer platform need not handle "request-format" and "reply-format" that have different versions. In such a unilateral schema compatibility policy world, services are just intrinsically compatible through schema compatibility.

Service+Schema Versioning: Flexible/Strict Strategy

In a few weeks time I will be giving a session at NNUG on SOA service and schema versioning strategies and practices. A central topic will be schema compatibility rules, where I will recommend creating a policy based on the "Strict", "Flexible" and "Loose" versioning strategies described in chapter 20.4 in the latest Thomas Erl series book Web Service Contract Design and Versioning for SOA. I guess David Orchard is the author/editor of part III in the book.

I recommend using a “Flexible/Strict” compatibility policy:

• Flexible: Safe changes to schemas are backwards compatible and cause just a point version


• Strict: All unsafe schema changes must cause a new schema version and thus a new service version


• Do not require forwards compatible schemas (Loose, wildcard schemas) - schemas should be designed for extensibility, not to avoid versioning


• Service interfaces should also have a Flexible/Strict policy

Safe changes is typically adding to schemas, while unsafe changes are typically modifying or removing schema components.

Note that forwards compatible schemas is not required to have forwards compatible services, as service compatibility is defined by the ability to validate messages. WCF uses a variant of 'validation by projection' (ignore unknown) for forwards compatibility, but also supports schema wildcards.


As Nicolai M. Josuttis shows in the book SOA in Practice (chapter 12.2.1 Trivial Domain-Driven Versioning), even simple backwards compatible changes might cause unpredicted side effects such as response times breaking SLAs and causing problems for consumers. It is much safer to provide a new service version with the new schema version, as if there is a problem, only the upgraded consumers that required the change will be affected.

Note that even adding backwards compatible schema components can be risky, but adding is typically safe. Josuttis recommends using "Strict" as it is a very simple and explicit policy, but I prefer "Flexible/Strict" as this gives more flexibility and less service versions to govern.

Avoid trying to implement some smart automagical mechanism for handling schema version issues in the service logic. Rather use backwards compatibility, explicit schema versions and support multiple active service versions. In addition, consider applying a service virtualization mechanism.

WCF: Core categories of data contracts

One of the famous SOA tenets is "services share contract, not class/implementation", meaning that it is the schema of your contract that is the main conveyor of how to consume the operations provided by your service. This has a huge impact on how you should design your contracts to provide for clear, understandable and comprehensive semantics, and also to minimize ambiguity in how to use your service. Contracts that have subtle or vague semantics are just more difficult to use and are thus more error prone. The same applies to contracts that are too flexible.

This post is about how to design data contracts that a simple to use, rather than easy to implement (simple vs easy); and at the same time keeping the number of data contracts to a minimum. The latter is important both for the consumers of your service and for the maintainability of your service. It is also important wrt SOA governance, the less stuff you have to govern, the better. Less schemas, less semantics, less maintenance, less governance.

Data contracts belong to one of these two groupings: altering state and querying information. Generally speaking, operations that modifies your system need to comply with stricter requirements and rules than operations that reads data from your system. This is because operations that can leave your system in an invalid state have greater technical impact on your business than operations that just returns information. Of course, if you disclose the incorrect information, your business could be in serious legal trouble.

The two data contract groupings can be further refined into several categories based on the different needs for expressing contract semantics and for being unambiguous. These five data contract core categories have manifested themselves through several more or less service-oriented solutions that I have implemented:

• Insert/update contracts: Typically one contract per domain object. Optional contained data contracts must be avoided or specifically handled.
• Delete contracts: Typically one contract per domain object.
• Specification/criteria contracts: Typically one contract per result contract, but it is not uncommon that a single specification can relate to multiple result contracts. Optional members are perfectly standard; the same applies to nullable criteria. Composite specifications are normal.
• Read/query result contracts: One or more contracts per domain object. Optional contained data contracts are allowed for flexibility and this is a key mechanism for keeping the number of result contracts to a minimum. Composite contracts are also allowed for the same reasons.
• Batch update/import contracts: Typically one contract per domain object batch operation type. Composite contracts are normal. Optional composite or contained contracts must be specifically handled.

These are core data contract categories for entity/core services. You will need to have more than just these core data contracts to provide good, event-driven, specialized business process services (EDA) in different contexts (sales, support, accounting, logistics, partners, suppliers, customers, etc).

The term ‘domain object’ also comprises complex objects (aggregate root objects) such as an order or a document card. The term ‘contained’ is used for complex objects. The term ‘composite’ is used for contracts that consists of several domain objects. The term ‘batch update’ includes insert, update and delete actions or a combination of these actions.

Note that I use CRUDy terms in the categories for simplicity (easier for me), to cover any real-life event that affects the state of a domain object. E.g. the “customer has moved” event falls into the “update” category.

A result contract will typically contain a composite structure of domain objects, defined by exactly the same unambiguous data contracts used for insert/update actions. The main reason for defining data contracts in the first place is to promote standardization and reuse across services and operations. To be able to support both the rigid insert/update data contact requirements and the flexible result contract requirements; it becomes a must to separate structure from data, isolating the structure/composition to the result set data contracts. Structural elements in a data contracts implicitly impose subtle semantics: how will the service handle the omission of composite/contained domain objects.

Insert/Update Contracts

It is important that insert/update contracts have little room for ambiguity, especially for complex domain objects. E.g. if the customer data contract contains a collection of addresses, what will happen if a customer update action is performed and no addresses are provided: does it mean that the customer no longer have any addresses or does it just mean that your can update a customers phone number without having to specify the addresses?

Such contained objects must be either A) required or B) specifically handled and by default optional/ignored. Controlled optional elements can be handled the way that the .NET 1.x XmlSerializer handled optional elements: using an extra property to indicate the state of the optional element. The XmlSerializer uses a Boolean XxxSpecified property for each optional element, e.g. OrderShippedDateSpecified.

Rather than using just a Boolean for the contained optional object, I recommend using an enumeration that contains Ignore (default value) and then some other applicable actions; much like cascading actions in SQL Server. The customer data contract should contain both an AddressList collection and an AddressListAction enumeration with e.g. the values Ignore, Replace, Alter, Purge. The point is that the user has to explicitly assign an action on the contained collection, rather than the service just assuming that an empty collection means deletion of the existing children. Assumptions are semantic coupling, and that is something you should strive to avoid.

Note that these 'insert/update' contract recommendations apply to entity/core services, which are not the services you want to expose publicly. Your public services need to reflect the events of your service-oriented business processes, and these "published" services belongs to the 'application to application services' category. By layering your services according to the four service categories, you will be able to expose more specialized operations with smaller contracts. Large contracts imply stronger coupling to the service, and as large contracts are more likely to change, your service will be more subject to breaking changes. Small contracts are simpler contracts, and simple contracts are important for the reusability, reliability, quality and robustness of your service (more about this in "Patterns for High-Integrity Data Consumption and Composition" by Dion Hinchcliffe).

You can still provide a very specific business operation that builds on the core service. E.g. the "customer has moved" event can be supported by an composite operation that takes only the customer key and the new postal address; which internally gets the complete customer, alters the address, and then stores the customer, in a single transaction using the core services. Services at the A2AS layer allows you to be "liberal in what you accept" as they shield the consumers from the details of the core services.

Read/Query Result Contracts

Result data contracts should be able to fit multiple needs and support several views of domain objects and composite result sets. At the same time, a consumer should be able to control how much information that gets returned from the service. E.g. one consumer might not be interested in address information when fetching customer data. Thus, a result data contract will most likely comprise optional elements, and consumers will not fail if some data is not present in the result set.

An empty collection does not normally imply the same ambiguity for reads as it does for insert/update contracts. A consumer will typically assume that if a fetched complex object contains no elements for a contained data contract, then the object does not have any such children; e.g. that a customer has no addresses if the customer AddressList collection is empty. An extra metadata property could be added to the result data contract as an indication of whether an optional element actually contains data even if not returned due to the processed query specification.

Note that ‘not present’ in the result set is not the same as ‘missing’ from the result set, which is clearly an error and should have caused a service fault.

Batch Update/Import Contracts

Batch update contracts are typically used to alter the state of a set of (related) domain objects. E.g. to update the TaskList collection of a project by sending a message that contains the tasks to add, modify and remove as one batch. Batch operations are a good way to avoid having to expose transactions outside your service; package all domain objects that must be altered in a transaction into a single message and perform the update using a single transacted operation.

Note that each data contract must still follow the rules described for ‘insert/update contracts’ even when used as part of a batch contract.

To be ideal objects for batch operations, domain objects should expose a “row-state” property; if they don’t, you need something like ‘Service Data Objects’ to make your batch contracts really simple to use. A message with one collection per action should be the last alternative.

WSSF v2 released: WCF contract first, contract versioning

Normally I wouldn't post just some links, but the release of the WCF web-service software factory version 2 merits an aggregation post.

Read about WSSF v2 details such as contract first supprt (WSCF), versioning guidance, etc, at Don Smith's blog; and download it from MSDN (not GotDotNet). The complete web-service versioning emerging guidance article is a must-read for anyone implementing "published" services.

Boy am I excited to check out the WSCF stuff, I just loved the WSCF tool provided by Christian Weyer/thinktecture a few years ago.

Note that the data contract wizard still assigns order attribute values incrementally, and not according to the best practices recommended by Microsoft:

"The Order property on the DataMemberAttribute should be used to make sure that all of the newly added data members appear after the existing data members. The recommended way of doing this is as follows: None of the data members in the first version of the data contract should have their Order property set. All of the data members added in version 2 of the data contract should have their Order property set to 2. All of the data members added in version 3 of the data contract should have their Order set to 3, and so on. It is okay to have more than one data member set to the same Order number."

Read Aaron Skonnard's Service Station article 'The Service Station for WCF' at MSDN for an introduction to the WCF WSSF.

Note that if you do not use VSTS, some of the new features such as the "WCF semantic code analysis" tool will not work (not even if you have VS2005 Pro + FxCop 1.35).

WCF Contracts: Separating Structure from Data

In my last post about designing WCF data contracts, I showed how to make a simple, straightforward hierarchical contract. In this post, I will refine the contract to make the building blocks of the data contract more reusable and how to adhere to WS-I basic profile best practices wrt naming of collections.

The data contract design of my last post is well suited from getting all data about a domain object structure in one go. It can also be used to add/modify nodes to/in the structure; but as each node data contract contains a combination of node metadata and collections of related items, it is not well suited for operations that inserts or updates items in the structure. The XSD schema of the data contract has no way of conveying what will happen with any collection items contained in the data when inserting a new ProjectDocuments node. The service might just ignore them (the 'easy' way), but this will add concealed semantics to the operations and violates the "share contract, not class/implementation" tenet of SOA. It would be better if the contract was unambiguous and easily understandable for the service consumers (the 'simple' way).

So to refactor the 'project documents' data contract into something that is well suited for both retrieving and modifying data, separating structure from data is needed. The metadata of each node in the hierarchy must be extracted into a new data contract, that becomes just another element in the node along with the collections. The new contract for the metadata is what we needed to make the schemas for insert and update operations simpler to use. In addition, the service will now have less subtle semantics.

The new ProjectDocuments contract using the new DocumentCategory data contract looks like this (click to enlarge):
The takeaway is that you should "normalize" your data contracts to some extent to make them more reusable across data contracts, messages and operations.

My colleague Anders Norås made me aware that naming collections ArrayOfXxx is not according to WS-I basic profile recommendations. WCF supports specifying the both collection element name and the collection item element name using the [CollectionDataContract] attribute. This attribute cannot, however, be applied to class members, only to a whole class. Thus, you will need a separate class for each type of collection in your data contracts:

namespace KjellSJ.Sample.Application.DataContracts
{
[CollectionDataContract(Namespace = "http://kjellsj.blogspot.com/2006/11/DocumentCard", Name = "DocumentCardCollection", ItemName = "DocumentCard")]
public class DocumentCardCollection : Collection<DocumentCard>{}
}

The use of the custom named collection is straightforward:

[System.Runtime.Serialization.DataMemberAttribute(IsRequired = true, Name = "DocumentCardList", Order = 3)]
private DocumentCardCollection _fieldDocumentCardList;

public DocumentCardCollection DocumentCardList
{
get { return _fieldDocumentCardList; }
}

The number of XSDs in your WSDL will increase significantly when applying custom names to collection. I recommend that you group related DataContract classes into a single namespace, e.g. the contract classes DocumentCategory, DocumentCard and DocumentCardCollection should have the same namespace. All contracts in a specific namespace must be versioned as one, so group your contracts wisely.

If you have code that uses the MetadataExchangeClient object to do MEX stuff, you must ensure that the MaximumResolvedReferences setting has room for all the new metadata. I use the MetaData Explorer from IDesign to inspect the service contract, and had to increase the resolve limit beyond the default 10.

WCF: Designing a Hierarchical Data Contract

We have now started designing the data contracts of our WCF services using the WSSF (Web Service Software Factory). All samples and demos I have seen so far shows simple entity objects such as a customer with a few basic value type fields. In this post I will show how to design a data contract for hierarchical data that uses value types, enums, entity objects and collections of entities.

The WSSF wizards require you to design the contracts bottom-up, as any object must exists before it can be added to the contract. Note also that a wizard cannot be rerun to expand data contracts, it can only create new contracts. This limitation is espesiaclly daft when it comes to service interfaces: you cannot add another operation to an existing service contract using the wizard. Thus, you must start by defining the leaf nodes of the structure and work your way up to the root element of the hierarchy.

I still recommend that you do a little design up front: start by identifying and writing down the use cases, identityfy the services (logical groups of operations) that you need to support the use cases, and finally write down a list of the logical operations that you think are sufficient to implement the business processes of the use cases. Use high level terms (verbs and nouns) to outline the set of operations (list, get, register, fulfill, etc; customer, offer, order, invoice, etc). Ensure that the operations in fact are business services, and not a RPC/CRUDy style programming model exposed as web-services.

The data in this example is a list of documents organized into a categorizing hierarchy. Each document again consists of metadata (the document card) and a list of files making up the document (classical DM stuff). Each file has to have an associated file type (enum).

I now prefer using the contract attributes over creating XSD schemas first, even if I still am a big fan of the contract first approach. I always extract the XSD schemas and inspect them with XmlSpy to ensure that the generated schemas looks the way I would like them to be.

This is how the data contract schema should look (split into two for readability, click to enlarge):

The file type is an XSD enum:
The data contract wizard makes it easy to add value types, enums and objects to the contract. It is not, however, possible to add a System.Collections.ObjectModel Collection<T> using the wizard. To add a collection, first add just the type (T) to the contract, then generate the contract class and modify it like this:

using System;
using System.Runtime.Serialization;
using System.Collections.ObjectModel;

namespace KjellSJ.Sample.Application.DataContracts
{

/// <summary>
/// Data Contract Class - ProjectDocuments
/// </summary>

[System.Runtime.Serialization.DataContractAttribute(Namespace = "http://kjellsj.blogspot.com/2006/11/ProjectDocuments", Name = "ProjectDocuments")]
public partial class ProjectDocuments
{
public ProjectDocuments()
{
_fieldChildNodes = new Collection<ProjectDocuments>();
_fieldDocumentCardList = new Collection<DocumentCard>();
}

private System.String _fieldId;

[System.Runtime.Serialization.DataMemberAttribute(IsRequired = true, Name = "Id", Order = 1)]
public System.String Id
{
get { return _fieldId; }
set { _fieldId = value; }
}

private System.String _fieldCode;

[System.Runtime.Serialization.DataMemberAttribute(IsRequired = true, Name = "Code", Order = 2)]
public System.String Code
{
get { return _fieldCode; }
set { _fieldCode = value; }
}

private System.String _fieldDescription;

[System.Runtime.Serialization.DataMemberAttribute(IsRequired = false, Name = "Description", Order = 3)]
public System.String Description
{
get { return _fieldDescription; }
set { _fieldDescription = value; }
}

[System.Runtime.Serialization.DataMemberAttribute(IsRequired = false, Name = "ChildNodes", Order = 4)]
private Collection<ProjectDocuments> _fieldChildNodes;

public Collection<ProjectDocuments> ChildNodes
{
get { return _fieldChildNodes; }
}

[System.Runtime.Serialization.DataMemberAttribute(IsRequired = false, Name = "DocumentCardList", Order = 5)]
private Collection<DocumentCard> _fieldDocumentCardList;

public Collection<DocumentCard> DocumentCardList
{
get { return _fieldDocumentCardList; }
}

}
}

As you can see, I have made the DocumentCardList a Collection<DocumentCard> and moved the DataMember attribute to the private member instead of the public property. The reason for moving the attribute is that I made the collection property read-only, a best practice, and the data contract serializer does not support read-only properties.

Note also the self reference to ProjectDocuments in the ChildNodes collection that makes this schema a hierachy. The collection becomes a XSD array:

<xs:complexType name="ArrayOfProjectDocuments">
<xs:sequence>
<xs:element name="ProjectDocuments" type="tns:ProjectDocuments" nillable="true" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>

Note that to adhere to the WS-I basic profile best practices, you should customize the collection element name using the [CollectionDataContract] attribute. Read more about using collections in the data contract here at MSDN.

I prefer not to mark as DataContract any enum that is referenced by a DataMember in the schemas. If you apply DataContracty to the enum, it will cause the fields using the enum to be interpreted a XSD string data type instead of a enum data type (xs:enumeration). Just leave the enum class as is and use it as any other datatype in the DataContract wizard. Read more about using enums in data contracts here at MSDN.

Web-services: interoperability & encoding

I am currently implementing MSCRM at a customer that is also implementing an enterprise portal using SiteVision delivered by Objectware's Java department. The need for providing the SiteVision portal with information about MSCRM accounts made me implement a few .NET XML web-services to be used by the Java guys. The web-service provides basic read, create and update services for accounts and contacts. The 'data transfer object' was simply implemented as a string that would contain XML according to a predefined XSD schema.

The enterprise application integration server used is BIE (Business Integration Engine; Java open source), chosen by the Java lead developer on this project. Their use of my web-services went along quite smootly as was to be expected as .NET web-services complies with WS-I BP1.0 (intro article here). I had even used the WS-I compliance tools to check my web-services.

What caused a major halt on the integration implementation was when we started testing with data containing Norwegian characters (ÆØÅ). To be more precise, this was tested and found to be OK by using MSIE as the test-client, and by using XmlSpy to retrieve data about e.g. an account, modifying the data, and finally updating the account. I strongly recommend the XmlSpy SOAP Debugger tool both for testing and debugging web-serives.

The problem was that although BIE supports web-servies using UTF-8, it does not differentiate between UTF-8 as a text encoding in the web-service and UTF-8 encoded XML and the W3C character encoding rules that dictates that a unicode character must be encoded as a #xHHHH numeric character reference (i.e. 16-bit unicode code points/units; UTF-16 NCR). For more details see 'Can XML use non-Latin characters?'.

The MSCRM 1.2 object model supports and returns XML using the #xHHHH character encoding. All strings in .NET are unicode.

The character encoding went through these phases (examples for Æ and æ):

• from our MSCRM web-service: Æ encoded as &#x00C6;
• back from BIE on create/update: Æ "encoded" as &#195;† (not even valid UTF-8 code units)
• data in MSCRM after operation: Æ

• from our MSCRM web-service: æ encoded as &#x00E6;
• back from BIE on create/update: æ encoded as &#195;¦ (which are valid UTF-8 code units)
• data in MSCRM after operation: æ

To make the problem even worse, by running the same BIE test case (workflow route) over and over, these 'funny' characters would double each time, as each XML encoded character from our service was sent back as two UTF-8 code units from BIE and not as XML UTF-16 code units (the W3C character encoding standard).

A nice unicode character encoding test tool is available here.

The Java lead developer had run into these encoding problems in BIE and modified the portal code to do some character replacing (Æ to Æ, etc) on all text to/from BIE on their side, and I asked them to change their integration implementation to comply with the WS-I and W3C standards. Unfortunately, this lead developer is more interested in hailing the glory of the application architecture & design and the open source movement than complying with standards "invented by Microsoft" :-)

Thus, I had to write a SoapExtension to modify the incoming SOAP message before it was deserialized, to change the "wrong" UTF-8 encoding into correct XML W3C UTF-8 encoding. I used this GotDotNet sample as the basis for my code and this is how I modify the incoming and outgoing SOAP messages:

public override void ProcessMessage(SoapMessage message)
{
switch (message.Stage)
{
//INCOMING
case SoapMessageStage.BeforeDeserialize:
this.ChangeIncomingEncoding();
break;
case SoapMessageStage.AfterDeserialize:
_isPostDeserialize = true;
break;

//OUTGOING
case SoapMessageStage.BeforeSerialize:
break;
case SoapMessageStage.AfterSerialize:
this.ChangeOutgoingEncoding();
break;
}
}

public override Stream ChainStream( Stream stream )
{
//http://hyperthink.net/blog/CommentView,guid,eafeef67-c240-44cc-8550-974f5d378a8f.aspx

if(!_isPostDeserialize)
{
//INCOMING
_inputStream = stream;
_outputStream = new MemoryStream();
return _outputStream;
}
else
{
//OUTGOING
_outputStream = stream;
_inputStream = new MemoryStream();
return _inputStream;
}
}

public void ChangeIncomingEncoding()
{
//at BeforeDeserialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;

TextReader reader = new StreamReader(_inputStream);
TextWriter writer = new StreamWriter(_outputStream);

string line;
while((line = reader.ReadLine()) != null)
{
writer.WriteLine( Utilities.FixBieEncoding (line) );
}
writer.Flush();

//reset the new stream to ensure that AfterDeserialize is called
if(_outputStream.CanSeek)
_outputStream.Position = 0L;
}

public void ChangeOutgoingEncoding()
{
//at AfterSerialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;

Regex regex = new Regex("utf-8", RegexOptions.IgnoreCase);
TextReader reader = new StreamReader(_inputStream);
//HACK: TextWriter writer = new StreamWriter(_outputStream);
TextWriter writer = new StreamWriter(_outputStream, System.Text.Encoding.GetEncoding(_encoding));

string line;
while((line = reader.ReadLine()) != null)
{
// change the encoding only is needed
if(_encoding != null && !_encoding.Equals("utf-8"))
line = regex.Replace(line, _encoding);
writer.WriteLine(line);
}
writer.Flush();
}

The central method here is the ChangeIncomingEncoding method that converts all the "wrong" UTF-8 encodings from BIE into correct XML W3C NCRs. Note the resetting of the position to zero on the output stream after modifying the message; this is important as forgetting to do so will cause the AfterDeserialize step of the SoapMessageStage in ProcessMessage not to be called.

After deploying the modified web-service and testing it with XmlSpy, it was time to test it with the BIE workflow dashboard. The BIE developer had set up some test cases for me, and they all worked as expected. No more funny farm in the MSCRM database.

What an illustrious victory for Java-.NET web-service interoperability!

Note that you cannot test/debug your SoapExtension code by using MSIE as the test-client as HTTP POST/GET will not trigger the SoapExtension. I used XmlSpy to test and debug my code; just set some breakpoints, start your web-service in debug mode and leave it running, then trigger the SoapExtention by making a SOAP request using XmlSpy.