In WCF you can use shared types from existing assemblies when generating a service proxy. Use the advanced options in VS2008, otherwise use SVCUTIL.EXE with the /reference /r option. Note that you can only use /reference when your WSDL operations, types and messages adheres to the DataContractSerializer rules that I described this spring.
I'm currently trying to research why we are not able to use shared types for our wsdl:fault message parts.
<wsdl:message name="GetCustomerRequest">
<wsdl:part element="tns:GetCustomerRequest" name="parameters" />
</wsdl:message>
<wsdl:message name="GetCustomerResponse">
<wsdl:part element="tns:GetCustomerResponse" name="parameters" />
</wsdl:message>
<wsdl:message name="FaultDetailList">
<wsdl:part element="fault:FaultDetailList" name="detail" />
</wsdl:message>
<wsdl:portType name="CustomerService">
<wsdl:operation name="GetCustomer">
<wsdl:input message="tns:GetCustomerRequest" name="GetCustomerRequest" />
<wsdl:output message="tns:GetCustomerResponse" name="GetCustomerResponse" />
<wsdl:fault message="tns:FaultDetailList" name="FaultDetailList" />
</wsdl:operation>
</wsdl:portType>
<wsdl:binding name="CustomerServiceSoap11" type="tns:CustomerService">
<soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http" />
<wsdl:operation name="GetCustomer">
<soap:operation soapAction="" />
<wsdl:input name="GetCustomerRequest">
<soap:body use="literal" />
</wsdl:input>
<wsdl:output name="GetCustomerResponse">
<soap:body use="literal" />
</wsdl:output>
<wsdl:fault name="FaultDetailList">
<soap:fault name="FaultDetailList" use="literal" />
</wsdl:fault>
</wsdl:operation>
</wsdl:binding>
Even if the fault details contains only xs:string elements, we get this error when generating the proxy:
Cannot import wsdl:portTypeDetail: An exception was thrown while running a WSDL import extension: System.ServiceModel.Description.DataContractSerializerMessageContractImporterError: Referenced type 'SharedDataContracts.FaultDetailType, MyService, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' with data contract name 'FaultDetailType' in namespace 'urn:kjellsj.blogspot.com/FaultDetail/1.0' cannot be used since it does not match imported DataContract. Need to exclude this type from referenced types.
I thought that writing a unit-test to import the WSDL using the underlying SVCUTIL importer directly should help me deduce the cause of the error. You will find all the code that you need to import WSDL using DataContractSerializerMessageContractImporter here at the "nV Framework for Enterprise Integration" project as CodePlex.
Add these lines to enforce the use of /namespace, /reference and /ct from code (note that these switches can be used multiple times):
xsdDataContractImporter.Options.Namespaces. Add(new KeyValuePair<string, string>("*", "MyService"));
xsdDataContractImporter.Options.ReferencedCollectionTypes. Add(typeof(List<>));
xsdDataContractImporter.Options.ReferencedTypes. Add(typeof(SharedDataContracts.FaultDetailList));
xsdDataContractImporter.Options.ReferencedTypes. Add(typeof(SharedDataContracts.FaultDetailType));
Sure enough, I get the exact same exception in my unit-test. Inspecting the imported contract shows that the operation has a fault with a "null" DetailType XML Schema type (watch: contracts[0].Operations[0].Faults[0].DetailType). I suspect that it is the "null" DetailType in the FaultDescription of the imported ServiceDescription for the WSDL that causes the error for the referenced shared types.
As it turns out this is a bug that has been fixed in .NET 3.5 SP1. The workaround for earlier versions of SVCUTIL is to add nillable="true" to all elements that are reference types (strings and complexTypes) in your wsdl:types XSD schemas. Or just drop the /reference switch and cut'n'paste the generated code to use your shared types.
Showing posts with label Serializing. Show all posts
Showing posts with label Serializing. Show all posts
Wednesday, June 18, 2008
Wednesday, April 16, 2008
WCF: Client Domain Objects, Decorator Pattern
The client domain objects (aka business entity - BE) must be related to the data contracts provided by the web-services. Still, they will need to be enhanced with e.g. dirty-tracking, validation, state, property conversions (xs:date), business rules, and other aspects needed to produce a fully working client application. This article describes how to achieve this when sharing domain objects across layers is not an option (i.e. interop with JEE services), and the same applies to AOP.
One option for implementing the BE objects, described in this article, is to apply the Decorator pattern to the WSDL scema-based DTO objects to enhance them into full-fledged domain objects. Read ‘Design of Client Domain Objects’ for an intro this option and the related Mapper option.
[DataMemberAttribute(Name = "NominatedValue", . . .) ]
private UnitValueType NominatedValueData
The next two issues are caused by adding new items to the data contract or changing some existing items. The latter is not recommended and is definitively not a recommended practice for published contracts:
[DataMemberAttribute(. . . , Order=<order number in XML xs:sequence element for item>) ]
For new major versions of data contracts, you must consider creating a new version of the BE class in a new .NET namespace. This allows you to do controlled versioning of your domain objects in synchronization with the versioning policy of the service contracts.
However, during the development of the web-services, the contracts are bound to get breaking changes as the data contracts evolve (which is OK as they are still not published). For this reason, wait as long as you can to make changes to the generated BE code, stick to working on the partial class used to decorate the BE. This will allow you to re-generate the schema-bound parts of your BE to keep up with the changes in the data contracts.
A side note: deriving your BE objects from the DTO objects is an alternative to partial classes that strictly enforces the separation of decorator code from generated code, and allows the generated code to be auto-generated over and over again - like for the Mapper option. Alas, there are drawbacks to inheriting the DTOs also, as none of the generated proxy artifacts are geared towards this. Look into using an AOP framework to do dynamic subclassing if chosing to derive your BE objects.
For data contracts under development, any combination of adding, changing, removing and reordering of data contract items can be expected to occur. The XML serialization attributes must then be updated accordingly.
One option for implementing the BE objects, described in this article, is to apply the Decorator pattern to the WSDL scema-based DTO objects to enhance them into full-fledged domain objects. Read ‘Design of Client Domain Objects’ for an intro this option and the related Mapper option.
If you do not like this Decorator option due to the weak [XML attribute] link to the data contracts (see end of article), you can always use a derived decorator class instead of a partial class. Make all your BE enhancements in the derived class using a combination of DTO overrides and new domain logic. If you prefer explicit, debuggable code, you're better off using the Mapper option.
Service Proxy
Use the SVCUTIL.EXE /REFERENCE switch when generating the proxy code and messages to reuse your BE objects as DTOs and to avoid getting new, duplicate DTO objects generated (might not compile due to name clashes). Alternatively, just remove the DTO code from the generated proxy. Make sure that the proxy code reference the .NET namespace and interface assembly that contains your BE objects.
Business Entities
You can use the data contract DTO objects generated by SVCTIL.EXE /DCONLY to get started. This will give you data-binding enabled (/EDB) anemic entity objects with all the properties in place, annotated with XML serialization attributes. Just ensure that the output files from SVCUTIL.EXE do not overwrite your code. The DTO objects become your generated BE objects - ready for decoration.
The BE objects must be manually maintained whenever you make changes to the generated BE code, e.g. to add EntLib3 validation attributes on the properties. Keep your additions to the generated BE in a partial class as far as possible, e.g. add new properties and methods in a partial class extending the decorated class. If all your decorator code can be isolated into a partial class, then the generated BE code can be re-generated without losing your extended BE code.
Our programming model for decorator-based client domain objects is based on using two partial classes; one T.schema.cs class file for all code based originally on the data contracts, and one T.cs class file for all added aspects. T is the type name of the BE.
Note that BE constructors will not be called during deserialization. Create a method to be used as a surrogate constructor to be run after deserialization and mark it with [OnDeserialized]. This is a good place for resetting e.g. the dirty flag:
[OnDeserialized]
void DeserializationCtor(StreamingContext context)
{
_isDirty = false;
}
Your BE class must still have a constructor to initialize the object for normal use. The constructor must initialize all properties that must have a defined initial value. In addition, the c'tor must initialize all child objects that are exposed as (de-normalized) public properties, to avoid null reference exceptions when using the BE (make it simple to use the BE). This also applies to child collections; create the collection to make it ready for use.
public NominationType()
{
//NOTE: WILL NOT BE CALLED ON DE-SERIALIZATION
//initialize all normalized valueobjects with members
this.NominatedValueField = new UnitValueType();
this.NominatedValueField.Unit = String.Empty;
}
Also initialize all fields that are required by the XSD schema, failure to initialize these fields will give serialization errors:
[DataMemberAttribute(IsRequired=true, . . .) ]
Don't break the DRY principle by adding the value object initialization code multiple places.
Dirty Tracking
If your BE implements INotifyPropertyChanged to enable data-binding, then the event is a good place to add dirty-tracking: Simple Dirty Tracking in Domain Objects
Converting XML Data Types to/from Strongly Typed Properties
Note that xs:date becomes string in .NET as there is no Date only type, just DateTime. Do not let this kind of weak type design leak into your BE objects. Contain the weak typing inside your BE object by converting to/from strongly typed properties.
private string _demandDateData; //XML xs:date datatype
public DateTime DemandDate
{
get
{
return DateTime.ParseExact(this._demandDateData, WSDateFormat, null);
}
set
{
string xsDate = value.ToString(WSDateFormat);
if (this._demandDateData.Equals(xsDate) != true)
{
this._demandDateData = xsDate;
this.RaisePropertyChanged("DemandDate");
}
}
}
This keeps your BE object within the "Simple vs Easy" principle.
De-normalization of BE Value Objects
The data contracts provided by the web-services will most likely contain some DDD value objects for numeric types annotated with some related attributes (e.g. money with currency) and other similar cases. These value objects will become child objects of the BE object, which is not very data-binding friendly as most controls (grids, etc) needs the data to be non-nested properties of the bound object. In addition, one control can only have one binding source. Thus, unlike master-details type binding that can be resolved using multiple binding sources, nested value objects must be de-normalized and projected as normal properties directly on the BE objects.
Start by making the value object private and give the property a new name (e.g. suffix with Data) while specifying the original name in the serialization attribute to keep the link to the XML. Keep the RaisePropertyChanged parameter equal to the property name.
[DataMemberAttribute(Name = "NominatedValue", IsRequired = true, EmitDefaultValue = false, Order = 2)]
private UnitValueType NominatedValueData
{
get { return this.NominatedValueField; }
set { . . . }
}
Then create the de-normalized public properties (in a partial class) as required by the BE and data-binding needs. The new public properties should be wrappers exposing an element of the original private member value object, using a friendly name and the correct data type. Make sure that the new-old value comparison is correct. Keep the RaisePropertyChanged parameter equal to the new property name.
public Decimal NominatedValue
{
get { return this.NominatedValueField.Value; }
set
{
if (this.NominatedValueField.Value.Equals(value) != true)
{
this.NominatedValueField.Value = value;
this.RaisePropertyChanged("NominatedValue");
}
}
}
public string NominatedValueUnit
{
get { return this.NominatedValueField.Unit; }
set { . . . this.RaisePropertyChanged("NominatedValueUnit"); }
}
As can be seen from the example, the new public properties are named by concatenating the value object name and the exposed property name; except for when the property name is just repeating the last part of the value object (i.e. NominatedValue.Value).
Remember to initialize all the value object children in the BE c'tor; both create the child objects and set their initial values.
Keeping the BE Synchronized with the Data-Contract Schema
It is imperative that the XML serialization attributes in code are correct according to the data contract XSD when using the decorator approach on proxy DTO objects to implement BE objects. There are three typical changes that can break the mapping between the BE code and the payload XML:
- Changes to the code by renaming properties without updating the XML serialization attributes accordingly
- Changes to the data contract XSD schema without adding properties with correct XML serialization attributes
- Changes to the data contract XSD schema without updating the XML serialization attributes accordingly
[DataMemberAttribute(Name = "NominatedValue", . . .) ]
private UnitValueType NominatedValueData
The next two issues are caused by adding new items to the data contract or changing some existing items. The latter is not recommended and is definitively not a recommended practice for published contracts:
- It is a proven practice that published contracts are not changed, except for non-breaking changes in minor versions.
- For new, non-breaking versions of the contract, items must only be added at the end of the existing contract.
[DataMemberAttribute(. . . , Order=<order number in XML xs:sequence element for item>) ]
For new major versions of data contracts, you must consider creating a new version of the BE class in a new .NET namespace. This allows you to do controlled versioning of your domain objects in synchronization with the versioning policy of the service contracts.
However, during the development of the web-services, the contracts are bound to get breaking changes as the data contracts evolve (which is OK as they are still not published). For this reason, wait as long as you can to make changes to the generated BE code, stick to working on the partial class used to decorate the BE. This will allow you to re-generate the schema-bound parts of your BE to keep up with the changes in the data contracts.
A side note: deriving your BE objects from the DTO objects is an alternative to partial classes that strictly enforces the separation of decorator code from generated code, and allows the generated code to be auto-generated over and over again - like for the Mapper option. Alas, there are drawbacks to inheriting the DTOs also, as none of the generated proxy artifacts are geared towards this. Look into using an AOP framework to do dynamic subclassing if chosing to derive your BE objects.
For data contracts under development, any combination of adding, changing, removing and reordering of data contract items can be expected to occur. The XML serialization attributes must then be updated accordingly.
Thursday, March 27, 2008
WCF: DataContractSerializer Schema Rules
Anyone who have tried to do contract/schema first design of data contracts or tried to generate proxies using SVCUTIL.EXE in an interop scenario with e.g. Spring-WS, will have run into how WCF magically decides to use the old XmlSerializer rather than the new DataContractsSerializer (DCS). And deducing which parts of a schema that cause this is not easy with real-life contracts.
The XML schema rules/limitations for DCS is well documented, but the proxy generator will not tell you which of the rules your XSD/WSDL does not adhere to. Even worse, if you specify /serializer:DataContractSerializer and the rules are broken; the proxy will get generated without any error, but also without any messages and data transfer objects.
Note that the SVCUTIL switches /collectionType /ct and /reference /r only work with the DataContractSerializer, not the XmlSerializer. Thus, if the proxy generator falls back to the XmlSerializer, you will not get collections (List / BindingList / etc) for arrays nor any reuse of existing shared object types.
The only way I've found that will tell you why DCS is not used, is by using the /DCONLY option to extract the data contracts from the WSDL. Using this switch will disclose data contract (XSD) errors one at a time, but it is still better than not knowing why. Note that it will not disclose any errors related to the messages not being document/literal wrapped (WSDL).
The most common XSD schema lapsus is not wrapping arrays in a two-level complex type, one for the list of items and one for the items themselves. This is not very well documented in the DCS rules. The array XSD schema must follow this construct:
<xs:element name="ItemList" type="tns:ItemListType" />
<xs:complexType name="ItemListType">
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" name="Item" type="tns:ItemType" />
</xs:sequence>
</xs:complexType>
<xs:complexType name="ItemType">
<xs:sequence>
<xs:element name="Id" type="xs:long" />
<xs:element name="Value" nillable="true" type="xs:string" />
</xs:sequence>
</xs:complexType>
Note that there is no need to use the prefix ArrayOf on the list element name.
The reason for wrapping the array might seem quite silly when the message contains elements of only one type. However, look at this XML segment that contains elements of multiple types:
<root><a/><b/><b/><b/></root>
The DCS does not like a XSD schema where the children of an element varies in numbers and are not of the same type; it requires that all child elements that varies in numbers must have the same type, thus the wrapper element:
<root><a/><itemList><b/><b/><b/></itemList></root>
This way the <root> element contains a fixed number of different elements, the <a> element and the <itemList> element; while the <itemList> element contains a varying number of <b> elements only. In addition, an empty/null array can still be represented in the XML.
The other typical WSDL lapsus wrt DCS is about the message part definitions and the related operation input and output elements plus the related binding section. The important rule here is to remember to use only one part in each message (WS-I Basic Profile) and to set the message part attribute @name = "parameters" (by convention) to make the messages document/literal wrapped. Other @name conventions are "header" for message headers and "detail" for fault message parts to be used as wsdl:fault elements in operations.
This is an example that will allow you to use the DataContractSerializer:
<wsdl:message name="GetItemRequest">
<wsdl:part element="tns:GetItemRequestType" name="parameters" />
</wsdl:message>
<wsdl:message name="GetItemResponse">
<wsdl:part element="tns:GetItemResponseType" name="parameters" />
</wsdl:message>
<wsdl:message name="FaultDetailList">
<wsdl:part element="fault:FaultDetailList" name="detail" />
</wsdl:message>
<wsdl:portType name="InspectValues">
<wsdl:operation name="GetItem">
<wsdl:input message="tns:GetItemRequest" name="GetItemRequest" />
<wsdl:output message="tns:GetItemResponse" name="GetItemResponse" />
<wsdl:fault message="tns:FaultDetailList" name="FaultDetailList" />
</wsdl:operation>
</wsdl:portType>
<wsdl:binding name="InspectValuesSoap11" type="tns:InspectValues">
<soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http" />
<wsdl:operation name="GetItem">
<soap:operation soapAction="" />
<wsdl:input name="GetItemRequest">
<soap:body use="literal" /></wsdl:input>
<wsdl:output name="GetItemResponse">
<soap:body use="literal" /></wsdl:output>
<wsdl:fault name="FaultDetailList">
<soap:fault name="FaultDetailList" use="literal"/></wsdl:fault>
</wsdl:operation>
</wsdl:binding>
Note that if your message part element is a wrapped array, the referenced element must be marked as nillable. This is a SVCUTIL WSDL requriement, and applies even if using /DCONLY generates the contracts just fine.
If the generated DCS-based proxy interface contains operations that has no input or output messages, then your WSDL does not adhere to the above rules. The same is true if you get any "cannot import" errors or warnings related to wsdl:binding, wsdl:portType or wsdl:port. Note also that the element referenced as a message part must be a DCS allowable <xs:complexType> with an <xs:sequence> child (can also be empty). Read more about message schema rules in Handcrafting WCF friendly WSDLs.
There are some known issues with SVCUTIL /reference regarding the mismatches between some XSD schema types and .NET types (e.g. xs:date, xs:positiveinteger), which will make /reference fail as the WSDL types don't correspond with the referenced .NET data contract types: .NET has no Date only data type, just DateTime - thus xs:date becomes string. Also watch out for enumerations being represented as pure strings without any defined enum items. These issues will cause /reference to fail.
Note also that your custom shared collection types must also be referenced using /ct even if already added using /reference. Forgetting this will give your duplicate classes for all your [CollectionDataContractAttribute] lists defined in your data contracts.
As you might have noticed, using nillable elements might be required for /reference to work properly. Always try setting nillable="true" if you can't get any code generated when using the DataContractSerializer.
Pre .NET 3.5 SP1: All xs:string elements are required to be nillable. The same applies to your custom collection types.
I wish the DCS rules documentation had included more XSD schema examples, both for allowable and invalid schema constructs.
The /reference option is very important when using an architecture like CAB/SCSF where the service implementation (proxy) is separated from the service definition (interface). Then, it is important that the data contracts are included in the service interface assembly for reuse across business modules; i.e. the DTO objects can not be inside the proxy implementation assembly. The proxy itself is most likely wrapped in a service agent and should never be exposed in the DDD service or repository pattern interface.
If you can't get the /dconly plus /reference combo to work, then you're stuck with generating the contracts and proxy into classes in a single file using a .Service.Interface namespace and then extracting the proxy code into a separate file in a .Service namespace using e.g. a RegEx script, followed by some namespace using/imports refactoring for the extracted proxy code file so that it references the data contracts in the interface namespace.
The XML schema rules/limitations for DCS is well documented, but the proxy generator will not tell you which of the rules your XSD/WSDL does not adhere to. Even worse, if you specify /serializer:DataContractSerializer and the rules are broken; the proxy will get generated without any error, but also without any messages and data transfer objects.
Note that the SVCUTIL switches /collectionType /ct and /reference /r only work with the DataContractSerializer, not the XmlSerializer. Thus, if the proxy generator falls back to the XmlSerializer, you will not get collections (List / BindingList / etc) for arrays nor any reuse of existing shared object types.
The only way I've found that will tell you why DCS is not used, is by using the /DCONLY option to extract the data contracts from the WSDL. Using this switch will disclose data contract (XSD) errors one at a time, but it is still better than not knowing why. Note that it will not disclose any errors related to the messages not being document/literal wrapped (WSDL).
The most common XSD schema lapsus is not wrapping arrays in a two-level complex type, one for the list of items and one for the items themselves. This is not very well documented in the DCS rules. The array XSD schema must follow this construct:
<xs:element name="ItemList" type="tns:ItemListType" />
<xs:complexType name="ItemListType">
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" name="Item" type="tns:ItemType" />
</xs:sequence>
</xs:complexType>
<xs:complexType name="ItemType">
<xs:sequence>
<xs:element name="Id" type="xs:long" />
<xs:element name="Value" nillable="true" type="xs:string" />
</xs:sequence>
</xs:complexType>
Note that there is no need to use the prefix ArrayOf on the list element name.
The reason for wrapping the array might seem quite silly when the message contains elements of only one type. However, look at this XML segment that contains elements of multiple types:
<root><a/><b/><b/><b/></root>
The DCS does not like a XSD schema where the children of an element varies in numbers and are not of the same type; it requires that all child elements that varies in numbers must have the same type, thus the wrapper element:
<root><a/><itemList><b/><b/><b/></itemList></root>
This way the <root> element contains a fixed number of different elements, the <a> element and the <itemList> element; while the <itemList> element contains a varying number of <b> elements only. In addition, an empty/null array can still be represented in the XML.
The other typical WSDL lapsus wrt DCS is about the message part definitions and the related operation input and output elements plus the related binding section. The important rule here is to remember to use only one part in each message (WS-I Basic Profile) and to set the message part attribute @name = "parameters" (by convention) to make the messages document/literal wrapped. Other @name conventions are "header" for message headers and "detail" for fault message parts to be used as wsdl:fault elements in operations.
This is an example that will allow you to use the DataContractSerializer:
<wsdl:message name="GetItemRequest">
<wsdl:part element="tns:GetItemRequestType" name="parameters" />
</wsdl:message>
<wsdl:message name="GetItemResponse">
<wsdl:part element="tns:GetItemResponseType" name="parameters" />
</wsdl:message>
<wsdl:message name="FaultDetailList">
<wsdl:part element="fault:FaultDetailList" name="detail" />
</wsdl:message>
<wsdl:portType name="InspectValues">
<wsdl:operation name="GetItem">
<wsdl:input message="tns:GetItemRequest" name="GetItemRequest" />
<wsdl:output message="tns:GetItemResponse" name="GetItemResponse" />
<wsdl:fault message="tns:FaultDetailList" name="FaultDetailList" />
</wsdl:operation>
</wsdl:portType>
<wsdl:binding name="InspectValuesSoap11" type="tns:InspectValues">
<soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http" />
<wsdl:operation name="GetItem">
<soap:operation soapAction="" />
<wsdl:input name="GetItemRequest">
<soap:body use="literal" /></wsdl:input>
<wsdl:output name="GetItemResponse">
<soap:body use="literal" /></wsdl:output>
<wsdl:fault name="FaultDetailList">
<soap:fault name="FaultDetailList" use="literal"/></wsdl:fault>
</wsdl:operation>
</wsdl:binding>
Note that if your message part element is a wrapped array, the referenced element must be marked as nillable. This is a SVCUTIL WSDL requriement, and applies even if using /DCONLY generates the contracts just fine.
If the generated DCS-based proxy interface contains operations that has no input or output messages, then your WSDL does not adhere to the above rules. The same is true if you get any "cannot import" errors or warnings related to wsdl:binding, wsdl:portType or wsdl:port. Note also that the element referenced as a message part must be a DCS allowable <xs:complexType> with an <xs:sequence> child (can also be empty). Read more about message schema rules in Handcrafting WCF friendly WSDLs.
There are some known issues with SVCUTIL /reference regarding the mismatches between some XSD schema types and .NET types (e.g. xs:date, xs:positiveinteger), which will make /reference fail as the WSDL types don't correspond with the referenced .NET data contract types: .NET has no Date only data type, just DateTime - thus xs:date becomes string. Also watch out for enumerations being represented as pure strings without any defined enum items. These issues will cause /reference to fail.
Note also that your custom shared collection types must also be referenced using /ct even if already added using /reference. Forgetting this will give your duplicate classes for all your [CollectionDataContractAttribute] lists defined in your data contracts.
As you might have noticed, using nillable elements might be required for /reference to work properly. Always try setting nillable="true" if you can't get any code generated when using the DataContractSerializer.
Pre .NET 3.5 SP1: All xs:string elements are required to be nillable. The same applies to your custom collection types.
I wish the DCS rules documentation had included more XSD schema examples, both for allowable and invalid schema constructs.
The /reference option is very important when using an architecture like CAB/SCSF where the service implementation (proxy) is separated from the service definition (interface). Then, it is important that the data contracts are included in the service interface assembly for reuse across business modules; i.e. the DTO objects can not be inside the proxy implementation assembly. The proxy itself is most likely wrapped in a service agent and should never be exposed in the DDD service or repository pattern interface.
If you can't get the /dconly plus /reference combo to work, then you're stuck with generating the contracts and proxy into classes in a single file using a .Service.Interface namespace and then extracting the proxy code into a separate file in a .Service namespace using e.g. a RegEx script, followed by some namespace using/imports refactoring for the extracted proxy code file so that it references the data contracts in the interface namespace.
Monday, May 07, 2007
WCF: Exception Handling with Logging
In my 'WCF Exception Shielding using Generics' post there was a "//TODO: add logging as applicable" comment, which I now have implemented using the EntLib3 Logging Application Block.
As part of the error information I wanted to write to the Windows Application Event Log was the actual request message causing the service operation exception. Thus, I googled for a solution for how to get the XML of the incoming message, so that I could log it along with the stack trace. There are a lot of options for getting at the message in WCF:
The code to get at the message XML from within an operation is quite simple:
private static string GetMessageXml(DefaultMessage request)
{
return OperationContext.Current. RequestContext.RequestMessage.ToString();
}
This approach does not work for operations with no RequestContext, such as one-way operations.
The logging code is added after creating the fault details, this makes it possible to use fault details data in the log. This is how my WCF exception shielding with logging looks like:
public static FaultException<T> NewFaultException<T>(Exception ex)
where T : FaultContracts.DefaultFaultContract, new()
{
return NewFaultException<T>(ex, null);
}
public static FaultException<T> NewFaultException<T>(Exception ex, DefaultMessage request)
where T : FaultContracts.DefaultFaultContract, new()
{
return CreateNewFaultException<T>(ex, request);
}
private static FaultException<T> CreateNewFaultException<T>(Exception ex, DefaultMessage request)
where T : FaultContracts.DefaultFaultContract, new()
{
StackFrame stackFrame = new StackFrame(2);
string methodName = stackFrame.GetMethod().Name;
T fault = new T();
Type type = fault.GetType();
type.InvokeMember("SetFaultDetails", BindingFlags.Public BindingFlags.Instance BindingFlags.InvokeMethod, null, fault, new object[] { ex });
string msg = String.Format("An unexpected error occurred in {0}.", methodName);
Dictionary<string, object> props = new Dictionary<string, object>();
props.Add("stackTrace", ex.ToString());
if(request!=null)
{
string xml = GetMessageXml(request);
props.Add("requestMessage", xml);
}
Logger.Write(msg, Category.General, Priority.Highest, (int)EventLogId.Exception, TraceEventType.Error, ex.Message, props);
string reasonText = methodName + " failed, check detailed fault information.";
#if DEBUG
reasonText = methodName + " error: " + ex.ToString();
#endif
return new FaultException<T>(fault, new FaultReason(reasonText));
}
Note that I now use an inner private method to create the new fault to be thrown and thus must skip two stack frames to get the details of the original method called. The DEBUG section is there to make it simple to see full exception details in NUnit or with Fiddler when debugging the service. This DEBUG code must never make it into release builds, as this is not exactly exception shielding.
I have not looked into the WCF app-blocks integration mechanisms added in the april release of EntLib3, that is next on my task list for quieter days. There is not much info about this to be found, but Guy Burstein provides some samples. Btw, you should check out the Patterns and Practices Guidance site, it contains useful stuff and tutorials about EntLib3.
As part of the error information I wanted to write to the Windows Application Event Log was the actual request message causing the service operation exception. Thus, I googled for a solution for how to get the XML of the incoming message, so that I could log it along with the stack trace. There are a lot of options for getting at the message in WCF:
- Write a custom message interceptor at the channel level
- Use the Message class directly in the service operations definitions
- Use CreateMessage internally to get at the Message class methods to reproduce the behind-the-scenes stuff of WCF
The code to get at the message XML from within an operation is quite simple:
private static string GetMessageXml(DefaultMessage request)
{
return OperationContext.Current. RequestContext.RequestMessage.ToString();
}
This approach does not work for operations with no RequestContext, such as one-way operations.
The logging code is added after creating the fault details, this makes it possible to use fault details data in the log. This is how my WCF exception shielding with logging looks like:
public static FaultException<T> NewFaultException<T>(Exception ex)
where T : FaultContracts.DefaultFaultContract, new()
{
return NewFaultException<T>(ex, null);
}
public static FaultException<T> NewFaultException<T>(Exception ex, DefaultMessage request)
where T : FaultContracts.DefaultFaultContract, new()
{
return CreateNewFaultException<T>(ex, request);
}
private static FaultException<T> CreateNewFaultException<T>(Exception ex, DefaultMessage request)
where T : FaultContracts.DefaultFaultContract, new()
{
StackFrame stackFrame = new StackFrame(2);
string methodName = stackFrame.GetMethod().Name;
T fault = new T();
Type type = fault.GetType();
type.InvokeMember("SetFaultDetails", BindingFlags.Public BindingFlags.Instance BindingFlags.InvokeMethod, null, fault, new object[] { ex });
string msg = String.Format("An unexpected error occurred in {0}.", methodName);
Dictionary<string, object> props = new Dictionary<string, object>();
props.Add("stackTrace", ex.ToString());
if(request!=null)
{
string xml = GetMessageXml(request);
props.Add("requestMessage", xml);
}
Logger.Write(msg, Category.General, Priority.Highest, (int)EventLogId.Exception, TraceEventType.Error, ex.Message, props);
string reasonText = methodName + " failed, check detailed fault information.";
#if DEBUG
reasonText = methodName + " error: " + ex.ToString();
#endif
return new FaultException<T>(fault, new FaultReason(reasonText));
}
Note that I now use an inner private method to create the new fault to be thrown and thus must skip two stack frames to get the details of the original method called. The DEBUG section is there to make it simple to see full exception details in NUnit or with Fiddler when debugging the service. This DEBUG code must never make it into release builds, as this is not exactly exception shielding.
I have not looked into the WCF app-blocks integration mechanisms added in the april release of EntLib3, that is next on my task list for quieter days. There is not much info about this to be found, but Guy Burstein provides some samples. Btw, you should check out the Patterns and Practices Guidance site, it contains useful stuff and tutorials about EntLib3.
Wednesday, February 07, 2007
WCF Streaming: Upload files over HTTP
We have a web application that requires a robust and reliable document upload mechanism for very large files, and are currently using Microsoft BITS technology to achieve this. With the introduction of WCF in the solution, I set out to implement an upload service prototype using WCF and a HTTP binding.
You have two main options for uploading files in WCF: streamed or buffered/chunked. The latter option is the one that provides reliable data transfer as you can use wsHttpBinding with WS-ReliableMessaging turned on. Reliable messaging cannot be used with streaming as the WS-RM mechanism requires processing the data as a unity to apply checksums, etc. Processing a large file this way would require a huge buffer and thus a lot of available memory on both client and server; denial-of-service comes to mind. The workaround for this is chunking: splitting the file into e.g. 64KB fragments and using reliable, ordered messaging to transfer the data.
If you do not need the robustness of reliable messaging, streaming lets you transfer large amount of data using small message buffers without the overhead of implementing chuncking. Streaming over HTTP requires you to use basicHttpBinding; thus you will need SSL to encrypt the transferred data. Buffered transfer, on the other hand, can use wsHttpBinding, which by default provides integrity and confidentiality for your messages; thus there is no need for SSL then.
Read more about 'Large Data and Streaming' at MSDN.
So I implemented my upload prototype following the 'How to: Enable Streaming' guidelines, using a VS2005 web application (Cassini) as the service host. I made my operation a OneWay=true void operation with some SOAP headers to provide meta-data:
The reason for using message headers for meta-data is that WCF requires that the stream object is the only item in the message body for a streamed operation. Headers are the recommended way for sending meta-data when streaming. Note that headers are always sent before the body, thus you can depend on receving the header data before processing the streamed data.
Note that you should provide a separate endpoint (address, binding, contract) for your streamed services. The main reason for this is that configuration such as transferMode = "Streamed" applies to all operations in the endpoint. The same goes for transferMode, maxBufferSize, maxReceivedMessageSize, receiveTimeout, etc.
I use MTOM as the encoding format for the streamed data and set the max file size to 64MB. I also set the buffer size to 64KB even if I read the input stream in 4KB chucks, this to avoid receive buffer underruns. My binding config looks like this:
I have set the transport security to none as I am testing the prototype without using SSL. I have not modified the service behavior; the default authentication (credentials) and authorization behaviors of the binding is used.
Note that the setting for transferMode does not propagate to clients when using a HTTP binding. You must manually edit the client config file to set transferMode = "Streamed" after using 'Add service reference' or running SVCUTIL.EXE. If you forget to do this, the transfer mode will be "Buffered" and you will get an error like this:
System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://localhost:1508/Host/FileTransferService.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.
After ensuring that the client basicHttpBinding config was correct, I ran the unit test once more. Now I got this error from VS2005 Cassini:
System.ServiceModel.ProtocolException: The remote server returned an unexpected response: (400) Bad Request.
System.Net.WebException: The remote server returned an error: (400) Bad Request.
I have inspected the HTTP traffic using Fiddler and can see nothing wrong with the MTOM encoded request. Googling lead me to a lot of frustrated "streamed" entries at the forums.microsoft.com "Indigo" group, but no solution to my problem. So I made a self-hosted service using a console application, using the code provided in the WCF samples (see below), and it worked like a charm. I expect that it is just VS2005 Cassini that does not support streamed transfers/MTOM encoding.
I then installed IIS to check whether HTTP streaming works better with IIS. The first error I ran into when uploading my 1KB test file was this:
System.ServiceModel.CommunicationException: The underlying connection was closed: The connection was closed unexpectedly.
System.Net.WebException: The underlying connection was closed: The connection was closed unexpectedly.
Inspecting the traffic with Fiddler, I found the HTTP request to be fine, but no HTTP response. Knowing that this is a sure sign of a server-side exception, I debugged the service operation. The error was caused by the IIS application pool identity (ASPNET worker process) lacking NTFS rights to store the uploaded file on disk. Note that WCF by default will not impersonate the caller, thus the identity used to run your service will need to have sufficient permissions on all resources that your service needs to access. This includes the temp folder, which is used to provide the WSDL file for HTTP GET.
As the UploadFile operation has [OperationContract(IsOneWay = true)], it cannot have a response and neither a fault contract. Thus, there will be no HTTP response when a server-side exception occurs, just the famous "The underlying connection was closed" message.
I assigned the correct NTFS rights to my upload folder and re-ran the unit test: streamed upload to a WCF service hosted by IIS worked. The next unit test used a 43MB file to check if uploading data larger than the WCF buffer size works. The test gave me this error:
System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:59.8590000'.System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
I was a bit puzzled as the file was less than the WCF maxReceivedMessageSize limit set to 64MB, and the stream read buffer was only 4KB as compared to the WCF maxBufferSize being 64KB. Knowing that IIS/ASP.NET has a system enforced limit on the size of a HTTP request to prevent denial-of-service attacks, which seems to be unrelated to WCF as I do not use the AspNetCompatibilityRequirements service setting, I decided to set the maxRequestLength limit to 64MB also (default: 4096KB). This time the "stream large file upload" test passed!
So for WCF streaming to IIS to work properly, you need this <system.web> config in addition to the <system.serviceModel> config in the IIS web.config file:
Make the maximum HTTP request size equal to the WCF maximum request size.
I have to recommend the 'Service Trace Viewer' (SvcTraceViewer.EXE) and the 'Service Configuration Editor' (SvcConfigEditor.EXE) provided in the WCF toolbox (download .NET3 SDK). These tools make it really simple to add tracing and logging to your service for debugging and diagnosing errors. Start the WCF config editor, open your WCF config file, select the 'Diagnostics' node and just click "Enable tracing" in the "Tracing" section to turn on tracing. The figure shows some typical settings (click to enlarge):
Save the config file and run the unit test to invoke the service. This will generate the web_tracelog.svclog file to use as input for the trace viewer. Open the trace file in the trace viewer and look for the errors (red text). Click an error to see the exception details. The figure shows some of the details available for the "Maximum request length exceeded" error (click to enlarge):
The WCF tools installed by the SDK is located here:
C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin\
The code to process the incoming stream is rather simple, storing the uploaded files to a single folder on disk and overwriting any existing files:
The 'Stream Sample' available on MSDN contains all the code you need to upload a file as a stream to a self-hosted WCF service and then save it to disk on the server by reading the stream in 4KB chunks. The download contains about 150 solutions with more than 4800 files, so there is a lot of stuff in it. Absolutely worth a look. The streaming sample is located here:
<unzip folder> \TechnologySamples\Basic\Contract\Service\Stream
Please let me know if you have been able to get HTTP streaming with MTOM encoding to work with Microsoft VS2005 Cassini.
Finally, a useful tip for streaming download: How to release streams and files using Disponse() on the the message contract.
You have two main options for uploading files in WCF: streamed or buffered/chunked. The latter option is the one that provides reliable data transfer as you can use wsHttpBinding with WS-ReliableMessaging turned on. Reliable messaging cannot be used with streaming as the WS-RM mechanism requires processing the data as a unity to apply checksums, etc. Processing a large file this way would require a huge buffer and thus a lot of available memory on both client and server; denial-of-service comes to mind. The workaround for this is chunking: splitting the file into e.g. 64KB fragments and using reliable, ordered messaging to transfer the data.
If you do not need the robustness of reliable messaging, streaming lets you transfer large amount of data using small message buffers without the overhead of implementing chuncking. Streaming over HTTP requires you to use basicHttpBinding; thus you will need SSL to encrypt the transferred data. Buffered transfer, on the other hand, can use wsHttpBinding, which by default provides integrity and confidentiality for your messages; thus there is no need for SSL then.
Read more about 'Large Data and Streaming' at MSDN.
So I implemented my upload prototype following the 'How to: Enable Streaming' guidelines, using a VS2005 web application (Cassini) as the service host. I made my operation a OneWay=true void operation with some SOAP headers to provide meta-data:
[OperationContract(Action = "UploadFile", IsOneWay = true)]
void UploadFile(ServiceContracts.FileUploadMessage request);
[MessageContract]
public class FileUploadMessage
{
[MessageHeader(MustUnderstand = true)]
public DataContracts.DnvsDnvxSession DnvxSession
[MessageHeader(MustUnderstand = true)]
public DataContracts.EApprovalContext Context
[MessageHeader(MustUnderstand = true)]
public DataContracts.FileMetaData FileMetaData
[MessageBodyMember(Order = 1)]
public System.IO.Stream FileByteStream
}
The reason for using message headers for meta-data is that WCF requires that the stream object is the only item in the message body for a streamed operation. Headers are the recommended way for sending meta-data when streaming. Note that headers are always sent before the body, thus you can depend on receving the header data before processing the streamed data.
Note that you should provide a separate endpoint (address, binding, contract) for your streamed services. The main reason for this is that configuration such as transferMode = "Streamed" applies to all operations in the endpoint. The same goes for transferMode, maxBufferSize, maxReceivedMessageSize, receiveTimeout, etc.
I use MTOM as the encoding format for the streamed data and set the max file size to 64MB. I also set the buffer size to 64KB even if I read the input stream in 4KB chucks, this to avoid receive buffer underruns. My binding config looks like this:
<basicHttpBinding>
<!-- buffer: 64KB; max size: 64MB -->
<binding name="FileTransferServicesBinding"
closeTimeout="00:01:00" openTimeout="00:01:00"
receiveTimeout="00:10:00" sendTimeout="00:01:00"
transferMode="Streamed" messageEncoding="Mtom"
maxBufferSize="65536" maxReceivedMessageSize="67108864">
<security mode="None">
<transport clientCredentialType="None"/>
</security>
</binding>
</basicHttpBinding>
I have set the transport security to none as I am testing the prototype without using SSL. I have not modified the service behavior; the default authentication (credentials) and authorization behaviors of the binding is used.
Note that the setting for transferMode does not propagate to clients when using a HTTP binding. You must manually edit the client config file to set transferMode = "Streamed" after using 'Add service reference' or running SVCUTIL.EXE. If you forget to do this, the transfer mode will be "Buffered" and you will get an error like this:
System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://localhost:1508/Host/FileTransferService.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.
After ensuring that the client basicHttpBinding config was correct, I ran the unit test once more. Now I got this error from VS2005 Cassini:
System.ServiceModel.ProtocolException: The remote server returned an unexpected response: (400) Bad Request.
System.Net.WebException: The remote server returned an error: (400) Bad Request.
I have inspected the HTTP traffic using Fiddler and can see nothing wrong with the MTOM encoded request. Googling lead me to a lot of frustrated "streamed" entries at the forums.microsoft.com "Indigo" group, but no solution to my problem. So I made a self-hosted service using a console application, using the code provided in the WCF samples (see below), and it worked like a charm. I expect that it is just VS2005 Cassini that does not support streamed transfers/MTOM encoding.
I then installed IIS to check whether HTTP streaming works better with IIS. The first error I ran into when uploading my 1KB test file was this:
System.ServiceModel.CommunicationException: The underlying connection was closed: The connection was closed unexpectedly.
System.Net.WebException: The underlying connection was closed: The connection was closed unexpectedly.
Inspecting the traffic with Fiddler, I found the HTTP request to be fine, but no HTTP response. Knowing that this is a sure sign of a server-side exception, I debugged the service operation. The error was caused by the IIS application pool identity (ASPNET worker process) lacking NTFS rights to store the uploaded file on disk. Note that WCF by default will not impersonate the caller, thus the identity used to run your service will need to have sufficient permissions on all resources that your service needs to access. This includes the temp folder, which is used to provide the WSDL file for HTTP GET.
As the UploadFile operation has [OperationContract(IsOneWay = true)], it cannot have a response and neither a fault contract. Thus, there will be no HTTP response when a server-side exception occurs, just the famous "The underlying connection was closed" message.
I assigned the correct NTFS rights to my upload folder and re-ran the unit test: streamed upload to a WCF service hosted by IIS worked. The next unit test used a 43MB file to check if uploading data larger than the WCF buffer size works. The test gave me this error:
System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:59.8590000'.System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
I was a bit puzzled as the file was less than the WCF maxReceivedMessageSize limit set to 64MB, and the stream read buffer was only 4KB as compared to the WCF maxBufferSize being 64KB. Knowing that IIS/ASP.NET has a system enforced limit on the size of a HTTP request to prevent denial-of-service attacks, which seems to be unrelated to WCF as I do not use the AspNetCompatibilityRequirements service setting, I decided to set the maxRequestLength limit to 64MB also (default: 4096KB). This time the "stream large file upload" test passed!
So for WCF streaming to IIS to work properly, you need this <system.web> config in addition to the <system.serviceModel> config in the IIS web.config file:
<!-- maxRequestLength (in KB) max size: 2048MB -->
<httpRuntime
maxRequestLength="65536"
/>
Make the maximum HTTP request size equal to the WCF maximum request size.
I have to recommend the 'Service Trace Viewer' (SvcTraceViewer.EXE) and the 'Service Configuration Editor' (SvcConfigEditor.EXE) provided in the WCF toolbox (download .NET3 SDK). These tools make it really simple to add tracing and logging to your service for debugging and diagnosing errors. Start the WCF config editor, open your WCF config file, select the 'Diagnostics' node and just click "Enable tracing" in the "Tracing" section to turn on tracing. The figure shows some typical settings (click to enlarge):
Save the config file and run the unit test to invoke the service. This will generate the web_tracelog.svclog file to use as input for the trace viewer. Open the trace file in the trace viewer and look for the errors (red text). Click an error to see the exception details. The figure shows some of the details available for the "Maximum request length exceeded" error (click to enlarge):
The WCF tools installed by the SDK is located here:
C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin\
The code to process the incoming stream is rather simple, storing the uploaded files to a single folder on disk and overwriting any existing files:
public void UploadFile(FileUploadMessage request)
{
FileStream targetStream = null;
Stream sourceStream = request.FileByteStream;
string uploadFolder = @"C:\work\temp\";
string filename = request.FileMetaData.Filename;
string filePath = Path.Combine(uploadFolder, filename);
using (targetStream = new FileStream(filePath, FileMode.Create,
FileAccess.Write, FileShare.None))
{
//read from the input stream in 4K chunks
//and save to output stream
const int bufferLen = 4096;
byte[] buffer = new byte[bufferLen];
int count = 0;
while ((count = sourceStream.Read(buffer, 0, bufferLen)) > 0)
{
targetStream.Write(buffer, 0, count);
}
targetStream.Close();
sourceStream.Close();
}
}
The 'Stream Sample' available on MSDN contains all the code you need to upload a file as a stream to a self-hosted WCF service and then save it to disk on the server by reading the stream in 4KB chunks. The download contains about 150 solutions with more than 4800 files, so there is a lot of stuff in it. Absolutely worth a look. The streaming sample is located here:
<unzip folder> \TechnologySamples\Basic\Contract\Service\Stream
Please let me know if you have been able to get HTTP streaming with MTOM encoding to work with Microsoft VS2005 Cassini.
Finally, a useful tip for streaming download: How to release streams and files using Disponse() on the the message contract.
Thursday, February 01, 2007
WCF: Importing InfoPath forms to Data Contracts
The solution that we are implementing includes a WCF service that allows a set of document metadata to be imported into our system. The operation is ImportProjectDocuments( ImportProjectDocumentsMessage request) and the message contains the ProjectDocuments data contract specifying the document structure. The data contract of course adhers to my 'separate structure from data' rule. The import operation is intended for application-to-application layer usage by our e-biz partners, for connecting our systems in a service-oriented distributed architecture.
Then came the need for allowing users to manually enter the document structure in a disconnected manner and submitting it to our service later on; i.e. a human-to-application layer service.
Enter InfoPath 2003 as the occasionally connected client (OCC), a perfect match for our existing data contract (see closing note) and a nice data entry application. Note that InfoPath is not used to submit the entered data directly to the WCF service endpoint, just to fill out a form and submitting it to our server as XML files - by e-mail or to a SharePoint form library when connected. You may think of the form library as the human friendly "message queue".
By the way - having a OCC service consumer is a rather good assessment of whether your services adheres to SOA best practices such as explicit boundaries, message based, share contract, and self-contained business event operations based on the paper form metaphor.
The submitted InfoPath forms must then be processed, typically by a workflow system such as K2.NET or even WWF, monitoring the form library. Each XML form must taken off the "message queue", deserialized using the DataContractSerializer and passed to our service, invoking the correct operation for the submitted XML data.
This is all quite trivial, but even if the XML generated by InfoPath adheres to the XSD schema, InfoPath adds some processing instructions that the deserializer chokes on:
System.Runtime.Serialization.SerializationException:
There was an error deserializing the object of type DNVS.DNVX.eApproval.DataContracts.ProjectDocuments.
Processing instructions (other than the XML declaration) and DTDs are not supported.
You must remove the InfoPath processing instructions before deserialization, and the easiest way to do this is by using a standard XmlTextReader instead of the WCF XmlDictionaryReader used in the MSDN documentation:
[Test]
public void DeserializeDataContractFromInfoPathXmlFileTest()
{
string fileName = @".\form1.xml";
ProjectDocuments documents = null;
DataContractSerializer xlizer = new DataContractSerializer(typeof(ProjectDocuments));
FileStream fs = new FileStream(fileName, FileMode.Open);
//NOTE: must get rid of InfoPath processing instructions
//XmlDictionaryReader reader = XmlDictionaryReader.CreateTextReader(fs, new XmlDictionaryReaderQuotas());
XmlTextReader reader = new XmlTextReader(fs);
documents = (ProjectDocuments) xlizer.ReadObject(reader, true);
reader.Close();
fs.Close();
Assert.IsNotNull(documents, "Could not deserialize the project documents XML");
}
The ReadObject() method is what actually converts the XML into a WCF data contract instance, ready for processing by the service layer just as if it had arrived through the WCF endpoint.
Note that the duration data type defined in the XSD schemas generated by WCF will cause an InfoPath 2003 parse error. You need to remove it before you start to design a new InfoPath form based on the data contract XSD schemas.
Then came the need for allowing users to manually enter the document structure in a disconnected manner and submitting it to our service later on; i.e. a human-to-application layer service.
Enter InfoPath 2003 as the occasionally connected client (OCC), a perfect match for our existing data contract (see closing note) and a nice data entry application. Note that InfoPath is not used to submit the entered data directly to the WCF service endpoint, just to fill out a form and submitting it to our server as XML files - by e-mail or to a SharePoint form library when connected. You may think of the form library as the human friendly "message queue".
By the way - having a OCC service consumer is a rather good assessment of whether your services adheres to SOA best practices such as explicit boundaries, message based, share contract, and self-contained business event operations based on the paper form metaphor.
The submitted InfoPath forms must then be processed, typically by a workflow system such as K2.NET or even WWF, monitoring the form library. Each XML form must taken off the "message queue", deserialized using the DataContractSerializer and passed to our service, invoking the correct operation for the submitted XML data.
This is all quite trivial, but even if the XML generated by InfoPath adheres to the XSD schema, InfoPath adds some processing instructions that the deserializer chokes on:
System.Runtime.Serialization.SerializationException:
There was an error deserializing the object of type DNVS.DNVX.eApproval.DataContracts.ProjectDocuments.
Processing instructions (other than the XML declaration) and DTDs are not supported.
You must remove the InfoPath processing instructions before deserialization, and the easiest way to do this is by using a standard XmlTextReader instead of the WCF XmlDictionaryReader used in the MSDN documentation:
[Test]
public void DeserializeDataContractFromInfoPathXmlFileTest()
{
string fileName = @".\form1.xml";
ProjectDocuments documents = null;
DataContractSerializer xlizer = new DataContractSerializer(typeof(ProjectDocuments));
FileStream fs = new FileStream(fileName, FileMode.Open);
//NOTE: must get rid of InfoPath processing instructions
//XmlDictionaryReader reader = XmlDictionaryReader.CreateTextReader(fs, new XmlDictionaryReaderQuotas());
XmlTextReader reader = new XmlTextReader(fs);
documents = (ProjectDocuments) xlizer.ReadObject(reader, true);
reader.Close();
fs.Close();
Assert.IsNotNull(documents, "Could not deserialize the project documents XML");
}
The ReadObject() method is what actually converts the XML into a WCF data contract instance, ready for processing by the service layer just as if it had arrived through the WCF endpoint.
Note that the duration data type defined in the XSD schemas generated by WCF will cause an InfoPath 2003 parse error. You need to remove it before you start to design a new InfoPath form based on the data contract XSD schemas.
Friday, December 22, 2006
How to: Expose WCF service also as ASMX web-service
In my last post, I wrote about my endeavors to consume a WCF service from an ASMX client to support non-WCF clients such as .NET 2 clients. The WCF service is exposed using a BasicHttpBinding endpoint and our main consumer is a Flex 2 web-application.
As the support for WS-I Basic Profile 1.x compliant web-services is quite limited in Flex, I had to make ASMX-style clients works. What made it worse, is that there is a bug in Flex regarding the support for xsd:import: Flex will repeatedly request the XSD files on .LoadWSDL(), causing bad performance for the web-service and the client. Use Fiddler to watch all the HTTP GETs issued by Flex. The solution is to ensure that the generated WSDL is a single file, without any xsd:import, and this is just what a classic .NET 2 ASMX web-service provides.
WCF has good support for migrating ASMX web-services to WCF, and this feature can also be used to actually expose a native WCF service also as an ASMX web-service. Start by adding the classic [WebService] and [WebMethod] attributes to your service interface:
Note that there is no need for adding [XmlSerializerFormat] to the [ServiceContract] attribute, just leave the WCF service as-is. Note also that you must specify the web-service binding Name parameter to avoid getting duplicate wsdl:port elements in the generated WSDL file. You should apply the [WebService] attribute using the same Name and Namespace parameters as for the [WebServiceBinding], to the class implementing the service. If not, you might get a wsdl:include and and xsd:import in the generated WSDL file.
Then add a new .ASMX file to the IIS host used for your WCF service, using the service implementation assembly to provide the web-services. I added a file called "ProjectDocumentWebService.asmx" with this content:
That's it. You now have a classic ASMX web-service at your disposal. WCF supports running WCF services and ASMX web-services side-by-side in the same IIS web-application: Hosting WCF Side-by-Side with ASP.NET.
Most likely you must also apply [XmlElement], [XmlArray] and [XmlIgnore] at relevant places in your message and data contracts to ensure correct serialization (XmlSerializer) and ensure consistent naming for both WCF and ASMX clients/proxies. Remember that the XmlSerializer works only with public members and that it does not support read-only properties. Do not mark message and data contract classes as [Serializable] as this changes the wire-format of the XML, which might cause problems in Flex 2.Note that the ASMX-style WSDL will not respect the [DataMember (IsRequired=true)] setting or any other WCF settings because it uses the XmlSerializer and not the DataContractSerializer. You will get minOccurs=0 in the generated WSDL for string fields, as string is a reference type. This is because the XmlSerializer by default makes reference types optional, while value types by default are mandatory. Read more about XmlSerializer 'MinOccurs Attribute Binding Support' at MSDN.
Note the difference between optional elements (minOccurs) and non-required data (nillable) in XSD schemas. This is especially useful in query specification objects, make all the elements optional and apply only the specified elements as criteria - even "is null" criteria.
The unsolved WSDL problem described in my last post is no longer an issue for ASMX-clients, as the WSDL generated by the classic ASMX web-service of course natively supports .NET 2 proxies generated by WSDL.EXE. The issue with the Name parameter for the [CollectionDataContract] attribute is still a problem, it must be removed.
Thanks to Mattavi for the "Migrating .NET Remoting to WCF (and even ASMX!)" article that got me started.
As the support for WS-I Basic Profile 1.x compliant web-services is quite limited in Flex, I had to make ASMX-style clients works. What made it worse, is that there is a bug in Flex regarding the support for xsd:import: Flex will repeatedly request the XSD files on .LoadWSDL(), causing bad performance for the web-service and the client. Use Fiddler to watch all the HTTP GETs issued by Flex. The solution is to ensure that the generated WSDL is a single file, without any xsd:import, and this is just what a classic .NET 2 ASMX web-service provides.
WCF has good support for migrating ASMX web-services to WCF, and this feature can also be used to actually expose a native WCF service also as an ASMX web-service. Start by adding the classic [WebService] and [WebMethod] attributes to your service interface:
[ServiceContractAttribute(Namespace = "http://kjellsj.blogspot.com/2006/11",
Name = "IProjectDocumentServices")]
[ServiceKnownTypeAttribute(
typeof(DNVS.DNVX.eApproval.FaultContracts.DefaultFaultContract))]
[WebService(Name = "ProjectDocumentServices")]
[WebServiceBinding(Name = "ProjectDocumentServices",
ConformsTo = WsiProfiles.BasicProfile1_1, EmitConformanceClaims = true)]
public interface IProjectDocumentServices
{
[OperationContractAttribute(Action = "GetDocumentCard")]
[FaultContractAttribute(
typeof(DNVS.DNVX.eApproval.FaultContracts.DefaultFaultContract))]
[WebMethod]
DocumentCardResponse GetDocumentCard(KeyCriteriaMessage request);
. . .
Note that there is no need for adding [XmlSerializerFormat] to the [ServiceContract] attribute, just leave the WCF service as-is. Note also that you must specify the web-service binding Name parameter to avoid getting duplicate wsdl:port elements in the generated WSDL file. You should apply the [WebService] attribute using the same Name and Namespace parameters as for the [WebServiceBinding], to the class implementing the service. If not, you might get a wsdl:include and and xsd:import in the generated WSDL file.
Then add a new .ASMX file to the IIS host used for your WCF service, using the service implementation assembly to provide the web-services. I added a file called "ProjectDocumentWebService.asmx" with this content:
<%@ WebService Language="C#" Class="DNVS.DNVX.eApproval.ServiceImplementation
.ProjectDocumentServices, DNVS.DNVX.eApproval.ServiceImplementation" %>
That's it. You now have a classic ASMX web-service at your disposal. WCF supports running WCF services and ASMX web-services side-by-side in the same IIS web-application: Hosting WCF Side-by-Side with ASP.NET.
Most likely you must also apply [XmlElement], [XmlArray] and [XmlIgnore] at relevant places in your message and data contracts to ensure correct serialization (XmlSerializer) and ensure consistent naming for both WCF and ASMX clients/proxies. Remember that the XmlSerializer works only with public members and that it does not support read-only properties. Do not mark message and data contract classes as [Serializable] as this changes the wire-format of the XML, which might cause problems in Flex 2.Note that the ASMX-style WSDL will not respect the [DataMember (IsRequired=true)] setting or any other WCF settings because it uses the XmlSerializer and not the DataContractSerializer. You will get minOccurs=0 in the generated WSDL for string fields, as string is a reference type. This is because the XmlSerializer by default makes reference types optional, while value types by default are mandatory. Read more about XmlSerializer 'MinOccurs Attribute Binding Support' at MSDN.
Note the difference between optional elements (minOccurs) and non-required data (nillable) in XSD schemas. This is especially useful in query specification objects, make all the elements optional and apply only the specified elements as criteria - even "is null" criteria.
The unsolved WSDL problem described in my last post is no longer an issue for ASMX-clients, as the WSDL generated by the classic ASMX web-service of course natively supports .NET 2 proxies generated by WSDL.EXE. The issue with the Name parameter for the [CollectionDataContract] attribute is still a problem, it must be removed.
Thanks to Mattavi for the "Migrating .NET Remoting to WCF (and even ASMX!)" article that got me started.
Wednesday, October 04, 2006
.NET deep clone - IsClone<T> using CloneFormatter
In some previous posts, I have written about how to implement an entity row state mechanism using .NET deep clone and an IsDirty method based on comparing two objects to see if they are clones of eachother. The IsClone<T> method was based on comparing the byte stream created by the BinaryFormatter, just as the Clone<T> method uses the BinaryFormatter to do deep cloning.
[UPDATE] A simple way to implement IsDirty when using IProperyNotificationChanged in data-binding enabled entity objects.
As I wrote in my last post, using the BinaryFormatter and the [NonSerialized] attribute in the IsClone<T> method is OK as long as you do not interfere with other stuff that use the BinaryFormatter, such as .NET remoting. The same applies to using the XmlFormatter, which will affect e.g. web-services. Thus, a separate formatter was needed to support the 'is clone' logic and fix the null/nothing string and the decimal problems. Enter the CloneFormatter based on the code in this article about Serialization Formatters.
I have made these changes to create the CloneFormatter:
private void WriteSerializableMembers(object obj, long objId)
{
System.Reflection.MemberInfo[] mi = FormatterServices.GetSerializableMembers(obj.GetType());
if (mi.Length > 0)
{
object[] od = FormatterServices.GetObjectData(obj, mi);
for (int i = 0; i < mi.Length; ++i)
{
System.Reflection.MemberInfo member = mi[i];
object data = od[i];
//KJELLSJ: do not serialize when marked with "CloneNonSerializedAttribute"
if (IsMarkedCloneNonSerialized(member) == true) continue;
if (member.MemberType == MemberTypes.Field)
{
FieldInfo fi = (FieldInfo)member;
if (data == null)
{
//KJELLSJ: ensure that null/nothing string gets "normalized"
if (fi.FieldType == typeof(string)) data = String.Empty;
}
else
{
//KJELLSJ: ensure that decimal gets "normalized"
if (fi.FieldType == typeof(decimal)) data = Convert.ToDecimal(Convert.ToDouble(data));
}
}
WriteMember(member.Name, data);
}
}
}
[UPDATE] A simple way to implement IsDirty when using IProperyNotificationChanged in data-binding enabled entity objects.
As I wrote in my last post, using the BinaryFormatter and the [NonSerialized] attribute in the IsClone<T> method is OK as long as you do not interfere with other stuff that use the BinaryFormatter, such as .NET remoting. The same applies to using the XmlFormatter, which will affect e.g. web-services. Thus, a separate formatter was needed to support the 'is clone' logic and fix the null/nothing string and the decimal problems. Enter the CloneFormatter based on the code in this article about Serialization Formatters.
I have made these changes to create the CloneFormatter:
private void WriteSerializableMembers(object obj, long objId)
{
System.Reflection.MemberInfo[] mi = FormatterServices.GetSerializableMembers(obj.GetType());
if (mi.Length > 0)
{
object[] od = FormatterServices.GetObjectData(obj, mi);
for (int i = 0; i < mi.Length; ++i)
{
System.Reflection.MemberInfo member = mi[i];
object data = od[i];
//KJELLSJ: do not serialize when marked with "CloneNonSerializedAttribute"
if (IsMarkedCloneNonSerialized(member) == true) continue;
if (member.MemberType == MemberTypes.Field)
{
FieldInfo fi = (FieldInfo)member;
if (data == null)
{
//KJELLSJ: ensure that null/nothing string gets "normalized"
if (fi.FieldType == typeof(string)) data = String.Empty;
}
else
{
//KJELLSJ: ensure that decimal gets "normalized"
if (fi.FieldType == typeof(decimal)) data = Convert.ToDecimal(Convert.ToDouble(data));
}
}
WriteMember(member.Name, data);
}
}
}
// Is the type attributed with [CloneNonSerialized]?
private bool IsMarkedCloneNonSerialized(MemberInfo mi)
{
object[] attributes = mi.GetCustomAttributes(typeof(CloneNonSerializedAttribute), false);
return attributes.Length > 0;
}
void ReadMember(long oid, MemberInfo mi, object o, SerializationInfo info)
{
//KJELLSJ: do not deserialize when marked with "CloneNonSerializedAttribute"
if (IsMarkedCloneNonSerialized(mi) == true) return;
// Read member name.
. . .
}
In addition, a new attribute was needed to be able to exclude class members from the 'is clone' logic:
[AttributeUsage(AttributeTargets.Field, AllowMultiple=false)]
public class CloneNonSerializedAttribute : System.Attribute
{
public CloneNonSerializedAttribute()
{
//only purpose is for marking fields as not-serialized by the CloneFormatter
}
}
Apply the [CloneNonSerialized] attribute to any class member that should not be part of the 'is clone' comparison.
The new IsClone<T> looks like this:
public static bool IsClone<T>(T sourceA, T sourceB)
{
if (sourceA == null sourceB == null)
return false;
MemoryStream streamA = new MemoryStream();
MemoryStream streamB = new MemoryStream();
IFormatter formatter = new CloneFormatter(); formatter.Serialize(streamA, sourceA);
//must create new formatter to reset the ObjectIDGenerator counter
formatter = new CloneFormatter();
formatter.Serialize(streamB, sourceB);
if (streamA.Length != streamB.Length) return false;
byte[] hashA = streamA.GetBuffer();
byte[] hashB = streamB.GetBuffer();
if (hashA.Length != hashB.Length) return false;
for (int i = 0; i < hashA.Length; i++)
{
if (hashA[i] != hashB[i]) return false;
}
//if here, objects are equal
return true;
}
Note that I have dropped the MD5 hashing in favor of plain byte-by-byte comparison as suggested by Nuri in a comment to the previous version. This should perform better, even if the for loop now gets longer that max 16 bytes.
Note also that the Clone<T> method still uses the BinaryFormatter to do the cloning. Use the CloneFormatter only for the IsClone<T> logic.
Note: serializing is not the most performant way to do cloning, read more at Anders Norås' blog.
Thursday, September 28, 2006
.NET Serialization: Custom Formatter, IFormatter
This spring I wrote about how to use cloning and serialization to implement an entity row-state mechanism. We used the BinaryFormatter in the 'clone' and 'is dirty' methods as web-services was the protocol between our tiers (XML serialization), and .NET remoting was not to be used (binary serialization).
Of course, this "absolute requirement" has now changed and using the BinaryFormatter and the NonSerialized attribute is no longer an option as all properties/fields must now be available through remoting. It was time for a custom formatter as the BinaryFormatter class is sealed/NotInheritable.
As the help topics on IFormatter contains only API level details, I though I should share this excellent documentation and working code with you: Serialization Formatters (published by Universität Karlsruhe).
The article also explains the little known ISerializationSurrogate mechanism; this allows you to provide your own serialization mechanism for specific classes/object types/value types, even when using one of the standard formatters. This is typically used to serialize classes that are not marked as [Serializable]. It can also be used to provide specific handling of null/nothing strings and for normalizing the serialization of decimal values. The former is a problem for the .IsDirty mechanism because a null/nothing string is different from a ""/String.Empty string, while the latter is a problem because the decimal value 1 might be different from the value 1.0 as the binary representation might change.
You might also find this Advanced Serialization MSDN article useful.
[UPDATE] Serializing object graphs that comprises inheritance requires some extra work. If you need to serialize inherited composite objects, the XmlInlcude(typeof(DerivedClass)) class attribute is the best solution when derived classes are know at compile-time. If you need to serialize "unkown" derived objects, the CodeProject XmlSerializer and 'not expected' Inherited Types article provides a really simple solution (the comments section includes a Generics NodeSerializer<T>).
Some of the MSDN help topics lacks working, real-life examples of how to implement an interface or abstract class; which is too bad and makes developers waste a lot of time in "the desert of desperation". I had the same problem last year looking for a working example of how to implement a SoapExtension. I wish Microsoft could do a litte more to help developers fall into "the pit of success".
Of course, this "absolute requirement" has now changed and using the BinaryFormatter and the NonSerialized attribute is no longer an option as all properties/fields must now be available through remoting. It was time for a custom formatter as the BinaryFormatter class is sealed/NotInheritable.
As the help topics on IFormatter contains only API level details, I though I should share this excellent documentation and working code with you: Serialization Formatters (published by Universität Karlsruhe).
The article also explains the little known ISerializationSurrogate mechanism; this allows you to provide your own serialization mechanism for specific classes/object types/value types, even when using one of the standard formatters. This is typically used to serialize classes that are not marked as [Serializable]. It can also be used to provide specific handling of null/nothing strings and for normalizing the serialization of decimal values. The former is a problem for the .IsDirty mechanism because a null/nothing string is different from a ""/String.Empty string, while the latter is a problem because the decimal value 1 might be different from the value 1.0 as the binary representation might change.
You might also find this Advanced Serialization MSDN article useful.
[UPDATE] Serializing object graphs that comprises inheritance requires some extra work. If you need to serialize inherited composite objects, the XmlInlcude(typeof(DerivedClass)) class attribute is the best solution when derived classes are know at compile-time. If you need to serialize "unkown" derived objects, the CodeProject XmlSerializer and 'not expected' Inherited Types article provides a really simple solution (the comments section includes a Generics NodeSerializer<T>).
Some of the MSDN help topics lacks working, real-life examples of how to implement an interface or abstract class; which is too bad and makes developers waste a lot of time in "the desert of desperation". I had the same problem last year looking for a working example of how to implement a SoapExtension. I wish Microsoft could do a litte more to help developers fall into "the pit of success".
Wednesday, February 08, 2006
Conditional XML serialization
In my post about implementing entity object row state using 'is dirty' and 'cloning' I described how to exclude properties during binary serialization using [NonSerialized]. The same can be done with XML serialization using [XmlIgnore]. In addition, properties can be temporarily excluded when serializing to XML. The mechanism behind conditionally excluding properties from XML serialization is not well known, thus this post to shed some light on it.
The bool <property name>Specified mechanism is the .NET serialization framework answer to XSD minOccurs="0" applied by code generators such as XSD.EXE, XsdObjectGen, and others. Most developers actually implicitly use this mechanism when they design and use typed datasets with optional fields (columns).
In short, whenever .NET serializes the properties of a class to XML, it always checks to see if there exists a boolean metod whose name is the name of the property suffixed with Specified. If such a method exists, and it returns false, the property is not included in the serialized XML; if it returns true, the property and value is serialized. The oppsite logic is applied on de-serialization of optional XML elements.
Thus, you can implement and use Specified methods at will to get conditional XML serialization in .NET. The serializer does not discriminate between methods implemented by code generators and methods implemented by you. Partial classes in .NET 2.0 also makes adding such methods to code generated classes much simpler.
I got to know the details of how XSD.EXE handles XSD minOccurs="0" and xsi:nil="true" very well when doing "web-service contract first" in combination with InfoPath last year. The support for doing WSCF in .NET 2.0 has improved with the support for e.g. interface based .ASMX web-services, but Visual Studio 2005 still has a way to go to support WSCF to its maximum extent. I expect that this will be finally solved when WCF (Indigo) is released.
The bool <property name>Specified mechanism is the .NET serialization framework answer to XSD minOccurs="0" applied by code generators such as XSD.EXE, XsdObjectGen, and others. Most developers actually implicitly use this mechanism when they design and use typed datasets with optional fields (columns).
In short, whenever .NET serializes the properties of a class to XML, it always checks to see if there exists a boolean metod whose name is the name of the property suffixed with Specified. If such a method exists, and it returns false, the property is not included in the serialized XML; if it returns true, the property and value is serialized. The oppsite logic is applied on de-serialization of optional XML elements.
Thus, you can implement and use Specified methods at will to get conditional XML serialization in .NET. The serializer does not discriminate between methods implemented by code generators and methods implemented by you. Partial classes in .NET 2.0 also makes adding such methods to code generated classes much simpler.
I got to know the details of how XSD.EXE handles XSD minOccurs="0" and xsi:nil="true" very well when doing "web-service contract first" in combination with InfoPath last year. The support for doing WSCF in .NET 2.0 has improved with the support for e.g. interface based .ASMX web-services, but Visual Studio 2005 still has a way to go to support WSCF to its maximum extent. I expect that this will be finally solved when WCF (Indigo) is released.
Friday, December 09, 2005
.NET deep clone - IsDirty check using IsClone<T>
Those of you that read my colleague Anders Norås' blog may have tried the deep clone using serialization method (see also article at MSDN Mag). This is really useful e.g. when implementing undo, transactions, and other mechanisms that need to make copies of objects.
[UPDATE] Refer to this post for a faster IsDirty check and more reliable IsClone method: IsClone using custom formatter.
Another useful appliance of the clone method is for implementing an IsDirty property in your business entity objects or in other areas of your application. I have implemented an IsClone method that checks whether two objects are identical or not:
public static bool IsClone<T>(T sourceA, T sourceB)
{
IFormatter formatter = new BinaryFormatter();
Stream streamA = new MemoryStream();
Stream streamB = new MemoryStream();
formatter.Serialize(streamA, sourceA);
formatter.Serialize(streamB, sourceB);
if(streamA.Length != streamB.Length) return false;
streamA.Seek(0, SeekOrigin.Begin);
streamB.Seek(0, SeekOrigin.Begin);
byte[] hashA = new System.Security.Cryptography.MD5CryptoServiceProvider().ComputeHash(streamA);
byte[] hashB = new System.Security.Cryptography.MD5CryptoServiceProvider().ComputeHash(streamB);
for (int i = 0; i < 16; i++)
{
if (hashA[i] != hashB[i]) return false;
}
//if here, objects have same hash = are equal
return true;
}
The "source" objects must of course be deep serializable. The method uses hashing to achieve good performance. The compare using MD5 is based on code by Corrado Cavalli for fast comparison of byte arrays.
[UPDATE] Refer to this post for a faster IsDirty check and more reliable IsClone method: IsClone using custom formatter.
Another useful appliance of the clone method is for implementing an IsDirty property in your business entity objects or in other areas of your application. I have implemented an IsClone
public static bool IsClone<T>(T sourceA, T sourceB)
{
IFormatter formatter = new BinaryFormatter();
Stream streamA = new MemoryStream();
Stream streamB = new MemoryStream();
formatter.Serialize(streamA, sourceA);
formatter.Serialize(streamB, sourceB);
if(streamA.Length != streamB.Length) return false;
streamA.Seek(0, SeekOrigin.Begin);
streamB.Seek(0, SeekOrigin.Begin);
byte[] hashA = new System.Security.Cryptography.MD5CryptoServiceProvider().ComputeHash(streamA);
byte[] hashB = new System.Security.Cryptography.MD5CryptoServiceProvider().ComputeHash(streamB);
for (int i = 0; i < 16; i++)
{
if (hashA[i] != hashB[i]) return false;
}
//if here, objects have same hash = are equal
return true;
}
The "source" objects must of course be deep serializable. The method uses hashing to achieve good performance. The compare using MD5 is based on code by Corrado Cavalli for fast comparison of byte arrays.
Wednesday, August 31, 2005
Web-services: interoperability & encoding
I am currently implementing MSCRM at a customer that is also implementing an enterprise portal using SiteVision delivered by Objectware's Java department. The need for providing the SiteVision portal with information about MSCRM accounts made me implement a few .NET XML web-services to be used by the Java guys. The web-service provides basic read, create and update services for accounts and contacts. The 'data transfer object' was simply implemented as a string that would contain XML according to a predefined XSD schema.
The enterprise application integration server used is BIE (Business Integration Engine; Java open source), chosen by the Java lead developer on this project. Their use of my web-services went along quite smootly as was to be expected as .NET web-services complies with WS-I BP1.0 (intro article here). I had even used the WS-I compliance tools to check my web-services.
What caused a major halt on the integration implementation was when we started testing with data containing Norwegian characters (ÆØÅ). To be more precise, this was tested and found to be OK by using MSIE as the test-client, and by using XmlSpy to retrieve data about e.g. an account, modifying the data, and finally updating the account. I strongly recommend the XmlSpy SOAP Debugger tool both for testing and debugging web-serives.
The problem was that although BIE supports web-servies using UTF-8, it does not differentiate between UTF-8 as a text encoding in the web-service and UTF-8 encoded XML and the W3C character encoding rules that dictates that a unicode character must be encoded as a #xHHHH numeric character reference (i.e. 16-bit unicode code points/units; UTF-16 NCR). For more details see 'Can XML use non-Latin characters?'.
The MSCRM 1.2 object model supports and returns XML using the #xHHHH character encoding. All strings in .NET are unicode.
The character encoding went through these phases (examples for Æ and æ):
A nice unicode character encoding test tool is available here.
The Java lead developer had run into these encoding problems in BIE and modified the portal code to do some character replacing (Æ to Æ, etc) on all text to/from BIE on their side, and I asked them to change their integration implementation to comply with the WS-I and W3C standards. Unfortunately, this lead developer is more interested in hailing the glory of the application architecture & design and the open source movement than complying with standards "invented by Microsoft" :-)
Thus, I had to write a SoapExtension to modify the incoming SOAP message before it was deserialized, to change the "wrong" UTF-8 encoding into correct XML W3C UTF-8 encoding. I used this GotDotNet sample as the basis for my code and this is how I modify the incoming and outgoing SOAP messages:
public override void ProcessMessage(SoapMessage message)
{
switch (message.Stage)
{
//INCOMING
case SoapMessageStage.BeforeDeserialize:
this.ChangeIncomingEncoding();
break;
case SoapMessageStage.AfterDeserialize:
_isPostDeserialize = true;
break;
//OUTGOING
case SoapMessageStage.BeforeSerialize:
break;
case SoapMessageStage.AfterSerialize:
this.ChangeOutgoingEncoding();
break;
}
}
public override Stream ChainStream( Stream stream )
{
//http://hyperthink.net/blog/CommentView,guid,eafeef67-c240-44cc-8550-974f5d378a8f.aspx
if(!_isPostDeserialize)
{
//INCOMING
_inputStream = stream;
_outputStream = new MemoryStream();
return _outputStream;
}
else
{
//OUTGOING
_outputStream = stream;
_inputStream = new MemoryStream();
return _inputStream;
}
}
public void ChangeIncomingEncoding()
{
//at BeforeDeserialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;
TextReader reader = new StreamReader(_inputStream);
TextWriter writer = new StreamWriter(_outputStream);
string line;
while((line = reader.ReadLine()) != null)
{
writer.WriteLine( Utilities.FixBieEncoding (line) );
}
writer.Flush();
//reset the new stream to ensure that AfterDeserialize is called
if(_outputStream.CanSeek)
_outputStream.Position = 0L;
}
public void ChangeOutgoingEncoding()
{
//at AfterSerialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;
Regex regex = new Regex("utf-8", RegexOptions.IgnoreCase);
TextReader reader = new StreamReader(_inputStream);
//HACK: TextWriter writer = new StreamWriter(_outputStream);
TextWriter writer = new StreamWriter(_outputStream, System.Text.Encoding.GetEncoding(_encoding));
string line;
while((line = reader.ReadLine()) != null)
{
// change the encoding only is needed
if(_encoding != null && !_encoding.Equals("utf-8"))
line = regex.Replace(line, _encoding);
writer.WriteLine(line);
}
writer.Flush();
}
The central method here is the ChangeIncomingEncoding method that converts all the "wrong" UTF-8 encodings from BIE into correct XML W3C NCRs. Note the resetting of the position to zero on the output stream after modifying the message; this is important as forgetting to do so will cause the AfterDeserialize step of the SoapMessageStage in ProcessMessage not to be called.
After deploying the modified web-service and testing it with XmlSpy, it was time to test it with the BIE workflow dashboard. The BIE developer had set up some test cases for me, and they all worked as expected. No more funny farm in the MSCRM database.
What an illustrious victory for Java-.NET web-service interoperability!
Note that you cannot test/debug your SoapExtension code by using MSIE as the test-client as HTTP POST/GET will not trigger the SoapExtension. I used XmlSpy to test and debug my code; just set some breakpoints, start your web-service in debug mode and leave it running, then trigger the SoapExtention by making a SOAP request using XmlSpy.
The enterprise application integration server used is BIE (Business Integration Engine; Java open source), chosen by the Java lead developer on this project. Their use of my web-services went along quite smootly as was to be expected as .NET web-services complies with WS-I BP1.0 (intro article here). I had even used the WS-I compliance tools to check my web-services.
What caused a major halt on the integration implementation was when we started testing with data containing Norwegian characters (ÆØÅ). To be more precise, this was tested and found to be OK by using MSIE as the test-client, and by using XmlSpy to retrieve data about e.g. an account, modifying the data, and finally updating the account. I strongly recommend the XmlSpy SOAP Debugger tool both for testing and debugging web-serives.
The problem was that although BIE supports web-servies using UTF-8, it does not differentiate between UTF-8 as a text encoding in the web-service and UTF-8 encoded XML and the W3C character encoding rules that dictates that a unicode character must be encoded as a #xHHHH numeric character reference (i.e. 16-bit unicode code points/units; UTF-16 NCR). For more details see 'Can XML use non-Latin characters?'.
The MSCRM 1.2 object model supports and returns XML using the #xHHHH character encoding. All strings in .NET are unicode.
The character encoding went through these phases (examples for Æ and æ):
- from our MSCRM web-service: Æ encoded as Æ
- back from BIE on create/update: Æ "encoded" as Æ (not even valid UTF-8 code units)
- data in MSCRM after operation: Æ
- from our MSCRM web-service: æ encoded as æ
- back from BIE on create/update: æ encoded as æ (which are valid UTF-8 code units)
- data in MSCRM after operation: æ
A nice unicode character encoding test tool is available here.
The Java lead developer had run into these encoding problems in BIE and modified the portal code to do some character replacing (Æ to Æ, etc) on all text to/from BIE on their side, and I asked them to change their integration implementation to comply with the WS-I and W3C standards. Unfortunately, this lead developer is more interested in hailing the glory of the application architecture & design and the open source movement than complying with standards "invented by Microsoft" :-)
Thus, I had to write a SoapExtension to modify the incoming SOAP message before it was deserialized, to change the "wrong" UTF-8 encoding into correct XML W3C UTF-8 encoding. I used this GotDotNet sample as the basis for my code and this is how I modify the incoming and outgoing SOAP messages:
public override void ProcessMessage(SoapMessage message)
{
switch (message.Stage)
{
//INCOMING
case SoapMessageStage.BeforeDeserialize:
this.ChangeIncomingEncoding();
break;
case SoapMessageStage.AfterDeserialize:
_isPostDeserialize = true;
break;
//OUTGOING
case SoapMessageStage.BeforeSerialize:
break;
case SoapMessageStage.AfterSerialize:
this.ChangeOutgoingEncoding();
break;
}
}
public override Stream ChainStream( Stream stream )
{
//http://hyperthink.net/blog/CommentView,guid,eafeef67-c240-44cc-8550-974f5d378a8f.aspx
if(!_isPostDeserialize)
{
//INCOMING
_inputStream = stream;
_outputStream = new MemoryStream();
return _outputStream;
}
else
{
//OUTGOING
_outputStream = stream;
_inputStream = new MemoryStream();
return _inputStream;
}
}
public void ChangeIncomingEncoding()
{
//at BeforeDeserialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;
TextReader reader = new StreamReader(_inputStream);
TextWriter writer = new StreamWriter(_outputStream);
string line;
while((line = reader.ReadLine()) != null)
{
writer.WriteLine( Utilities.FixBieEncoding (line) );
}
writer.Flush();
//reset the new stream to ensure that AfterDeserialize is called
if(_outputStream.CanSeek)
_outputStream.Position = 0L;
}
public void ChangeOutgoingEncoding()
{
//at AfterSerialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;
Regex regex = new Regex("utf-8", RegexOptions.IgnoreCase);
TextReader reader = new StreamReader(_inputStream);
//HACK: TextWriter writer = new StreamWriter(_outputStream);
TextWriter writer = new StreamWriter(_outputStream, System.Text.Encoding.GetEncoding(_encoding));
string line;
while((line = reader.ReadLine()) != null)
{
// change the encoding only is needed
if(_encoding != null && !_encoding.Equals("utf-8"))
line = regex.Replace(line, _encoding);
writer.WriteLine(line);
}
writer.Flush();
}
The central method here is the ChangeIncomingEncoding method that converts all the "wrong" UTF-8 encodings from BIE into correct XML W3C NCRs. Note the resetting of the position to zero on the output stream after modifying the message; this is important as forgetting to do so will cause the AfterDeserialize step of the SoapMessageStage in ProcessMessage not to be called.
After deploying the modified web-service and testing it with XmlSpy, it was time to test it with the BIE workflow dashboard. The BIE developer had set up some test cases for me, and they all worked as expected. No more funny farm in the MSCRM database.
What an illustrious victory for Java-.NET web-service interoperability!
Note that you cannot test/debug your SoapExtension code by using MSIE as the test-client as HTTP POST/GET will not trigger the SoapExtension. I used XmlSpy to test and debug my code; just set some breakpoints, start your web-service in debug mode and leave it running, then trigger the SoapExtention by making a SOAP request using XmlSpy.
Subscribe to:
Posts (Atom)