WCF: Importing InfoPath forms to Data Contracts

The solution that we are implementing includes a WCF service that allows a set of document metadata to be imported into our system. The operation is ImportProjectDocuments( ImportProjectDocumentsMessage request) and the message contains the ProjectDocuments data contract specifying the document structure. The data contract of course adhers to my 'separate structure from data' rule. The import operation is intended for application-to-application layer usage by our e-biz partners, for connecting our systems in a service-oriented distributed architecture.

Then came the need for allowing users to manually enter the document structure in a disconnected manner and submitting it to our service later on; i.e. a human-to-application layer service.

Enter InfoPath 2003 as the occasionally connected client (OCC), a perfect match for our existing data contract (see closing note) and a nice data entry application. Note that InfoPath is not used to submit the entered data directly to the WCF service endpoint, just to fill out a form and submitting it to our server as XML files - by e-mail or to a SharePoint form library when connected. You may think of the form library as the human friendly "message queue".

By the way - having a OCC service consumer is a rather good assessment of whether your services adheres to SOA best practices such as explicit boundaries, message based, share contract, and self-contained business event operations based on the paper form metaphor.

The submitted InfoPath forms must then be processed, typically by a workflow system such as K2.NET or even WWF, monitoring the form library. Each XML form must taken off the "message queue", deserialized using the DataContractSerializer and passed to our service, invoking the correct operation for the submitted XML data.

This is all quite trivial, but even if the XML generated by InfoPath adheres to the XSD schema, InfoPath adds some processing instructions that the deserializer chokes on:

System.Runtime.Serialization.SerializationException:
There was an error deserializing the object of type DNVS.DNVX.eApproval.DataContracts.ProjectDocuments.
Processing instructions (other than the XML declaration) and DTDs are not supported.

You must remove the InfoPath processing instructions before deserialization, and the easiest way to do this is by using a standard XmlTextReader instead of the WCF XmlDictionaryReader used in the MSDN documentation:

[Test]
public void DeserializeDataContractFromInfoPathXmlFileTest()
{
string fileName = @".\form1.xml";
ProjectDocuments documents = null;

DataContractSerializer xlizer = new DataContractSerializer(typeof(ProjectDocuments));

FileStream fs = new FileStream(fileName, FileMode.Open);
//NOTE: must get rid of InfoPath processing instructions
//XmlDictionaryReader reader = XmlDictionaryReader.CreateTextReader(fs, new XmlDictionaryReaderQuotas());

XmlTextReader reader = new XmlTextReader(fs);

documents = (ProjectDocuments) xlizer.ReadObject(reader, true);

reader.Close();
fs.Close();

Assert.IsNotNull(documents, "Could not deserialize the project documents XML");
}


The ReadObject() method is what actually converts the XML into a WCF data contract instance, ready for processing by the service layer just as if it had arrived through the WCF endpoint.

Note that the duration data type defined in the XSD schemas generated by WCF will cause an InfoPath 2003 parse error. You need to remove it before you start to design a new InfoPath form based on the data contract XSD schemas.

Conditional XML serialization

In my post about implementing entity object row state using 'is dirty' and 'cloning' I described how to exclude properties during binary serialization using [NonSerialized]. The same can be done with XML serialization using [XmlIgnore]. In addition, properties can be temporarily excluded when serializing to XML. The mechanism behind conditionally excluding properties from XML serialization is not well known, thus this post to shed some light on it.

The bool <property name>Specified mechanism is the .NET serialization framework answer to XSD minOccurs="0" applied by code generators such as XSD.EXE, XsdObjectGen, and others. Most developers actually implicitly use this mechanism when they design and use typed datasets with optional fields (columns).

In short, whenever .NET serializes the properties of a class to XML, it always checks to see if there exists a boolean metod whose name is the name of the property suffixed with Specified. If such a method exists, and it returns false, the property is not included in the serialized XML; if it returns true, the property and value is serialized. The oppsite logic is applied on de-serialization of optional XML elements.

Thus, you can implement and use Specified methods at will to get conditional XML serialization in .NET. The serializer does not discriminate between methods implemented by code generators and methods implemented by you. Partial classes in .NET 2.0 also makes adding such methods to code generated classes much simpler.

I got to know the details of how XSD.EXE handles XSD minOccurs="0" and xsi:nil="true" very well when doing "web-service contract first" in combination with InfoPath last year. The support for doing WSCF in .NET 2.0 has improved with the support for e.g. interface based .ASMX web-services, but Visual Studio 2005 still has a way to go to support WSCF to its maximum extent. I expect that this will be finally solved when WCF (Indigo) is released.

InfoPath: Add a row to a repeating table with JScript

One of the most typical questions on the InfoPath newsgroup is how to programmatically add a new row to a repeating table. The XML DOM is very fine-grained when it comes to adding new elements and attributes, demanding a lot of boring code especially when adding stuff like a new row to a table. Unfortunately, MSXML does not allow for adding an XML fragment from a string, which would have made things very simple. InfoPath script uses the MSXML component.

The trick is an old one with MSXML and bad horror movies: cloning !

This code shows how to create a new row, reset the values, and add it to the XML document:

//get parent row
var parent = XDocument.DOM.selectSingleNode("/dfs:myFields/dfs:dataFields//s1:InvoiceCommission");

//get first row
var rowOne = parent.selectSingleNode("./s1:InvoiceDetailsRow");

// Create xsi:nil attribute with the proper namespace.
var xmlNil = parent.ownerDocument.createNode(2, "xsi:nil", "http://www.w3.org/2001/XMLSchema-instance");
xmlNil.text = "true";

//clone the first row
var rowClone = rowOne.cloneNode(true);

//reset values
var description = rowClone.selectSingleNode("s1:Description");
description.text = "CLONED";
rowClone.selectSingleNode("s1:IsVatCharged").text = 0;
rowClone.selectSingleNode("s1:VatAmount").text = 0;

//nillable: The order is important. Attribute must be removed when setting actual value.
var amount = rowClone.selectSingleNode("s1:NetAmount");
amount.text = "";
amount.setAttributeNode(xmlNil);

//append row to XML document
parent.insertBefore(rowClone, rowOne);

Your XML must ofcourse contain a "seed" row in the table, otherwise there will be nothing to clone. Do not use parent.appendChild() as this can make your XML document invalid against your XSD schema. This will happen when the XSD defines a <xs:sequence> and the row element you add is not at the very end in the schema definition.

Note that MSXML does not have an .insertAfter() method. The .NET assembly System.Xml does, but this will require you to use managed code in your form, which makes deployment more complicated than a script based solution.

Do not use XSD default values with InfoPath

I have had some strange behavior in InfoPath with numeric fields that became locked even if they were not configured to be read-only. These fields were never locked when using the form to enter new data, but sometimes when modifying existing data.

After a bit of investigation, I found out that when an element in the XSD has e.g. default="0" and the user does not change it to something else, then this value will not be included in the XML when submitting to the web service. When the user later on opens the saved data, the element will not be in the XML returned by the web service. This will cause InfoPath to lock the field in the form, as InfoPath cannot bind to an element that is not there. This is very annoying, and setting default values in the InfoPath form will not make this go away. A field that contains a value different than the XSD default value, will be in the XML and thus fully editable.

In short, do not use the XSD default attribute in schemas intended for usage with InfoPath. It is sad that a client application imposes its weaknesses on the contract schema design, but the best is the enemy of the good, so I changed the schema to suite InfoPath.

Note that I have only tested this in combination with a WSCF web service.

InfoPath text box handling of TAB and other whitespace

Today I got a surprise with regards to how InfoPath handled whitespace in strings. Our data contained text that was nicely aligned using TAB characters and sometimes mono-spaced font with multiple space characters. While testing my form, I noticed that all the formatting was gone when displayed in InfoPath even if the whitespace was preserved in the XML data. All TABs and all sequences of spaces got converted into a single space.

In addition, I could not enter TABs in the text box using the keyboard. The same thing happend with \t characters added to strings in the form code (JScript).

I inspected the properties of the text box, and the only formatting option available is "Paragraph breaks". As my options were limited, I chose that option and re-loaded the XML data. I was mildly delighted when all the alignment stuff in the string data were displayed correctly, including the TAB characters. I can now also enter TABs from the keyboard using CTRL-TAB.

The InfoPath team has blogged about line breaks (CR LF \r \n), and they outline how to make these formatting characters available in rules. Their solution should also be applicable to TAB.

Submitting xsi:nil="true" values from InfoPath to ASP.NET 1.x web services

In my ongoing quest to build an InfoPath solution based on a WSCF web service implemented with ASP.NET 1.1, I have finally been able to:

• make non-string data types such as xs:date and xs:decimal truly optional in InfoPath
• submit data to the web service by modifing the XML data in OnSubmitRequest to circumvent the missing support for nillable in .NET 1.x ASP.NET web services

The first issue is caused by the fact that ASP.NET 1.x web service by design do not support nillable elements in the web service parameters. Check the <wsdl:types> element of any ASMX WSDL, and you will not find any nillable="true" elements even if your data XSD contains such elements. The reason for this is the lack of support for NULL in .NET 1.x value types. ASP.NET 2.0 web services will support nillable="true".

After modifying the underlying XSD schemas of your InfoPath as outlined in my previous post to manually re-introduce the nillable="true" attributes, the fields can be left empty in your form. But you will not be able to sumbit the form to the web service. The SOAP request will fail like this:

The SOAP response indicates that an error occurred:
Server was unable to read request. --> There is an error in XML document (47, 104). --> Input string was not in a correct format.

This is caused by this kind of element in the submitted XML:

<s1:Amount xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"></s1:Amount>

The web service is not able to deserialize this element into a value type due to the lack of support for nillable. Thus, the XML data must be modified before submitting the data to the web service.

InfoPath allows you to use the OnSubmitRequest event to write your own script for submitting the forms data. This event is only available by using Tools-Submitting Forms and selecting "Custom submit using form code" in the 'Submit to' dropdown. Check the 'Edit Form Code' checkbox and click OK to add an event handler for OnSubmitRequest.

This example shows how to look for all instances of a specific element that is NULL, remove the nil attribute and set a dummy value, and finally submitting the form using code:

function XDocument::OnSubmitRequest(eventObj)
{
try
{
var submitDataSource = "Main submit";

//debugger
//get a collection of nodes with xsi:nil="true"
var nodeList = XDocument.DOM.selectNodes("/dfs:myFields/dfs:dataFields//s1:Amount[@xsi:nil='true']");

//iterate the list and
//set values acceptable to ASP.NET 1.x web service
for(var i=0; i < nodeList.length; i++)
{
//get node (element)
var xmlNode = nodeList(i);

//remove nil attribute and
//put zero into the element
//(uses code from InfoPath common.js)
// The xsi:nil needs to be removed before we set the value.
xmlNode.removeAttribute("xsi:nil");
// Setting the value will mark the document as dirty.
xmlNode.text = -1; //biz logic will interpret this as NULL
}

//call the Submit method of the
//primary data source to send the data
XDocument.DataAdapters(submitDataSource).Submit();

eventObj.ReturnStatus = true;
}
catch(ex)
{
XDocument.UI.Alert("Failed while sending the request.\r\n" + ex.number + " - " + ex.description);
eventObj.ReturnStatus = false;
}
}

You will now be able to submit the form's XML data to the web service. The data will of course contain dummy values for all elements that are really NULL (blank), and your 'save' business logic must handle this accordingly. In addition, you will need to add the same type of logic to the form's request operation and your 'read' business logic. Otherwise the users will see the dummy values when viewing existing data from your web service

PS! remember to set the script language of the form before adding any code, as it is not trivial to change the language afterwards. This KB article explains how to change the language, note that you will need to delete all existing scripts.

Optional numeric fields in InfoPath

In an earlier post I wrote about optional and non-required elements in XSD schemas, especially date and decimal (double) elements, and how InfoPath deals with such elements. I was annoyed that I could not get them to be optional when included in the InfoPath main data source (schema scope).

Today I took a closer look at this problem, and instead of starting with the service contract (WSCF) and creating an InfoPath form based on the generated web service, I designed a new form from scratch by building the data source (XSD schema) using InfoPath. And, lo and behold, the decimal fields added manually to the data source are by default optional! The 'Cannot be blank' checkbox is enabled and unchecked!

I extracted the form files and inspected the XSD schema, and the element had both minOccurs="0" and nillable="true". Just as I had defined the decimal elements in the XSD schemas used in my service contract. So, why the different behavior ?

To find out why, I opened one of the InfoPath forms made by connecting to a WSCF web service and then extracted the form files. When I inspected the schemas, the nillable="true" was not in the XSD schema induced by InfoPath. By adding the attribute manually to the schema and then opening the form definition file (.XSF) with InfoPath, the problem was solved and my decimal fields are now truly optional in InfoPath. Note that this fix will be gone as soon as you re-induce the data source schemas.

Is this a bug with regards to nillable in the generated WSDL, web service or just the way it is ? Or, more likely, a weakness in InfoPath ? To be continued...

Using XPath preceding-sibling in InfoPath rules

Several of our InfoPath forms have a repeating table that contains details about the entity being edited, e.g. an order with zero, one or more order lines for the products that makes up the order.

In this example I will outline how to add a rule to the repeating table that assigns incremental values to an element in a row based on the value of the preceding row's same element. I will also outline how to add a rule to the form's 'Open behavior' to set the seed value for this element in the first row of the table. This will ensure that the rules of the table will work as expected.

The XML used in this example has this structure:
<PaymentPlan>
<PaymentPlanRow>
<Year/><Product/><Amount/>
</PaymentPlanRow>
<PaymentPlanRow>
<Year/><Product/><Amount/>
</PaymentPlanRow>
</PaymentPlan>

To add a rule to the 'new table row' event, start by opening the properties dialog for the repeating table and click 'Rules'. Do not add the rule to the table cell, ensure that you add it to the table. Add a rule called 'SetRowYear' with a condition that the 'Year' field is blank. Then add an action of type 'Set a field's value' to calculate the next value for the 'Year' element. Enter this formula to look up the value of the preceding row's 'Year' element, convert it to a number and add one:

number(preceding-sibling::PaymentPlanRow[1]/Year) + 1

The XPath preceding-sibling axis consists of all nodes that have the same parent as the current node, from the current node and up to the start. The following-sibling axis consists of the nodes from the current node and down to the end of the parent's child nodes set. The important thing to notice is the index into the node set: [1], as when XPath is used to get a value from a set of nodes, it will always take the first node in document order when an index is not applied. Thus, if you do not specify the node set index, the value returned will always be that of the first row in the table, not the preceding row. Note that the index is relative from the current node; i.e. preceding-sibling[1] is the previous row, while preceding-sibling[2] is the row before that again.

There are several ways to set the value for the first row's 'Year' field to ensure that the above formula has a seed value. I have used a rule when the form is opened. Start by opening the 'Form Options' dialog and go to the 'Open and Save' folder, then click on 'Rules' in the 'Open Behavior' section. Add a rule called 'SetInitialYear' with a condition that the 'Year' field is blank. The condition will ensure that the value will not be set when opening an existing form. Then add an action of type 'Set a field's value' to calculate the seed value for the 'Year' element. Enter this formula to use the current year:

substring(today(); 1; 4)

These examples shows how you can use rules to perform node set operations, calculations and setting of values, without having to use form programming and without having to manipulate the XML DOM with JScript.

A few annoyances with InfoPath

InfoPath is a good application for controlled viewing and form entry of data into XML documents and web services, but there are a few things that could have been much better or logical.

What has annoyed me most this week in InfoPath are the handling of optional and non-required date and numerical elements in XSD (e.g. xs:date and xs:decimal), and the handling of default values based on formulas.

First the optional (minOccurs="0") and non-required (nillable="true") "object" types in XSD: they will still be mandatory in InfoPath when using the WSCF (web services contract first) approach. What you gain is the option to leave these elements out of the InfoPath form's scope. If you include these fields in the scope, you must yourself handle the fields in an OnSubmit event handler, setting a dummy date into the fields. Likewise, you must clear the dummy dates when loading existing data into InfoPath.

Then the default value mechanism of InfoPath: we have some fields that are calculated based on other fields, but the user can override the calculated value by typing a number of their own choice into the field. This seems to function OK with InfoPath, and when you submit the form to the web service, the user's number is there alright. However, when the form loads the stored data later on, InfoPath actually reapplies the default value formula, even when the field already has content. Not exactly the way I expected it to work. The solution is to use InfoPath rules. But even rules has a small quirck; they only trigger on changing the content of a field, thus it is hard to apply "default value when user leaves field empty" logic in InfoPath.

Debug the InfoPath submit SOAP message

I am currently building a prototype for using web services contract first (WSCF from Thinktecture) to build web services to support InfoPath as the client. Setting up InfoPath (SP1) to query and submit to the web service is straightforward, and getting data from the web service worked immediately. I could not, however, get submitting the form to the web service to work, and I suspected my setup of which part of the XML to post to be incorrect.

During my attempts to get submit working, I needed to spy on the SOAP messages generated by InfoPath to see why the posted XML could not be deserialized according to my message schema. As I am a long time fan of Altova XmlSpy, I decided to try out their XmlSpy SOAP Debugger. The XmlSpy integration with VS.NET makes working with XSD schemas quite easy.

[UPDATE] XmlSpy is still great, but you might find that Fiddler is sufficient for inspecting the HTTP request/response traffic to between InfoPath and your web-service.

Follow the steps outlined in the above link to configure and start the SOAP debugger, noting that the debugger is a proxy process that sets up a new HTTP port (:8080) that your SOAP traffic must be sent to. The debugger proxy then does its magic and finally forwards the traffic to the original port (:80).

So to be able to intercept and spy on the SOAP message submitted by InfoPath, you must open the form and change the data connection used by submit. In the first step of the 'Data Connection' wizard, you must change the location in 'web service details (submit data)' to use the SOAP debugger proxy at port :8080. Then complete the remaining steps of the wizard, selecting the correct data to use as the 'submit operation' parameters. Close the wizard, save the form and then test it using 'Preview Form'. When you now submit the form, XmlSpy will break on the selected operation and you can review the SOAP request made by InfoPath.

My mistake was that I had chosen to include 'XML subtree...' instead of 'Text and child elementes only', thus InfoPath posted my main parameter element twice, nested within itself (<message><dto><dto>...). I would most likely have found the correct configuration anyway, but using XmlSpy certainly saved me a lot of time.