Web-services: interoperability & encoding

I am currently implementing MSCRM at a customer that is also implementing an enterprise portal using SiteVision delivered by Objectware's Java department. The need for providing the SiteVision portal with information about MSCRM accounts made me implement a few .NET XML web-services to be used by the Java guys. The web-service provides basic read, create and update services for accounts and contacts. The 'data transfer object' was simply implemented as a string that would contain XML according to a predefined XSD schema.

The enterprise application integration server used is BIE (Business Integration Engine; Java open source), chosen by the Java lead developer on this project. Their use of my web-services went along quite smootly as was to be expected as .NET web-services complies with WS-I BP1.0 (intro article here). I had even used the WS-I compliance tools to check my web-services.

What caused a major halt on the integration implementation was when we started testing with data containing Norwegian characters (ÆØÅ). To be more precise, this was tested and found to be OK by using MSIE as the test-client, and by using XmlSpy to retrieve data about e.g. an account, modifying the data, and finally updating the account. I strongly recommend the XmlSpy SOAP Debugger tool both for testing and debugging web-serives.

The problem was that although BIE supports web-servies using UTF-8, it does not differentiate between UTF-8 as a text encoding in the web-service and UTF-8 encoded XML and the W3C character encoding rules that dictates that a unicode character must be encoded as a #xHHHH numeric character reference (i.e. 16-bit unicode code points/units; UTF-16 NCR). For more details see 'Can XML use non-Latin characters?'.

The MSCRM 1.2 object model supports and returns XML using the #xHHHH character encoding. All strings in .NET are unicode.

The character encoding went through these phases (examples for Æ and æ):

• from our MSCRM web-service: Æ encoded as Æ
• back from BIE on create/update: Æ "encoded" as Æ (not even valid UTF-8 code units)
• data in MSCRM after operation: Æ

• from our MSCRM web-service: æ encoded as æ
• back from BIE on create/update: æ encoded as æ (which are valid UTF-8 code units)
• data in MSCRM after operation: æ

To make the problem even worse, by running the same BIE test case (workflow route) over and over, these 'funny' characters would double each time, as each XML encoded character from our service was sent back as two UTF-8 code units from BIE and not as XML UTF-16 code units (the W3C character encoding standard).

A nice unicode character encoding test tool is available here.

The Java lead developer had run into these encoding problems in BIE and modified the portal code to do some character replacing (Æ to Æ, etc) on all text to/from BIE on their side, and I asked them to change their integration implementation to comply with the WS-I and W3C standards. Unfortunately, this lead developer is more interested in hailing the glory of the application architecture & design and the open source movement than complying with standards "invented by Microsoft" :-)

Thus, I had to write a SoapExtension to modify the incoming SOAP message before it was deserialized, to change the "wrong" UTF-8 encoding into correct XML W3C UTF-8 encoding. I used this GotDotNet sample as the basis for my code and this is how I modify the incoming and outgoing SOAP messages:

public override void ProcessMessage(SoapMessage message)
{
switch (message.Stage)
{
//INCOMING
case SoapMessageStage.BeforeDeserialize:
this.ChangeIncomingEncoding();
break;
case SoapMessageStage.AfterDeserialize:
_isPostDeserialize = true;
break;

//OUTGOING
case SoapMessageStage.BeforeSerialize:
break;
case SoapMessageStage.AfterSerialize:
this.ChangeOutgoingEncoding();
break;
}
}

public override Stream ChainStream( Stream stream )
{
//http://hyperthink.net/blog/CommentView,guid,eafeef67-c240-44cc-8550-974f5d378a8f.aspx

if(!_isPostDeserialize)
{
//INCOMING
_inputStream = stream;
_outputStream = new MemoryStream();
return _outputStream;
}
else
{
//OUTGOING
_outputStream = stream;
_inputStream = new MemoryStream();
return _inputStream;
}
}

public void ChangeIncomingEncoding()
{
//at BeforeDeserialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;

TextReader reader = new StreamReader(_inputStream);
TextWriter writer = new StreamWriter(_outputStream);

string line;
while((line = reader.ReadLine()) != null)
{
writer.WriteLine( Utilities.FixBieEncoding (line) );
}
writer.Flush();

//reset the new stream to ensure that AfterDeserialize is called
if(_outputStream.CanSeek)
_outputStream.Position = 0L;
}

public void ChangeOutgoingEncoding()
{
//at AfterSerialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;

Regex regex = new Regex("utf-8", RegexOptions.IgnoreCase);
TextReader reader = new StreamReader(_inputStream);
//HACK: TextWriter writer = new StreamWriter(_outputStream);
TextWriter writer = new StreamWriter(_outputStream, System.Text.Encoding.GetEncoding(_encoding));

string line;
while((line = reader.ReadLine()) != null)
{
// change the encoding only is needed
if(_encoding != null && !_encoding.Equals("utf-8"))
line = regex.Replace(line, _encoding);
writer.WriteLine(line);
}
writer.Flush();
}

The central method here is the ChangeIncomingEncoding method that converts all the "wrong" UTF-8 encodings from BIE into correct XML W3C NCRs. Note the resetting of the position to zero on the output stream after modifying the message; this is important as forgetting to do so will cause the AfterDeserialize step of the SoapMessageStage in ProcessMessage not to be called.

After deploying the modified web-service and testing it with XmlSpy, it was time to test it with the BIE workflow dashboard. The BIE developer had set up some test cases for me, and they all worked as expected. No more funny farm in the MSCRM database.

What an illustrious victory for Java-.NET web-service interoperability!

Note that you cannot test/debug your SoapExtension code by using MSIE as the test-client as HTTP POST/GET will not trigger the SoapExtension. I used XmlSpy to test and debug my code; just set some breakpoints, start your web-service in debug mode and leave it running, then trigger the SoapExtention by making a SOAP request using XmlSpy.

What's new in MSCRM 3.0

Microsoft has finally made public some white papers that describes the new features and new customization options of MSCRM 3.0, of which:

• campaigns & marketing
• creating new business entities; with offline support
• adding new relationships to entitites (not just new attributes)
• client-side validators and scripting support
• customizing activities
• workflow for activities and custom entities
• better CRM e-mail integration with Outlook 'inbox' and 'sent items'
• separate tables for custom entity attributes/relations

are the most needed improvements based on my MSCRM experience at several customers.

The use of a separate custom attribute/relation table for each entity extends the number of fields you can add to an entity in version 3.0, thus improving on the current limitation which is kind of an "official secret". MSCRM 1.2 is limited by the SQL Server 8K row-size restriction because all fields are added to the entity table (actually less because of replication overhead). This is especially hurtful for the contact entity as it is 80% full out-of-the-box in version 1.2.

To make a long story short, Mattew Wittemann has published a nice summary of the white papers, with several sceenshots, which is available here. Recommended reading!

The Microsoft MSCRM 3.0 white papers can be downloaded here (feature overview) and here (discloses new customization options).

Outlook recipients: AD contact postal address data

For our MSCRM customers we have made a small service that monitors the MSCRM database for changes to accounts and contacts, and maintains a specific Active Directory (AD) container that contains AD contacts that shadow the MSCRM accounts and contacts and their e-mail addresses. All these AD contacts are made available to Outlook through a new address book entry configured in Exchange System Manager. This ensures that all users have access to the e-mail information stored in MSCRM, even those that do not use Sales For Outlook. They can use the Outlook address book to pick or search for e-mail addresses and see details about an e-mail address such as contact name, company, postal address, etc.

The AD container is also used to hold AD distribution lists (mailing lists) built from the AD contacts. These distribution lists are also made available to Outlook through Exchange 2003. In addition, we have made a .NET add-in for Outlook that allows the users to preview the members of a list before sending e.g. the weekly newsletter e-mail (using an add-in toolbar in the e-mail Inspector window). The add-in allows the users to see the extra information stored in AD, and remove those members that should not get a mail this time. The users (ship brokers) typically decide that the e-mail should be sent to all members except those in Greece, and use the add-in to sort by country, multi-select the applicable members in a WinForms checkbox ListView and finally remove the selected members. This 'explodes' the mailing list into recipients, in addition to moving them to BCC to ensure that recipients do not see who else got this e-mail.

The code that maintains the AD contacts and the AD distribution lists runs on the Exchange Server 2003 to be able to modify data in both AD and Exchange (details here). Adding and maintaining AD contacts with .NET C# is quite easy, there are just a few pitfalls to be aware of when using AD contacts in combination with Exchange (details here).

The users recently requested the possibility to see the country of the MSCRM account/contact in the Outlook address book and when removing distribution list members. "No problem", I responded quickly and started checking postal address fields in Outlook and AD. I added full address information for an AD contact I knew was in the Outlook address book, waited for the RUS, and found the entered data in Outlook. I was ready to start coding !

First I added a 'country' column to my list view and then inspected the AddressEntry properties for access to postal address data. Unfortunately, Microsoft chose not to expose these properties in the Outlook object model. Fortunately, we were already using Redemption for other purposes in our add-in:

Redemption.MAPIUtils mapiUtils = new Redemption.MAPIUtils();
const int PR_COUNTRY = 0x3A26001E;
const int PR_EMAIL = 0x39FE001E;
foreach (Outlook.AddressEntry entry in list.Members)
{
string country = "";
object mapiCountry = mapiUtils.HrGetOneProp(entry.MAPIOBJECT, PR_COUNTRY);
if (mapiCountry != null) country = mapiCountry.ToString();

string smtp = entry.Address;
//check if Exchange address
if (smtp.StartsWith("/o="))
{
//get SMTP address from MAPI
smtp = mapiUtils.HrGetOneProp(entry.MAPIOBJECT, PR_EMAIL).ToString();
}

//add to listview
string[] items = new string[] { entry.Name, smtp, country };
ListViewItem itemX = new ListViewItem(items);
//keep name and address for later matching against recipients
itemX.Text = entry.Name;
itemX.Tag = entry.Address;
lstRecipients.Items.Add(itemX);
}
mapiUtils.Cleanup();

You will find a list of relevant MAPI property keys at OutlookCode.com.

Then I modified our AD updater service to read the country of each MSCRM account/contact to be able to set it on each AD contact. I used the ADSIEdit MMC snap-in to inspect the properties of my 'guinea pig' AD contact to see which property was used for storing the country. The 'co' property contained the country name, and the property 'countryCode' seemed to have something to do with a contact's country. Some googling lead me to MSDN, which revealed that these three properties must be set according to ISO 3166:

1. c: two letter country designation
2. co: country name
3. countryCode: country code (integer)

You will find the ISO 3166 list here. Copy and save the list to a .TXT file, open it in Excel as a 'fixed width' file to convert the text into a useful .XLS table. Use SQL Server 2000 DTS to import the .XLS to a new table in your database, enabling you to lookup MSCRM country names in the ISO 3166 country list.

The code to set MSCRM data into an AD contact properties looks like this:

//set properties
adContact.Properties["DisplayName"].Value = displayName;
adContact.Properties["mail"].Value = mailAddress;
//NOTE: AD fails on empty string, "invalid attribute syntax"
if(firstName.Length!=0) adContact.Properties["givenName"].Value = firstName;
if(lastName.Length!=0) adContact.Properties["sn"].Value = lastName;
if(company.Length!=0) adContact.Properties["company"].Value = company;
if(department.Length!=0) adContact.Properties["department"].Value = department;

if(country.IndexOf(";")>0)
{
//format: name;code;number
country += ";;;"; //just to be sure
string[] parts = country.Split(new char[]{';'});
string countryName = parts[0];
string countryA2 = parts[1];
string countryCode = parts[2];
if(countryName.Length!=0) adContact.Properties["co"].Value = countryName;
if(countryA2.Length!=0) adContact.Properties["c"].Value = countryA2;
if(countryCode.Length!=0) adContact.Properties["countryCode"].Value = Convert.ToInt32(countryCode);
}

// Flush to the directory
adContact.CommitChanges();

Note that I chose to require the country information to be provided as a semicolon separated string just for "simplicity" as I have not used entity objects (data xfer objects) in my AD service. I think refactoring of my service interface is needed, soon the users will want "just one more field" to be added...

Programming tip: use a reverse for-loop (i--) when removing entries from the Outlook.MailItem.Recipients collection, because the collection changes when a recipient is removed and this messes up the iteration if not performed end-to-beginning.

Use a toolbar in multiple Outlook 2003 inspectors with VSTO

Making .NET add-ins for Outlook 2003 has become straightforward with VSTO 2005 (VSTO-O), especially of you only add your menus and toolbars and event handlers to the main window of Outlook (Outlook.Application.ActiveExplorer). Most of the samples available with VSTO-O do just this.

Adding toolbars and event handlers to the popup windows (Outlook.Application.Inspectors) that Outlook uses to show details about a mail, an appointment, a task, etc, will at first seem to be trivial, but there are some pitfalls. Things might work fine with a single inspector, but you need to test your Inspector add-in stuff by opening multiple inspectors at once and clicking your toolbar in all of them to test your event handler. Typically, only the first inspector will trigger the event, or your event will trigger once for each open inspector. Not to forget the event working for some time, then stop working due to the well-known 'garbage collector ate my event handler' mistake.

What you need to make your code support Outlook inspectors correctly, is an inspector wrapper class that gives your code one custom object per open inspector at run-time. The wrapper lets you add code that ensures that you handle only inspectors for specific item types; e.g. mails, but not contacts, tasks and appointments (check the Outlook.OlItemType of the Inspector.CurrentItem). The wrapper also ensures that the correct instance of your event handler code gets called when your toolbar is clicked in one of the multiple open inspectors. Finally, the wrapper keeps a reference to your toolbar and event handler for the lifetime of each inspector, solving the garbage collector problem.

I have used the inspector wrapper code written by Helmut Obertanner, which is available here at OutlookCode.com. The code is for .NET C# pre VSTO-O, but will work with a few modifications. Refer to the related discussions in the forum for how to solve diverse add-in problems.

[UPDATE] Helmut has provided an updated version of the explorer and inspector wrapper at his site, including applicable Marshal.ReleaseComObject() calls: download the X4UTools.

[UPDATE] If Outlook hangs around in the background when closed, then you have missed calling Marshal.ReleaseComObject() for some Outlook objects created or referenced by your add-in. This can also be the cause of the "The operation failed due to network or other communication problems. Check your connections and try again." message, be sure to release all Outlook (COM) objects you create.

I have modified the code slightly to work with "temporary" Outlook toolbars and to ensure that multiple inspectors functions correctly:


public class XMailItem
{
private const string _TOOL_EDITMAILINGLIST = "OW_EDITMAILINGLIST";
private const string _BTN_EDITMAILLIST = "Choose mailing list members";
private DateTime _createdDts = DateTime.Now;
private Office.CommandBar _toolBar;
private Office.CommandBarButton _btnEditMailingList;

. . .

private void MailItem_Open(ref bool Cancel)
{
#if DEBUG
DateTime tmp = _createdDts; //inspect to check which run-time inspector object this is
#endif
// event isn't needed anymore
_mailItem.Open -= new Microsoft.Office.Interop.Outlook. ItemEvents_10_OpenEventHandler(MailItem_Open);

// get the Inspector here
_inspector = (Outlook.InspectorClass)_mailItem.GetInspector;

// register for the Inspector events
_inspector.InspectorEvents_Event_Close += new Microsoft.Office.Interop.Outlook. InspectorEvents_CloseEventHandler(Inspector_InspectorEvents_Close);

//create the toolbar
this.InitializeMailToolbar();
}



private void InitializeMailToolbar()
{
try
{

if (_mailItem is Outlook.MailItem)
{
//find existing toolbar (same toolbar in all inspectors), even when temporary
try
{
_toolBar = _inspector.CommandBars[_TOOL_EDITMAILINGLIST];
}
catch (Exception)
{
//add toolbar
_toolBar = _inspector.CommandBars.Add(_TOOL_EDITMAILINGLIST, Office.MsoBarPosition.msoBarTop, false, true);
}

//find existing toolbar button
try
{
_btnEditMailingList = (Office.CommandBarButton)_inspector. CommandBars[_TOOL_EDITMAILINGLIST].Controls[_BTN_EDITMAILLIST];
}
catch (Exception)
{
//add button
_btnEditMailingList = (Office.CommandBarButton)_toolBar.Controls.Add(Office.MsoControlType.msoControlButton, Type.Missing, Type.Missing, 1, true);
_btnEditMailingList.Caption = _BTN_EDITMAILLIST;
_btnEditMailingList.Style = Office.MsoButtonStyle.msoButtonCaption;
}

_toolBar.Visible = true;
_btnEditMailingList.Visible = true;
//add event handler to button; each open inspector adds itself to the event handler chain (+=)
_btnEditMailingList.Click += new Microsoft.Office.Core._CommandBarButtonEvents_ClickEventHandler(_btnEditMailingList_Click);

}
}
catch (Exception ex)
{
MessageBox.Show("An unexpected error occurred during toolbar init: " + ex.Message, CONST.MSGBOX_TITLE, MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
}
}


void _btnEditMailingList_Click(Microsoft.Office.Core.CommandBarButton Ctrl, ref bool CancelDefault)
{
#if DEBUG
DateTime tmp = _createdDts; //inspect to check which run-time inspector object this is
#endif
this.ShowEditMailingListDialog();
}


private void Inspector_InspectorEvents_Close()
{
#if DEBUG
DateTime tmp = _createdDts; //inspect to check which run-time inspector object this is
#endif
try
{
//raise event, to remove us from active items collection
if (Item_Closed != null)
{
Item_Closed(this, new XEventArgs(_mailItem.GetHashCode()));
}

//cleanup resources; remove this from event handler chains
_btnEditMailingList.Click -= new Microsoft.Office.Core. _CommandBarButtonEvents_ClickEventHandler(_btnEditMailingList_Click);

_inspector.InspectorEvents_Event_Close -= new Microsoft.Office.Interop.Outlook. InspectorEvents_CloseEventHandler(Inspector_InspectorEvents_Close);

//release Outlook COM objects as applicable
Marshal.ReleaseComObject(_inspector);
Marshal.ReleaseComObject(_mailItem);
}
catch (System.Exception ex)
{
MessageBox.Show("An unexpected error occurred during inspector close: " + ex.Message, CONST.MSGBOX_TITLE, MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
}
}

. . .

} //XMailItem

It is important to know that the key of the Controls[] collection is actually the Caption of the button. Failure to use the same text in both places will cause multiple buttons to be added to the toolbar in some circumstances (e.g. use the next and previous mail buttons a couple of times to move back and forth, use the move to folder button, etc).

Note how the toolbar button click event handler for each wrapper is added (+=) to the button's event delegate chain. With no further code than this, Outlook (.NET) will be able to call only the event handler in the wrapper object of the inspector that triggered the click event of the common toolbar. Remember to remove your event handler from the event chain when the inspector closes.

Note that the toolbar is shared between all open inspectors, thus you must never delete it when an inspector closes as this will remove the toolbar from all open inspectors. Still, it is recommended to remove the toolbar when the add-in unloads, as a best practise, should the "temporary" flag not apply to your Outlook configuration (temporary has no effect when using Word as the mail editor).

If your toolbar opens dialog boxes, I strongly suggest that they are modal, .ShowDialog(), to avoid confusing your users with which dialog belongs to which inspector.

[UPDATE] Ken Slovak has published example code including explorer and inspector wrappers for Outlook 2007: templates from Professional Outlook 2007 Programming.

.NET add-ins for both Office 2000 and XP/2003

My current project is implementing MSCRM and SharePoint with some enhancements such as archive documents and e-mails directly from Office/Outlook to SharePoint document archives integrated into the MSCRM account, contact and case modules. The enhancements are Office add-ins .NET components that we deliver as part of our MSCRM and/or SharePoint projects. The add-ins were implemented with the Office XP PIAs and as such support only Office XP and 2003 (version 10.x and 11.x) out-of-the-box.

This customer unfortunately has Office 2000 and Outlook 2000 (version 9.x), and has no money or willingness to upgrade to Office 2003. Thus, I had to figure out how to make the .NET add-ins support Office 2000. It turned out that this is actually quite possible, following these guidelines:

• Use the Office XP PIAs, always install them to GAC (download from MSDN)
• Use the Office object model interfaces, not the classes
• Implement version switch code whenever Office 9 methods have different signatures than Office 10/11
• Use .NET reflection and .GetType().InvokeMember(...) to call the Office 9 methods that are different from Office 10/11

Microsoft provides no PIAs for Office 2000, only for Office XP and for Office 2003.

The Office object model provides you with both interfaces and classes, typically in pairs. E.g. Application/ApplicationClass, Document/DocumentClass, MailItem/MailItemClass. Always declare your object references using the interface, not the class. The class is version specific, and you might get cast errors when your add-in runs on a different Office version.

Some methods have different number of parameters in Office 2000 than in XP/2003. This applies e.g. to Word.Documents.Open(...) and Word.Document.SaveAs(...). Thus, you cannot use these methods directly in your code as it will cause run-time errors when loaded in e.g. Word 2000. Note that the code will compile OK, afterall the code references the Office XP PIAs. Your code must check the version number at run-time and switch between calling the Office XP/2003 or the Office 2000 methods.

To be able to call some of the Office 2000 methods from your code, you must use .NET reflection to call the methods using .InvokeMember(...), which is similar to COM "late binding". The code will look like this:

string version = this._application.version;
if(version.StartsWith("9."))
{
Object[] params = new Object[]{fileUrl, Type.Missing, ...};
doc.GetType().InvokeMember("SaveAs", BindingFlags. InvokeMethod, null, doc, params);
}
else
{
doc.SaveAs(fileUrl, ref param1, ref param2, ...);
}

Deploying at this customer also required some changes to our installers. The .NET 1.1 add-ins (not VSTO 2005 add-ins) read settings from our Objectware.OfficeAddin.dll.config file which has to be installed to the Office executable folder, which differs between the versions:

C:\program files\microsoft office\
plus the applicable folder office\ or office10\ or office11\

In addition, this customer runs both Windows XP and Windows 2000, which requires the add-in registry settings to be modified to refer to the correct location of MSCorEE.DLL:

C:\winNT\system32\MSCorEE.DLL for Win2000Pro
C:\windows\system32\MSCorEE.DLL for WinXPPro

A final issue about using .NET web-services in your Office add-in: if the web-service uses serializable objects as parameters in the WSDL, the .NET framework will try to auto-generate and -compile classes for these objects on the client side at run-time. As your add-in runs within the Office process, this will most likely cause hard to track and resolve exceptions. I recommend using only basic .NET value types and array types (e.g. ArrayList) in any web-service that is to be consumed by an Office add-in.

Office 2000 object model reference at MSDN.
Office XP object model reference at MSDN.
Office 2003 object model reference at MSDN.
(Navigate to the VBA language reference using the treeview if the link does not)