A lot of the themes that I have blogged about here stems from a MSCRM and Exchange based solution that utilizes SharePoint, Office and Outlook as the front-end. I worked as the lead developer on the project and did a lot of cool MSCRM, AD, Outlook and Exchange stuff, plus some VSTO-O alpha development. Mads Nissen did the SharePoint stuff and lately an extra VSTO-O add-in.
This shipbroker solution was chosen by Microsoft to become one of a few worldwide customer evidence videos! Watch the video here. I am (unfortunately?) not in the video.
The project would not have succeeded without the effort of a bunch of other people not mentioned here, but then noone gets forgotten.
PS! if the video won't start, download the launcher and start it using MediaPlayer (afterall, it is a Microsoft video).
Thursday, September 15, 2005
Wednesday, August 31, 2005
Web-services: interoperability & encoding
I am currently implementing MSCRM at a customer that is also implementing an enterprise portal using SiteVision delivered by Objectware's Java department. The need for providing the SiteVision portal with information about MSCRM accounts made me implement a few .NET XML web-services to be used by the Java guys. The web-service provides basic read, create and update services for accounts and contacts. The 'data transfer object' was simply implemented as a string that would contain XML according to a predefined XSD schema.
The enterprise application integration server used is BIE (Business Integration Engine; Java open source), chosen by the Java lead developer on this project. Their use of my web-services went along quite smootly as was to be expected as .NET web-services complies with WS-I BP1.0 (intro article here). I had even used the WS-I compliance tools to check my web-services.
What caused a major halt on the integration implementation was when we started testing with data containing Norwegian characters (ÆØÅ). To be more precise, this was tested and found to be OK by using MSIE as the test-client, and by using XmlSpy to retrieve data about e.g. an account, modifying the data, and finally updating the account. I strongly recommend the XmlSpy SOAP Debugger tool both for testing and debugging web-serives.
The problem was that although BIE supports web-servies using UTF-8, it does not differentiate between UTF-8 as a text encoding in the web-service and UTF-8 encoded XML and the W3C character encoding rules that dictates that a unicode character must be encoded as a #xHHHH numeric character reference (i.e. 16-bit unicode code points/units; UTF-16 NCR). For more details see 'Can XML use non-Latin characters?'.
The MSCRM 1.2 object model supports and returns XML using the #xHHHH character encoding. All strings in .NET are unicode.
The character encoding went through these phases (examples for Æ and æ):
A nice unicode character encoding test tool is available here.
The Java lead developer had run into these encoding problems in BIE and modified the portal code to do some character replacing (Æ to Æ, etc) on all text to/from BIE on their side, and I asked them to change their integration implementation to comply with the WS-I and W3C standards. Unfortunately, this lead developer is more interested in hailing the glory of the application architecture & design and the open source movement than complying with standards "invented by Microsoft" :-)
Thus, I had to write a SoapExtension to modify the incoming SOAP message before it was deserialized, to change the "wrong" UTF-8 encoding into correct XML W3C UTF-8 encoding. I used this GotDotNet sample as the basis for my code and this is how I modify the incoming and outgoing SOAP messages:
public override void ProcessMessage(SoapMessage message)
{
switch (message.Stage)
{
//INCOMING
case SoapMessageStage.BeforeDeserialize:
this.ChangeIncomingEncoding();
break;
case SoapMessageStage.AfterDeserialize:
_isPostDeserialize = true;
break;
//OUTGOING
case SoapMessageStage.BeforeSerialize:
break;
case SoapMessageStage.AfterSerialize:
this.ChangeOutgoingEncoding();
break;
}
}
public override Stream ChainStream( Stream stream )
{
//http://hyperthink.net/blog/CommentView,guid,eafeef67-c240-44cc-8550-974f5d378a8f.aspx
if(!_isPostDeserialize)
{
//INCOMING
_inputStream = stream;
_outputStream = new MemoryStream();
return _outputStream;
}
else
{
//OUTGOING
_outputStream = stream;
_inputStream = new MemoryStream();
return _inputStream;
}
}
public void ChangeIncomingEncoding()
{
//at BeforeDeserialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;
TextReader reader = new StreamReader(_inputStream);
TextWriter writer = new StreamWriter(_outputStream);
string line;
while((line = reader.ReadLine()) != null)
{
writer.WriteLine( Utilities.FixBieEncoding (line) );
}
writer.Flush();
//reset the new stream to ensure that AfterDeserialize is called
if(_outputStream.CanSeek)
_outputStream.Position = 0L;
}
public void ChangeOutgoingEncoding()
{
//at AfterSerialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;
Regex regex = new Regex("utf-8", RegexOptions.IgnoreCase);
TextReader reader = new StreamReader(_inputStream);
//HACK: TextWriter writer = new StreamWriter(_outputStream);
TextWriter writer = new StreamWriter(_outputStream, System.Text.Encoding.GetEncoding(_encoding));
string line;
while((line = reader.ReadLine()) != null)
{
// change the encoding only is needed
if(_encoding != null && !_encoding.Equals("utf-8"))
line = regex.Replace(line, _encoding);
writer.WriteLine(line);
}
writer.Flush();
}
The central method here is the ChangeIncomingEncoding method that converts all the "wrong" UTF-8 encodings from BIE into correct XML W3C NCRs. Note the resetting of the position to zero on the output stream after modifying the message; this is important as forgetting to do so will cause the AfterDeserialize step of the SoapMessageStage in ProcessMessage not to be called.
After deploying the modified web-service and testing it with XmlSpy, it was time to test it with the BIE workflow dashboard. The BIE developer had set up some test cases for me, and they all worked as expected. No more funny farm in the MSCRM database.
What an illustrious victory for Java-.NET web-service interoperability!
Note that you cannot test/debug your SoapExtension code by using MSIE as the test-client as HTTP POST/GET will not trigger the SoapExtension. I used XmlSpy to test and debug my code; just set some breakpoints, start your web-service in debug mode and leave it running, then trigger the SoapExtention by making a SOAP request using XmlSpy.
The enterprise application integration server used is BIE (Business Integration Engine; Java open source), chosen by the Java lead developer on this project. Their use of my web-services went along quite smootly as was to be expected as .NET web-services complies with WS-I BP1.0 (intro article here). I had even used the WS-I compliance tools to check my web-services.
What caused a major halt on the integration implementation was when we started testing with data containing Norwegian characters (ÆØÅ). To be more precise, this was tested and found to be OK by using MSIE as the test-client, and by using XmlSpy to retrieve data about e.g. an account, modifying the data, and finally updating the account. I strongly recommend the XmlSpy SOAP Debugger tool both for testing and debugging web-serives.
The problem was that although BIE supports web-servies using UTF-8, it does not differentiate between UTF-8 as a text encoding in the web-service and UTF-8 encoded XML and the W3C character encoding rules that dictates that a unicode character must be encoded as a #xHHHH numeric character reference (i.e. 16-bit unicode code points/units; UTF-16 NCR). For more details see 'Can XML use non-Latin characters?'.
The MSCRM 1.2 object model supports and returns XML using the #xHHHH character encoding. All strings in .NET are unicode.
The character encoding went through these phases (examples for Æ and æ):
- from our MSCRM web-service: Æ encoded as Æ
- back from BIE on create/update: Æ "encoded" as Æ (not even valid UTF-8 code units)
- data in MSCRM after operation: Æ
- from our MSCRM web-service: æ encoded as æ
- back from BIE on create/update: æ encoded as æ (which are valid UTF-8 code units)
- data in MSCRM after operation: æ
A nice unicode character encoding test tool is available here.
The Java lead developer had run into these encoding problems in BIE and modified the portal code to do some character replacing (Æ to Æ, etc) on all text to/from BIE on their side, and I asked them to change their integration implementation to comply with the WS-I and W3C standards. Unfortunately, this lead developer is more interested in hailing the glory of the application architecture & design and the open source movement than complying with standards "invented by Microsoft" :-)
Thus, I had to write a SoapExtension to modify the incoming SOAP message before it was deserialized, to change the "wrong" UTF-8 encoding into correct XML W3C UTF-8 encoding. I used this GotDotNet sample as the basis for my code and this is how I modify the incoming and outgoing SOAP messages:
public override void ProcessMessage(SoapMessage message)
{
switch (message.Stage)
{
//INCOMING
case SoapMessageStage.BeforeDeserialize:
this.ChangeIncomingEncoding();
break;
case SoapMessageStage.AfterDeserialize:
_isPostDeserialize = true;
break;
//OUTGOING
case SoapMessageStage.BeforeSerialize:
break;
case SoapMessageStage.AfterSerialize:
this.ChangeOutgoingEncoding();
break;
}
}
public override Stream ChainStream( Stream stream )
{
//http://hyperthink.net/blog/CommentView,guid,eafeef67-c240-44cc-8550-974f5d378a8f.aspx
if(!_isPostDeserialize)
{
//INCOMING
_inputStream = stream;
_outputStream = new MemoryStream();
return _outputStream;
}
else
{
//OUTGOING
_outputStream = stream;
_inputStream = new MemoryStream();
return _inputStream;
}
}
public void ChangeIncomingEncoding()
{
//at BeforeDeserialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;
TextReader reader = new StreamReader(_inputStream);
TextWriter writer = new StreamWriter(_outputStream);
string line;
while((line = reader.ReadLine()) != null)
{
writer.WriteLine( Utilities.FixBieEncoding (line) );
}
writer.Flush();
//reset the new stream to ensure that AfterDeserialize is called
if(_outputStream.CanSeek)
_outputStream.Position = 0L;
}
public void ChangeOutgoingEncoding()
{
//at AfterSerialize
if(_inputStream.CanSeek)
_inputStream.Position = 0L;
Regex regex = new Regex("utf-8", RegexOptions.IgnoreCase);
TextReader reader = new StreamReader(_inputStream);
//HACK: TextWriter writer = new StreamWriter(_outputStream);
TextWriter writer = new StreamWriter(_outputStream, System.Text.Encoding.GetEncoding(_encoding));
string line;
while((line = reader.ReadLine()) != null)
{
// change the encoding only is needed
if(_encoding != null && !_encoding.Equals("utf-8"))
line = regex.Replace(line, _encoding);
writer.WriteLine(line);
}
writer.Flush();
}
The central method here is the ChangeIncomingEncoding method that converts all the "wrong" UTF-8 encodings from BIE into correct XML W3C NCRs. Note the resetting of the position to zero on the output stream after modifying the message; this is important as forgetting to do so will cause the AfterDeserialize step of the SoapMessageStage in ProcessMessage not to be called.
After deploying the modified web-service and testing it with XmlSpy, it was time to test it with the BIE workflow dashboard. The BIE developer had set up some test cases for me, and they all worked as expected. No more funny farm in the MSCRM database.
What an illustrious victory for Java-.NET web-service interoperability!
Note that you cannot test/debug your SoapExtension code by using MSIE as the test-client as HTTP POST/GET will not trigger the SoapExtension. I used XmlSpy to test and debug my code; just set some breakpoints, start your web-service in debug mode and leave it running, then trigger the SoapExtention by making a SOAP request using XmlSpy.
Friday, August 26, 2005
What's new in MSCRM 3.0
Microsoft has finally made public some white papers that describes the new features and new customization options of MSCRM 3.0, of which:
The use of a separate custom attribute/relation table for each entity extends the number of fields you can add to an entity in version 3.0, thus improving on the current limitation which is kind of an "official secret". MSCRM 1.2 is limited by the SQL Server 8K row-size restriction because all fields are added to the entity table (actually less because of replication overhead). This is especially hurtful for the contact entity as it is 80% full out-of-the-box in version 1.2.
To make a long story short, Mattew Wittemann has published a nice summary of the white papers, with several sceenshots, which is available here. Recommended reading!
The Microsoft MSCRM 3.0 white papers can be downloaded here (feature overview) and here (discloses new customization options).
- campaigns & marketing
- creating new business entities; with offline support
- adding new relationships to entitites (not just new attributes)
- client-side validators and scripting support
- customizing activities
- workflow for activities and custom entities
- better CRM e-mail integration with Outlook 'inbox' and 'sent items'
- separate tables for custom entity attributes/relations
The use of a separate custom attribute/relation table for each entity extends the number of fields you can add to an entity in version 3.0, thus improving on the current limitation which is kind of an "official secret". MSCRM 1.2 is limited by the SQL Server 8K row-size restriction because all fields are added to the entity table (actually less because of replication overhead). This is especially hurtful for the contact entity as it is 80% full out-of-the-box in version 1.2.
To make a long story short, Mattew Wittemann has published a nice summary of the white papers, with several sceenshots, which is available here. Recommended reading!
The Microsoft MSCRM 3.0 white papers can be downloaded here (feature overview) and here (discloses new customization options).
Friday, August 12, 2005
Outlook recipients: AD contact postal address data
For our MSCRM customers we have made a small service that monitors the MSCRM database for changes to accounts and contacts, and maintains a specific Active Directory (AD) container that contains AD contacts that shadow the MSCRM accounts and contacts and their e-mail addresses. All these AD contacts are made available to Outlook through a new address book entry configured in Exchange System Manager. This ensures that all users have access to the e-mail information stored in MSCRM, even those that do not use Sales For Outlook. They can use the Outlook address book to pick or search for e-mail addresses and see details about an e-mail address such as contact name, company, postal address, etc.
The AD container is also used to hold AD distribution lists (mailing lists) built from the AD contacts. These distribution lists are also made available to Outlook through Exchange 2003. In addition, we have made a .NET add-in for Outlook that allows the users to preview the members of a list before sending e.g. the weekly newsletter e-mail (using an add-in toolbar in the e-mail Inspector window). The add-in allows the users to see the extra information stored in AD, and remove those members that should not get a mail this time. The users (ship brokers) typically decide that the e-mail should be sent to all members except those in Greece, and use the add-in to sort by country, multi-select the applicable members in a WinForms checkbox ListView and finally remove the selected members. This 'explodes' the mailing list into recipients, in addition to moving them to BCC to ensure that recipients do not see who else got this e-mail.
The code that maintains the AD contacts and the AD distribution lists runs on the Exchange Server 2003 to be able to modify data in both AD and Exchange (details here). Adding and maintaining AD contacts with .NET C# is quite easy, there are just a few pitfalls to be aware of when using AD contacts in combination with Exchange (details here).
The users recently requested the possibility to see the country of the MSCRM account/contact in the Outlook address book and when removing distribution list members. "No problem", I responded quickly and started checking postal address fields in Outlook and AD. I added full address information for an AD contact I knew was in the Outlook address book, waited for the RUS, and found the entered data in Outlook. I was ready to start coding !
First I added a 'country' column to my list view and then inspected the AddressEntry properties for access to postal address data. Unfortunately, Microsoft chose not to expose these properties in the Outlook object model. Fortunately, we were already using Redemption for other purposes in our add-in:
Redemption.MAPIUtils mapiUtils = new Redemption.MAPIUtils();
const int PR_COUNTRY = 0x3A26001E;
const int PR_EMAIL = 0x39FE001E;
foreach (Outlook.AddressEntry entry in list.Members)
{
string country = "";
object mapiCountry = mapiUtils.HrGetOneProp(entry.MAPIOBJECT, PR_COUNTRY);
if (mapiCountry != null) country = mapiCountry.ToString();
string smtp = entry.Address;
//check if Exchange address
if (smtp.StartsWith("/o="))
{
//get SMTP address from MAPI
smtp = mapiUtils.HrGetOneProp(entry.MAPIOBJECT, PR_EMAIL).ToString();
}
//add to listview
string[] items = new string[] { entry.Name, smtp, country };
ListViewItem itemX = new ListViewItem(items);
//keep name and address for later matching against recipients
itemX.Text = entry.Name;
itemX.Tag = entry.Address;
lstRecipients.Items.Add(itemX);
}
mapiUtils.Cleanup();
You will find a list of relevant MAPI property keys at OutlookCode.com.
The AD container is also used to hold AD distribution lists (mailing lists) built from the AD contacts. These distribution lists are also made available to Outlook through Exchange 2003. In addition, we have made a .NET add-in for Outlook that allows the users to preview the members of a list before sending e.g. the weekly newsletter e-mail (using an add-in toolbar in the e-mail Inspector window). The add-in allows the users to see the extra information stored in AD, and remove those members that should not get a mail this time. The users (ship brokers) typically decide that the e-mail should be sent to all members except those in Greece, and use the add-in to sort by country, multi-select the applicable members in a WinForms checkbox ListView and finally remove the selected members. This 'explodes' the mailing list into recipients, in addition to moving them to BCC to ensure that recipients do not see who else got this e-mail.
The code that maintains the AD contacts and the AD distribution lists runs on the Exchange Server 2003 to be able to modify data in both AD and Exchange (details here). Adding and maintaining AD contacts with .NET C# is quite easy, there are just a few pitfalls to be aware of when using AD contacts in combination with Exchange (details here).
The users recently requested the possibility to see the country of the MSCRM account/contact in the Outlook address book and when removing distribution list members. "No problem", I responded quickly and started checking postal address fields in Outlook and AD. I added full address information for an AD contact I knew was in the Outlook address book, waited for the RUS, and found the entered data in Outlook. I was ready to start coding !
First I added a 'country' column to my list view and then inspected the AddressEntry properties for access to postal address data. Unfortunately, Microsoft chose not to expose these properties in the Outlook object model. Fortunately, we were already using Redemption for other purposes in our add-in:
Redemption.MAPIUtils mapiUtils = new Redemption.MAPIUtils();
const int PR_COUNTRY = 0x3A26001E;
const int PR_EMAIL = 0x39FE001E;
foreach (Outlook.AddressEntry entry in list.Members)
{
string country = "";
object mapiCountry = mapiUtils.HrGetOneProp(entry.MAPIOBJECT, PR_COUNTRY);
if (mapiCountry != null) country = mapiCountry.ToString();
string smtp = entry.Address;
//check if Exchange address
if (smtp.StartsWith("/o="))
{
//get SMTP address from MAPI
smtp = mapiUtils.HrGetOneProp(entry.MAPIOBJECT, PR_EMAIL).ToString();
}
//add to listview
string[] items = new string[] { entry.Name, smtp, country };
ListViewItem itemX = new ListViewItem(items);
//keep name and address for later matching against recipients
itemX.Text = entry.Name;
itemX.Tag = entry.Address;
lstRecipients.Items.Add(itemX);
}
mapiUtils.Cleanup();
You will find a list of relevant MAPI property keys at OutlookCode.com.
Then I modified our AD updater service to read the country of each MSCRM account/contact to be able to set it on each AD contact. I used the ADSIEdit MMC snap-in to inspect the properties of my 'guinea pig' AD contact to see which property was used for storing the country. The 'co' property contained the country name, and the property 'countryCode' seemed to have something to do with a contact's country. Some googling lead me to MSDN, which revealed that these three properties must be set according to ISO 3166:
- c: two letter country designation
- co: country name
- countryCode: country code (integer)
You will find the ISO 3166 list here. Copy and save the list to a .TXT file, open it in Excel as a 'fixed width' file to convert the text into a useful .XLS table. Use SQL Server 2000 DTS to import the .XLS to a new table in your database, enabling you to lookup MSCRM country names in the ISO 3166 country list.
The code to set MSCRM data into an AD contact properties looks like this:
//set properties
adContact.Properties["DisplayName"].Value = displayName;
adContact.Properties["mail"].Value = mailAddress;
//NOTE: AD fails on empty string, "invalid attribute syntax"
if(firstName.Length!=0) adContact.Properties["givenName"].Value = firstName;
if(lastName.Length!=0) adContact.Properties["sn"].Value = lastName;
if(company.Length!=0) adContact.Properties["company"].Value = company;
if(department.Length!=0) adContact.Properties["department"].Value = department;
if(country.IndexOf(";")>0)
{
//format: name;code;number
country += ";;;"; //just to be sure
string[] parts = country.Split(new char[]{';'});
string countryName = parts[0];
string countryA2 = parts[1];
string countryCode = parts[2];
if(countryName.Length!=0) adContact.Properties["co"].Value = countryName;
if(countryA2.Length!=0) adContact.Properties["c"].Value = countryA2;
if(countryCode.Length!=0) adContact.Properties["countryCode"].Value = Convert.ToInt32(countryCode);
}
// Flush to the directory
adContact.CommitChanges();
Note that I chose to require the country information to be provided as a semicolon separated string just for "simplicity" as I have not used entity objects (data xfer objects) in my AD service. I think refactoring of my service interface is needed, soon the users will want "just one more field" to be added...
Programming tip: use a reverse for-loop (i--) when removing entries from the Outlook.MailItem.Recipients collection, because the collection changes when a recipient is removed and this messes up the iteration if not performed end-to-beginning.
Subscribe to:
Posts (Atom)