Outlook recipient autocomplete, AD contacts sharing SMTP address

We have implemented an extension to MSCRM that replicates accounts and contacts to Active Directory as AD contacts for use as an Outlook address book (see this post). It is not very common, but some accounts and contacts do actually share the same SMTP mail address. E.g. several departments of a large corporation might share the same mailbox, and your user expects to find each department in the MSCRM address book in Outlook as separate contacts.

A few days ago, the operations department at one of our customers raised a concern about the support in Exchange Server 2003 for resolving mail recipients against the set of AD contacts when some of them share the same SMTP address. "This is not allowed in AD, and you should fix your code to not add the same SMTP address multiple times to AD" he stated. What ? As "proof" he sent me a screenshot of an error he got when trying to add a new contact with a duplicate SMTP address using the AD Users and Computers admin tool:

This e-mail address already exists in this organization (c10312e7)

This is a known limitation in the Active Directory Users and Computers snap-in. To resolve this, do not create the contact with AD Users and Computers, but use the ADSI-edit MMC snap-in or the ADSI component from your code to add multiple AD contacts that must share the same SMTP address. The AD attributes to set are "mail", "mailNickname", "proxyAddress", and "targetAddress". Those fields can be modified to enter a SMTP address that exist for other contacts without error (more info at MSDN).

Note that you should not create Exchange mailboxes for any external AD contact, and never create AD contacts duplicating the mail addresses of internal users. This will lead to NDR errors during Exchange resolving and mail delivery. Also ensure that you use correct settings for the Exchange Recipient Update Service (RUS) on your AD contacts, as RUS will automatically create Exchange entries for your AD contacts when not disabled as applicable. Read more in this post.

A related problem to Exchange NDRs for unreliable AD contact data, is the Outlook 2003 recipient cache. This autocomplete mechanism is heavensent when adding recipients that you send e-mail to often, but it is a p.i.t.a when your favorite recipients change their e-mail address or their data change in AD (the source of Outlook contact groups, a.k.a Exchange Global Address Lists). E.g. removing contacts from AD might lead to errors when Exchange tries to resolve cached recipients.

It is not easy to change the cached recipients list in Outlook using Outlook. The only option you have is to delete cached entries, but not many users have been able to find where to do that or clear the cache. It is actually quite simple, but not very intuitive; just select the entry you want to remove in the autocomplete suggestion list and press 'Delete' on your keyboard. The whole list of cached recipients is stored in a .NK2 file in your application data folder. Delete this file to clear the cache completely.

Read this article at Outlook Exchange for more info about the Outlook recipient autocomplete cache (Outlook .NK2 file). Also check out the Ingressor NK2 Management tool for editing and managing the .NK2 file.

MSCRM 3.0 in a multi AD forest infrastructure

MSCRM 3.0 by default supports a single AD domain (really a single AD forest) and a single Exchange 'organization'. The full spectrum of MSCRM functionality will be available to your users when your infrastructure adheres to these requirements. I will call the AD domain into which you install MSCRM, SQL Server 2000/2005 and SRS, the native domain. The same term is used for the Exchange organization of the native AD domain.

These are most likely infrastructure challenges you will encounter outside the native domain:

• Outlook desktop client: access to MSCRM platform services
• Outlook laptop client: desktop + go offline and online
• Exchange: Routing of incoming e-mails to MSCRM users and queues
• SQL Server Reporting Services: access to SRS services for reporting

First an overview of what sould work and what should not, dependent on some 'unsupported' infrastructure scenarios:

If you have multiple AD forests without explicit trusts, then the users not in the native domain will get only basic MSCRM functionality; the web client over HTTPS with basic authentication. These users will not be able to use neither the online nor the offline Outlook client (MSCRM desktop / laptop client) as they are not logged on to the domain. Note that such users will not get full reporting functionality with SQL Server Reporting Services (SRS) in this scenario.

If you have multiple Exhange organizations without explicit trusts; then the users not in the native organization (forest) will get only basic send e-mail functionality, the 'e-mail router' will not be able to automatically route incoming e-mails as the mailboxes are not in the native organization. In addition, the router cannot access the native AD domain when not explicitly trusted.

If you have users in an NT4 domain with a one-way trust from the native domain, these users will be able to use both the web and desktop client. They will not be able to use the laptop client as they cannot go off/online, incoming e-mail will not be routed to them, and they will not get full reporting functionality.

Then an overview of how can a multi AD forest and Exchange organization be configured to support full MSCRM functionality:

First of all, forget getting full MSCRM functionality for NT4 domains. Microsoft does not support NT4 anymore, so your're on your own.

The good news is that your users across several AD forests will be able to get the full spectrum of functionality available in MSCRM 3.0. This will just require some configuration of your infrastructure.

The most important aspect is that you have to add at least one-way trusts from the native MSCRM domain to the other domains. Trusting requires a LAN, WAN, or VPN connection between your domains. Support for full MSCRM over plain HTTPS is not possible.

The MSCRM Outlook client requires Windows Authentication / Kerberos against the native AD domain and usage of the default security credentials on the client PC. Thus, by adding one-way trusts, your users will be able to use both the MSCRM desktop client and the laptop client. Sending e-mail will of course work, while routing of incoming e-mails to users and queues will require some more configuration (see below).

Note that basic SRS reporting functionality will be available with just one-way trusts. For full reporting functionality, two-way trusts are needed between the AD forests. Alternatively, you need to configure a fixed identity on the clients for accessing the SRS reports (KB article to be published).

MSCRM 3.0 now supports having multiple Exchange servers in your native Exchange 'organization', including Exchange clusters. It is no longer required that you have a single Exchange server handling all incoming internet e-mails for your 'organization', as the functionality of the MSCRM e-mail router has changed in v3.0.

The v1.2 router had this limitation, which made it impossible to have one common MSCRM database in a company with multiple Exchange organizations (mail domains). E.g. I work in a company with several daughter companies and thus mail domains (itera.no, objectware.no, gazette.no, etc). This meant that with v1.2 we could not get full mail functionality in MSCRM. With MSCRM 3.0, we finally can.

The router no longer inspects all incoming mail messages, but rather a specific MSCRM mailbox.

The inspection of all incoming mails have been replaced by Exchange rules that must be deployed to each Exchange server that contain one or more mailboxes of MSCRM users and queues.
Click to enlarge figure

The Exchange rules, the MSCRM mailbox and the E-mail Router by default require mailboxes to be in the native domain and native 'organization', as the router must be able to access the MSCRM platform services to do its work.

You can deploy the mail routing rules and components to other Exchange organizations, provided that you configure the routing service to use an identity that has access to the MSCRM platform services. This will of course require that you have at least a one-way trust between the AD domains.

Outlook recipients: AD contact postal address data

For our MSCRM customers we have made a small service that monitors the MSCRM database for changes to accounts and contacts, and maintains a specific Active Directory (AD) container that contains AD contacts that shadow the MSCRM accounts and contacts and their e-mail addresses. All these AD contacts are made available to Outlook through a new address book entry configured in Exchange System Manager. This ensures that all users have access to the e-mail information stored in MSCRM, even those that do not use Sales For Outlook. They can use the Outlook address book to pick or search for e-mail addresses and see details about an e-mail address such as contact name, company, postal address, etc.

The AD container is also used to hold AD distribution lists (mailing lists) built from the AD contacts. These distribution lists are also made available to Outlook through Exchange 2003. In addition, we have made a .NET add-in for Outlook that allows the users to preview the members of a list before sending e.g. the weekly newsletter e-mail (using an add-in toolbar in the e-mail Inspector window). The add-in allows the users to see the extra information stored in AD, and remove those members that should not get a mail this time. The users (ship brokers) typically decide that the e-mail should be sent to all members except those in Greece, and use the add-in to sort by country, multi-select the applicable members in a WinForms checkbox ListView and finally remove the selected members. This 'explodes' the mailing list into recipients, in addition to moving them to BCC to ensure that recipients do not see who else got this e-mail.

The code that maintains the AD contacts and the AD distribution lists runs on the Exchange Server 2003 to be able to modify data in both AD and Exchange (details here). Adding and maintaining AD contacts with .NET C# is quite easy, there are just a few pitfalls to be aware of when using AD contacts in combination with Exchange (details here).

The users recently requested the possibility to see the country of the MSCRM account/contact in the Outlook address book and when removing distribution list members. "No problem", I responded quickly and started checking postal address fields in Outlook and AD. I added full address information for an AD contact I knew was in the Outlook address book, waited for the RUS, and found the entered data in Outlook. I was ready to start coding !

First I added a 'country' column to my list view and then inspected the AddressEntry properties for access to postal address data. Unfortunately, Microsoft chose not to expose these properties in the Outlook object model. Fortunately, we were already using Redemption for other purposes in our add-in:

Redemption.MAPIUtils mapiUtils = new Redemption.MAPIUtils();
const int PR_COUNTRY = 0x3A26001E;
const int PR_EMAIL = 0x39FE001E;
foreach (Outlook.AddressEntry entry in list.Members)
{
string country = "";
object mapiCountry = mapiUtils.HrGetOneProp(entry.MAPIOBJECT, PR_COUNTRY);
if (mapiCountry != null) country = mapiCountry.ToString();

string smtp = entry.Address;
//check if Exchange address
if (smtp.StartsWith("/o="))
{
//get SMTP address from MAPI
smtp = mapiUtils.HrGetOneProp(entry.MAPIOBJECT, PR_EMAIL).ToString();
}

//add to listview
string[] items = new string[] { entry.Name, smtp, country };
ListViewItem itemX = new ListViewItem(items);
//keep name and address for later matching against recipients
itemX.Text = entry.Name;
itemX.Tag = entry.Address;
lstRecipients.Items.Add(itemX);
}
mapiUtils.Cleanup();

You will find a list of relevant MAPI property keys at OutlookCode.com.

Then I modified our AD updater service to read the country of each MSCRM account/contact to be able to set it on each AD contact. I used the ADSIEdit MMC snap-in to inspect the properties of my 'guinea pig' AD contact to see which property was used for storing the country. The 'co' property contained the country name, and the property 'countryCode' seemed to have something to do with a contact's country. Some googling lead me to MSDN, which revealed that these three properties must be set according to ISO 3166:

1. c: two letter country designation
2. co: country name
3. countryCode: country code (integer)

You will find the ISO 3166 list here. Copy and save the list to a .TXT file, open it in Excel as a 'fixed width' file to convert the text into a useful .XLS table. Use SQL Server 2000 DTS to import the .XLS to a new table in your database, enabling you to lookup MSCRM country names in the ISO 3166 country list.

The code to set MSCRM data into an AD contact properties looks like this:

//set properties
adContact.Properties["DisplayName"].Value = displayName;
adContact.Properties["mail"].Value = mailAddress;
//NOTE: AD fails on empty string, "invalid attribute syntax"
if(firstName.Length!=0) adContact.Properties["givenName"].Value = firstName;
if(lastName.Length!=0) adContact.Properties["sn"].Value = lastName;
if(company.Length!=0) adContact.Properties["company"].Value = company;
if(department.Length!=0) adContact.Properties["department"].Value = department;

if(country.IndexOf(";")>0)
{
//format: name;code;number
country += ";;;"; //just to be sure
string[] parts = country.Split(new char[]{';'});
string countryName = parts[0];
string countryA2 = parts[1];
string countryCode = parts[2];
if(countryName.Length!=0) adContact.Properties["co"].Value = countryName;
if(countryA2.Length!=0) adContact.Properties["c"].Value = countryA2;
if(countryCode.Length!=0) adContact.Properties["countryCode"].Value = Convert.ToInt32(countryCode);
}

// Flush to the directory
adContact.CommitChanges();

Note that I chose to require the country information to be provided as a semicolon separated string just for "simplicity" as I have not used entity objects (data xfer objects) in my AD service. I think refactoring of my service interface is needed, soon the users will want "just one more field" to be added...

Programming tip: use a reverse for-loop (i--) when removing entries from the Outlook.MailItem.Recipients collection, because the collection changes when a recipient is removed and this messes up the iteration if not performed end-to-beginning.

The Exchange RUS and external AD contacts

One thing that the Exchange 2003 SDK does not warn you about when adding mail enabled external Active Directory contacts is the Exchange Recipient Update Service (RUS), and this will cause strange effects when Exchange tries to resolve mail addresses. I.e. you will not be able to send e-mail to external mail contacts. Note that these effects will not happen immediately due to the periodical processing schedule of the RUS, thus you can test and believe that your code is correct, only to get hit in the face a few hours later.

The problem is that even if you set only a SMTP mail address on the new AD contact as shown in the SDK, the RUS will apply all the other types of mail addresses defined in your Exchange policies during its processing. This will give the external contact an internal Exhcange address, an X.400 address, etc. Thus, Exhcange will not be able to resolve or deliver e-mails to these contacts as most of these addresses will be bogus.

To correctly mail enable an external AD contact you need to set a GUID in two specific Exchange properties on the AD contact:

DirectoryEntry adContact = adContainter.Children.Find("CN=" + contactId, CONST.AD_CONTACT);

//exlude from RUS
adContact.Properties["msExchPoliciesExcluded"].Value = "{26491CFC-9E50-4857-861B-0CB8DF22B5D7}";
adContact.Properties["msExchPoliciesIncluded"].Value = "{26491CFC-9E50-4857-861B-0CB8DF22B5D7}";

//set mail AD property to e-mail address
//must set again after exluding from RUS
adContact.Properties["mail"].Value = mailAddress;

// Write Exchange information to the directory.
adContact.CommitChanges();

The policy GUID must be set on both the Exchange RUS policies excluded list and the policies included list. The specified GUID is valid for Exchange Server 2003. Refer to my previous post for details about how to write code for Exchange Server 2003.

You should also ensure that you never add an internal user's mail address as an AD contact through code. If your code does not prevent this, it will prevent Exchange from delivering incoming e-mail to the user's mailbox as it cannot resolve the mailbox when both an AD user and an AD contact has the same address.

Note that you should not exclude mail enabled AD distribution group (mailing lists) from the RUS, as this will cause the mailing lists to stop working. They must be processed by the RUS to function properly.

Outlook .NET add-ins connected to Exchange services

In some solutions we implement Outlook COM add-ins with .NET using the templates provided by Microsoft on MSDN. We typically add some menus, toolbars and dialog boxes that allow the users to manage stuff in their mailboxes or public folders, using client-side CDO. To avoid the Outlook security popup we utilize the Redemption component, which is widely recommended on the 'net for this purpose.

In addition, we sometimes need to access extended MAPI properties that are not exposed in CDO, and Redemption gives us easy access to these properties and other stuff that is not supported by client-side CDO. Redemption is a component that is really useful, so is the OutlookSpy tool. Recommended!

In an earlier posting, I wrote about how we prefer to implement code for managing ActiveDirectory and Exchange and its mailbox and public folder stores as server-side services. These services uses e.g. CDOEX and CDOEXM to perform the logic provided by our services.

One of our service methods is to copy a mail message from a mailbox store to a public folder store in response to an action initiated by an Outlook user through our COM add-in. This is not easy to do using CDO because it involves separate stores and also involves having sufficient permissions on the stores. This operation is, however, quite easy to implement using Exchange OleDB on the server-side and impersonating a specific identity that has the required rights. The ExOleDB methods for operating on different items in the store requires either the HTTP: or the File: URL Scheme, of which Microsoft recommends the HTTP: scheme.

The HTTP: URL scheme for mail messages uses the .EML name of the message item, and this URL is easy to see when using Outlook Web Access (OWA). This URL is available only through extended MAPI, thus we use Redemption to get it from the message item:

//get MAPI IMessage propery["PR_URL_COMP_NAME"]
int PR_URL_COMP_NAME = 0x10F3001E;
Redemption.SafeMailItem message = new Redemption.SafeMailItemClass();
message.Item = this._mailItem;
string url = message.get_Fields(PR_URL_COMP_NAME).ToString();

The string returned by Redemption is just the 'name' component of the URL (i.e. the message item subject). It must be encoded before passing it to the Exchange service for processing by ExOleDB:

//removed code from sample:
//handle that Redemption returns '?' for '/'
//and other quircks with the char encoding

//encode message item URL
url = url.Replace("/", "_xF8FF_");
url = Util.MailUrlEncode(url);
//clean up
Redemption.MAPIUtils util = new Redemption.MAPIUtilsClass();
util.Cleanup();

Note that not all characters in the URL needs to be encoded, and I have some small utility functions to handle the encoding:

public static string MailUrlEncode(string input)
{
string output="";
char[] uniChars = input.ToCharArray();
foreach(char uniChar in uniChars)
{
if(uniChar <= 'z') output += uniChar.ToString(); else output += Util.ExchangeEncode(uniChar.ToString());
}
return output;
}

public static string ExchangeEncode(string uniChar)
{
string output="";
byte[] encoded = System.Text.Encoding.UTF8.GetBytes(uniChar);
foreach(byte utf8Char in encoded)
{
output += "%" + utf8Char.ToString("X2");
}
return output;
}

Finally, the URL must be prepended with the path to the message item store (mailbox or public folder) before it becomes a complete URL that ExOleDB can operate on.

The server-side ExOleDB code for copying the message item and its attachment is not included here. I might publish a posting on this later on.

Programming Exchange 2003 using .NET

We have several projects where we need to manage e.g. Exchange public folders and distribution lists from our own code, such as SPS web parts (really ASP.NET user controls hosted in a web part such as SmartPart). There are 21 different technologies available from Microsoft for programming Exchange 2003 (see official list), thus a choice must be made, also with regards to selected API being supported by .NET (see supported list).

We do not want to use the common, but not good, approach to install Outlook on the web server or on the Exchange Server to use CDO to implement our logic. Outlook is a client tier application, and CDO belongs on the client tier. The same applies to all Office applications, just try to use Word as an automation server in a web solution to produce documents from templates. Very bad application design indeed.

After a bit of experimenting with a selection of the 21 APIs, we now use these APIs against Exchange 2003 and ActiveDirectory:

• CDOEXM
• ADSI
• System.DirectoryServices (the only native .NET component, the others are COM)
• CDOEX
• Exchange OleDB (ExOleDB)

You might wonder why we use the ActiveDirectory components (ADSI and DirectoryServices). Several of the entities that an Outlook user thinks of as part of Outlook or Exchange, are actually spit into two different storages: Exchange for mailboxes and public folders, and ActiveDirectory for contacts, distribution groups and more. AD also handles the authorization for the different objects. To be able to make code that e.g. creates a mail enabled contact (AD contact, remember), you need to use both AD and Exchange components. I recommend checking out the ADSIEdit MMC snap-in to explore the structure of AD and Exchange.

Note that CDOEXM/CDOEX must run on the Exchange server, or on a computer where the Exchange Administrative Tools have been installed (and in the same Active Directory organization). The Exchange server by default has CDOEXM/CDOEX installed (note that just copying and registering the DLLs will not work). We have chosen the former, because the latter have a long list of prerequsites for the target computer, thus using the Exchange server as the location of our .NET components is simply the easiest approach.

In order to call the CDOEXM code on the Exchange server from your WebForm or WinForm application, you can use .NET Remoting or .NET web services. We have chosen the latter, as it is simply the easiest way to implement a distributed solution for intra-system "remote procedure calls". This approach with a specific service for all our Exchange and AD logic also have several other benefits, such as making it easy to set the identity used to access AD, mailboxes, etc. Use the above mentioned ADSIEdit tool to view and set permissions for AD and Exchange objects.

Always remember that the Exchange domain (myco.com) might not be the same as your AD domain (myco.local), as this might cause you some grief when trying to understand and use the rather poor code samples provided on MSDN.

Warning: If you try to use CDOEXM code on a computer that do not have the Exchange Administrative Tools properly installed, you will at run-time get a cast exception (InvalidCastException, "specified cast is not valid") when trying to use .NativeObject to get a CDOEXM object from a .NET AD object, e.g. when your code must call MailEnable() on an AD user, AD contact or AD distribution group.

A final warning: Watch out for the Exchange RUS when mail enabling external contacts, more on this in a later posting.