Use Azure VMs with On Premises Gateway as SP2010 Branch Office Farm

Here are some notes from my ongoing experience of setting up a SharePoint 2010 "branch office" farm in Azure using the current preview of persistent Azure virtual machines in a Azure virtual network connected to the on premises Active Directory using an Azure gateway to a Juniper VPN device.

Useful TechEd Europe 2012 web-casts that show how things work in general:
VM+VNET in Azure: http://channel9.msdn.com/Events/TechEd/Europe/2012/AZR208
AD in Azure: http://channel9.msdn.com/Events/TechEd/Europe/2012/SIA205
SP in Azure: http://channel9.msdn.com/Events/TechEd/Europe/2012/OSP334

Step by step instructions that I followed:
How to Create a Virtual Network for Cross-Premises Connectivity using the Azure gateway.
How to Install a Replica Active Directory Domain Controller in Windows Azure Virtual Networks.

Creating a virtual network (vnet) is easy and simple using the Azure preview management portal. I recommed creating the local network first, as the vnet wizard otherwise might fail - without giving any useful exception message. We had an issue caused by naming the local network "6sixsix" which didn't work due to the name starting with a digit. Also note that the VPN gateway only supports one LAN subnet in the current preview.

Plan your subnets upfront and make sure that they don't overlap with the on premises subnets. Register both the existing on premises DNS server and the planned vnet DNS server when configuring the vnet. A tip here is that the first VM created in a subnet will get .4 as the last part of the IP address, so if your ADDNSSubnet is 10.3.4.0/24, then the vnet DNS will get 10.3.4.4 as its IP address. Note that you can't change the DNS configuration after adding the first VM to the network, this includes creating the Azure gateway which adds devices to the gateway subnet.

After creating the Azure virtual network, we created and started the Azure gateway for connecting to the on premises LAN using a Site-to-Site VPN tunnel using a secure IPSec connection. Creating the gateway takes some time as some devices or VMs are provisioned in the gateway subnet you specified. We then sent the public IP address of the gateway, plus the shared key and the configuration script for the Juniper VPN device to our network admin. The connection wouldn't come up, and to make a long story short, the VPN configuration needs the 'peerid' to be set to an IP address of a device in the gateway subnet. Our gateway subnet was 10.3.1.0/24 and after trying 10.3.1.4 first (see above tip), the network admin tried 10.3.1.5 and that worked. I'll come back to this below when telling you about our incident when our trial Azure account was deactivated by Microsoft.

With the Azure virtual network up and running and connected to the on premises LAN, I created the AD DNS virtual machine using the preview portal "create from gallery" option. As SP2010 is not supported on WS2012 yet, I decided to use the WS2008R2 server image in this Azure server farm. Note that you should use size "large" for hosting AD as you need to attach two data disks for storing the AD database, log files and system state backup.

I did not use powershell for creating this first VM, instead I manually changed the DNS setting on the network adapter (both IPv4 and IPv6) and then manually joined the to-be AD DNS VM to the existing domain. Note that while you're at it, also set the advanced DNS option "Use this connection's DNS suffix in DNS registration" for both network adapters. Otherwise you will get the "Changing the Primary Domain DNS name of this computer to "" failed" error when trying to join the domain.

Following the how-to for setting up a replica AD in Azure work fine, we only had some minor issues due to the existing AD being on WS2003. For instance, we found no DEFAULTIPSITELINK when creating a new site in AD, so we had to create a new site link first, then create the site and finally modify the site link so that it linked the Azure "CloudSite" and the LAN site. Then the dcpromo wizard step for AD site detection didn't manage to resolve against our WS2003 domain controller, just click "ok" on the error message and manually select the "CloudSite" in the "Select a site" page.

I really wanted to set up a read-only domain controller (RODC) to save some outgoing (egress) traffic and thus save some money, as this branch farm don't need a full fidelity domain controller. However, it is not possible to create a RODC when the existing DC is on WS2003, because RODC is a WS2008 feature. So for "Additional Domain Controller Options" we went with DNS and "Global Catalog" (GC). GC isn't required, but if not installed then all authentication traffic on login need to go all the way to the on premises DC. So to save some traffic (and money), and get faster authN in the branch farm, we added the GC - even if the extra data will drive up Azure storage cost.

The next servers in the farm were added using powershell to ensure that 1) the VM is domain joined on boot, and 2) that the DNS settings for the VM is automatically configured.

Here are some tips for using New-AzureDNS, New-AzureVMConfig and New-AzureVM:

• You can use the Azure vnet DNS server or the on premises DNS server with New-AzureDNS. I used the former.
• The New-AzureVMConfig Name parameter is used when naming and registering the server in AD and DNS. Make sure that the full computer name is unique across the domain.
• The New-AzureVM ServiceName parameter is used for the cloud DNS name prefix in the .cloudapp.net domain. It is also used to provision a "Cloud Service" in the Azure preview management portal. Even if multiple VMs can be added to the same service name (shared workload), I used uniqe names for the farm VMs (standalone virtual machine), connected using the vnet for load balancing.
• To get the built-in Azure image names, use this powershell to go through the images in the gallery until you find the one you're looking for:
         (Get-AzureVMImage)[1].ImageName

After adding the SQL Server 2012, web server and application server VMs using powershell, I logged in using RDP and verified that each server was up and running, domain joined and registered in the DNS. Note that the SQL Server image is not by default configured with separate data disks for data and log files. This means that the SQL Server 2012 master database etc is stored on the OS disk in this preview. You need to add data disks and then change the SQL Server file location configuration your self. Adding two data disks will require that the SQL Server VM is of size "large".

The next step was to intall SharePoint 2010 on the farm the next day. Thats when the trial account was deactivated because all the computing hours was spent. Even if you then reactivate the account, all your VM instances are deleted, keeping only the VHD disks. As Microsoft support says, it is easy to recreate the VMs, but they also tell you that the AD virtual machine needs a static IP which you can only get in Azure if you never delete the VM. Remember to recreate the VMs in the correct order so that they get the same IP addresses as before.

What is worse is that they also delete the virtual network and the gateway. Even if it is also easy to recreate these, your gateway will get a new public IP address and a new shared key, so you need to call your network provider again to make them reconfigure the VPN device.

I strongly recommend not using a spending capped trial account for hosting your Azure branch office farm. Microsoft deleted the VMs and the network to stop incurring costs, which was fine with non-persistent Azure VM Roles (PaaS) anyway, but not as nice for a IaaS service with a persistent server farm.

I recommend exporting your VMs using Export-AzureVM so that you don't have to recreate the VMs manually if something should happen. The exported XML will contain all the VM settings, including the attached data disks.

How to deatch Azure VMs to move or save cost: http://michaelwasham.com/2012/06/18/importing-and-exporting-virtual-machine-settings/

When we recreated the Azure virtual network and the gateway, the VPN connection would not come back up again. The issue was that this time the gateway devices had got different IP addresses, so now the "peerid" had to be configured as 10.3.4.4 to make things work.

Now the gateway is back up again, and next week I'll restore the VMs and continue with installing SP2010 on the Azure "branch office" farm. More notes to come if I run into other issues.

- - - continued - - -

Installing the SharePoint 2010 bits went smooth, but running the config wizard did not. First you need to allow incoming TCP traffic on port 1433 on the Azure SQL Server. Then the creation of the SharePoint_Config database failed with:

    Could not find stored procedure 'sp_dboption'.

...even if I had downloaded and installed SP2010 with SP1 bits. So I downloaded and installed SharePoint Server 2010 SP1 and June CU from 2011 due to the issue caused by using SQL Server 2012 and that fixed the problem. Got "Configuration Successfull" without any further issues.

Finally, I tested and verified it all by creating a SP2010 web-application with a team site, creating a self-signed certificate with IIS7 and adding an Azure port mapping for SSL (virtual machine endpoint, TCP 443 to 443), allowing me to login to the team site using my domain account over HTTPS from anywhere.

A note on the VM firewall config is that ping is by default blocked, thus you can't ping other machines in the vnet unless you configure the firewall to allow it. Also note that you can't ping addresses outside of the virtual network and the connected LAN anyway; even if you can browse to www.puzzlepart.com, you can't ping us.

SharePoint ResolvePrincipal for ADFS users

Granting permissions in SharePoint 2010 by code is done by assigning roles to user or group principals, or for claims-based application, to a claim type instance. As you will find out when implementing a claims-based applications against ADFS, the SPUtility ResolvePrincipal method that you can use against the Windows identity provider and also against forms-based authentication (FBA), don't work for ADFS users. That is no surprise when looking at the SPPrincipalSource enum that lists the site's user info list, the Windows provider, and then the FBA membership and role providers.

The solution is rather simple, pass the user identity claim value such as the user's e-mail address used in ADFS to the SPClaimProviderManager CreateUserClaim method using the TrustedProvider issuer type. Then pass the generated claim to EnsureUser to get a SPPrincipal wrapping the user identity claim:

public static SPUser ResolveAdfsPrincipal(SPWeb web, string email, string issuerIdentifier)

{

    SPUser user = null;

    if (!SPClaimProviderManager.IsEncodedClaim(email))

    {

        SPClaim claim = SPClaimProviderManager.CreateUserClaim(email, SPOriginalIssuerType.TrustedProvider, issuerIdentifier);

        if (claim != null)

        {

            string userClaim = claim.ToEncodedString();

            user = web.EnsureUser(userClaim);

        }

    }

    return user;

}

public static SPUser ResolvePrincipal(SPWeb web, string email)

{

    SPUser user = null;

    SPPrincipalInfo identity = SPUtility.ResolvePrincipal(web, email, SPPrincipalType.All, SPPrincipalSource.All, null, true);

    if (identity != null)

    {

        user = web.EnsureUser(identity.LoginName);

    }

    return user;

}

Use the name of your ADFS identity provider as the issuer identifer parameter. If you're unsure of what this string should be, add a user to a SharePoint group using the UI and look at the identity claim value generated by SharePoint.

The identity claim value follows a specific format, and with ADFS (trusted provider) the issuer identifier name is always the 2nd pipe-separated value. The 1st part starting "i:" is for "identity" type claim, while the ending ".t" is for "trusted" provider (".f" is FBA, ".w" is Windows). In between is an encoding of the claim type used as the IdentifierClaim specified when registering the ADFS trusted identity provider. See the full list of claim type and claim value type encoding characters here. The 3rd part of the claim is the actual user claim value. This was an oversimplified explanation, refer to the SP2010 claims whitepaper for complete details.

Side note: The claim type encoding logic is important when using a custom claims provider (CCP) to augment the claimset with your own custom claim types. If you have CCPs on multiple SharePoint farms, it’s very important that you deploy all the CCPs in the same sequence across farms to ensure that the claim type mapping and encoding character for a custom claim type becomes the same across all farms. The best approach for ensuring parity across farms might be using Powershell to add the claim mappings first.

The code really doesn't resolve the user against ADFS, it just creates a claim type instance that authorization later on can validate against the logged in user's token (claim set). If the user's token contains the correct value for the claim type assigned to the securable object, then the user is authorized according to the role assignment.

Outlook recipient autocomplete, AD contacts sharing SMTP address

We have implemented an extension to MSCRM that replicates accounts and contacts to Active Directory as AD contacts for use as an Outlook address book (see this post). It is not very common, but some accounts and contacts do actually share the same SMTP mail address. E.g. several departments of a large corporation might share the same mailbox, and your user expects to find each department in the MSCRM address book in Outlook as separate contacts.

A few days ago, the operations department at one of our customers raised a concern about the support in Exchange Server 2003 for resolving mail recipients against the set of AD contacts when some of them share the same SMTP address. "This is not allowed in AD, and you should fix your code to not add the same SMTP address multiple times to AD" he stated. What ? As "proof" he sent me a screenshot of an error he got when trying to add a new contact with a duplicate SMTP address using the AD Users and Computers admin tool:

This e-mail address already exists in this organization (c10312e7)

This is a known limitation in the Active Directory Users and Computers snap-in. To resolve this, do not create the contact with AD Users and Computers, but use the ADSI-edit MMC snap-in or the ADSI component from your code to add multiple AD contacts that must share the same SMTP address. The AD attributes to set are "mail", "mailNickname", "proxyAddress", and "targetAddress". Those fields can be modified to enter a SMTP address that exist for other contacts without error (more info at MSDN).

Note that you should not create Exchange mailboxes for any external AD contact, and never create AD contacts duplicating the mail addresses of internal users. This will lead to NDR errors during Exchange resolving and mail delivery. Also ensure that you use correct settings for the Exchange Recipient Update Service (RUS) on your AD contacts, as RUS will automatically create Exchange entries for your AD contacts when not disabled as applicable. Read more in this post.

A related problem to Exchange NDRs for unreliable AD contact data, is the Outlook 2003 recipient cache. This autocomplete mechanism is heavensent when adding recipients that you send e-mail to often, but it is a p.i.t.a when your favorite recipients change their e-mail address or their data change in AD (the source of Outlook contact groups, a.k.a Exchange Global Address Lists). E.g. removing contacts from AD might lead to errors when Exchange tries to resolve cached recipients.

It is not easy to change the cached recipients list in Outlook using Outlook. The only option you have is to delete cached entries, but not many users have been able to find where to do that or clear the cache. It is actually quite simple, but not very intuitive; just select the entry you want to remove in the autocomplete suggestion list and press 'Delete' on your keyboard. The whole list of cached recipients is stored in a .NK2 file in your application data folder. Delete this file to clear the cache completely.

Read this article at Outlook Exchange for more info about the Outlook recipient autocomplete cache (Outlook .NK2 file). Also check out the Ingressor NK2 Management tool for editing and managing the .NK2 file.

MSCRM 3.0 in a multi AD forest infrastructure

MSCRM 3.0 by default supports a single AD domain (really a single AD forest) and a single Exchange 'organization'. The full spectrum of MSCRM functionality will be available to your users when your infrastructure adheres to these requirements. I will call the AD domain into which you install MSCRM, SQL Server 2000/2005 and SRS, the native domain. The same term is used for the Exchange organization of the native AD domain.

These are most likely infrastructure challenges you will encounter outside the native domain:

• Outlook desktop client: access to MSCRM platform services
• Outlook laptop client: desktop + go offline and online
• Exchange: Routing of incoming e-mails to MSCRM users and queues
• SQL Server Reporting Services: access to SRS services for reporting

First an overview of what sould work and what should not, dependent on some 'unsupported' infrastructure scenarios:

If you have multiple AD forests without explicit trusts, then the users not in the native domain will get only basic MSCRM functionality; the web client over HTTPS with basic authentication. These users will not be able to use neither the online nor the offline Outlook client (MSCRM desktop / laptop client) as they are not logged on to the domain. Note that such users will not get full reporting functionality with SQL Server Reporting Services (SRS) in this scenario.

If you have multiple Exhange organizations without explicit trusts; then the users not in the native organization (forest) will get only basic send e-mail functionality, the 'e-mail router' will not be able to automatically route incoming e-mails as the mailboxes are not in the native organization. In addition, the router cannot access the native AD domain when not explicitly trusted.

If you have users in an NT4 domain with a one-way trust from the native domain, these users will be able to use both the web and desktop client. They will not be able to use the laptop client as they cannot go off/online, incoming e-mail will not be routed to them, and they will not get full reporting functionality.

Then an overview of how can a multi AD forest and Exchange organization be configured to support full MSCRM functionality:

First of all, forget getting full MSCRM functionality for NT4 domains. Microsoft does not support NT4 anymore, so your're on your own.

The good news is that your users across several AD forests will be able to get the full spectrum of functionality available in MSCRM 3.0. This will just require some configuration of your infrastructure.

The most important aspect is that you have to add at least one-way trusts from the native MSCRM domain to the other domains. Trusting requires a LAN, WAN, or VPN connection between your domains. Support for full MSCRM over plain HTTPS is not possible.

The MSCRM Outlook client requires Windows Authentication / Kerberos against the native AD domain and usage of the default security credentials on the client PC. Thus, by adding one-way trusts, your users will be able to use both the MSCRM desktop client and the laptop client. Sending e-mail will of course work, while routing of incoming e-mails to users and queues will require some more configuration (see below).

Note that basic SRS reporting functionality will be available with just one-way trusts. For full reporting functionality, two-way trusts are needed between the AD forests. Alternatively, you need to configure a fixed identity on the clients for accessing the SRS reports (KB article to be published).

MSCRM 3.0 now supports having multiple Exchange servers in your native Exchange 'organization', including Exchange clusters. It is no longer required that you have a single Exchange server handling all incoming internet e-mails for your 'organization', as the functionality of the MSCRM e-mail router has changed in v3.0.

The v1.2 router had this limitation, which made it impossible to have one common MSCRM database in a company with multiple Exchange organizations (mail domains). E.g. I work in a company with several daughter companies and thus mail domains (itera.no, objectware.no, gazette.no, etc). This meant that with v1.2 we could not get full mail functionality in MSCRM. With MSCRM 3.0, we finally can.

The router no longer inspects all incoming mail messages, but rather a specific MSCRM mailbox.

The inspection of all incoming mails have been replaced by Exchange rules that must be deployed to each Exchange server that contain one or more mailboxes of MSCRM users and queues.
Click to enlarge figure

The Exchange rules, the MSCRM mailbox and the E-mail Router by default require mailboxes to be in the native domain and native 'organization', as the router must be able to access the MSCRM platform services to do its work.

You can deploy the mail routing rules and components to other Exchange organizations, provided that you configure the routing service to use an identity that has access to the MSCRM platform services. This will of course require that you have at least a one-way trust between the AD domains.

MSCRM: issues with one-way trusts between domains

At one of our customer we had to setup a new Active Directory domain for MSCRM 1.2 as their existing domain was NT4. Thus, all the users and their mailboxes stayed in the NT4 domain, while MSCRM and SQL Server were installed in the new AD domain. This deployment is "supported" by Microsoft, but beware of the small print and ommisions.

First of all, "go offline" in Sales for Outlook (SFO) does not work when the users recide in a trusted NT4 domain. We never got to test "go online" for obvious reasons. This might be due to v1.2 using SQL Replication, which in v3.0 has been replaced by the good, old BCP tool. Note that v3.0 still uses MSDE as the offline database and not SQL Express. Both SQL Server 2000 and 2005 are supported by MSCRM 3.0 as the master database.

Then the famous "E-mail Router": setting up routing of incoming e-mails as shown in the implementation guide works, sort of. Install a new Exchange Server in the AD domain and use either a CRM subdomain or forwarding of non-CRM e-mails to the original Exchange Server. Beware of the small print, however! Only e-mails to mailboxes registered in the native AD domain of MSCRM will be processed by the router. Thus, mails to a user will not be routed, even when a reply to a MSCRM e-mail, as they are in the NT4 domain. The only AD mailboxes we had were for queues (support@myco.com, etc), and routing of incoming e-mails to these queues works like a breeze.

We are currently deploying MSCRM 3.0 in a simmilar scenario, this time with five customer divisions, each with its own AD domain that are not within a single, common AD forest. Each domain (customer division) has its own Exchange server. I will post our experiences on the limitations with this infrastructure later on.

Note that an Exchange 'organization' cannot span AD forests, and that MSCRM is limited to one Exchange 'organization'. This restricts MSCRM with full Exchange e-mail functionality to a single AD forest.

Outlook recipients: AD contact postal address data

For our MSCRM customers we have made a small service that monitors the MSCRM database for changes to accounts and contacts, and maintains a specific Active Directory (AD) container that contains AD contacts that shadow the MSCRM accounts and contacts and their e-mail addresses. All these AD contacts are made available to Outlook through a new address book entry configured in Exchange System Manager. This ensures that all users have access to the e-mail information stored in MSCRM, even those that do not use Sales For Outlook. They can use the Outlook address book to pick or search for e-mail addresses and see details about an e-mail address such as contact name, company, postal address, etc.

The AD container is also used to hold AD distribution lists (mailing lists) built from the AD contacts. These distribution lists are also made available to Outlook through Exchange 2003. In addition, we have made a .NET add-in for Outlook that allows the users to preview the members of a list before sending e.g. the weekly newsletter e-mail (using an add-in toolbar in the e-mail Inspector window). The add-in allows the users to see the extra information stored in AD, and remove those members that should not get a mail this time. The users (ship brokers) typically decide that the e-mail should be sent to all members except those in Greece, and use the add-in to sort by country, multi-select the applicable members in a WinForms checkbox ListView and finally remove the selected members. This 'explodes' the mailing list into recipients, in addition to moving them to BCC to ensure that recipients do not see who else got this e-mail.

The code that maintains the AD contacts and the AD distribution lists runs on the Exchange Server 2003 to be able to modify data in both AD and Exchange (details here). Adding and maintaining AD contacts with .NET C# is quite easy, there are just a few pitfalls to be aware of when using AD contacts in combination with Exchange (details here).

The users recently requested the possibility to see the country of the MSCRM account/contact in the Outlook address book and when removing distribution list members. "No problem", I responded quickly and started checking postal address fields in Outlook and AD. I added full address information for an AD contact I knew was in the Outlook address book, waited for the RUS, and found the entered data in Outlook. I was ready to start coding !

First I added a 'country' column to my list view and then inspected the AddressEntry properties for access to postal address data. Unfortunately, Microsoft chose not to expose these properties in the Outlook object model. Fortunately, we were already using Redemption for other purposes in our add-in:

Redemption.MAPIUtils mapiUtils = new Redemption.MAPIUtils();
const int PR_COUNTRY = 0x3A26001E;
const int PR_EMAIL = 0x39FE001E;
foreach (Outlook.AddressEntry entry in list.Members)
{
string country = "";
object mapiCountry = mapiUtils.HrGetOneProp(entry.MAPIOBJECT, PR_COUNTRY);
if (mapiCountry != null) country = mapiCountry.ToString();

string smtp = entry.Address;
//check if Exchange address
if (smtp.StartsWith("/o="))
{
//get SMTP address from MAPI
smtp = mapiUtils.HrGetOneProp(entry.MAPIOBJECT, PR_EMAIL).ToString();
}

//add to listview
string[] items = new string[] { entry.Name, smtp, country };
ListViewItem itemX = new ListViewItem(items);
//keep name and address for later matching against recipients
itemX.Text = entry.Name;
itemX.Tag = entry.Address;
lstRecipients.Items.Add(itemX);
}
mapiUtils.Cleanup();

You will find a list of relevant MAPI property keys at OutlookCode.com.

Then I modified our AD updater service to read the country of each MSCRM account/contact to be able to set it on each AD contact. I used the ADSIEdit MMC snap-in to inspect the properties of my 'guinea pig' AD contact to see which property was used for storing the country. The 'co' property contained the country name, and the property 'countryCode' seemed to have something to do with a contact's country. Some googling lead me to MSDN, which revealed that these three properties must be set according to ISO 3166:

1. c: two letter country designation
2. co: country name
3. countryCode: country code (integer)

You will find the ISO 3166 list here. Copy and save the list to a .TXT file, open it in Excel as a 'fixed width' file to convert the text into a useful .XLS table. Use SQL Server 2000 DTS to import the .XLS to a new table in your database, enabling you to lookup MSCRM country names in the ISO 3166 country list.

The code to set MSCRM data into an AD contact properties looks like this:

//set properties
adContact.Properties["DisplayName"].Value = displayName;
adContact.Properties["mail"].Value = mailAddress;
//NOTE: AD fails on empty string, "invalid attribute syntax"
if(firstName.Length!=0) adContact.Properties["givenName"].Value = firstName;
if(lastName.Length!=0) adContact.Properties["sn"].Value = lastName;
if(company.Length!=0) adContact.Properties["company"].Value = company;
if(department.Length!=0) adContact.Properties["department"].Value = department;

if(country.IndexOf(";")>0)
{
//format: name;code;number
country += ";;;"; //just to be sure
string[] parts = country.Split(new char[]{';'});
string countryName = parts[0];
string countryA2 = parts[1];
string countryCode = parts[2];
if(countryName.Length!=0) adContact.Properties["co"].Value = countryName;
if(countryA2.Length!=0) adContact.Properties["c"].Value = countryA2;
if(countryCode.Length!=0) adContact.Properties["countryCode"].Value = Convert.ToInt32(countryCode);
}

// Flush to the directory
adContact.CommitChanges();

Note that I chose to require the country information to be provided as a semicolon separated string just for "simplicity" as I have not used entity objects (data xfer objects) in my AD service. I think refactoring of my service interface is needed, soon the users will want "just one more field" to be added...

Programming tip: use a reverse for-loop (i--) when removing entries from the Outlook.MailItem.Recipients collection, because the collection changes when a recipient is removed and this messes up the iteration if not performed end-to-beginning.

The Exchange RUS and external AD contacts

One thing that the Exchange 2003 SDK does not warn you about when adding mail enabled external Active Directory contacts is the Exchange Recipient Update Service (RUS), and this will cause strange effects when Exchange tries to resolve mail addresses. I.e. you will not be able to send e-mail to external mail contacts. Note that these effects will not happen immediately due to the periodical processing schedule of the RUS, thus you can test and believe that your code is correct, only to get hit in the face a few hours later.

The problem is that even if you set only a SMTP mail address on the new AD contact as shown in the SDK, the RUS will apply all the other types of mail addresses defined in your Exchange policies during its processing. This will give the external contact an internal Exhcange address, an X.400 address, etc. Thus, Exhcange will not be able to resolve or deliver e-mails to these contacts as most of these addresses will be bogus.

To correctly mail enable an external AD contact you need to set a GUID in two specific Exchange properties on the AD contact:

DirectoryEntry adContact = adContainter.Children.Find("CN=" + contactId, CONST.AD_CONTACT);

//exlude from RUS
adContact.Properties["msExchPoliciesExcluded"].Value = "{26491CFC-9E50-4857-861B-0CB8DF22B5D7}";
adContact.Properties["msExchPoliciesIncluded"].Value = "{26491CFC-9E50-4857-861B-0CB8DF22B5D7}";

//set mail AD property to e-mail address
//must set again after exluding from RUS
adContact.Properties["mail"].Value = mailAddress;

// Write Exchange information to the directory.
adContact.CommitChanges();

The policy GUID must be set on both the Exchange RUS policies excluded list and the policies included list. The specified GUID is valid for Exchange Server 2003. Refer to my previous post for details about how to write code for Exchange Server 2003.

You should also ensure that you never add an internal user's mail address as an AD contact through code. If your code does not prevent this, it will prevent Exchange from delivering incoming e-mail to the user's mailbox as it cannot resolve the mailbox when both an AD user and an AD contact has the same address.

Note that you should not exclude mail enabled AD distribution group (mailing lists) from the RUS, as this will cause the mailing lists to stop working. They must be processed by the RUS to function properly.

Outlook .NET add-ins connected to Exchange services

In some solutions we implement Outlook COM add-ins with .NET using the templates provided by Microsoft on MSDN. We typically add some menus, toolbars and dialog boxes that allow the users to manage stuff in their mailboxes or public folders, using client-side CDO. To avoid the Outlook security popup we utilize the Redemption component, which is widely recommended on the 'net for this purpose.

In addition, we sometimes need to access extended MAPI properties that are not exposed in CDO, and Redemption gives us easy access to these properties and other stuff that is not supported by client-side CDO. Redemption is a component that is really useful, so is the OutlookSpy tool. Recommended!

In an earlier posting, I wrote about how we prefer to implement code for managing ActiveDirectory and Exchange and its mailbox and public folder stores as server-side services. These services uses e.g. CDOEX and CDOEXM to perform the logic provided by our services.

One of our service methods is to copy a mail message from a mailbox store to a public folder store in response to an action initiated by an Outlook user through our COM add-in. This is not easy to do using CDO because it involves separate stores and also involves having sufficient permissions on the stores. This operation is, however, quite easy to implement using Exchange OleDB on the server-side and impersonating a specific identity that has the required rights. The ExOleDB methods for operating on different items in the store requires either the HTTP: or the File: URL Scheme, of which Microsoft recommends the HTTP: scheme.

The HTTP: URL scheme for mail messages uses the .EML name of the message item, and this URL is easy to see when using Outlook Web Access (OWA). This URL is available only through extended MAPI, thus we use Redemption to get it from the message item:

//get MAPI IMessage propery["PR_URL_COMP_NAME"]
int PR_URL_COMP_NAME = 0x10F3001E;
Redemption.SafeMailItem message = new Redemption.SafeMailItemClass();
message.Item = this._mailItem;
string url = message.get_Fields(PR_URL_COMP_NAME).ToString();

The string returned by Redemption is just the 'name' component of the URL (i.e. the message item subject). It must be encoded before passing it to the Exchange service for processing by ExOleDB:

//removed code from sample:
//handle that Redemption returns '?' for '/'
//and other quircks with the char encoding

//encode message item URL
url = url.Replace("/", "_xF8FF_");
url = Util.MailUrlEncode(url);
//clean up
Redemption.MAPIUtils util = new Redemption.MAPIUtilsClass();
util.Cleanup();

Note that not all characters in the URL needs to be encoded, and I have some small utility functions to handle the encoding:

public static string MailUrlEncode(string input)
{
string output="";
char[] uniChars = input.ToCharArray();
foreach(char uniChar in uniChars)
{
if(uniChar <= 'z') output += uniChar.ToString(); else output += Util.ExchangeEncode(uniChar.ToString());
}
return output;
}

public static string ExchangeEncode(string uniChar)
{
string output="";
byte[] encoded = System.Text.Encoding.UTF8.GetBytes(uniChar);
foreach(byte utf8Char in encoded)
{
output += "%" + utf8Char.ToString("X2");
}
return output;
}

Finally, the URL must be prepended with the path to the message item store (mailbox or public folder) before it becomes a complete URL that ExOleDB can operate on.

The server-side ExOleDB code for copying the message item and its attachment is not included here. I might publish a posting on this later on.

Get AD objects by GUID using System.DirectoryServices

All objects in ActiveDirectory are uniquely identified by their distinguishedName or objectGuid attributes. One of these identifiers are typically used to store references to an AD object and to retrieve it. The strenght of the GUID is that it stays the same even when an object is moved and thus gets a new distinguishedName. We typically stores an AD object GUID in a SQL Server database in the standard .NET GUID format (DirectoryEntry.Guid) in uniqueidentifier columns.

Searching for an object by a standard .NET GUID using the DirectorySearcher class is not as straightforward as you may think. A GUID must be formatted as an "octet string" (ActiveDirectory native format) to be able to use it in a DirectorySearcher filter as a filter clause. I have used this function to convert a standard GUID string to an octet string:

public static string Guid2OctetString(string objectGuid)
{
System.Guid guid = new Guid(objectGuid);
byte[] byteGuid = guid.ToByteArray();
string queryGuid = "";
foreach(byte b in byteGuid)
{
queryGuid += @"\" + b.ToString("x2");
}
return queryGuid;
}

Note that if you have a DirectoryEntry object instead of just a standard GUID string, then the .NativeGuid will give you the octet string.

All AD objects have a specific class such as "user", "contact" or "group", which helps on narrowing the number of entries to search through. Always apply an objectClass filter in your AD searches. With the GUID, octet string method and the objectClass at hand, looking up the AD object is done like this:

. . .
string adPath = "LDAP://DC=myco,DC=local"
DirectoryEntry adRoot = new DirectoryEntry(adPath);
Object adsiObj = adRoot.NativeObject; //Bind to the native AdsObject to force authentication

DirectorySearcher adSearch = new DirectorySearcher(adRoot);
string queryGuid = Guid2OctetString(objectGuid);
adSearch.Filter = "(&(objectClass=" + objectClass + ")(objectGUID=" + queryGuid + "))";

SearchResult adResults = adSearch.FindOne();
adObject = adResults.GetDirectoryEntry();
. . .

Applying a GUID filter clause to the objectGUID attribute in this example is just for illustration purposes.

To lookup the AD object identified by the GUID simply use the octet string (.NativeGuid) in the DirectoryEntry constructor (see Binding Using GUID in the DirectoryService SDK):

DirectoryEntry adObject= new DirectoryEntry("LDAP://<" + "GUID=" + nativeGuid + ">")

Please let me know if the .NET Guid .ToString() has a format argument I have missed that produces the octet string format.

Using SQL to query ActiveDirectory

Finding information in ActiveDirectory using the standard Windows 2003 Server tools is not as easy as database programmers would prefer, like using SQL select statements to read data from tables.

If you have SQL Server 2000 you can use AD as a linked server, and then apply all your SQL knowledge to query AD using SQL select statements with where, order by, group by and having clauses.

SQL Server 2000 makes AD accessible as a SQL view against the linked server using the ADSI OleDB provider.

MSDN describes how to configure and use this distributed query mechanism. Use the ADSIEdit MMC snap-in to explore the attributes for different AD objects.

Programming Exchange 2003 using .NET

We have several projects where we need to manage e.g. Exchange public folders and distribution lists from our own code, such as SPS web parts (really ASP.NET user controls hosted in a web part such as SmartPart). There are 21 different technologies available from Microsoft for programming Exchange 2003 (see official list), thus a choice must be made, also with regards to selected API being supported by .NET (see supported list).

We do not want to use the common, but not good, approach to install Outlook on the web server or on the Exchange Server to use CDO to implement our logic. Outlook is a client tier application, and CDO belongs on the client tier. The same applies to all Office applications, just try to use Word as an automation server in a web solution to produce documents from templates. Very bad application design indeed.

After a bit of experimenting with a selection of the 21 APIs, we now use these APIs against Exchange 2003 and ActiveDirectory:

• CDOEXM
• ADSI
• System.DirectoryServices (the only native .NET component, the others are COM)
• CDOEX
• Exchange OleDB (ExOleDB)

You might wonder why we use the ActiveDirectory components (ADSI and DirectoryServices). Several of the entities that an Outlook user thinks of as part of Outlook or Exchange, are actually spit into two different storages: Exchange for mailboxes and public folders, and ActiveDirectory for contacts, distribution groups and more. AD also handles the authorization for the different objects. To be able to make code that e.g. creates a mail enabled contact (AD contact, remember), you need to use both AD and Exchange components. I recommend checking out the ADSIEdit MMC snap-in to explore the structure of AD and Exchange.

Note that CDOEXM/CDOEX must run on the Exchange server, or on a computer where the Exchange Administrative Tools have been installed (and in the same Active Directory organization). The Exchange server by default has CDOEXM/CDOEX installed (note that just copying and registering the DLLs will not work). We have chosen the former, because the latter have a long list of prerequsites for the target computer, thus using the Exchange server as the location of our .NET components is simply the easiest approach.

In order to call the CDOEXM code on the Exchange server from your WebForm or WinForm application, you can use .NET Remoting or .NET web services. We have chosen the latter, as it is simply the easiest way to implement a distributed solution for intra-system "remote procedure calls". This approach with a specific service for all our Exchange and AD logic also have several other benefits, such as making it easy to set the identity used to access AD, mailboxes, etc. Use the above mentioned ADSIEdit tool to view and set permissions for AD and Exchange objects.

Always remember that the Exchange domain (myco.com) might not be the same as your AD domain (myco.local), as this might cause you some grief when trying to understand and use the rather poor code samples provided on MSDN.

Warning: If you try to use CDOEXM code on a computer that do not have the Exchange Administrative Tools properly installed, you will at run-time get a cast exception (InvalidCastException, "specified cast is not valid") when trying to use .NativeObject to get a CDOEXM object from a .NET AD object, e.g. when your code must call MailEnable() on an AD user, AD contact or AD distribution group.

A final warning: Watch out for the Exchange RUS when mail enabling external contacts, more on this in a later posting.