Sunday, August 28, 2016

Exchange 2010 (SP3) - Message Tracking - 03

After my first blog post on message tracking, I wanted to confirm the sequence of operations when one of "our" users (test user Alan Reid in this case) sends an email to an external recipient (a Gmail user).

The results confirmed the sequence in the first blog post.

In the scenario where we have two mail servers in a Database Availability Group, and when the mailbox database holding our user's mailbox (Alan Reid) is "active" on our first Exchange server (EX13-1), the following operations take place:
  • SUBMIT (EX13-1) : EX13-1 submits the message to EX13-2

  • RECEIVE (EX13-2) : EX13-2 receives the message from EX13-1
  • TRANSFER (EX13-2) : EX13-2 determines the route for the message (and performs content conversion).
  • SEND (EX13-2) : EX13-2 sends the message. We can obtain details of the session with the remote server from the protocol logs.

If I move the active copy of the mailbox database to EX13-2, the process is the exact opposite: EX13-2 submits the message to EX13-1 and the RECEIVE, TRANSFER and SEND operations take place on EX13-1.

The location of the DAG Primary Active Manager (PAM) does not seem to play a role.

I sent a total of 12 test messages, with the active copy of the mailbox database on EX13-1 for the first 3 tests, on EX13-2 for the next 3 tests, on EX13-1 for the following 3 tests and on EX13-2 for the last 3 tests. I see little usefulness in posting output or screenshots for each of these tests.


Now I want to send a message from Gmail to Alan Reid.

What entries does that produce in the Message Tracking logs ?

Before I proceed, I want to clarify the path taken by inbound messages.

Messages from Gmail first arrive at a hardware firewall that performs 1-to-1 NAT and forwards these messages to a Citrix NetScaler VPX load balancer. This device then forwards the messages to either of the two Exchange servers (EX13-1 or EX13-2).


Gmail -> [ CLOUD] -> perimeter firewall (1-to-1 NAT) -> NetScaler VPX -> Exchange servers


First Message (Gmail - INBD - 001)

It looks like the incoming message from Gmail passes through the Netscaler load-balancer ( and then is received by EX13-2.

Since the mailbox of recipient Alan Reid is active on EX13-1, EX13-2 passes the message to EX13-1 where it it is delivered.

Get-Messagetrackinglog -Server "EX13-2" -Start "8/27/2016 5:15:00 PM" -End "8/27/2016 5:30:00 PM" | fl

Note: you can click on the images to enlarge.

Note: I have edited out some of the parameters for simplicity and concision, especially parameters with no values.

It seems (since this is the first time I analyze this) that unlike the send operation to an external recipient with the following sequence of operations...


  • SEND

We have a simple Receive and Deliver combination (both happening on EX13-2):


But which server actually delivers the message?

Apparently, EX13-1 plays no role in the delivery of the message. In any case, no operations are recorded in the logs:

[PS] C:\>Get-Messagetrackinglog -Server "EX13-1" -Start "8/27/2016 5:15:00 PM" -End "8/27/2016 5:30:00 PM" | fl
[PS] C:\>

In the example above, there is no result for EX13-1 so it seems that EX13-2 peforms the delivery operation (even though the DELIVER operation entry lists EX13-1 for the ServerHostname).

Second Message (Gmail - INBD - 002)

Same thing for second message...

[PS] C:\>get-messagetrackinglog -Server "EX13-1" -Start "8/27/2016 5:45:00 PM" -End "8/27/2016 6:00:00 PM" | fl
[PS] C:\>
[PS] C:\>get-messagetrackinglog -Server "EX13-2" -Start "8/27/2016 5:45:00 PM" -End "8/27/2016 6:00:00 PM" | fl

Third Message (Gmail - INBD - 003)

Here it looks like EX13-1 receives the message (not EX13-2, contrary to the first two messages) but then sends it to EX13-2, even though the mailbox resides on EX13-1 (it is part of a single mailbox database that is now "active" on EX13-1):

[PS] C:\>get-messagetrackinglog -Server "EX13-1" -Start "8/27/2016 5:53:00 PM" -End "8/27/2016 5:55:00 PM" | fl

So, in this case, the sequence is:

  • SEND

And EX13-1 sends the message to EX13-2, only for it to deliver the message to the mailbox that is "active" on EX13-1 (in fact, it is the copy of mailbox database that is "active" on EX13-1):

Here (above) we see the same Receive - Deliver combination that we observed for the first two messages.

Fourth Message (Gmail - INBD - 004)

As for the 4th message, EX13-2 receives it from the NetScaler as for the Message 1 and 2 above, and then passes it to EX13-1 where it is delivered. Since the sequence is exactly the same, I will not waste space by posting screenshots that illustrate exactly the same thing.

The load balancer seems to play a role here:

It is configured to use "Round Robin" between the two Exchange servers and could deliver to either one, although the alternance between the two does not seem to be strict. One would have thought that 2 messages would go to EX13-1 and 2 messages to EX13-2 but in fact (or in this case) it is 3 messages for EX13-2 and only 1 for EX13-1.

I will not test moving the PAM. I doubt very much that this clustering component has any effect at all on the forwarding decisions of the load balancer or even on the inter-Exchange server routing.


I changed the active copy of the database from EX13-1 to EX13-2. For the next two test messages, I observed the same behavior.
  1. If the server that does not hold the active copy of the mailbox database receives the message directly from the load balancer, it delivers the message. This happened for the first of two test messages.
  2. If the server that DOES hold the active copy of the mailbox database receives the message directly from the load balancer, it sends it to its partner which in return delivers the message (to the mailbox currently "active" on the first server). This happened for the second of two test messages.
After some reflection, I have opted not to post the screenshots of the operations that took place on the servers after moving the active copy of the mailbox database, since it would simply be a repetition of the sequences examined earlier.

No comments:

Post a Comment