Wednesday, November 28, 2018

Exchange 2016 - HCW and MX records

Two blog posts ago, I used a new version of the Hybrid Configuration Wizard (HCW) to update the connection between my Exchange on-premises environment and Exchange Online (EXO). You may recall that the new version, downloaded through the Exchange 2016 Admin Center (EAC), updated a hybrid configuration first established some time ago with Exchange 2010.

Exchange 2016 - Hybrid Configuration Wizard

That post interrupted my series of blog posts about migrating from Exchange 2010 to Exchange 2016 but since I now had an Exchange 2016 server to work with, I could not resist trying a newer version of the HCW before finishing the migration. After running the new wizard, I wanted to verify something: does email sent to the Internet from a mailbox still on-premises pass through Exchange Online (and in particular the antimalware / antispam filters of Exchange Online Protection - not to mention DLP)?

I discovered that it does not. Even the new HCW does not adjust send connectors to make general outbound traffic transit though Exchange Online. Only email for users with mailboxes migrated to Exchange Online goes to Exchange Online (logical). So I wanted to examine how the send connectors are configured after running the HCW. And so much that I'm interrupting my series on the Exchange 2010 to 2016 migration once again. Mea culpa.


***


First of all, send connectors are organization wide and not specific to a particular server. I have two:

  • MYNET-SendConn-1 (original default send connector)
  • Outbound to Office 365 (send connector specific to Office 365 - configured with the HCW) 

When I run the HCW on EX1, I observe that only EX1 (my Exchange 2016 server) is a source server for the connector "Outbound to Office 365". It must have replaced EX13-1 (the 2010 server) since mail flow was functional before the addition of EX1 in the context of the migration project.

Note: we need at least one "source server" for a send connector.

Note: you are unfamiliar with an Exchange "send connector" and concepts such as "address spaces", "source servers" and "smart hosts", I would recommend consulting sources online (or a good Exchange book) since I am assuming knowledge of these concepts and do not intend to present or explain them here. I also suppose the reader would know how to read the header of an email and view all the technical details.


On the other hand, EX13-1 is the sole source server for "MYNET-SendConn-1". So I reiterate my question from the introduction above: do we even need this connector (and the associated smart host)? After we run the HCW, isn't all outbound mail flow supposed to transit via Office 365 and Exchange Online Protection (EOP) in particular?

Let's test this.

One of my test users sends an email to an external address from Outlook. The message arrives. But did the message transit via Exchange Online (because of the hybrid configuration) or did it use the MYNET connector which in turn uses a No-IP smart host?

Note: No-IP is a vendor that provides smart host services (among others).

Once the message arrives, I look at the properties in the email header and notice it arrived from IP adress 8.23.224.60. A quick look at the ARIN registry shows this IP address belongs to "Vitalwerks Internet Solutions". I happen to know Vitalworks owns No-IP but even if I did not, the address indicated for reporting email abuse is quite clear: abuse (at) no-ip.com.

Note: we can query the ARIN database different ways. I happen to use MXTOOLBOX:

https://mxtoolbox.com/arin.aspx

So despite the successful completion of the HCW (and even the most recent version at this time), it does not appear that outbound email transits via EOP.

Rather, we have (simplified):

EX13-1 -> No-IP (Postfix) -> mx.google.com

Because our MX records point to EXO (or EOP), we would expect the response to transit through O365 and that is the case:

mail-lj1-f175.google.com -> BY2NAM01FT009.mail.protection.outlook.com -> EX1 -> EX13-1

Reminder: the on-premises mailbox is still hosted on the Exchange 2010 server.


We can also determine the path taken by posting the header information in the Microsoft Remote Connectivity Analyzer which will provide us with a more organized view:

https://testconnectivity.microsoft.com/

Message Analyzer tab:




On the other hand, I confirm that messages for recpients with mailboxes migrated to EXO do transit through the hybrid connection:

EX13-1 -> EX1 -> SN1NAM01FT008.mail.protection.outlook.com

So No-IP is not involved in the delivery of messages within the hybrid environment (as we would expect).

For the reply we have:

NAM04-CO1-obe.outbound.protection.outlook.com -> EX1 -> EX13-1



***


Let's see what happens if I disable "MYNET-SendConn-1".

My prediction: the outbound message will not leave because the "Outbound for Office 365" is scoped to handle only messages for... Office 365.

And that's what happens.

I disable the connector:



I send a message (from Outlook - not illustrated) and the message is stuck in the queue:



When I re-enable the connector, the message leaves the queue:





I found a solution here:

Outbound Email Via Exchange Online Protection When Using Hybrid Exchange Online


We need to create another send connector with * for the address space but with the MX record provided by Microsoft for our tenant. The other parameters are shown below:

New-SendConnector -Name <DescriptiveName> -AddressSpaces * -CloudServicesMailEnabled $true -Fqdn <CertificateHostNameValue> -RequireTLS $true -DNSRoutingEnabled $false -SmartHosts <YourDomain_MX_Value> -TlsAuthLevel  CertificateValidation -Usage Internet


More information can be found here, with a script that is almost identical:

Set up connectors to route mail between Office 365 and your own email servers

Note: you can see your MX record (and other DNS records for your tenant) in the Microsoft 365 Admin Center. Click on the "Setup" tab and then select the domain for which you want to view the DNS records.

Note: Brian Reid tests with specific domains first and in a production environment that is what we should do. In my test environment, there is no risk of interrupting mail flow so I will use * for the address space from the start.


And it works!

I disable "MYNET-SendConn-1" once again and attempt to send a message after creating a new send connector with this command: 


The message arrives as expected, following this path:

EX13-1 -> EX1 -> BY2NAM01FT030.mail.protection.outlook.com ->
CY4PR0601CA0093.outlook.office365.com -> DM6PR06MB4778.namprd06.prod.outlook.com ->
NAM04-SN1-obe.outbound.protection.outlook.com (mail-eopbgr700060.outbound.protection.outlook.com) ->
mx.google.com


I leave "MYNET-SendConn-1" in a disabled state and optionally can remove it once I have confirmed (after so many days or weeks) that mail is indeed flowing correctly through Exchange Online.

Friday, November 23, 2018

Exchange 2010 > 2016 migration - Part 6

A migration from one version of Exchange to another (2010 to 2016 in this case) often involves more than the Exchange servers themselves. In a previous blog post, we adjusted load balancer settings so the load balancer would direct client requests to the new Exchange 2016 server rather than the old Exchange 2010 server.

In most networks, incoming mail (and possibly incoming client access requests) arrive at some sort of perimeter security device such as a firewall. Access rules and 1 to 1 NAT rules (or something comparable) allow access to the Exchange servers while ensuring a certain level of security. For example, in the context of a hybrid environment with Exchange Online, access to the internal services may be limited to certain IP ranges (those related to Exchange Online).

In this blog post, I will not demonstrate how to create these rules but rather how to modify existing rules so HTTPS and SMTP traffic would be redirected to the new Exchange server. I will use a Cisco ASA device in my case. Obviously, the exact process would be more or less different with some other appliance.


***


With the Cisco ASA (5505 - using ASDM), the Exchange server is represented by a "network object" with the IP address being one of the properties. I will not insist on accessing the management interface and assume that the person in charge of making these changes would know how to do this (and once again, your device may be completely different from mine, so I will concentrate on the general concepts rather than step-by-step details).

Once we open the ASDM, we can access the network objects here:




They can also be accessed from the firewall access rules section (also visible in the screenshot above). When we open the access rules section, we see this:





If I double-click on the object in the access rule (obj-10.0.0.23-smtp), an ellipsis (three dots) appears and if I click on it...



Another window opens with a similar representation of the firewall rules. If I highlight the rule I want to modify and click on "Edit"...




I can modify the properties of the network object. In this case, I would change the IP address and then for clarity, the name of the object itself, which references the IP address:







I repeat the operation for HTTPS and then save the modified configuration. In my case, I click on Apply (not shown in the screenshot) and then on the "Save" icon:



Note: we should see something about the running configuration being saved to flash.

Note: in my case, NAT uses the same network objects so the changes made to the network objects in the firewall section of the interface will also apply elsewhere (for NAT and for whatever else uses these network objects).


***


I will not insist on the details (since every device will be different) but reiterate that, in reality, an Exchange migration often requires changes to more than just the Exchange servers themselves. Depending on your environment, you may be responsible for making these changes or some other team will take care of them.

Wednesday, November 21, 2018

Exchange 2016 - Hybrid Configuration Wizard

In this blog post, I take a look at the latest hybrid configuration wizard for Exchange (using Exchange 2016 CU11). This is not directly related to the Exchange 2010 to 2016 migration but I had a reason to interrupt that series of blog posts so I could see the most recent HCW. I'll return to the migration in the following post.


***


I attempted to start the wizard two ways. Both failed.

I thought I could download the latest HCW here:

Introducing the Microsoft Office 365 Hybrid Configuration Wizard


The download itself went fine (see file above) but I could not execute it (we will see why in a moment).


So I tried from the EAC:



Still no luck : nothing happens.


I found the solution here:

Office 365 Hybrid Configuration Wizard won’t launch

Once I made Internet Explorer the default program for opening .application files (or a "manifest" in fact), the HCW finally opened.


When you configure the wizard from the EAC, you'll will first be prompted to log into your Office 365 account (as a global administrator):









Once logged in, you can install the application:







You will be prompted to confirm you really want to run the program:





If you click "Run" (above), the HCW finally starts:







If you have been reading my blog, you know that Exchange 2016 was recently installed and therefore is still using the 180 day evaluation license:



Automatic licensing of what people sometimes call the "hybrid server" is a feature of more recent versions of the HCW. With Exchange 2010, I was not able to automatically license the server with this role.

If I click on "License server now", the HCW starts to request a product key and almost immediately requests that I login to Office 365 again:





The installation of the license (or product key) continues:






At one point, it looks like the process has finished:




I verify that the Exchange 2016 server is indeed licensed:




If I remember correctly, I had to click back and next again to see the change in licensing:








Next, we provide both our on-premises administrator credentials (the account you used to login by default) and global administrator credentials for our tenant: 




The wizard attempts the connection (and succeeds)...





And then, we have to choose between a "Minimal Hybrid Configuration" and a "Full Hybrid Configuration". We can click on the "learn more" link if we are not sure what to choose. I select the "Full" option:




Note: with the most recent versions of the HCW, we now have the "Organization Configuration Transfer" (OCT) tool. This feature is not especially useful in my scenario so I will leave the box unchecked. You can "learn more" (link) if you think the tool might be useful for you.


We also have to make a choice concerning mail flow between Exchange Online (EXO) and Exchange on-premises. The default (or "typical") option is to establish direct communications between EXO and Exchange on-premises with a change in DNS MX records so incoming mail flows directly to EXO. For mailboxes still located on-premises, EXO redirects incoming messages to the local Exchange server(s). Outgoing mail also passes through EXO (where DLP policies can be applied).

I will keep the default option and simply mention that we could configure Edge Transport servers as the link with EXO or even maintain mail flow directly to the local site (the "centralized mail transport" option):





Then we select a server for the receive and send connectors (I just use my 2016 server "EX1"):






I select a certificate for securing mail transport between EXO and my on-premises server:




And then indicate a FQDN for the outbound mail connector (outbound from the Microsoft perspective):




In my case, I arrive at a page that says "Ready for Update", probably because I already have a hybrid Exchange environment. I imagine the wording would be different if we were running the wizard for the first time:



The wizard runs...






And complete successfully:





Saturday, November 17, 2018

Exchange 2010 > 2016 migration - Part 5

In Exchange 2016, all client access now takes place via HTTPS. Even with Outlook, RPC is no longer used. This means that we must configure three elements correctly:

  • Virtual directory URLs
  • Certificates
  • DNS

We do not necessarily have to proceed in that order. In my case (please see the previous blog posts), I configured the virtual directories first, then installed a third-party public certificate with names matching those in the virtual directory URLs and I will now reconfigure DNS. One of the primary rules (or guidelines) for Exchange coexistence is to direct client access requests to server(s) running the latest version of Exchange. At present, DNS is pointing to the Exchange 2010 server - or rather a load balancer that directs requests to this server.

Note: obviously a load balancer only really makes sense when there is more than one server. In the past, I had installed a second Exchange server to learn Database Availability Group (DAG) operations but this server no longer exists (in my test network servers "come and go" quite often). Even so, I still maintain the load balancer to simulate, as much as possible, a real network where this type of material would likely be present.

So instead of changing our DNS records, we have them still point to the load balancer but adjust the virtual server settings so client requests are directed to the new Exchange 2016 server. If you do not have a load balancer, you would make the appropriate changes in DNS itself. 


***


In our DNS, observe that the records for "mail" and "autodiscover" point to IP address:

10.0.0.38



This is a virtual IP (VIP) on our load balancer, configured for services like OWA and Exchange Web Services that use HTTPS on port 443. There is also a VIP for SMTP traffic (10.0.0.36) but it does not come into play here.

So DNS directs any requests for "autodiscover" or "mail" to VIP 10.0.0.38 on the load balancer and it is there that we must make the necessary changes.

In my case, the load balancer is a Citrix Netscaler VPX (the "Freemium" version). There are three sections to consider:

  • Virtual Servers
  • Services (or optionally Service Groups)
  • Servers

We need to create a new "server" to represent our Exchange 2016 server, EX1.

Note: of course, the exact procedure will vary depending of the brand and type of load balancer. I will illustrate the procedure for a Citrix VPX. I will also assume the person managing the load balancer would know how to access the management interface and go to the appropriate sections.


 In the Citrix VPX, we go to the "Servers" section of "Traffic Management" and select the option "Add":





We enter a name for the server and its IP address (the other parameters are optional). I use a naming convention where "srv_" is prefixed to the name of the server but this is not a technical requirement. You can choose whatever name makes sense in your environment:





Now we have two servers: EX13-1 (the old Exchange 2010 server) and EX1 (the new Exchange 2016 server):





Next, we create a service for HTTPS traffic (OWA, EWS and now Outlook for that matter) associated with the new server EX1:





Note that we select the server we created in the previous step and then the correct protocol (SSL for HTTPS traffic). The port is automatically populated:




We can do the same for other services (like SMTP) but I will not repeat the procedure for each service here. In any case, we now have a service for HTTPS traffic associated with each of our two Exchange servers:




In the end, we want to disable the services associated with the old server (and ultimately can delete them) because client access traffic should be directed to the Exchange server(s) running the latest version of Exchange. The screenshot above reflects that reality.

Note: we make this change before we migrate mailboxes and select a time (possibly outside business hours) when the changes can be made without disrupting normal operations.

Note: we do not need to direct SMTP traffic to the Exchange server(s) running the latest version of Exchange. However, if we are going to remove the older server(s), we would adjust load balancer settings for this reason.



Lastly, in the "Virtual Servers" section...




We do not need to add an entry here but we do need to bind the new HTTPS service for EX1 to the HTTPS virtual server. So we select that virtual server and click on "Edit". In the virtual server properties, we open the bindings section...




We click on "Add Binding"...




We click to select the service...




We select the service (svc_HTTPS_EX1)...




And bind the service to the virtual server:





We can then close the section and return to the virtual server properties.


Attention! Remember to save the changes to the VPX configuration!




***


We can now test client access with a user that has a mailbox hosted on the Exchange 2010 server. They should be able to open Outlook and access their messages and their calendar without errors (even though they transit through the Exchange 2016 server).

Once the Exchange 2010 server is removed from service, we can remove any related entries from the load balancer properties shown above.