Sunday, December 9, 2018

Exchange 2010 > 2016 migration - Part 8 (uninstall Exchange 2010)

In this blog post, I will remove an Exchange 2010 server from a DAG (Database Availability Group) and then uninstall Exchange from the server. The server being removed is EX13-2. In many examples, you will see the roles being moved to server EX13-1.


DAG operations

We can use either the EMC or the EMS. I will use the latter option.

First, we have to ensure that the server being removed does not hold the active copy of the databases and is not the PAM (Primary Active Manager) of the DAG. In my case, this is server EX13-2. We can verify that with the following command:

Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus


Note: the command Get-MailboxDatabase alone does not show passive copies. The active copy of the database is the "mounted" copy and the passive copy is the "healthy" copy (if everything is normal).

Note: you can click on the images to enlarge.


We can view the PAM with this command:

Get-DatabaseAvailabilityGroup -status | format-list *primary*


Note: the command is abbreviated in the screenshot.


If necessary, we can move the active copy of the mailbox database with this command:

Move-ActiveMailboxDatabase DB01 -ActivateOnServer EX1

Note: DB01 and EX1 are simple examples.


As for the PAM, we can place the server in DAG maintenance mode to move the role to another server (and remove it from maintenance mode immediately thereafter if needed).


There are other methods that vary depending on the OS version. With Server 2012 and later, we can use a PowerShell cmdlet, for example:

Move-ClusterGroup "Cluster Group" -Node EX13-1

This command does not work with 2008/R2. For this OS, I was able to use the following command instead:

cluster dag1.mynet.lan group "Cluster Group" /moveto:EX13-2


We should also make sure that the server that will host the active copy of the mailbox is not in maintenance mode (although we would know soon enough from the inevitable error message if it was):


The status for the activation policy should be "unrestricted" rather than "blocked". In addition, the command below should show that no servers are in maintenance mode:




After these verifications, we can remove the server from the DAG.




Step 1: remove the mailbox database copy from the DAG

We remove the copy of the mailbox database with this command:

Remove-MailboxDatabaseCopy DB1\EX13-2




If we need to identify the name of the mailbox database copy, we must use this cmdlet (already mentioned earlier):

Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus



Note: there is not a corresponding "Get-MailboxDatabaseCopy" cmdlet.




Step 2: remove the server from the DAG

Next, we remove the server from the DAG with this command:

Remove-DatabaseAvailabilityGroupServer DAG1 -MailboxServer EX13-2




Note: DAG1 is the name of the DAG in my test environment.


Once again, if we want to verify the name of the servers in the DAG, we have to use this command:

Get-DatabaseAvailabilityGroup



There is NOT a Get-DatabaseAvailabilityGroupServer cmdlet (in Exchange 2010).

Note: if the absence of operational servers surprises you, remember that you have to add the -status parameter to see certain details:

Optionally, we can remove EX13-1 from the DAG and then the DAG itself at this time (or wait until later).




Step 3: remove the DAG

First, we have to remove the last server (EX13-1). The command is the same as above...

Remove-DatabaseAvailabilityGroupServer DAG1 -MailboxServer EX13-2


But if you look quickly, there is also a message indicating the cluster has been removed because EX13-1 is the last server of the DAG:



Note: you can click on the images to enlarge.


If we view the properties of the DAG now, we see a DAG with no servers:



We remove the DAG itself with this command:

Remove-DatabaseAvailabilityGroup DAG1


We can confirm the removal of the DAG with the command:

Get-DatabaseAvailabilityGroup

There should be no results (see screenshot directly above).




Other uninstall prerequisites


Source server for Send Connectors

We can remove the server as a source server for the "Send Connectors" with this command:

Set-SendConnector MYNET-SendConn-1 -SourceTransportServers EX13-1

Because of the way the command functions, if we indicate only the server(s) we want to retain, the other (unmentioned) server is automatically eliminated.




We also need to verify that the server is not that one responsible for the generation of the Offline Address Book (OAB). If that is the case, we can move the OAB generation server with this command:

Move-OfflineAddressBook "Default Offline Address Book" -Server EX13-1



Note: since the server hosts no more databases, it logically cannot hold mailboxes and there is no need to move system mailboxes as we might have to do in other scenarios.



Uninstall Exchange


For the uninstall of Exchange itself, we can go to "Programs", select Exchange 2010 and then uncheck all the boxes below, including "Management Tools":




If all goes well, we should obtain these results:




***

I have concentrated on Exchange. We may also want to remove the server from the domain and disable monitoring of the system by management tools such as SCOM (or PowerAdmin) and any centrally managed antimalware solution. The removal of the server from such consoles exceeds the scope of this blog post.

As for removing the last Exchange 2010 server (EX13-1), we would do what we did above but moving any mailbox databases or other Exchange components to our Exchange 2016 server(s).

Tuesday, December 4, 2018

Exchange 2010 > 2016 migration - Part 7

Now that I've prepared my environment (please see previous blog posts of this series), I'll migrate mailboxes from Exchange 2010 to 2016. In fact, I have only one user mailbox that I had intended to migrate to Exchange Online. I'll use it to demonstrate the Exchange 2010 to 2016 mailbox migration procedure. I also have some system mailboxes to migrate.

We can use either the Exchange Admin Center (GUI) or the Exchange Management Shell (PowerShell command line interface) to migrate mailboxes. I'll demonstrate the use of the EAC for a user mailbox and then the EMS for the system mailboxes.


***


Mailbox migration with the EAC


We login to the EAC and go to the recipients tab and then select migration. We click on the + sign and select "Move to a different database":





We can use a .CSV file with a list of users or simply select them manually. I'll use this latter option. Once again, I click on a + sign to select the users (a single user in my case)...




And select them in the list before clicking add:





The user (test user Alan Reid) has now been selected. We click on Next to continue:





In the following step, we name what is called the "migration batch", decide if the archive mailbox will be moved with the primary mailbox (if applicable) and then select the target database (click on Browse):





I select and add the target database for the migration:






I did not select an archive database (this was not necessary). I can adjust the "bad item limit". If the number of bad items is exceeded, the migration will fail:





Now we can start the batch, configuring the report setting and then the schedule:




Once we click on new (in the screenshot above), the migration process starts and we can view the status. The duration of the migration can depend on a number of factors (mailbox size, network latency). You may have to click on the refresh icon from time to time:






If we click on view details (screenshot above), more information on the individual mailboxes is provided. Below, the status is "Completed" and we can see the number of items synced:





That's quite a few screenshots. The command line equivalent is much more concise. Here is one variation:

Get-Mailbox Alan.Reid | New-MoveRequest -TargetDatabase DB01




Mailbox migration with the EMS


Moving system mailboxes can be more challenging. First of all, some are not visible in the GUI (EAC). Second, the name can be difficult to type. Concerning the arbitration mailboxes, I proceeded as shown in the screenshots below.

We need to add the parameter -Arbitration to display these mailboxes. I also limited the output to mailboxes on the old Exchange 2010 server (EX13-1) using the where-object cmdlet (abbreviated below as "where"):



Next I pipeline the output to the cmdlet New-MoveRequest with the target database DB01. This avoids having to type the long alpha-numerical names:



Afterwards, I can re-run the first command to verify that no mailboxes remain on the old server (EX13-1).

Of course, we can use the command line to move simple user mailboxes as well. In addition to the example at the end of the previous section, we could do this:

New-MoveRequest Alan.Reid -TargetDatabase DB01

These mailboxes moved very quickly and the migration had already completed when I executed the cmdlet Get-Request. Consequently, I did not examine all the options of this cmdlet but for those interested, you can find more details here:

Get-Request

Wednesday, November 28, 2018

Exchange 2016 - HCW and MX records

Two blog posts ago, I used a new version of the Hybrid Configuration Wizard (HCW) to update the connection between my Exchange on-premises environment and Exchange Online (EXO). You may recall that the new version, downloaded through the Exchange 2016 Admin Center (EAC), updated a hybrid configuration first established some time ago with Exchange 2010.

Exchange 2016 - Hybrid Configuration Wizard

That post interrupted my series of blog posts about migrating from Exchange 2010 to Exchange 2016 but since I now had an Exchange 2016 server to work with, I could not resist trying a newer version of the HCW before finishing the migration. After running the new wizard, I wanted to verify something: does email sent to the Internet from a mailbox still on-premises pass through Exchange Online (and in particular the antimalware / antispam filters of Exchange Online Protection - not to mention DLP)?

I discovered that it does not. Even the new HCW does not adjust send connectors to make general outbound traffic transit though Exchange Online. Only email for users with mailboxes migrated to Exchange Online goes to Exchange Online (logical). So I wanted to examine how the send connectors are configured after running the HCW. And so much that I'm interrupting my series on the Exchange 2010 to 2016 migration once again. Mea culpa.


***


First of all, send connectors are organization wide and not specific to a particular server. I have two:

  • MYNET-SendConn-1 (original default send connector)
  • Outbound to Office 365 (send connector specific to Office 365 - configured with the HCW) 

When I run the HCW on EX1, I observe that only EX1 (my Exchange 2016 server) is a source server for the connector "Outbound to Office 365". It must have replaced EX13-1 (the 2010 server) since mail flow was functional before the addition of EX1 in the context of the migration project.

Note: we need at least one "source server" for a send connector.

Note: you are unfamiliar with an Exchange "send connector" and concepts such as "address spaces", "source servers" and "smart hosts", I would recommend consulting sources online (or a good Exchange book) since I am assuming knowledge of these concepts and do not intend to present or explain them here. I also suppose the reader would know how to read the header of an email and view all the technical details.


On the other hand, EX13-1 is the sole source server for "MYNET-SendConn-1". So I reiterate my question from the introduction above: do we even need this connector (and the associated smart host)? After we run the HCW, isn't all outbound mail flow supposed to transit via Office 365 and Exchange Online Protection (EOP) in particular?

Let's test this.

One of my test users sends an email to an external address from Outlook. The message arrives. But did the message transit via Exchange Online (because of the hybrid configuration) or did it use the MYNET connector which in turn uses a No-IP smart host?

Note: No-IP is a vendor that provides smart host services (among others).

Once the message arrives, I look at the properties in the email header and notice it arrived from IP adress 8.23.224.60. A quick look at the ARIN registry shows this IP address belongs to "Vitalwerks Internet Solutions". I happen to know Vitalworks owns No-IP but even if I did not, the address indicated for reporting email abuse is quite clear: abuse (at) no-ip.com.

Note: we can query the ARIN database different ways. I happen to use MXTOOLBOX:

https://mxtoolbox.com/arin.aspx

So despite the successful completion of the HCW (and even the most recent version at this time), it does not appear that outbound email transits via EOP.

Rather, we have (simplified):

EX13-1 -> No-IP (Postfix) -> mx.google.com

Because our MX records point to EXO (or EOP), we would expect the response to transit through O365 and that is the case:

mail-lj1-f175.google.com -> BY2NAM01FT009.mail.protection.outlook.com -> EX1 -> EX13-1

Reminder: the on-premises mailbox is still hosted on the Exchange 2010 server.


We can also determine the path taken by posting the header information in the Microsoft Remote Connectivity Analyzer which will provide us with a more organized view:

https://testconnectivity.microsoft.com/

Message Analyzer tab:




On the other hand, I confirm that messages for recpients with mailboxes migrated to EXO do transit through the hybrid connection:

EX13-1 -> EX1 -> SN1NAM01FT008.mail.protection.outlook.com

So No-IP is not involved in the delivery of messages within the hybrid environment (as we would expect).

For the reply we have:

NAM04-CO1-obe.outbound.protection.outlook.com -> EX1 -> EX13-1



***


Let's see what happens if I disable "MYNET-SendConn-1".

My prediction: the outbound message will not leave because the "Outbound for Office 365" is scoped to handle only messages for... Office 365.

And that's what happens.

I disable the connector:



I send a message (from Outlook - not illustrated) and the message is stuck in the queue:



When I re-enable the connector, the message leaves the queue:





I found a solution here:

Outbound Email Via Exchange Online Protection When Using Hybrid Exchange Online


We need to create another send connector with * for the address space but with the MX record provided by Microsoft for our tenant. The other parameters are shown below:

New-SendConnector -Name <DescriptiveName> -AddressSpaces * -CloudServicesMailEnabled $true -Fqdn <CertificateHostNameValue> -RequireTLS $true -DNSRoutingEnabled $false -SmartHosts <YourDomain_MX_Value> -TlsAuthLevel  CertificateValidation -Usage Internet


More information can be found here, with a script that is almost identical:

Set up connectors to route mail between Office 365 and your own email servers

Note: you can see your MX record (and other DNS records for your tenant) in the Microsoft 365 Admin Center. Click on the "Setup" tab and then select the domain for which you want to view the DNS records.

Note: Brian Reid tests with specific domains first and in a production environment that is what we should do. In my test environment, there is no risk of interrupting mail flow so I will use * for the address space from the start.


And it works!

I disable "MYNET-SendConn-1" once again and attempt to send a message after creating a new send connector with this command: 


The message arrives as expected, following this path:

EX13-1 -> EX1 -> BY2NAM01FT030.mail.protection.outlook.com ->
CY4PR0601CA0093.outlook.office365.com -> DM6PR06MB4778.namprd06.prod.outlook.com ->
NAM04-SN1-obe.outbound.protection.outlook.com (mail-eopbgr700060.outbound.protection.outlook.com) ->
mx.google.com


I leave "MYNET-SendConn-1" in a disabled state and optionally can remove it once I have confirmed (after so many days or weeks) that mail is indeed flowing correctly through Exchange Online.

Friday, November 23, 2018

Exchange 2010 > 2016 migration - Part 6

A migration from one version of Exchange to another (2010 to 2016 in this case) often involves more than the Exchange servers themselves. In a previous blog post, we adjusted load balancer settings so the load balancer would direct client requests to the new Exchange 2016 server rather than the old Exchange 2010 server.

In most networks, incoming mail (and possibly incoming client access requests) arrive at some sort of perimeter security device such as a firewall. Access rules and 1 to 1 NAT rules (or something comparable) allow access to the Exchange servers while ensuring a certain level of security. For example, in the context of a hybrid environment with Exchange Online, access to the internal services may be limited to certain IP ranges (those related to Exchange Online).

In this blog post, I will not demonstrate how to create these rules but rather how to modify existing rules so HTTPS and SMTP traffic would be redirected to the new Exchange server. I will use a Cisco ASA device in my case. Obviously, the exact process would be more or less different with some other appliance.


***


With the Cisco ASA (5505 - using ASDM), the Exchange server is represented by a "network object" with the IP address being one of the properties. I will not insist on accessing the management interface and assume that the person in charge of making these changes would know how to do this (and once again, your device may be completely different from mine, so I will concentrate on the general concepts rather than step-by-step details).

Once we open the ASDM, we can access the network objects here:




They can also be accessed from the firewall access rules section (also visible in the screenshot above). When we open the access rules section, we see this:





If I double-click on the object in the access rule (obj-10.0.0.23-smtp), an ellipsis (three dots) appears and if I click on it...



Another window opens with a similar representation of the firewall rules. If I highlight the rule I want to modify and click on "Edit"...




I can modify the properties of the network object. In this case, I would change the IP address and then for clarity, the name of the object itself, which references the IP address:







I repeat the operation for HTTPS and then save the modified configuration. In my case, I click on Apply (not shown in the screenshot) and then on the "Save" icon:



Note: we should see something about the running configuration being saved to flash.

Note: in my case, NAT uses the same network objects so the changes made to the network objects in the firewall section of the interface will also apply elsewhere (for NAT and for whatever else uses these network objects).


***


I will not insist on the details (since every device will be different) but reiterate that, in reality, an Exchange migration often requires changes to more than just the Exchange servers themselves. Depending on your environment, you may be responsible for making these changes or some other team will take care of them.

Wednesday, November 21, 2018

Exchange 2016 - Hybrid Configuration Wizard

In this blog post, I take a look at the latest hybrid configuration wizard for Exchange (using Exchange 2016 CU11). This is not directly related to the Exchange 2010 to 2016 migration but I had a reason to interrupt that series of blog posts so I could see the most recent HCW. I'll return to the migration in the following post.


***


I attempted to start the wizard two ways. Both failed.

I thought I could download the latest HCW here:

Introducing the Microsoft Office 365 Hybrid Configuration Wizard


The download itself went fine (see file above) but I could not execute it (we will see why in a moment).


So I tried from the EAC:



Still no luck : nothing happens.


I found the solution here:

Office 365 Hybrid Configuration Wizard won’t launch

Once I made Internet Explorer the default program for opening .application files (or a "manifest" in fact), the HCW finally opened.


When you configure the wizard from the EAC, you'll will first be prompted to log into your Office 365 account (as a global administrator):









Once logged in, you can install the application:







You will be prompted to confirm you really want to run the program:





If you click "Run" (above), the HCW finally starts:







If you have been reading my blog, you know that Exchange 2016 was recently installed and therefore is still using the 180 day evaluation license:



Automatic licensing of what people sometimes call the "hybrid server" is a feature of more recent versions of the HCW. With Exchange 2010, I was not able to automatically license the server with this role.

If I click on "License server now", the HCW starts to request a product key and almost immediately requests that I login to Office 365 again:





The installation of the license (or product key) continues:






At one point, it looks like the process has finished:




I verify that the Exchange 2016 server is indeed licensed:




If I remember correctly, I had to click back and next again to see the change in licensing:








Next, we provide both our on-premises administrator credentials (the account you used to login by default) and global administrator credentials for our tenant: 




The wizard attempts the connection (and succeeds)...





And then, we have to choose between a "Minimal Hybrid Configuration" and a "Full Hybrid Configuration". We can click on the "learn more" link if we are not sure what to choose. I select the "Full" option:




Note: with the most recent versions of the HCW, we now have the "Organization Configuration Transfer" (OCT) tool. This feature is not especially useful in my scenario so I will leave the box unchecked. You can "learn more" (link) if you think the tool might be useful for you.


We also have to make a choice concerning mail flow between Exchange Online (EXO) and Exchange on-premises. The default (or "typical") option is to establish direct communications between EXO and Exchange on-premises with a change in DNS MX records so incoming mail flows directly to EXO. For mailboxes still located on-premises, EXO redirects incoming messages to the local Exchange server(s). Outgoing mail also passes through EXO (where DLP policies can be applied).

I will keep the default option and simply mention that we could configure Edge Transport servers as the link with EXO or even maintain mail flow directly to the local site (the "centralized mail transport" option):





Then we select a server for the receive and send connectors (I just use my 2016 server "EX1"):






I select a certificate for securing mail transport between EXO and my on-premises server:




And then indicate a FQDN for the outbound mail connector (outbound from the Microsoft perspective):




In my case, I arrive at a page that says "Ready for Update", probably because I already have a hybrid Exchange environment. I imagine the wording would be different if we were running the wizard for the first time:



The wizard runs...






And complete successfully: