Thursday, April 28, 2016

NetScaler VPX - load balance Exchange - Part 6 (SSL Offloading - Security Risks)

In my previous blog post, I enabled SSL Offloading which has one remarkable disadvantage: traffic between the NetScaler SNIP (subnet address of the NetScaler) and the Exchange server(s) is not encrypted. How much of a risk does this represent? It all depends on your security requirements. Some organizations may consider (accurately or not) that their internal network is secure and that unencrypted data is not a significant risk. Other organizations may have policies that do not allow data to be decrypted elsewhere than at the two endpoints exchanging the data.

But what exactly could we "see" if the traffic (for Outlook Web App in this case) is unencrypted? Usernames and especially passwords in clear text would be the worst scenario. But could we see anything else?

I attempted to respond to this question by capturing packets between the NetScaler and Exchange with WireShark.


I installed WireShark 2.0.2 on the Exchange server and created what is known as a "capture filter" so only data passing between the NetScaler SNIP and the Exchange server would be collected.

I will not present the use of WireShark in detail but here is the text I entered for the capture filter:


This is the "SNIP" (subnet IP) of the NetScaler VPX.

Note: for those not familiar with the basic use of WireShark (how to start and stop a capture, how to create capture and display filters, how to search for various elements (strings for example) in captured packets, I would recommend you consult one of the numerous tutorials or videos online.

Likewise, I assume basic knowledge of Exchange (2010) administration. In the paragraphs below, I have screenshots of the OWA properties page in the Exchange Management Console (EMC): Server Configuration > Client Access. I will not present a step-by-step path to these pages or how to change settings. This is also true of the IIS Manager and browsing among the different properties of the different virtual directories.

In each case (see below) I started the capture just before attempting to login to OWA with Internet Explorer 11. Once the test user was logged in, I stopped the capture.

The test username and password are, respectively:
  • alan.reid
  • MyPassword123#

Yes, it is useful to know these credentials beforehand but I learned that they can be easily detected in the captured data regardless, by simply searching for the string "password".

Forms-based authentication

For my first attempt, the OWA virtual directory was configured with "forms-based authentication". This is how the authentication settings appear...

In Exchange:

And in IIS Manager:

Remember! SSL terminates at the NetScaler. SSL is not enabled between the NetScaler and the Exchange server where the packet capture was performed.

Searching for the string "password" (without quotes) in the packet details produced the following results:

User credentials are easily visible (no need to highlight in bright yellow).

Reminder: when forms-based authentication is selected, the user will see a logon page similar to this:

Integrated Windows Authentication

What about Windows Authentication? I thought that with this type of authentication, at least the password would be encrypted. With this hypothesis in mind, I changed the OWA authentication settings so they looked like this...

In Exchange:

In IIS Manager:

Note: after this change to the virtual directory settings, a message displays, reminding us to execute the command "iisreset /noforce" so the new settings take effect. Make sure you do this. Otherwise, results may be inconsistent and confusing.

Note: make changes to authentication settings with the EMC in Exchange (or the EMS if you prefer the command line). I only show the IIS Manager settings for reference.

Windows Authentication is an improvement. The password is not sent in clear text. If we search the packet details, with the same settings as those above, we observe that no such string can be found:

We can observe the name of the user but the NTLM authentication does protect the password from packet capture and analysis. Below, we can see the NTLM activity but not the password itself (only the username):

Note: NTLM authentication and Windows Authentication are more or less synonymous - at least for this experiment.

Reminder: when (Integrated) Windows Authentication is selected, the user will either see a logon page like this...

Or... OWA will open automatically with no need to re-enter user credentials.

Some clarification: Integrated Windows Authentication should allow us to open OWA without having to enter our credentials a second time. This type of authentication uses the credentials that we used to login to the client machine itself. There are some conditions however.
  • The (Windows) client machine must be a domain member.
  • Under the Internet Options "Advanced" tab, the setting "Enable integrated Windows authentication" must be checked.
  • The URL for the OWA website must be added to Local Intranet sites under the "Security" tab of Internet Explorer.
Even then, I was only able to make this work after a complete reboot of both the client machine and the Exchange servers (although this may not be absolutely necessary).


With forms-based authentication, SSL Offloading exposes both the username and the password. Even with integrated Windows authentication, SSL offloading exposes the username. Moreover, elements of the OWA webpage are also visible, for example the names of persons with whom the user is corresponding. Wireshark allows us to export elements such as images using "File - Export Object - HTTP" and is apparently able to reconstitute an entire webpage (although some elements may be absent). In summary, SSL Offloading represents a security risk if we cannot guarantee that the network between the NetScaler and the Exchange server(s) is indeed secure. Since SSL Offloading is a feature on most load balancers, this remark is valid for load balancers in general (and not just the NetScaler product suite).

Saturday, April 23, 2016

NetScaler VPX - load balance Exchange - Part 5 (SSL Offloading)

SSL Offloading with Exchange 2010

By default, SSL connections to the Exchange server (such as OWA) remain encrypted until they reach the Exchange server itself. This is a very secure configuration because the data is never unprotected between the two end-points. In some cases, in finance or in the defense industry for example, such a configuration may be mandatory.

Like other load balancing solutions, the NetScaler allows this type of configuration, usually known as "SSL pass through". The encrypted data arrives from the remote end-point, passes through the NetScaler, and is redirected to the local end-point, remaining in an encrypted state from start to finish.

The advantage of SSL pass through is the high level of security that it provides.

Another configuration is possible however: the NetScaler can decrypt the SSL traffic as it arrives and relay it to the local end-point (Exchange, for example) in "clear text". This configuration is less secure but has some advantages:

  • It relieves the Exchange servers of the encryption and decryption workload.
  • It allows the NetScaler to process data at Layer 7 (that would otherwise be obfuscated by encryption) and use other types of persistence (COOKIEINSERT for example) rather than the basic SOURCEIP persistence type that has notable limitations - especially when clients connect to resources via NAT (Network Address Translation).
  • The NetScaler can analyze the content of the packets and peform various actions based on this content (directing traffic to a certain virtual server IPs or rewriting a URL).

Yes, that summary of SSL offloading benefits does assume some knowledge of load balancing concepts such as "persistence". If these terms are unfamiliar, I would recommend that you consult either Citrix documentation on the subject or perform an online search with your preferred search engine.

In either case (SSL pass through or SSL offloading), we must import and install the certificate used by the Exchange server for encryption/decryption so the NetScaler can perform this task instead. I completed this process in an earlier blog post.


I will now configure SSL offloading for OWA (it can also be configured for other forms of client access such as Outlook Anywhere or Active Sync). We must make some changes on the NetScaler and then some changes on the Exchange server(s).

NetScaler changes for SSL Offload

First, we go to the following section in the NetScaler management GUI:

NetScaler > Traffic Management > Load Balancing > Services

We click on "Add":

Note: the first service for offloaded SSL has already been created. You can choose a different name, one that makes the most sense for you, or that respects whatever naming convention you may have in your organization.

I configure the service with the following settings (yes, this was for the first HTTP service) and then I repeat as needed. In other words, if I have a second Exchange server, I will create a service for it as well.

Now I have the following services. I will replace the last two services (that use SSL - port 443) with the first two services (that use HTTP - port 80):

The services, each of which is associated with a backend resource server (Exchange in this case), are bound to a "virtual server" to which clients connect using its "VIP" or virtual IP.

I will adjust the service bindings (replacing the SSL services with the HTTP services) at this location:

NetScaler > Traffic Management > Load Balancing > Virtual Servers

I hightlight the lb_vs_OWA virtual server and click on "Edit":

Click on the "Load Balancing Virtual Server Service Bindings":

We will have to remove the SSL bindings first. Otherwise, this is what will happen...

Click on "Add Binding":

"Click to select" the new bindings:

Select the HTTP services for OWA (SSL Offload):

Click on Bind:

An error message displays:

We must first unbind each of the SSL services...

And then we can bind the HTTP services.

The service bindings for the lb_vs_OWA virtual server should look like this:

Exchange changes for SSL Offload

Out of curiosity, I wanted to see what would happen if I tried to establish an OWA connection without making any adjustments on the Exchange side.

What happens? A "403 - Forbidden: Access is denied" error:

If I change the NetScaler services to SSL again, the error does not occur, so I know that the requirements for SSL on the Exchange server are the problem. Indeed, we have to uncheck these requirements for SSL offloading to work.

In fact, we have to make two changes on each of our Exchange servers (EX13-1 and EX13-2):

  • Add a "SSLOffloaded" REG_DWORD key to the registry
  • Disable the SSL requirement on the OWA virtual directory.

We add this key ("SSLOffloaded") at this location in the registry:

Note: right-click on "MSExchange OWA", select "New" and create a new DWORD key with the name "SSLOffloaded" and a value of 1.

Note: the path is visible at the bottom of the screenshot above.

Next, we open IIS Manager, navigate to the "owa" virtual directory and open "SSL Settings":

We uncheck the "Require SSL" box, click on "Apply" (under Actions) and then execute the command "iisreset /noforce":

Note: we repeat the process on the other load balanced Exchange servers.

Now when I attempt to access OWA, I can not only access the page but also login successfully:

Tuesday, April 19, 2016

NetScaler VPX - load balance Exchange - Part 5 (monitors)

The NetScaler VPX has numerous built-in "monitors" that check the state of the servers for which we are load balancing.

For example, there is a "ping-default" monitor and a "tcp-default" monitor (among many others not shown in the screenshot below):

If we highlight a monitor and select the Action "Show Bindings", we can see the services to which the monitor is bound:

By default (unless we designate another monitor), the RPC services use the "ping-default" monitor to check the status of the Exchange servers:

On the other hand, the OWA services use "tcp-default".

Note: this is shown in a screenshot below.

If we examine the properties of a service (or service group) we can also determine the monitor that is being used from this perspective:

These default monitors check the availability of a server: if the server does not respond to a ping (for example), the load balancer will not send packets (in the broadest use of this term) to the server that appears to be unavailable. However, it is possible that the server is available (functional) but the actual services are stopped. Therefore, we can optionally configure a monitor if we want to fine-tune the awareness of service availability).

Note: in the paragraph above, the word service refers to the Windows services running on the server (or the Unix/Linux equivalent) as opposed to the "services" (or "service groups") that we create on the NetScaler. 

The NetScaler VPX has a number of pre-configured monitors capable of checking the status of a particular service. For OWA (SSL) we have the following possibilities:
  • http
  • https
  • http-ecv
  • https-ecv

Note: we can also create custom monitors.

With http, the NetScaler sends a http request ("GET" for example) to the target server and waits for a http status code in response ("200" for example).

In the properties of the http monitor, the property for "Secure" is unchecked:

This seems to be incompatible with a target server requiring SSL connections, which would be the case if we use SSL pass-through or a SSL Bridge.

When I tested this for OWA, I obtained the following result:

However, if I use the "https" monitor (with "Secure" checked in the monitor properties), I obtain this result:

As for the other options (with the ecv suffix), they request a particular html page (for example) from the target server and search for a particular message in the response. In other words, the request contains a particular string and the response must contain a particular string as well.


But how do we select another monitor?

We go to the "Monitors" section of the service properties (highlight the service and click on Edit) and then click on "Service Load Balancing Monitor Binding" where we click on "Add Binding":

"Click to select" a new monitor:

Select a new monitor (for this example, where we are using OWA, I will select https):

Note: yes, click on "Select" once you have made your selection.

When we return to the "Load Balancing Monitor Binding" screen, "https" should have replaced the previous selection and we click on "Bind":

Back on the service properties page, click on "Done" at the bottom of the page (you may have to scroll).

Lastly, be sure to click the floppy icon in the upper right hand corner to save the configuration!


Now the "OWA" services on each of the two Exchange servers use the "https" monitor rather than the default "tcp-default" monitor:

Monday, April 11, 2016

NetScaler VPX - load balance Exchange - Part 4 (load balance Outlook Web App)

After configuring load balancing for SMTP traffic and Outlook (RPC) in my two previous blog posts, I will configure it for Outlook Web App (formerly "Access") - or OWA - in this post. As for the previous exercises, we need to create (if they are not already created):
  • Servers
  • Services
  • Virtual Servers (with a "VIP")
  • Monitors (optional).

We have already created two server entries, representing our two Exchange servers, so there is no need to repeat this process. Please refer to my previous blog post if you need to see how to create these entries. Otherwise, we will create two entries for the OWA service and then a virtual server (with a VIP) to which OWA clients will be directed.

OWA uses SSL and the NetScaler offers a number of features that optimize this type of connection, for example:
  • SSL offloading: the NetScaler decrypts the incoming SSL connection and thus lessens the workload on the backend servers (in this case, the Exchange servers).
  • Rewrite (of URLs): users can enter a shortened URL for OWA and the NetScaler, analyzing the packets, sees that it is for OWA and (for example) adds "s" to http and "/owa" and the end of the URL.

In this blog post, I will opt for a simple configuration where the SSL traffic passes through the NetScaler and where users would have to enter the full URL (or use either a shortcut or a Internet favorite).


As mentioned above, we already have server entries for both Exchange servers and will start with the creation of services for OWA/SSL. Here is the list of existing services:

NetScaler | Traffic Management | Load Balancing | Services

We click on "Add" (see above) and create the following services for OWA/SSL:

We now have the following services:

Now let's create a virtual server (and VIP) to which OWA clients will connect. We go to...

NetScaler | Traffic Management | Load Balancing | Services

And click on "Add" (note the existing virtual servers for SMTP and Outlook (RPC)):

I enter the following values (use values appropriate for your network) and click on OK:

As with the other virtual servers, we need to bind the appropriate service(s) to the virtual server. Click on the binding link shown below (under "Services and Service Groups"):

I "Click to select" a service...

(Here I check the OWA services and click on "Select"):

Then click on "Bind":

I click "OK" and "Done" as needed to return to the list of Virtual Servers:

The State (and effective State) of the virtual server is "down".

However, we still need to configure the load balancing method and the type of persistence. Let's see if configuring these features changes the state of the virtual server.

We highlight the lb_vs_OWA virtual server as show above and click on the "Edit" button.

In the resulting menu that appears on the far right, we select Method (and later Persistence):

For the load balancing method, we will select ROUNDROBIN:

Note: other choices may be more appropriate. If we have two servers, one with more capacity than the other, we may want to select a method that directs a greater percentage of connections to the more powerful server.

For persistence, Citrix documentation recommends COOKIEINSERT for OWA with a 30 minute timeout (2 minutes was the default):

Note: click "OK" and then "Done" to return to the list of virtual servers.

However, the state of the virtual server remains "down":

This is because I have opted not to use (at least for now) SSL offloading. I want the OWA SSL traffic to "pass through" the NetScaler and remain encrypted until it reaches the Exchange server(s). This method requires us to import the SSL certificate used to secure OWA into the NetScaler.

If you need directions on the procedure to export a certificate from Exchange, I would direct you to my blog posts where I perform this operation (there are several). In this blog post, I export a certificate from an Exchange 2007 server for use on ISA/TMG:

Exchange 2007 (SP3) - ISA - Publish OWA - Export/Import SSL certificates

For other examples, I would search my blog posts for "export certificate" (use the search function in the upper left-hand corner):

Other option: refer to sources online ("export certificate Exchange 2010" or "2013" depending on your version of Exchange).

Once we have the certificate, we must import it and install it on the NetScaler VPX.

Note: I will perform each of these steps separately (the process seemed somewhat confusing. In the end, however, the certificate was usable).

First (after we have accessed the NetScaler web interface), go to the following section:

NetScaler > Traffic Management > SSL > SSL Certificates

The yellow exclamation mark indicates that the feature is not enabled. We need to right-click on SSL to enable the feature:

Now that SSL is enabled, we will import the certificate. In the Tools section, click on Import PKCS#12

We create what is called an "Output File Name" (simply a name - we can call it what we want - but make sure to use the .pem file extension) and then browse to the location of the exported certificate file. We enter the password created when we exported the certificate from the Exchange server (this is necessary since the exported certificate file also contains the private key associated with the public key in the certificate). I was able to leave the Encoding Format field blank and proceed all the same: 

We click on OK and return to the list of SSL options. If we click on the Manage Certificates / Keys / CSR link...

We can see that both the exported .pfx file and the .pem certificate we created:

Next, we move to this location to install the certificate:

NetScaler > Traffic Management > SSL > SSL Certificates

Click on "Install":

I enter a name for the Certificate-Key Pair and then browse to the "ns-SSL-Exch.pem" file I created in the previous step:

Note: make sure PEM is selected. Yes, I designated the same file for Certificate File Name and Key File Name. It was not necessary to re-enter a password. Leave the other settings as they are. Also, when you open "Browse", you can select the option "Appliance" (meaning the NetScaler) since the certificate (with the private key) has already been exported and (apparently) converted into the .pem format:

This is what we see when we browse the "Appliance" (essentially the content of the /nsconfig/ssl folder):

Once we click "Install" (Install Certificate screenshot above), we return to the list of installed SSL certificates:

We can now (finally!) bind the certificate to the OWA virtual server. Go to the list of virtual servers...

We will highlight the lb_vs_OWA virtual server and click on "Edit". In the Certificates section of the virtual server settings, click on "No Server Certificate". This is where we will bind the certificate to the virtual server:

I select my imported (and installed) "imp-SSL-OWA" certificate:

I click on "Bind":

Now the "State" and "Effective State" of the OWA virtual server are both "Up":

IMPORTANT: as explained in the previous blog post for Outlook (RPC), we have to adjust the DNS record for OWA so clients are directed to the VIP (virtual IP) of the virtual server rather than to the Exchange servers themselves.

So what happens when a user attempts to connect to OWA?

On my initial attempts, the credentials are accepted but the connections time out immediately (yes, instantaneously).

This is because the COOKIEINSERT persistence type assumes that SSL offloading is being used, which is not the case here. The solution is to use the SOURCEIP method instead:


Once I make this adjustment, users can connect to OWA via the NetScaler VIP/virtual server without any problem. However, we should note that if clients are connecting through NAT (a single IP) the SOURCEIP persistence type may not be the best choice. This is not a problem if we use SSL offloading and are able to use the COOKIEINSERT persistence type.

The advantages and disadvantages of these various peristence options are compared in this discussion on the Citrix NetScaler forums: