Sunday, December 18, 2016

PKI - ADCS: certificate revocation (review)

In my last two blog posts, I reviewed methods for the verification of PKI / ADCS health and then encountered (and later resolved) a problem with the publication of the CRL (certificate revocation list). I wanted to take that opportunity to review some of the fundamental aspects of certificate revocation, which is what I will do in the following paragraphs.


The "revocation" of a certificate prevents the use of the certificate in question.

The revoked certificate can no longer be used for its assigned role: client authentication, server authentication, code signing, etc..

When we revoke a certificate, the serial number of the certificate is added to a list called the "certificate revocation list" or "CRL"). This list is digitally signed by the certificate authority (CA) and published to the locations indicated in the CDP properties. This is usually a web share (the "http" location) and optionally in Active Directory (the "ldap" location):

Just like a certificate, the CRL has a validity period. However, this period is much shorter, often only a single day or several days. The CA must publish a new CRL at regular intervals to replace the previous CRL that has expired.

There are two types of revocation lists:
  • Base CRL: a complete list of certificate serial numbers revoked by the CA.
  • Delta CRL: a list of certificates revoked since the last base CRL publication.

Applications will verify certificates presented to them by consulting the CRL and reject any certificates on this list.

We can revoke a certifcate in the Certificate Authority MMC:

Unlike certificate expiration, which prevents the use of a certificate outside its validity period (and notably after), certificate revocation prevents the use of a certificate even if it has not yet expired.

Reasons for certificate revocation

A certificate may be revoked for a number of reasons. Here are some examples.

Loss of the private key (by theft or negligence)

The mobile device (laptop, tablet, smartphone) of a user is lost or stolen. The private key, stored on the device (by default), could be recuperated by someone with malicious intent. Since PKI consists in the interaction of the private and public key, we prevent this interaction by revoking the certificate containing the public key of the computer or user.

Compromise of the issuing CA

If the CA has been compromised, and notably its private key, an attacker could use that key to set up a new CA and issue certificates trusted by clients that trust the original or "real" CA. We would revoke the certificate of the compromised CA which would invalidate all certificates issued by the CA (since no certificate issued by this CA - or its clone - should be trusted).

Retired users and devices

When users retire or equipment is retired, it is best practice to revoke the associated certificates.

Replacement of old certificates with new certificates

We may need or want to replace old certificates with new ones. In this case, it is best practice to revoke the old certificates.

When we revoke a certificate, we can (and should) indicate the reason for which we are revoking the certificate:

Revocation is an irreversible act for all these reasons except "Certificate Hold". Although possible, it is not recommended to revoke certificates temporarily because it is difficult to determine at what time(s) the certificate was valid.

Note: it is better to revoke certificates based on a policy.

We must not decommission a CA if the certificates it issued are still meant to be used. Those less familiar with PKI may ask: "But why not? I'll just use some other CA to issue certificates in the future." We should remember that a CA not only issues certificates but also publishes a CRL of certificates it has revoked. If the CRL is not renewed at the interval determined at installation, the CA will no longer function. This is what we observed in a previous blog post. Please see the "Remarks" section below for more details.

Who can revoke a certificate?

Users granted the "Issue and Manage Certificates" permission. By default, administrators of the CA have this right. Domain and Enterprise administrators posess it as well.


The CRL is signed by the CA that publishes it.

This is very important concerning redundancy. When checking the revocation status of a certificate, we can only use the CRL published by the CA that issued that certificate. We cannot simply fabricate a list of serial numbers and make it available on the network. Once again, the CRL must be signed and only the signature of the same CA that issued the certificates to start with is valid. Clients cannot simply consult the CRL published by another CA.

If the CA is unavailable for a time exceeding the CRL validity period (plus whatever grace period - or "CRL overlap" - we have configured) the CA will go offline and stop functioning.

Note: we can obtain redundancy if we configure CA clustering which is an entirely different subject.

The Delta CRL is almost never necessary.

The client downloads the CRL associated with the certificate that it must validate. The file is so small (in KB) that there should be no effect on bandwidth, especially with 1 Gb or 10 Gb links. It is only if the CRL is exceptionnally large (this is apparently the case in some defense sectors) that a delta CRL may be useful.

Clients cache the CRL locally

 They only download the new CRL when the cached CRL expires.

This is advantageous concerning network bandwidth but creates a gap during which a very recently revoked certificate could still be accepted.

Imagine that the CRL is published at 12:01 AM (00:01) every day (so every 24 hours which is rather frequent). If a certificate is revoked at 12:05 AM, clients will only be informed almost 24 hours later. In the meantime, they will continue to use the cached CRL which does not contain the serial number of the very recently revoked certificate.

Each organization must evaluate the risk of such a delay versus the load on resources and notably bandwidth. Unless the CRL is very large, the effect on current systems should be negligeable.

We can see what is in the CRL cache with this command:

certutil -URLcache CRL

We can purge the CRL cache with this command:

 certutil –setreg chain\ChainCacheResyncFiletime @now

CRL overlap

When we set up our PKI, we can configure what is called "CRL overlap" with the post installation script . This is a sort of grace period during which our PKI will continue to function even though the CRL has expired. This might provide sufficient additional time to resolve a problem.

These are the CRL overlap settings that I configured for my subordinate CA in my first series of PKI blog posts:

certutil -setreg CA\CRLOverlapUnits 2
certutil -setreg CA\CRLOverlapPeriod "Days"

The overlap period cannot be longer than the validity period of the CRL. If the validity period was 3 days (which is what I configured elsewhere in the script), the overlap period could be 2 days at most.

OCSP (Online Certificate Status Protocol)

With this protocol, a client can validate single certificates as needed, via a web service, instead of downloading a CRL with the entire list of revoked certificates (most of which may not interest the client).  This is an optional function and I will not present it in this blog post.

Tuesday, December 13, 2016

PKI - ADCS: Health Check - part 2 (problem resolution)

We encountered a problem with Active Directory Certificate Services in my previous blog post:

In this blog post, we will attempt to resolve it.


So what was the problem?

It looks like the CRL of the root CA has expired (it was published in two places - on a web share (http) and in Active Directory (ldap)):

What about the CRL of the subordinate CA?

It is hard to see because the subordinate CA (issuing CA) is offline and no details are displayed:

But let's look in some other places, like "Manage AD Containers" (please consult the previous blog post if you do not know what or where this is). Everything looks fine until I come to the "CDP Container" tab:

I notice that both the CRL published by the root CA and the CRL published by the subordinate CA have expired. Both CAs will have to publish a new CRL and we will have to copy (by one method or another) the offline root CA's CRL to the subordinate CA.

Even so, I'm wondering if I cannot start the ADCS service and obtain more information?

The short answer is... No.

At one point, I attempted to start the service. This fails: 

Note: if we had not already determined that the expired CRL(s) is the problem, the reference to the revocation function is yet another pointer in this direction.

I attempt to publish the CRL on the subordinate CA. Maybe we can update the CRL at this level and make at least some progress?

Note: seriously, I would not have bet much on that option since the ADCS service is not even running...


We could (but should not) disable revocation checking.

First of all, does it even resolve the problem?

We disable revocation checking with this command (and then restart the ADCS service):

certutil -setreg ca\CRLFlags +CRLF_REVCHECK_IGNORE_OFFLINE

At first glance, this seems to improve matters.

After restarting the service, the subordinate CA is able to publish a new CRL (I repeated the same process as above) and I can expand the PKI Health Check tool:

Note the green check on the "Machlinkit Issuing CA" and compare with the previous screenshots.

At least the CRL of the issuing CA is up-to-date (status is OK from top to bottom):

But this is not a good solution for several reasons:

It only disables revocation checking on our subordinate CA. All other clients will continue to attempt to consult the CRL (which will result in failure since the CRL of the root CA is still not available).

Even if we were stubborn and disabled revocation checking on all clients (perhaps with a registry edit via Group Policy), we would prevent a critical security function from taking place. It would be impossible to reject compromised certificates. If the credibility of our PKI is important, disabling CRL would precisely destroy this credibility.

As it is very poor practice to disable revocation checking, I will re-enable it with this command:

certutil -setreg ca\CRLFlags -CRLF_REVCHECK_IGNORE_OFFLINE

As you might have expected (if you realize that both CRLs must be valid - 1 out of 2 is not enough), the subordinate CA goes offline again, once revocation checking is re-enabled and the ADCS service (certsvc) restarted.

We must publish an updated CRL from the root CA. There is no (acceptable) method to avoid this.

So I start and logon to the offline root CA (in my case, this is a virtual machine with no network adapters).


Once we have logged into the root CA, we can trigger the publication of a new CRL with one of two methods...

At the command line:

certutil -crl

Or in the Certificate Authority MMC:

We attempted this in one of the screenshots above - it's the same place and process on the root CA. We then copy the CRL file to the subordinate CA with removable media ((virtual) floppy, USB, etc.).

Note: it is worth specifying that the new CRL published on the root CA can be found in this location:


We will copy (or publish) the CRL file to three locations on the subordinate (issuing) CA:
  1. The (local computer) certificate store of the subordinate CA
  2. The web share indicated in the CDP record.
  3. Active Directory

The web share is the most simple operation. It is a matter of copying the CRL file to the location indicated:

I predict that if the CRL is available even in one location, the subordinate CA will detect it when I restart the ADCS service.

This is indeed what happens.

If we expand PKI View to the issuing CA, we notice, first of all, that we can descend to this level now, even though revocation checking was re-enabled. We also notice (see the screenshot below) that the CRL in the http location has a status of OK:

However, the CRL published to Active Directory is still in the "expired" state.

We still need to publish the CRL of the root CA to the local computer store of the subordinate CA as well as Active Directory. The commands below allow us, respectively, to achieve that double objective:

certutil -addstore -f Root C:\PKI\PKI-ROOT-CA.crl

certutil -dspublish -f C:\PKI\PKI-ROOT-CA.crl

Of course, you would adjust the path to the CRL to indicate the location to which you copied the CRL of the root CA.

Here is a screenshot of the operation:

Now if we consult the PKI Health Check tool, the status is OK for the entire system and the Machlinkit issuing CA is online (note the green check):

Sunday, December 11, 2016

PKI - ADCS: Health Check - part 1

In February and March 2015, I dedicated nine blog posts to the subject of Public Key Infrastructure with Active Directory Certificate Services.

This was the first of nine posts:

PKI (Public Key Infrastructure) with ADCS, Part 1: Introduction

At this time, I am reviewing some PKI best practices and verification of PKI health with various tools, in particular the PKI Health Tool (or "PKI View").

The subject was in fact already adressed, albeit indirectly, in my original series on posts on PKI:

PKI (Public Key Infrastructure) with ADCS, Part 7: configuration of the subordinate CA

Although I had addressed (very briefly) troubleshooting errors revealed by the PKI Health Check tool, I wanted to take advantage of this review to present a problem that will occur a year or so after the initial PKI installation if we do not manage our certificate services with diligence (expiration of the Root certificate CRL - "certificate revocation list"). In a blog post to follow, I will demonstrate the solution to this problem.


We should verify our PKI after installation (which I did as shown in the second link provided above) and at regular intervals later.

In particular, we want to determine if clients can access certification authority (CA) certificates and certificate revocation lists (CRLs).

Note: I'll use the terms "CA" and "CRL" in the rest of this blog post for concision.


Clients need to access the certificate of the CA that issued the certificate being verified as well as the certificate that provided the issuing CA with its own certificate. This process establishes a "chain of trust" and validates the "certification path" from the certificate being verified to the root CA of the PKI that produced the certificate in question.

We can see this "chain" when we open a certificate and look at the "Certification Path" tab.

Moreover, clients need to access the CRL to see if the certificate has been revoked.

  • The client uses the URL indicated in the certificate's AIA extension (Authority Information Access) to download the certificates needed to establish the chain of trust.
  • The client uses the URL indicated in the certificate's CDP extension (CRL Distribution Point) to access the list of revoked certificates that can no longer be trusted.

PKI Health Tool

If we want to verify the health of the PKI overall (rather than a specific certificate) we can use the "PKI Health Tool". Previously a down-loadable utility, it is part of the Active Directory Certificate Services (ADCS) role since Windows Server 2008.

We can access it in various ways.

We can enter pkiview.msc in the run dialog box and open the tool.

We can also open Server Manager, expand the Roles section: ADCS | Enterprise PKI

At the top level (PKI-ROOT-CA) we can see (in the first example below) that the status is OK. This takes into account all layers below.

Note: there is a green check on the icon of the issuing CA as well. This is always a good sign.

But what if there is a problem? Here is another example:

But what, exactly, is not right?

We need more details. At what level is the problem happening? Is it a problem with one of the CA certificates or with one of the CRLs?

So let's expand the structure and descend to the next level...

We can see the certificate of the Root CA (which is OK). We can also see that the Root CA certificate (necessary for certificate chaining) is available at both the http and ldap locations. On the other hand, the CRL for the Root CA, while available, has expired. Such an error has serious consequences as we can see when attempting to view the details of the subordinate (issuing) CA:

The issuing CA is offline and our PKI is no longer functional.

Note: I demonstrate how to correct this in a future blog post.

Some remarks...

The error icons have color codes that reflect (more or less) the severity of the problem:
  • Yellow - the CRL is expiring (but not yet expired).
  • Red - the CRL has expired or the AIA/CDP location cannot be accessed.
  • Red X - the CA is offline.

The status column provides more details about the error. Besides "OK", we may see:
  • Expiring - the certificate or CRL in question is expiring.
  • Expired - the certificate or CRL in question is expired.
  • Unable to download - the certificate or CRL in question could not be downloaded.

We should also note that we can adjust the number of days or hours for the certification expiration status.

We right-click on the Enterprise PKI icon and select Options

Besides viewing the Certificate Templates (which I would usually access through the Certification Authority MMC), we can view the "AD Containers" that hold certificates:

This can be useful for troubleshooting. In the screenshot above, the status is OK for all certificates in the AIA container. In the next blog post, we will see errors for a CRL in the CDP container.

Server Manager - Active Directory Certificate Services role

In the Active Directory Certificate Services role section of Server Manager, we have access to three useful tools:

  1. Event Viewer (filtered for ADCS related entries).
  2. System Services (where we can verify that services necessary for ADCS are running). Like for Event Viewer, this is the Services administrative tool (services.msc) filtered for ADCS related entries.
  3. ADCS Best Practices Analyzer.

Here is a view of the first two tools (yes, "Events" can be expanded):

The ADCS BPA is rather limited (at least in Windows 2008). There are only 8 checks, for example:


The tools presented so far are graphical tools. We can also use the certutil command line utility, especially if we require a high level of detail.

For example, we viewed a certificate earlier and validated the "chain of trust" to the certificate of the root CA. We can view (and save or output to a text file) much of the background certification validation process with certutil and in particular the following commands:

  • certutil -verify -urlfetch NameOfCert.cer
  • certutil -URL NameOfCert.cer

The output of the first command is quite verbose, possibly as long as this entire blog post. Therefore, I will only present a very brief sample here:

In the section above, certutil validates the AIA and CDP information of the certificate that was designated.

The second command opens a window where we can select various options and verify the URL for the AIA and the CDP (these were presented above):


This concludes my review of some of the tools we can use to verify the health of our Windows ADCS PKI. Although not presented in this blog post, we should remember that ADCS has numerous logging options which could also prove useful in the management and troubleshooting of a PKI. In a future blog post (probably the next one), I will demonstrate how to resolve the problem that caused our subordinate CA to go offline.

Monday, December 5, 2016

Exchange 2016 CU3 - updated pre-requisites, Active Directory preparation... and a word of caution.

This is my first blog post about Exchange 2016 CU3 (cumulative update 3). In fact, I had composed a blog post about the installation of the pre-release version of Exchange 2016 on July 26, 2015. In  an effort to avoid needless repetition, I will refer the reader to that blog post for general information about hardware and software pre-requisites. Only updates or corrections will be presented below.

Exchange 2016 (Preview) - original blog post on preview version

Here is the link to the Exchange 2016 system requirements:

Exchange 2016 system requirements

In the following paragraphs, I will concentrate on changes in system requirements since my original post in July 2015.



No significant changes

Operating system

Besides Windows Server 2012 and 2012 R2, the most recent Microsoft documentation lists Windows Server 2016 (Standard or Data Center) as a supported operating system on which Exchange 2016 can be installed. However, for Server 2016, the minimal version of Exchange 2016 is CU3.

In the title to this post, I implied there would be a word of caution (a warning) and here it is...

Although Windows 2016 Server is officially supported, a very serious problem affects Exchange 2016 on this platform when the server is part of a Database Availability Group (or "DAG"), so serious that Microsoft recommends that we do not install Exchange 2016 (CU3) on Windows 2016 Server. We should use Windows 2012 R2 instead.

You can read the statement made by the Microsoft Exchange team on their blog "You Had Me At EHLO":

The message was posted on November 4th, 2016 and as of December 5th no solution has been announced (besides the recommendation to use Windows 2012 R2 instead).


Windows 10 is now supported for Management Tools.

Active Directory

No significant changes except the addition of Windows 2016 domain controllers as supported domain controllers.

Coexistence with previous versions of Exchange

The minimal versions of Exchange 2010 and 2013 are currently:

  • Exchange 2010 SP3 Rollup 11 (Rollup 15 is the latest rollup at the time of this writing)
  • Exchange 2013 CU10



I will change the installation order for the Exchange 2016 pre-requisites and Active Directory schema extensions (compared to the order in my blog post on the Exchange 2016 pre-release).

All the actions below were taken on the future Exchange 2016 server (unless otherwise indicated).

Note: this server is a domain member with TCP/IP settings configured for such an environment. I will not detail the basic configuration of the server here. We should also make sure that we connect as a user that not only has administrative rights to the server itself but also has the status of domain, enterprise and schema administrators. In particular, this is necessary for the preparation of Active Directory. 

Step 1 - Install required roles and features

On the future Exchange server, we open a PowerShell prompt and execute the following command:

Install-WindowsFeature AS-HTTP-Activation, Desktop-Experience, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, RSAT-Clustering-Mgmt, RSAT-Clustering-PowerShell, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation, RSAT-ADDS

Note: yes, you can copy and paste. No need to retype!

Step 2 - Install .NET Framework 4.5.2

My test server is running Windows 2012 R2. If you are using a different operating system, check the system requirements for the proper .NET version. I have found that Windows Update installs this version of .NET on Windows 2012 R2 (at least if you select it...). Otherwise, download the offline installer.

.NET 4.5.2 Offline Installer

Step 3 - Install "Unified Communications Managed API"

Unified Communications Managed API 4.0 Runtime

Step 4 - Prepare the Active Directory schema

First, we download the Exchange 2016 CU3 installation files on the future Exchange 2016 server...

Cumulative Update 3 for Exchange Server 2016 (KB3152589)

After navigating to the location where we saved the files, we execute this command:

setup /ps /IAcceptExchangeServerLicenseTerms

Note: /ps is a shortcut for /PrepareSchema.

At first, I attempted to install Exchange 2016 CU3 in an existing Exchange 2010 SP3 / Office 365 hybrid environment and encountered an error:

A hybrid deployment with Office 365 has been detected. Please ensure that you are running setup with the /TenantOrganizationConfig switch. To use the TenantOrganizationConfig switch you must first connect to your Exchange Online tenant via PowerShell and execute the following command: "Get-OrganizationConfig | Export-Clixml -Path MyTenantOrganizationConfig.XML". Once the XML file has been generated, run setup with the TenantOrganizationConfig switch as follows "/TenantOrganizationConfig MyTenantOrganizationConfig.XML".

Error when you run Setup /PrepareSchema to prepare the schema for an existing Exchange hybrid environment

So we have to do two things:

  1. Create the MyTenantOrganizationConfig.xml file.
  2. Re-run the command with the /p (or /PrepareAD switch) instead of the /ps switch AND add the /TenantOrganizationConfig switch, designating the .xml file created in step 1.

This was the detailed solution:

Exchange 2010 hybrid environment - Get-OrganizationConfig

But I was still not able to upgrade the schema and prepare Active Directory.

Once again, I will refer the reader to the TechNet discussion that finally resulted in a solution:

MyTenantOrganizationConfig error

However, I will summarize...

After consultation with someone from Microsoft, I disabled the hybrid detection function with a registry change at the following location on the (future) Exchange server:


Note: create the keys when necessary.

Here is the registry change (yes, REG_SZ):

Disabling hybrid detection allowed the setup /PrepareAD command to complete.

I have to hope this will not cause other problems in the future.


Now I am ready to install Exchange 2016 CU3.

However, I will wait to do so in my hybrid environment because my priorities have very recently changed and I need to experiment with some aspects of Office 365. I'm not sure what effect installing Exchange 2016 would have. If it forced me to manage the hybrid environment from Exchange 2016 (and not Exchange 2010 anymore), my testing with Office 365 may produce inaccurate results.

Because of this, I will probably (after all the above...) opt to install Exchange 2016 in another ("clean") test environment so I can continue experimenting with the functions of Office 365 that interest me.