Tuesday, May 31, 2016

Exchange 2010 (SP3) - Search index problems

As an Exchange administrator, we may encounter the "Slow FindRow Rate" error in the Application log of our servers.

The Microsoft KB 2764440 article explains that "large values indicate that applications are crawling or searching mailboxes and that server performance is being affected."

We will usually notice this because some monitoring software alerts us that a certain threshold has been exceeded. In this case, it is the "\MSExchange IS Mailbox(*)\Slow FindRow Rate" performance counter that is over "10". In my case, the values were sometimes at 11 or 12.

This threshold exists at the level of the mailbox database (as opposed to a particular mailbox or the entire server). So the problem could affect one mailbox database but not the others. We should also remember that each mailbox database has its own "search index". 

Based on my research, it looks like this can be due to applications that search the mailbox database directly and do not use the Exchange search index.

This article, for example, is rather old (2008), refers to Exchange 2003 but describes perfectly what I was observing:

I quote:

"This could lead up to the dreaded RPC dialog box or the balloon stating the Outlook is requesting data from the server."

This is exactly what was being observed on the client side and, indeed, Outlook was in online mode (not to be confused with "Exchange Online" (Office 365)).

Otherwise, I understand that there is a "Slow FindRow" method and (since Exchange 2003?) a more efficient "Fast FindRow" method but the underlying hardware must be able to take advantage of the new method (or else Exchange reverts to the older less efficient method).

Other factors:
  • A high number of items (over 5000) in the Inbox, Sent Items, Deleted Items, Calendar and Contacts folder. Note: some users keep years and years of appointments in their calendar.
  • Apparently shared calendars can play a role.
  • Third-party applications that access the mailbox directly can be the source of the problem.

Some of the sources are old (2006-2008) so I am not sure to what extent the concepts presented would apply to the latest versions of Exchange (2013 and 2016) of which the IOPS performance is supposed to be vastly improved over earlier versions. On the other hand, it is on Exchange 2010 that I have observed references to "Slow FindRow".

It also appears that this error can be related to corruption of the search index (refer to Microsoft KB 2764440 cited above). In the case of a corrupt search index, we have a number of possible corrective actions.

"Manual" recreation of the search index

On all versions of Exchange (to my knowledge), we can delete the search index manually after stopping certain services. When we restart those services, the search index will be recreated.

On Exchange 2010, we could proceed as follows:

1. Stop services with the command...

net stop MsExchangeSearch

or with PowerShell:

Stop-Service MsExchangeSearch

2. Delete the subfolder named "CatalogData-<some-GUID-here>" that contains the search index. This subfolder is in the same location as the other files that constitute the mailbox database. For example:

And the content looks like this:

3. Start services with the command...

net start MsExchangeSearch

or (PowerShell)

Start-Service MsExchangeSearch

Note: of course, we can also use services.msc (the graphic interface) to stop and start the services.

Apparently (I have not experimented with this myself), there are two services to stop (and then start) with Exchange 2013 (and 2016?).
  • MSExchangeFastSearch
  • HostControllerService

Recreation of the search index with a script

In Exchange 2010, we can use the ResetSearchIndex.ps1 script to rebuild the index. We have a number of options...

After designating the mailboxdatabase with the Get-MailboxDatabase cmdlet:

Get-MailboxDatabase “DB01" | .\ResetSearchIndex.ps1 [-force]

Note: click to enlarge.

With the -force parameter, we avoid the prompt:

We also have this option:

ResetSearchIndex.ps1 [-force] “DB01"

Note: in all cases, execute the command from the Scripts directory, that is, by default:

C:\Program Files\Microsoft\Exchange\v14\Scripts

It may be necessary to replace "Exchange" with "Exchange Server". Of course, the path could be entirely different depending on what was chosen at the time of installation.

If you examined the execution of the script in the screenshots above attentively, you may have noticed that it uses the same three-step process we completed earlier, but manually:
  1. Stop services.
  2. Delete the search index folder.
  3. Start services.
Note: this script is no longer available with Exchange 2013 (and 2016?).

Update the "catalog" (DAG only)

If the mailbox database is part of a Database Availability Group (DAG) we have one more option: we can overwrite the catalog from another source (the partner Exchange server if we have a simple pair (there can be up to 16 Exchange servers in a DAG)).

Update-MailboxDatabaseCopy DB1\EX13-2 -CatalogOnly

Note: rebuilding the search index may take a certain amount of time and may use a good amount of system resources. You may want to schedule this accordingly.

No comments:

Post a Comment