Product DocsMenu

Configuring Near Real-Time Indexing

The Near Real-Time Indexing (NRTI) feature can significantly reduce the index pipeline processing time for indexes of two million documents or more (see About Near Real-Time Indexing). Smaller indexes already offer a short index pipeline processing time, so enabling NRTI when your index is small has no noticeable effect.

When your Coveo deployment includes more than one back-end server, NRTI must be enabled on all Mirrors that are serving queries.

Example: Your Coveo deployment consists of a Master server and two load-balanced Mirrors. The Front-End servers are configured to send queries only to the Mirror servers, not to the Master server. In this scenario, you must enable NRTI only on the two Mirrors. There is no point in enabling NRTI on the Default Mirror (Master server) because it does not serve queries.

To configure Near Real-Time Indexing

  1. Ensure that your Coveo Master and Mirror (if applicable) servers meet the requirements for the size of your index.

    Consider following the recommendation to add a NRTI dedicated disk on each server serving queries (see Coveo Platform Hardware and Software Requirements).

  2. On the Coveo server, access the Administration Tool (see Opening the Administration Tool).

  3. Before enabling NRTI on a Mirror server:

    1. Ensure that the Mirror is Online and In Sync (see Determining the Status of a Mirror or Slice).

    2. Synchronize the Mirror when it is not the case (see Applying an Action to a Mirror).

  4. On the Administration Tool menu, select Index > Mirrors.

  5. In the Mirrors page, in the Name column table, click the mirror for which you want to enable NRTI.

    Repeat the following steps for each mirror for which you want to enable NRTI.

  6. In the Mirror: [name] General page, in the navigation panel on the left, click Near Real-Time Indexing.

  7. In the Mirror: [name] Near Real-Time Indexing page:

    1. If not already the case, select the Enabled check box.

      For CES 7.0.7022+, NRTI is enabled by default for both new and existing indexes.

    2. In the Index Folder box, enter the full path where you want to save the NRTI index files.

      Note: The default folder for new or existing indexes like when the box is empty is [Index_Path]\Index\Default\RealtimeIndexing so by default, the NRTI files are stored on the same disk as the normal index slices.

      Sharing the same disk will work, but because NRTI is an I/O intensive process, to achieve optimal performances, it is recommended to add one dedicated disk for the NRTI files on each Coveo server (Master or Mirror) that are serving queries, and point to this disk with the Index Folder parameter. The dedicated disk specifications depend on the size of your index (see Coveo Platform Hardware and Software Requirements).

    3. In the Temporary Files Folder box, enter the path where the NRTI temporary files are saved.

      For an existing index, the default folder that applies when the box is empty is [Index_Path]\Temp.

    4. In the Maximum Number of Documents per Slice box, leave the default value (200000) unless Coveo Support instructs you to change it.

      This NRTI temporary slice size optimally minimizes the indexing pipeline processing time and the required NRTI resources.

    5. CES 7.0.7022+ (September 2014) In the Minimum Number of Documents to Activate box, enter the index size specified in number of documents, above which the enabled NRTI will start to operate. The default and recommended value is 1000000. The NRTI process starts to improve performances for indexes of about this size. Consult Coveo Support before changing this value.

    Note: The documents that are in the index pipeline at the time NRTI is enabled must be indexed before NRTI becomes effective. In a healthy index, this delay will be relatively short, such as 30 minutes for a 40-million document index. In a bad case scenario where the Coveo server is lacking resources and cannot manage to close pending transactions, this delay could be significantly longer.

  8. Estimate the index pipeline processing time.

    The following steps give a method to estimate how long it takes to complete the index pipeline process.

    1. On the Coveo server, using Windows Explorer, open the folder of the regular slice. If there is more than one slice, open the folder of each slice.

      Example: By default, the first slice is called Default and its folder is: [CES_Path]\Index\Default\Default

    2. In the slice folder(s), look for pre-transaction and transaction files that are named respectively using the following patterns: 

      • Index-nnnnnnnnnn.ptn

      • Index-nnnnnnnnnn.trn

      where nnnnnnnnnn is an incremental transaction ID number.

      Note: When no documents are being indexed, no pre-transaction and transaction files will be present in the slice folder(s). Either force a source refresh/rebuild or wait for some source to get refreshed to see new transaction files appear.

    3. At a given time, note the highest transaction ID number (without the leading zeroes) from the slice folder(s), and also note the current time.

    4. Using an account and a Coveo search interface having access to all the indexed content, in the search box, repeatedly submit the following query until you start getting results: 

      @systransactionid >= [highest_transaction_id]

      Example: At 14:05, the pre-transaction or transaction file in the slice folders with the highest transaction number is Index-0000005044.trn.

      In the search interface, you type the following query:

      @systransactionid >= 5044

    5. Note the time at which you start getting results, and then subtract the time at which you noted the highest transaction ID to get an estimate of the index pipeline process time.

What's Next?

Ensure that all CES aspects are optimized to minimize the searchability delay (see Minimizing the Searchability Delay).

People who viewed this topic also viewed