What Should Be the Frequency of Source Refresh Schedules?
The source refresh time interval that you select determines the freshness of the index for this source, it should be following the rate of change of documents in the source, but should be optimized to avoid useless connections between the CES and repository servers that can contribute to load both servers.
Note: You can create new or modify existing source refresh schedules (see Creating or Modifying a Source Schedule).
You can schedule the following types of refresh:
Note: For repositories with API call restrictions (such as cloud repositories), consider setting your source refresh frequencies to optimally consume your available API calls.
Incremental Refresh
Many Coveo connectors come with the incremental refresh feature that allows to constantly maintain the index up-to-date with new, modified, or deleted source documents. When available for a source type, it is strongly recommended that you schedule an incremental refresh for your sources (see Scheduling a Source Incremental Refresh).
Choose an incremental refresh time interval that is of the order of magnitude of the rate at which documents change in the source.
-
For a team collaboration Confluence source with a large number of active users where every hour some wiki pages change and several comments are added, select Every 30 minutes to make these changes searchable not too long after they have been saved.
-
For a relatively static Intranet source where only a few documents are added, modified, or deleted on average everyday, selecting Every weekday may be appropriate.
-
For a permanently static legacy repository where no document will ever change, do not configure an incremental refresh.
Note: Incremental refresh may be not available for a connector type either because the connector rather offers live monitoring (see About Incremental Refresh and Live Monitoring) or because the repository or the connector does not support it.
Full Refresh
When incremental refresh or live monitoring is configured, it is generally recommended to set a source full refresh to a weekly time interval such as Every Sunday.
A weekly full refresh schedule is a safety net to ensure the index stays up-to-date with your source. A weekly full refresh will pick up corner case changes that could be missed by incremental refresh or live monitoring. Also, the incremental refresh of some connectors are known to have limitations (often due to repository API limitations) such as not being able to track deleted documents or permission changes.
When neither incremental refresh or live monitoring is available for a source, choose a full refresh time interval in relation with the rate of change of source content to ensure an acceptable index freshness without overloading the CES and repository servers.
A repository source without incremental refresh contains a relatively small number of documents and several documents are added, modified, or deleted every day, selecting a refresh schedule of Every 6 hours would be appropriate.
For a permanently static legacy repository where no document will ever change, select (none) as no regular refresh is appropriate.
Rebuild
It is generally not recommended to create refresh schedules of Rebuild type, particularly for large sources, because each rebuild temporarily increases the size of the index and increases the load on the index compaction process (see About the Index Self-Optimization Process).
Rebuilding a source should be done manually when it is needed, for example following a source configuration change (see Applying an Action to a Collection or a Source).