Configuring and Indexing a Twitter v2 Source

A source defines a set of configuration parameters for a specific Twitter account.

To configure and index a Twitter v2 source

1. On the Coveo server, access the Administration Tool (see Opening the Administration Tool).

2. Select Index > Sources and Collections.

3. In the Collections section:

   • Select an existing collection to which you want to add the new source.

     OR

   • Click Add to create a new collection (see Adding a Collection).

4. In the Sources section, click Add.

   The Add Source page that appears is organized in two sections.

5. In the General Settings section of the Add Source page:

   

   1. Enter the appropriate value for the following required parameters:

      Name

      Enter a descriptive name of your choice for the connector source.

      Example: Twitter v2

      Source Type

      Select the connector used by this source. In this case, select Twitter v2.

      Note: If you do not see Twitter v2, your environment does not meet the requirements (see Twitter v2 Connector Requirements).

      Addresses

      Starting addresses are source locations in your Twitter account. The Twitter v2 connector supports four types of starting addresses. Enter all required starting addresses:

      • users: a list of comma-separated usernames.

        Example: users:twitterapi,coveo,support

      • list: a list's name and its creator's username in the format {username}/{listname}. These values can be obtained in the list's URL (https://twitter.com/{username}/lists/{listname}).

        Example: list:twitterapi/meetup-20100301

      • query: a search query. Only Tweets posted within the last 7 days will be retrieved with this starting address, and the results are not comprehensive (some Tweets posted in the last week might be missing).

        Examples:
        

        • query:#coveo

        • query:$GE

        • query:puppy filter:media

      • collection: a collection's ID, which can be obtained from the collection's URL (https:twitter.com/{username}/timelines/{ID}).

        Example: collection:719516944244817923

   2. Review the value for the following and seldom-modified parameters:

      Rating

      Change this value only when you want to globally change the rating associated with this source relative to the rating of other sources (see Understanding Search Results Ranking).

      Example: When a source replaces a legacy system, you may want to set this parameter to High so that, in the search interface, results from this source appear higher in the list compared to those from legacy system sources.

      Document Types

      If you defined a custom document type set for this source, select it (see What Are Document Type Sets?).

      Active Languages

      If you defined custom active language sets, make sure that you select the most appropriate for this source (see Adding and Configuring a Language Set).

      Fields

      Coveo Enterprise Search (CES) 7.0 does not require the presence of custom field sets to extract metadata from Twitter. However, if you elected to use a specific field set, you can select it in the Fields drop-down list. Otherwise, leave it at Default Scheme (see Exporting and Importing a Field Set and Adding or Modifying Custom Fields).

      Refresh Schedule

      Time interval at which index content is automatically updated. By default, the Every day option instructs CES to refresh the source every day at 12 AM. Since the incremental refresh takes care of updating the source, you can select a longer interval such as Every Sunday (see What Should Be the Frequency of Source Refresh Schedules?).

      Note: Since deleted items are not taken into account during normal incremental refresh operations (see Limitations), it is recommended to force a full refresh, or even a source rebuild, to update the status of those deleted items.

6. In the Specific Connector Parameters & Options section of the Add Source page:

   

   1. Enter the previously obtained values [more]. These values allow the Twitter v2 connector to access the content of your managed Twitter users.

      Note: These values are long series of alphanumeric characters.

      API Key

      The API key provides the identity of your corporate Twitter application when crawling the data of managed Twitter users whose Twitter account needs to be indexed.

      API Secret

      The API secret is used to authenticate the API key above.

      Access Token

      The access token is used to make API requests on behalf of your corporate Twitter application.

      Note: This access token is used to retrieve a new OAuth 1.0a access token.

      Access Token Secret

      The access token secret is used to authenticate the access token above.

   2. The default value for the following parameters often does not need to be changed:

      Number of refresh threads

      Determines the number of threads that the connector can refresh simultaneously. The default and recommended value is 2.

      Mapping File

      Indicates the path to the default mapping file that defines how the connector handles metadata. The installer creates the file indicated is this field.

   3. Select additional content to be indexed using the following options:

      Index User's Followings

      Indexes the Tweets of users that a managed user is following. With this check box selected, users followed by users specified in users:user1,user2,... starting addresses will be retrieved. By default, this check box is not selected.

      Index Mentions

      Indexes the Tweets mentioning a user specified in a users:user1,user2... starting address. By default, this check box is not selected.

      Note: Only Tweets from the last 7 days will be indexed. Mention Tweets are retrieved by searching for "@username", so parameters set for search query results will also affect the mention Tweets retrieved.

      Example: if SearchResultIsoLanguageCode is set to "fr", only mention Tweets written in French will be retrieved.

      Index Retweets

      Indexes all retweets when retrieving Tweets from a user timeline. By default, this check box is selected.

      Index Replies

      Indexes all replies when retrieving Tweets from a user timeline. By default, this check box is selected.

      Note: When the Index Retweets and Index Replies check boxes are not selected, the number of Tweets retrieved per page will be up to the number specified with the TweetsPerTimelinePage parameter, because results are filtered after being retrieved.

      Keep old items

      With this check box selected, items are not deleted, even if they are not recrawled. By default, this check box is selected. Items are not deleted unless this check box is cleared.

      Note: Old items can be removed by manually removing a folder or an item from the index, clearing this check box or deleting the source.

   4. (Optional) Click Add Parameter when you want to show and change the value of advanced source parameters (see Modifying Hidden Twitter v2 Source Parameters).

   5. The Option check boxes generally do not need to be changed:

      Index Subfolders

      When selected, CES indexes subfolders recursively. By default, this check box is selected.

      Index the document's metadata

      When selected, CES indexes all the document metadata, even metadata that is not associated with a field. Orphan metadata is added to the body of the document so that it can be searched using free text queries.

      When this check box is cleared (default), only system and custom field values for which the Free Text Queries attribute is selected will be searchable without using a field query (see Adding a Field to Search On and What Are Field Queries and Free Text Queries?) .

      Example: A document has the following metadata:

      • LastEditedBy containing the value Hector Smith

      • Department containing the value RH

      In CES, the custom field CorpDepartment is bound to the metadata Department and its Free Text Queries attribute is selected.

      When the Index the document's metadata option is cleared, searching for RH returns the document because a field is indexing this value. Searching for hector does not return the document because no field is indexing this value.

      When the Index the document's metadata option is selected, searching for hector also returns the document because CES indexed orphan metadata.

      Document's addresses are case-sensitive

      Determines if, when two addresses differ only by their casing, they are considered as two different addresses. This check box is cleared by default.

      Generate a cached HTML version of indexed documents

      At indexing time, CES creates HTML versions of indexed documents. In search interfaces, users can then more quickly review the content by clicking the Quick View link rather than opening the original document with the original application. By default, this check box is selected. Consider clearing this check box only when you neither want to use Quick View links nor save resources when indexing the source.

      Open results with cached version

      Leave this check box cleared (recommended) so that, in search interfaces, the main search result link opens the original document with the original application. Consider selecting this check box only when you do not want users to be able to open the original document but only see the HTML version of the document as a Quick View. In this case, you must also select Generate a cached HTML version of indexed documents.

7. Click Save to save the source configuration. You are brought back to the Source page.

8. In the Permissions section of the Source page, select Specifies the security permissions to index and click Apply Changes. You are brought back to the Source page.

   

9. When you are ready to start indexing the Twitter v2 source, click Start.

10. Validate that the source building process is executed without errors:

    • In the navigation panel on the left, click Status, and then validate that indexing proceeds without errors.

      OR

    • Open the CES Console to monitor source building activities (see Using the CES Console).