Indexing all Knowledge Base Versions and States

A Knowledge Base (KB) article has a state and a version. By default, the connector fetches the latest version of an article with the online (published) status.

The fields PublishStatus and IsLatestVersion respectively control which article version and state are fetched. Fetching all the versions and states means:

• Fetch the online articles

• Fetch the drafts articles

• Fetch the latest archived articles

• Fetch other archived articles

The following ObjectsToGet configuration file example for an article of type Documentation gets Draft, Online, and Archived articles.

Depending on your CES version:

• CES 7.0.7914+ (December 2015)

  Note: When you choose to not index articles of a certain publish status (draft, online, archived), articles are deleted during an incremental refresh whether the isLatestVersion parameter value is true or false. The deletion also occurs when an article changes status.

  Example: You choose to only index draft articles, so when an article changes status from draft to online, the article is deleted during the next incremental refresh.

  <?xml version="1.0" encoding="utf-8"?>
  <ArrayOfQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <Query>
      <ObjectName>Documentation__ka</ObjectName>
    </Query>
    <Query>
      <ObjectName>Documentation__kav</ObjectName>
      <Fields>
        <string>Id</string>
      </Fields>
      <Conditions>
        <InCondition>
          <Field>PublishStatus</Field>
          <AllowedValues>
            <SoqlString>Draft</SoqlString>
            <SoqlString>Online</SoqlString>
            <SoqlString>Archived</SoqlString>
          </AllowedValues>
        </InCondition>
      </Conditions>
    </Query>
    <Query>
      <ObjectName>Documentation__kav</ObjectName>
      <Fields>
        <string>Id</string>
      </Fields>
      <Conditions>
        <QueryCondition>
          <Field>IsLatestVersion</Field>
          <Operator>=</Operator>
          <Value>False</Value>
        </QueryCondition>
      </Conditions>
    </Query>
  </ArrayOfQuery>

• CES 7.0.7914– (October 2015)

  <?xml version="1.0" encoding="utf-8"?>
  <ArrayOfQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <Query>
      <ObjectName>Documentation__ka</ObjectName>
      <Fields>
        <string>Id</string>
        <string>LastPublishedDate</string>
      </Fields>
    </Query>
    <Query>
      <ObjectName>Documentation__kav</ObjectName>
      <Fields>
        <string>Id</string>
        <string>Title</string>
        <string>PublishStatus</string>
        <string>IsLatestVersion</string>
      </Fields>
      <Conditions>
        <QueryCondition>
          <Field>PublishStatus</Field>
          <Operator>=</Operator>
          <Value>'Archived'</Value>
        </QueryCondition>
        <QueryCondition>
          <Field>IsLatestVersion</Field>
          <Operator>=</Operator>
          <Value>False</Value>
        </QueryCondition>
      </Conditions>
    </Query>
    <Query>
      <ObjectName>Documentation__kav</ObjectName>
      <Fields>
        <string>Id</string>
        <string>Title</string>
        <string>PublishStatus</string>
        <string>IsLatestVersion</string>
      </Fields>
      <Conditions>
        <QueryCondition>
          <Field>PublishStatus</Field>
          <Operator>=</Operator>
          <Value>'Archived'</Value>
        </QueryCondition>
        <QueryCondition>
          <Field>IsLatestVersion</Field>
          <Operator>=</Operator>
          <Value>True</Value>
        </QueryCondition>
      </Conditions>
    </Query>
    <Query>
      <ObjectName>Documentation__kav</ObjectName>
      <Fields>
        <string>Id</string>
        <string>Title</string>
        <string>PublishStatus</string>
        <string>IsLatestVersion</string>
      </Fields>
      <Conditions>
        <QueryCondition>
          <Field>PublishStatus</Field>
          <Operator>=</Operator>
          <Value>'Draft'</Value>
        </QueryCondition>
      </Conditions>
    </Query>
    <Query>
      <ObjectName>Documentation__kav</ObjectName>
      <Fields>
        <string>Id</string>
        <string>Title</string>
        <string>PublishStatus</string>
        <string>IsLatestVersion</string>
      </Fields>
      <Conditions>
        <QueryCondition>
          <Field>PublishStatus</Field>
          <Operator>=</Operator>
          <Value>'Online'</Value>
        </QueryCondition>
      </Conditions>
    </Query>
  </ArrayOfQuery>

Excluding Archived Knowledge Base Articles

CES 7.0.7914– (October 2015)

Archived KB articles can be removed in an incremental refresh using a destructive query as shown in the following ObjectsToGet configuration file example. This query will remove previously indexed documents from the index instead of adding/updating them.

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Query>
    <ObjectName>Documentation__ka</ObjectName>
    <Fields>
      <string>Id</string>
      <string>LastPublishedDate</string>
    </Fields>
  </Query>
  <Query>
    <ObjectName>Documentation__kav</ObjectName>
    <Fields>
      <string>Id</string>
      <string>Title</string>
    </Fields>
  </Query>
  <Query>
    <ObjectName>Documentation__kav</ObjectName>
    <FoundRecordsAreDeleted>true</FoundRecordsAreDeleted>
    <Conditions>
      <QueryCondition>
        <Field>PublishStatus</Field>
        <Operator>=</Operator>
        <Value>'Archived'</Value>
      </QueryCondition>
      <QueryCondition>
        <Field>IsLatestVersion</Field>
        <Operator>=</Operator>
        <Value>True</Value>
      </QueryCondition>
    </Conditions>
  </Query>
  <Query>
    <ObjectName>Documentation__kav</ObjectName>
    <FoundRecordsAreDeleted>true</FoundRecordsAreDeleted>
    <Conditions>
      <QueryCondition>
        <Field>PublishStatus</Field>
        <Operator>=</Operator>
        <Value>'Archived'</Value>
      </QueryCondition>
      <QueryCondition>
        <Field>IsLatestVersion</Field>
        <Operator>=</Operator>
        <Value>False</Value>
      </QueryCondition>
    </Conditions>
  </Query>
</ArrayOfQuery>

Indexing Knowledge Base Attachments

The Salesforce connector processes attachments on KB articles differently compared to normal objects (which rely on the Attachment object). Attachments on KB articles are automatically indexed when fields ending with __Name__s are specified in the ObjectsToGet configuration file such is in the following example to index the attachment of an article of type Documentation.

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Query>
    <ObjectName>Documentation__ka</ObjectName>
    <Fields>
      <string>Id</string>
    </Fields>
  </Query>
  <Query>
    <ObjectName>Documentation__kav</ObjectName>
    <Fields>
      <string>Id</string>
      <string>Attachment__Name__s</string>
    </Fields>
  </Query>
</ArrayOfQuery>

An entry in the mapping file must also exist. As you can see below, the name of the mapping is [NameOfArticleType] Attachment:

<?xml version="1.0" encoding="Windows-1252" ?>
<SalesForce>
  ...
  <Mapping type="Documentation Attachment">
    <ContentType>binarydata</ContentType>
    ...
  </Mapping>
  ...
</SalesForce>

Fetching Records With Parent Relationships That Must Be Incrementally Updated

It is not uncommon to need to index a record with fields referring to a parent of a child relationship. The traditional way to achieve this is to use the <ChildRelationships/>, <ParentRelationships/> or <PolymorphicRelationships/> elements in a Query definition. However, using those element will give you records that will not be refreshed correctly. When record A references a field of the record B and record B changes, record A won't be updated in an incremental refresh.

Let's take a specific example, where an Opportunity is referring to an Account name. The following ObjectsToGet configuration file example produces one Opportunity record with the metadata ["Id", "Account.Name"]:

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Query>
    <ObjectName>Opportunity</ObjectName>
    <Fields>
      <string>Id</string>
    </Fields>
    <ParentRelationships>
      <ParentRelationship>
        <RelationshipName>Account</RelationshipName>
        <Fields>
          <string>Name</string>
        </Fields>
      </ParentRelationship>
    </ParentRelationships>
  </Query>
</ArrayOfQuery>

You can solve this problem using the foreign keys (see About Foreign Keys). You must create two queries instead of one. The following ObjectsToGet configuration file example produces one Opportunity record with the metadata ["AccountId"] and one Account record with the metadata ["Id", "Name"].

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Query>
    <ObjectName>Opportunity</ObjectName>
    <Fields>
      <string>AccountId</string>
    </Fields>
  </Query>
  <Query>
    <ObjectName>Account</ObjectName>
    <Fields>
      <string>Id</string>
      <string>Name</string>
    </Fields>
  </Query>
</ArrayOfQuery>

With metadata mapped to fields as follow:

Opportunity
  AccountId: sfaccountid
Account
  Id: sfaccountid
  Name: sfaccountname

<PhysicalIndex>
  ...
  <ForeignKeys>
    <ForeignKey ID="1">
      <KeyField>sfaccountid</KeyField>
      <ValueField>sfaccountname</ValueField>
      <FreeTextSearch>false</FreeTextSearch>
    </ForeignKey>
  </ForeignKeys>
  ...
</PhysicalIndex>

Fetching Child Relationship to Use for Folding

Fetching child relationships which are used in folding (aka: no facet) so they play well with incremental refresh is done through results folding (see About Results Folding).

Let's take a specific example, where a Case is referring to CaseComments. Similarly to the fetching parent relationship case, avoid using the <ChildRelationships/> element as shown below and producing one Case record with the metadata ["Id", "CaseComments.CommentBody"].

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Query>
    <ObjectName>Case</ObjectName>
    <Fields>
      <string>Id</string>
    </Fields>
    <ChildRelationships>
      <Query>
        <ObjectName>CaseComments</ObjectName>
        <Fields>
          <string>CommentBody</string>
        </Fields>
      </Query>
    </ChildRelationships>
  </Query>
</ArrayOfQuery>

Rather use two or more queries:

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Query>
    <ObjectName>Case</ObjectName>
    <Fields>
      <string>Id</string>
    </Fields>
  </Query>
  <Query>
    <ObjectName>CaseComment</ObjectName>
    <Fields>
      <string>Id</string>
      <string>ParentId</string>
    </Fields>
  </Query>
</ArrayOfQuery>

Now all that is left to do is to use the user interface folding component to fold CaseComments under Case documents like in the Coveo JavaScript Search (see Folding Component).

Fetch Child Relationship to Use to Increase or Reduce Results From Queries

Fetching child relationships which are used to increase or reduce the number of results (aka: no facet) so they play well with incremental refresh is done through nested queries.

Like in other cases, avoid using the <ChildRelationships/> element and rather use two or more queries. You can use the example from the previous recipe for the ObjectsToGet configuration file.

You can then use the nested query in the query parameters or the query extension language of the user interface.

A simple, but real use case is to query the Cases AND CaseComments documents but only return Cases as follows: 

@sfid=[[@syssfcaseid] @objecttype=CaseComment q] // Where q is the keyword.

Minimizing API Calls and Object Description Prefetching

By default, the connector prefetches the descriptions of every object in the Salesforce organization to minimize the number of API calls used based on the following assumptions:

• Fetching an object costs 1 API call.

• Prefetching 200 objects costs 1 API call.

• An organization typically has 500-1000 objects.

• An ObjectsToGet configuration file typically queries 10-20 objects.

Prefetching object descriptions does not always minimize API calls.

You can turn prefetching off by setting the hidden Salesforce source parameter PrefetchObjectDescriptions to False (see PrefetchObjectDescriptions (Boolean)). The formula to determine the optimal value for the PrefetchObjectDescriptions parameter is: 

PrefetchObjectDescriptions = ObjectsToGet.Objects.Count > Organization.Objects.Count / 200

Fetching Records Faster Using the Turbo Mode Runner

For some queries, this process can take days. The alternative solution in such cases is to use the TurboMode runner.

<Query>
  <Runner>TurboMode</Runner>
  ...
</Query>

SELECT Id, Subject, (SELECT Email FROM Shares) FROM Account

SELECT Id FROM Account

SELECT Id, Subject, (SELECT Email FROM Shares) FROM Account WHERE Id IN (...)

Reducing the Metadata Package Size by Scoping Parents of ContentVersion and Attachment Objects

ContentVersion and Attachment objects look up their parent objects to resolve record permissions. The default behavior of the connector is to fetch the metadata package for all parents of those objects to determine their sharing models. For some Salesforce organizations, the fetched metadata package can become too big.

You can specify a special InCondition on the field ParentObjectType in the ObjectsToGet configuration file to scope the parents of a record to specific values and consequently, reduce the size of the metadata package to fetch.

<Query>
  <ObjectName>ContentVersion</ObjectName>
  <Fields>
    <string>Id</string>
  </Fields>
  <Conditions>
    <InCondition>
      <Field>ParentObjectType</Field>
      <AllowedValues>
        <SoqlString>Case</SoqlString>
        <SoqlString>Opportunity</SoqlString>
      </AllowedValues>
    </InCondition>
  </Conditions>
</Query>

Choosing the Optimal Record Modification Date Field

When indexing records, you can select from multiple fields such as the following to set the record modification date: 

• SystemModstamp: Updated when a user or a script modifies the record.

• LastModifiedDate: Updated when a user modifies the record.

• CreatedDate: Set when the record is created.

The preferred field is SystemModstamp because it is updated more often.

Removing Leading Zeroes of a Field

When a field is an Auto Number, the connector produces an extra metadata with the suffix __stripped that contains the value without the leading zeros. You can use this field when you want to show or use the values without the leading zeroes.

MyField__c: 0000123
MyField__c__stripped: 123

Using the Currency Field Converter

The connector is able to interpret currencies. When the user crawling the source is configured with a Single Currency mode, extra metadata is generated for records.

When a field is a currency, its value is converted to the user currency and a new metadata is created with the suffix _converted.

Using the FiscalYearResolver

When a field is a date and has a value, the fiscal year resolver creates additional metadata on the records with the following suffixes:

__fiscal_year
__fiscal_quarter
__fiscal_month
__fiscal_week
__fiscal_pretty_quarter

You can turn off the creation of these extra metadata by setting the LoadFiscalYearMetadata hidden source parameter to False (see LoadFiscalYearMetadata (Boolean)).

Fixing the Feed Tracking Error

You can get a typical Salesforce source error that looks like:

Error with ID 'SALESFORCE_INVALID_QUERY': Cannot find child relationship 'Feeds' on object 'Products/Licenses' ('Case'). Make sure 'Feed Tracking' is enabled for this object in Salesforce.

When indexing Chatter items, the connector checks the parent related to field on all Chatter objects to differentiate a Chatter object from a normal object. Some objects do not have the child relationship 'Feeds' because the feed tracking is disable. This relationship is how the connector obtains related Chatter object.

You can resolve this error type by activating feed tracking (see Customizing Chatter Feed Tracking).

Indexing More Than the Built-in FeedItem Types

By default, the connector indexes the following Chatter feed types: TextPost, LinkPost, ContentPost, and PollPost.

In the ObjectsToGet configuration file, you can override this behavior using an InCondition in the query definition of FeedItem.

<Query>
  <ObjectName>FeedItem</ObjectName>
  <Fields>
    <string>Id</string>
  </Fields>
  <Conditions>
    <InCondition>
      <Field>Type</Field>
      <AllowedValues>
        <SoqlString>TextPost</SoqlString>
        <SoqlString>TrackedChange</SoqlString>
      </AllowedValues>
    </InCondition>
  </Conditions>
</Query>