BigPanda Unified Data Connector
The BigPanda Unified Data Connector syncs data from your ITSM platforms to provide context and insight for the AI Incident Assistant, Incident Prevention, and AI Detection and Response. Ingested data is securely stored and made available in the IT Knowledge Graph, powering accurate answers, deep analytics and trend analysis, and advanced capabilities.
The Unified Data Connector can connect to ServiceNow and Confluence.
Assistance required
Setting up a Unified Data Connector requires close coordination with the BigPanda team. Contact your BigPanda account manager if you’d like to enable this feature.
Unified Data Connector Configuration
The unified data connector setup process is multi-step and involves close coordination with BigPanda support. Contact BigPanda support when you are ready to set up a data connector.
ServiceNow Data Connector Set Up
By default, the ServiceNow data connector syncs data from the following ServiceNow tables:
Table | Contents |
|---|---|
change_request | Requests for system modifications |
cmdb_ci | Configuration items in the CMDB |
incident | Incidents |
kb_knowledge | Knowledge base articles |
problem | Root causes of incidents |
sys_attachment | Attachment changes and metadata |
sys_attachment_doc | Attachment binaries |
task_ci | Task-related configuration items |
change_task | Tasks for change requests |
The Unified Data Connector is highly customizable and can sync additional tables.
To begin, reach out to BigPanda support and provide credentials for your ServiceNow instance.
The user account provided must have at least read permission access to all of the tables to sync from ServiceNow.
The account time zone must be set to GMT, and must have the date format set to yyyy-MM-dd in the sys_user table.
These credentials are required:
Field | Description |
|---|---|
Instance | The name of your ServiceNow instance. (i.e. |
Username | Username of the account used to access ServiceNow. |
Password | Password of the account used to access ServiceNow. |
The BigPanda team will use your ServiceNow credential information to set up the data connector. Certain fields and options are customizable depending upon your organization’s preferences and requirements.
Provide the following information about your sync preferences to BigPanda:
Required Configuration
Option | Description |
|---|---|
Fields | Determine which fields you’d like to include from each ServiceNow table. You can also choose to include all fields. |
Mode | Determine whether you’d like to set up an incremental or historical sync. You can also choose to do both sync modes. |
Optional Configuration
Option | Description |
|---|---|
Related Tables | For each table, you can choose to include a related table. For example, if an incident has CIs, you can include the task_ci related table to collect them along with the incident. See the Related Tables section below for a list of related tables and join fields. |
Filter | You can apply query filters to specific tables to granularly choose what data is extracted. Each table’s filter is applied independently. Tables without a filter will load all records. Filters use ServiceNow's encoded query syntax with ^ as AND and OR as OR. We recommend using ServiceNow's query builder to test filters before adding them to the configuration. See the ServiceNow documentation for more information. |
Start Date | If you’re setting up a historical sync, provide the date of when you’d like the sync to begin. If no start date is provided, all data will be synced. We recommend syncing one year of data. |
End Date | Used for historical syncs only. Provide an end date to backfill historical syncs. |
Read Replica | ServiceNow supports read replicas through the If you’d like to set this up, provide your replica name. |
Performance-Related Configuration
Option | Description |
|---|---|
Query Limit | The maximum number of queries that can be sent per minute. The default is 60 queries per minute. |
Rate Limit Timeout (MS) | The timeout period in milliseconds after the query limit has been reached. The default is 1000 ms (1 second). |
Confluence Data Connector Set Up
The Confluence Data Connector syncs data from your Confluence pages. The data connector can be set up with both cloud and server instances of Confluence.
To begin, reach out to BigPanda support and provide credentials for your Confluence instance. The user account provided must have at least read permission access to all of the pages that will be synced from Confluence.
These credentials are required:
Field | Description |
|---|---|
URL | The URL of your Confluence instance ( |
Username | Username of the account used to access Confluence. |
Password | Password of the account used to access Confluence. |
Cloud | Whether or not your instance is Confluence Cloud. (True or False) |
The BigPanda team will use your Confluence credential information to set up the data connector. Certain fields and options are customizable depending upon your organization’s preferences and requirements.
Provide the following information about your sync preferences to BigPanda:
Option | Description |
Space keys | The unique key(s) of your Confluence space(s). The key can be found in your Confluence URL after /spaces/ (i.e.: |
Mode | Determine whether you’d like to set up an incremental or historical sync. You can also choose to do both sync modes. |
Start date | If you’re setting up a historical sync, provide the date of when you’d like the sync to begin. We recommend syncing one year of data. |
CQL filters (Optional) | You can use Confluence Query Language (CQL) to set up content filters. This allows you to extract only the content that matches your criteria. |
Historical Mode vs. Incremental
You can choose to set up either a historical or incremental sync for both the ServiceNow and Confluence data connectors.
Historical syncs allow you to load only data from a selected start date and after. This mode works well for one-time data loads or when you need to backfill data that was previously missed. Historical mode is the default, recommended method of syncing data.
Historical sync
Historical syncs can take minutes or days, depending on the amount of data sent.
Incremental syncs run periodically and load all new or updated records. This mode works well for scheduled, recurring data loads. Depending on what functionality you are using within BigPanda or Biggy, your incremental sync time period may vary.
Frequently Asked Questions
General
Q: How does the Unified Data Connector protect sensitive data and PII?
A: You can create filters for your tables to ensure that sensitive data is not included in the sync. Additionally, a Data Redaction feature will soon be available that will automatically scan for PII and remove it.
Q: How long does the initial sync take?
A: It depends on the rate limit in place, the amount of data synced, and the sync start date. The initial sync can take anywhere between a few hours to multiple weeks.
Q: How often does the data sync?
A: The default sync frequency is set to 2 minutes, but it can be modified. You can set the frequency from 1 minute to 24 hours.
ServiceNow
Q: How do you ensure that APIs don’t bypass manual review or approval processes when writing back into ServiceNow (such as during incident closure), and that audit trails are preserved?
A: The Unified Data Connector does not write anything to ServiceNow. It only uses GET APIs to pull data. Just like other user activity, you can track the user activity of the Unified Data Connector user in the ServiceNow system logs.
Q: What controls are in place to ensure that ServiceNow performance is not affected by large data pulls or frequent polling?
A: The Unified Data connector has the following features in place to control and limit the number of calls and load on the server:
Batch size control - Larger batch sizes result in fewer overall calls to the server.
Max calls per minute - This setting controls the maximum number of calls sent to ServiceNow. After it reaches the maximum, it pauses for a configured amount of time.
Rate limits - ServiceNow blocks users who reach the rate limit. If the rate limit is reached, the Unified Data Connector waits the amount of time returned by ServiceNow in the request header before sending additional queries.
Incremental loads - Every incremental load starts from the state that the last load had completed.
Q: Will customizations break if ServiceNow upgrades or changes its APIs?
A: Only a small number of unique calls are used for the sync.Calls made to ServiceNow are core REST APIs that don’t change often, such as now/table/{table_name}. If ServiceNow makes a change, the fixes to the Unified Data Connector will be minor and happen quickly.