Skip to main content

BigPanda Biggy AI and ServiceNow Context Sync

The BigPanda ServiceNow Context Sync app syncs ITSM data from specific tables in ServiceNow to provide context to Biggy AI. Ingested data is securely stored and made available to Biggy AI, allowing it to accurately answer questions and provide analytics and trend analysis capabilities.

Once installed, an initial data load is performed based on the configured settings, and then the data is incrementally synced based on the selected interval frequency.

Role

The ServiceNow role x_bip_bigpanda_ai.bigpanda is required to complete the installation.

Throttling

To minimize any impact on ServiceNow instance performance, we have optimized the context sync process using the following measures:

  • Sequential Processing - The sync runs on a single thread, sequentially processing tables in batches of 1,000 records.

  • Rate-limited data loads 

    • Initial loads - Data loads are limited by year, processing one year per execution.

    • Execution cap - Executions are capped at a maximum of 300,000 rows, regardless of the time range. If 300,000 rows are exceeded, the load will continue in the next execution. Only one execution occurs at a time.

  • Memory efficiency 

    • Table data - Transactions are limited to approximately 16MB per transfer to conserve memory on the processing instance.

    • Attachments - For attachments (if enabled), we use the native transfer facility available via setBodyFromAttachment.

By balancing throughput with resource constraints, this approach ensures minimal compute and memory usage, allowing the sync process to run without a noticeable impact on the ServiceNow instance. This results in longer initial sync times, but ensures system performance integrity. 

The one-time initial sync sequentially sends the records based on the configured historical timeframe. Subsequent executions run once every 2 hours by default (this is configurable) and will only send the new/updated records for the interval duration (for example, the last 2 hours). These incremental syncs rarely include more than 1000 records.

Install the App

Dev installation

We recommend initially installing the app in your dev ServiceNow instance to verify data connectivity before installing it in production.

Download the Biggy AI Context Sync app from the ServiceNow Developer Portal. Then, install the Update Set in your ServiceNow instance.

Update the App Settings

After you have installed the app, navigate to the App Settings page to configure the app, select what is synced, and how often.

To update the app settings:

  1. In the ServiceNow top navigation bar, go to the right side of the page, click the Globe icon , and select BigPanda Context Sync to assume the correct application scope. 

    snow_biggy_applicationscope.png

  2. In the BigPanda Context Sync page, enter x_bip_bigpanda_ai.bigpanda context sync into the Suffix and Name fields and click Save.

  3. In the left navigation, click the filter box and enter bigpanda context. Under BigPanda Context Sync, select Configuration.

    snow_biggy_config.png

  4. In the API Key field, enter your BigPanda API Key. If you need an API Key, contact your BigPanda admin.

    Sandbox instances

    When configuring the app in your dev ServiceNow instance, use an API key from your sandbox BigPanda instance.

  5. (Optional) In the Tables and Views section, the currently supported tables will be present by default. You can add additional tables by clicking Add New. This allows the additional tables to be synced. 

    Ensure that the user adding additional tables has the role x_bip_bigpanda_ai.bigpanda context sync.

  6. (Optional) You can add Query Filters for each table to limit what is sent to BigPanda. For example, you could limit the incident table to only send P1, P2, and P3 incidents.

    Test queries

    We recommend testing your query filters before adding them to the context sync. You can validate a query using the Search function in ServiceNow. 

    After you've determined that the query works as expected, right-click it and select Copy Query to copy the query and paste it into the app settings.

  7. (Optional) In the Related Tables section, you can define [related_table]:[field_name] values in a comma-separated list. The sync mechanism will include the associated records from the related tables when it pushes data by matching the sys_id from the field_name of the primary table with the sys_id of the records in the related_table.

    Related tables values

    We recommend using the default values for the Related Tables column and consulting with the BigPanda team before changing them to ensure the desired outcome is achieved.

  8. In the Months to Sync field, select the number of months in the past that you want the initial sync to load.

  9. (Optional) In the Sync Execution Interval field, you can define the frequency that incremental data pushes will be performed, in minutes.

    For example, if set to 30, it will run every 30 minutes, sending any new or updated records from the previous 30 minutes.

    Default settings

    We recommend keeping all other settings as the default values.

  10. Click Submit to update the settings.

  11. After updating the settings, reach out to BigPanda to verify that data is flowing properly.

Initial data load

The initial data load may take a few hours depending on how much data is included. Once complete, the incremental syncs will happen automatically at the configured interval.

Release Notes

V2.7 (4/2/2025) - Current Version

  • Improved performance on initial data load

  • Improved reliability, introduced backoff retry logic to handle rate-limiting scenarios

  • Optimized telemetry capture for more efficient auditability

V2.7 (12/6/2024)

  • Improved restart and continue logic (initial load)

  • Added large file chunking. Files larger than 16mb-4k are now split.

  • Expanded query filter support for data exclusion

V2.5.5 (11/2/2024)

  • Added initial load throttling

    • Spreads the initial load over multiple sequential executions to ensure minimal performance impact on the ServiceNow instance

  • Added support for complex query filters

    • Nested OR and AND operations supported

    • Added multiple wildcard support

V2.4.0 (10/20/2024)

  • Support attachment syncing for all tables (optional setting)

    • Added safeguard limitations on excessively large file sizes

  • Improved configuration page UX

V2.0.3 (10/3/2024)

  • Implemented optimizations to accommodate large-size payloads and a high volume of records. This reduces overall number of requests being sent and improves speed 16x over v1.

  • Syncing to associated task tables (ie, change task) is now supported.

  • Improved overall UX of the Settings page:

    • More clearly defined labels, disclaimers, and warnings.

    • Added an initial execution button to run the historical sync on-demand.

    • Updated the sync interval format to use minutes instead of hours, allowing for more granular control.

  • Optimized internal data storage structure used during mid-flight storage and increased data isolation.

  • Enabled optional streaming of errors, execution durations, and record quantities for improved telemetry statistics and traceability.

  • Added compatibility with the ServiceNow Xanadu SP1 release and Washington DC SP5+ releases.

V1.0.0 (8/15/2024)

  • Initial implementation.

  • Created settings page for configurable sync options.

  • Added secure data transmission and storage.

In this section: