Unified Data Connector self-service configuration
The BigPanda Unified Data Connector syncs data from your tools to provide context and insight for the AI Incident Assistant, AI Incident Prevention, and AI Detection and Response. Ingested data is securely stored and made available in the IT Knowledge Graph, powering accurate answers, deep analytics, trend analysis, and advanced capabilities.
Self-service configuration lets you set up and manage Unified Data Connector syncs directly. Add a connection to store your tool's credentials, then create one or more pipelines that use that connection to collect data on a schedule.
Self-service configuration is available for every tool that has a Unified Data Connector. All data connectors use the same connection and pipeline workflows. Each tool has different tool-specific settings, such as authentication methods and the data each pipeline collects.
Data connectors for the following tools are available:
BMC Helix
BMC Remedyforce
Confluence
Downdetector
HaloITSM
Ivanti Neurons for ITSM
Jira
Jira Service Management
OpenText ITSM
OpenText SMAX
Salesforce
ServiceNow
SFTP
Sharepoint
Snowflake
Zendesk
Data Redaction
You can redact sensitive information ingested via the Unified Data Connector, including PII, PHI, and PCI, giving you confidence that your confidential data remains exclusively within your approved channels. See the Data Redaction documentation for details on managing sensitive data.
Connections and pipelines
Self-service configuration is built on two objects:
Connection: Stores the credentials BigPanda uses to reach your tool. You add a connection once and reuse it across many pipelines. Each connection is tested before it is saved.
Pipeline: Defines what data to collect, how often to collect it, and how far back the first sync reaches. Each pipeline uses one connection.
For example, you can create one connection to your Confluence instance, then create separate pipelines for different groups of spaces, each with its own schedule.
Before you start
Make sure you have the required permissions in BigPanda to manage integrations. See the Roles Management documentation for more information.
Create or identify a service account in your tool with read access to all of the data you want to sync. The connector only collects data that the service account can read.
Gather the connection details for your tool, such as the instance URL, authentication method, and credentials. See the individual data connector page for the exact fields your tool requires.
If your tool already has a real-time integration, you can reuse its stored credentials instead of entering them again.
Access UDC Configuration
UDC configuration can be accessed from the BigPanda console or the Biggy web app.
Access UDC configuration from the BigPanda console
To access the UDC configuration from the BigPanda console:
Click the Integrations tab.
Select UDC Connections.
Find the desired tool in the list, and click Configure.
You can search for a specific tool using the search bar, or filter the list of tools by Category or Status (With active pipelines or No active pipelines).
Access UDC configurations from the web app
To access UDC configuration from the web app:
In the left navigation, go to Manage > Integrations > Manage Integrations.
Find the desired tool in the list. See the Integrations documentation for information about filtering the integrations list.
At the top of the integration page, click the UDC Connections tab.
UDC setup
The UDC Connections tab includes a Setup guide with tool-specific preparation notes and a summary of the four setup stages: prepare the tool, add a connection, create a pipeline, and activate and monitor the pipeline. Click the arrow on the right-hand side of the setup guide to expand or collapse it.
Below the setup guide, the Connections panel shows:
The number of configured connections and pipelines
The BigPanda products currently receiving data from this integration's pipelines, listed next to Works with
A list of existing connections
Each connection in the list shows its name, instance URL, number of pipelines, number of active pipelines, and the date it was last edited.
Add a connection
You can create a connection in one of two ways: copy the credentials from an existing real-time integration, or enter new credentials.
On the UDC Connections tab, click Add Connection. The New connection modal opens.
At the top of the modal, select one of the following options:
Copy from Real Time integration: Reuse credentials BigPanda already stores for this tool.
Enter credentials: Enter credentials specifically for the Unified Data Connector.
Complete the fields for the option you selected.
Click Test connection to confirm BigPanda can reach your tool.
Click Save connection.
When you save, BigPanda tests the credentials first, then stores them as a reusable connection. The new connection appears in the Connections list.
Connections in the list can be edited by clicking the pencil icon, or deleted by clicking the trash can icon.
Create a pipeline
Each pipeline belongs to a connection. You can create a pipeline from the Connections list or from an existing pipeline page.
On the UDC Connections tab, click the arrow on the right-hand side of the connection you want to use. The connection expands to show its pipelines.
Below the list of pipelines, click New pipeline. The New pipeline page opens.
Configure the pipeline settings, schedule, and tool-specific settings. See the following sections for details.
Click Create pipeline.
If any required field is missing, a message next to the Create pipeline button lists what you still need to enter.
New pipelines start paused
A new pipeline does not collect data until you activate it. See the Activate a pipeline section.
At the top of the New pipeline page, an information banner shows setup notes specific to your tool.
Pipeline configuration
In the Pipeline configuration section, enter the following:
Pipeline name: Enter a descriptive name for the pipeline. For example,
Production incidents.Connection: Select the connection whose credentials this pipeline uses. The list defaults to the connection you expanded.
Click Test connection to confirm the selected connection can reach your tool before you create the pipeline. If you need a new connection, click New connection to add one without leaving the page.
Schedule
In the Schedule section, choose how often the pipeline runs.
Select one of the following schedule types:
Interval: Runs the pipeline at a fixed interval. In the Every fields, enter a number and select a unit of time. For example,
30minutes.Daily: Runs the pipeline once per day. Select the time when the run should start.
Time range: Runs the pipeline only during a defined window. Select the window, and how often it should run during that window.
Custom cron: Runs the pipeline on a schedule defined by a cron expression.
Next to the schedule types, a summary shows how often the pipeline will run. For example, Every 30 minutes · 48 runs per day.
Below the schedule type, select the Timezone that should be used for the schedule.
In the Initial sync window list, select how far back the first sync reaches.
The initial sync collects historical data from the selected window. After the first run, each scheduled run collects only new or updated records.
Sync at least 12 months of data
For the best results with AI-driven insights and trend analysis, sync at least 12 months of historical data.
Choose the initial sync carefully
The initial sync window is locked once the pipeline has run. To collect older data after that, use the Backfill tab. See the Backfill older data section.
Tool-specific settings
Below the Schedule section, the New pipeline page shows settings that control what data the pipeline collects. These settings are different for each tool. For example, a Confluence pipeline asks for a list of space keys and whether to collect page attachments.
See the individual data connector page for your tool for a description of each setting.
Activate a pipeline
Click Create Pipeline when you're ready to save your settings. The Paused toggle appears at the top of the page; toggle it off to activate the pipeline.
After you activate a pipeline, its status changes from Paused to Active in the expanded connection list.
Monitoring
Use the Monitoring tab to track a pipeline's sync activity, see where its data goes, and review recent runs. The Monitoring tab shows the same layout for every data connector. The table or object names shown in each section depend on what the pipeline collects.
On the right-hand side of the tab bar, select the time period for the metrics on the tab: Last 7 days, Last 30 days, or Last 90 days.
Summary
At the top of the Monitoring tab, three cards summarize pipeline activity for the selected time period:
Records synced: The total number of records the pipeline collected, with a count for each table or object listed under By table.
Last record: The date and time of the most recently synced record, with the latest timestamp for each table or object listed under By table.
Success rate: The percentage of pipeline runs that completed successfully, with the number of Total runs and Successful runs.
Data lineage
The Data lineage section shows where the pipeline's data comes from and which BigPanda platforms it reaches. Each stage of the flow appears as a node, from the source tool on the left to the BigPanda platforms on the right. The lines between nodes show the number of records passed from one stage to the next.
A status icon above each node shows the health of that stage. For example, a check mark indicates the stage is working successfully.
Click any stage in the diagram to open a window showing additional details.
Pipeline activity
The Records per day section shows the number of records the pipeline synced each day during the selected time period. If the pipeline didn't sync any records during that time, the section shows No records synced in this window.
Recent runs
The Recent runs section lists the pipeline's most recent runs, with the newest run first. Each run includes the following information:
Status: The result of the run. For example, Success.
Started: The date and time the run started.
Duration: How long the run took.
Records: The number of records collected during the run.
Size: The amount of data collected during the run.
Error: The error message, if the run failed.
Successful runs with zero records
A run can be completed successfully and still collect zero records. This is expected when there are no new or updated records since the previous run. If runs consistently return zero records when you expect new data, confirm that the service account has read access to the data the pipeline collects
Backfill
Use the Backfill tab to collect data older than the pipeline's initial sync window, or to re-collect data for a specific time period. For example, after a data correction or migration in your tool, or after a gap caused by an outage.
A backfill pulls records from a past date range using the pipeline's existing connection and settings. The backfill splits the date range into smaller chunks and processes them one chunk at a time.
Backfills start immediately
A backfill starts as soon as you click Start backfill. Backfills aren't part of the pipeline configuration, so you don't need to click Save pipeline, and saving the pipeline doesn't start or change a backfill.
To start a backfill:
On the UDC Connections tab, click the arrow on the right-hand side of the connection to show its pipelines.
Click the pipeline you want to backfill. The pipeline page opens.
At the top of the pipeline page, click the Backfill tab.
In the Historical backfill section, enter the following:
Start date: The first date of the range to backfill.
End date: The last date of the range to backfill.
Chunk size: The number of days of data to process in each chunk. For example, enter
7to process the date range one week at a time.Tables to backfill: The tables or objects to include in the backfill. The list includes the tables or objects the pipeline collects. To remove one, click the X next to its name.
(Optional) Turn on Custom schedule to control when the backfill runs. Define the cadence for the job.
Click Start backfill.
Audit log
Use the Audit log tab to see a history of changes made to a pipeline, including who made each change and when. The audit log helps you track configuration changes, confirm when a pipeline was paused or activated, and investigate unexpected changes in sync behavior.
The Audit log lists every change to the pipeline, with the most recent change first. Each entry includes the following information:
When: The date and time of the change.
User: The email address of the user who made the change.
Action: The type of change. For example, created when the pipeline was created, or updated when its settings changed.
Changes: The fields that changed, with the previous value and the new value.
Troubleshoot self-service configuration
For common connection errors, such as 401 Unauthorized responses, zero-record results, and drops in record volume, see the Troubleshoot connection issues section of the BigPanda Unified Data Connector documentation.
If Test connection fails:
Confirm the instance URL is correct and reachable.
Confirm the credentials are current and haven't been rotated, locked, or expired.
Confirm the service account has read access to the data the pipeline is configured to collect.
If you copied credentials from a Real Time integration, confirm that the Real Time integration is still working.
Next steps
BigPanda Unified Data Connector: Learn about streaming modes and view the full list of available data connectors.
Confluence Data Connector Set Up: Configure tool-specific connection and pipeline settings for Confluence.
Data Redaction with UDC: Redact sensitive information, including PII, PHI, and PCI, from ingested data.