BigPanda maintains raw data in standard objects that can be accessed through the BigPanda ETL pipeline, or can be exported in CSV format and processed through your own external tools.

Both BigPanda Analytics and the ETL Pipeline make data from the platform available in near real-time. Data will be available on average in 1-2 minutes, but during high data-volume periods may take as many as 6-8 minutes to be available.

❗

️

BigPanda is able to assist with exporting raw data. We recommend working with a data specialist on your team to process the data once it is exported. BigPanda is unable to assist with any raw data processing or sync beyond what is mentioned on this page.

🚧

The exported CSV will not have headers and the source data may change at any time. Ensure your ETL process is able to handle changes in the CSV

To connect to the BigPanda ETL Pipeline, your data tools will need to first be configured.

Files can be accessed through the ETL by syncing or copying the raw files into a bucket of your own for your own ETL to transform and store the data.

Schema

Each of the objects below are available for ETL sync or can be downloaded in CSV format.

• Activities - a single action that a user has been doing
• Change Relations Activities - changes that have been marked as matched or not matched to an Incident
• Changes - Changes that have been sent to and processed by BigPanda
• Change Tags - the tags associated to a single change in BigPanda
• Entities - a single aggregated alert in BigPanda (e.g, a row in the BigPanda timeline view)
• Events - all status changes for a single raw event in BigPanda
• Incidents - a single BP Incident
• Incident Tags - the incident tags associated with a single BigPanda incident
• Maintenance Events - events that matched to BigPanda maintenance plans
• Maintenance Plans - a single BigPanda maintenance plan
• Matchers Log - the correlation patterns that matched and applied to a single BigPanda incident
• RCC Related Changes - a single change suspected as a potential root cause by the BigPanda algorithm for a single incident
• Tags - a single tag of an entity
• User - a single user

In the ETL process, 2 additional columns are added to each table with ETL information:

• _sys_import_time - the timestamp of the sync
• _sys_execution_id - The unique identifier of the sync execution

📘

When using the BigPanda ETL, incident_id and tag_id are the primary identifiers

For details on the attributes for each available object, see the Raw Data ETL Schema documentation.

AWS S3-Based Sync

An S3 Sync can be configured to mirror BigPanda BI analytics information in your organization’s S3 bucket.

When the process is enabled in BigPanda, it starts generating the raw data files in CSV format into the S3 bucket on the BigPanda side. Replication privileges then allow the data to be replicated into the S3 bucket for your organization.

AWS S3 Configuration

1. Contact BigPanda Support and request a product change
2. Create an S3 bucket with versioning enabled
3. Provide BigPanda Support the ARN and AWS account ID
4. BigPanda Support will provide a bucket policy
5. Add the bucket policy to your bucket to begin to receive objects through replication
   a. Click on the bucket
   b. Click on the Permissions tab.
   c. Select Bucket Policy and enter the policy provided by support
   d. Click on the Management tab.
   e. Choose Replication and click More -> Receive objects
   f. Add BigPanda AWS Account ID in the source and click Done
6. Confirm that the objects are received in your S3 bucket

Encryption

There is no encryption on the S3 bucket on BigPanda side. When the objects are replicated into the Customer bucket, they assume the destination privileges/permissions as well as encryption requirements.

If you use the default encryption on your S3 bucket, you need to add the following statement to the KMS key used on the bucket:

{
  "Sid": "Allow BigPanda to Generate Data Key",
  "Action": "kms:GenerateDataKey",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::103749124141:root"
  },
  "Resource": "*"
}

Next Steps

Review the Raw Data ETL Schema

Dig into additional reporting options with BigPanda Analytics