Raw Data ETL Schema

Raw data in standard objects can be accessed via the BigPanda ETL pipeline, or can be exported in CSV format and processed through your own external tools.

BigPanda maintains raw data in standard objects available for export as needed for reporting or building predictive models.

ETL Raw CSV SchemaETL Raw CSV Schema

ETL Raw CSV Schema

The raw data can be accessed through the BigPanda ETL pipeline, or can be exported in CSV format and processed through your own external tools.

Some columns may appear in different order when accessed through a reporting tool or from the ETL Warehouses. When using the Raw CSV files, refer to the CSV Index # Column to ensure the correct fields are parsed.

🚧

The exported CSV will not have headers and the source data may change at any time. Ensure your ETL process is able to handle changes in the CSV.

In the ETL process, additional columns may be added with ETL information:

  • _sys_import_time - the timestamp of the sync
  • _sys_execution_id - The unique identifier of the sync execution

To learn more about configuring an ETL sync for your data, see the ETL Sync Documentation.

The ETL schema tables can also be downloaded in an XLSX file. Contact the support team for a copy of the file.

📘

When using the BigPanda ETL, incident_id and tag_id are typically the primary identifiers. The tables below have bolded the column names that make up the primary keys for each reporting table

activities

A single action a user performed in BigPanda.

Column NameDescriptionData TypeCSV Index #Notes
activity_id The unique identifier of the ActivityString00
typeThe type of activityString1Format is: incident#ACTION

Assigned, Commented, External-Resolve, Manual-Resolve, Merge, Shared, Snoozed, Split, Unassigned, Unsnoozed, Become-Flapping
created_timeThe unix time the action was takenFloat2
incident_idThe unique identifier of the incident the action was taken onString3
creator_user_idThe unique identifier of the user who performed the activityString4
environmentThe name of the environment the activity took place inString5
annotationThe comment that was included with the activityString6
_sys_updated_dateThe last time the database record was updated in unix timeFloat7
assignee_idThe user assigned by the activityString8“Assign”-activity specific
share_targetThe name of the system the related incident was shared toString9“Share”-activity specific

Can be used to filter
_sys_offsetSystem field - can be ignoredInteger10
is_flappingWhether or not the related incident is in the flapping stateBoolean11
_sys_import_timeThe time of the last import to redshiftStringThis is the field used for redshift syncing

change_relations_activity

Actions in BigPanda that mark a change as related to an incident.

Column NameDescriptionData TypeCSV Index #Notes
change_relations_idThe unique identifier of the relationString11
change_idThe unique identifier of the related changeString0
incident_idThe unique identifier of the related incidentString1
user_idThe unique identifier of the user who performed the actionString2
created_atThe unix time when the change relation was marked in BigPandaFloat3
updated_atThe unix time when the change relation was last updated in BigPandaFloat4
match_typeThe type of matchString5MANUAL
match_certaintyThe confidence level of the matchString6None, Suspect, or Match
suggested_by_bigpandaNot currently in useBoolean7
commentA comment included with the actionString8
_sys_updated_dateThe last time the database record was updated in unix timeFloat9
_sys_offsetSystem field - can be ignoredInteger10
_sys_import_timeThe time of the last import to redshiftStringThis is the field used for redshift syncing

changes

A change that was sent to and processed by BigPanda.

Column NameDescriptionData TypeCSV Index #Notes
change_idThe unique identifier of the changeString0
created_atThe unix time when the change was created in BigPandaFloat1
updated_atThe unix time when the change was last updated in BigPandaFloat2
change_typeWhether a new change record was created or an existing change record was updated.String3change#create, or change#update
identifierSystem identifier for the changeString4
source_systemThe system that sent this change to BigPandaString5
ticket_urlThe URL to the change in the change control system, if applicableString6
start_timeThe unix time for the change startFloat7
end_timeThe unix time for the change endFloat8
summaryA summary of the changeString9
statusThe status of the changeString10Planned, In Progress, Done, or Canceled
_sys_updated_dateThe last time the database record was updated in unix timeFloat11
_sys_offsetSystem field - can be ignoredInteger12
_sys_import_timeThe time of the last import to redshiftStringThis is the field used for redshift syncing

change_tags

A tag related to a single change in BigPanda.

Column NameDescriptionData TypeCSV Index #Notes
change_idThe unique identifier of the changeString0
tag_nameThe tag nameString1
tag_valueThe tag valueString2
_sys_updated_dateThe last time the database record was updated in unix timeFloat3
_sys_offsetSystem field - can be ignoredInteger4
_sys_import_timeThe time of the last import to redshiftStringThis is the field used for redshift syncing

entities

Each entity is a single aggregated alert in BigPanda. Each entity appears as their own row in the BigPanda timeline view. If incidents are merged, the incident id will be updated to the destination incident id for all merged entities.

Column NameDescriptionData TypeCSV Index #Notes
entity_id The unique identifier of the entityString0
incident_idThe unique identifier of the incident to which this entity belongsString1
source_systemThe monitor system that sent this alertString2
start_timeThe unix time the alert startedFloat3
last_change_timeThe unix time for the last change made to an alertFloat4
end_timeThe unix time for the resolution of the alertFloat5
statusThe current status of the entityString6Ok, Critical, or Warning
highest_severityThe highest status of the entityString7Can be used to find entities that were critical at any point
last_severity_change_timeThe unix time for the last severity changeFloat8
num_status_changesThe number of times the entity has changed statusInteger9Limited to ~70 events
Changes to the “ok” statuses are not counted
_sys_updated_dateThe unix time that BigPanda extracted this record to the ETL pipelineFloat10
_sys_offsetSystem field - can be ignoredInteger11
primary_propertyThe main title of the related incident in the UIString12The tag key treated as the primary property
secondary_propertyThe secondary title of the related incident in the UIString13The tag key treated as the secondary property
incident_keyA unique id BigPanda uses to recognize if two events are related to each otherString14Splice the primary property’s value and the secondary property’s value
_sys_import_timeThe time of the last import to redshiftStringThis is the field used for redshift syncing

events

A raw event arriving in BigPanda. Only status changes are included: for instance, if a duplicate event for an already critical alert fires, it will not appear in this table.

Column NameDescriptionsData TypeCSV Index #Notes
event_idThe unique identifier of the eventString0
entity_idThe unique identifier of the entity to which this event belongsString1
timestampThe unix time of the eventFloat2
descriptionThe raw description sent with the eventString4
statusThe status for the eventString3
_sys_updated_dateThe last time the database record was updated in unix timeFloat5
_sys_offsetSystem field - can be ignoredInteger6
primary_propertyThe main title of the event in the UIString7
secondary_propertyThe secondary title of the incident in the UIString8
incident_keyA unique id BigPanda uses to recognize if two events are related to each otherString9
_sys_import_timeThe time of the last import to redshiftStringThis is the field used for redshift syncing

incidents

A single BigPanda incident.

Column NameDefinitionData TypeCSV Index #Notes
incident_idThe unique identifier of the incidentString0
start_timeThe unix time the incident startedFloat1
last_change_timeThe unix time for the last status changeFloat2
end_timeThe unix time for the resolution of the incidentFloat3
statusThe current status of the incidentString4Ok, Critical, or Warning
highest_severityThe highest status of the incidentString5Can be used to differentiate incidents that were critical at any point
_sys_updated_dateThe last time the database record was updated in unix timeFloat6
_sys_offsetSystem field - can be ignoredInteger7
_sys_import_timeThe time of the last import to redshiftStringThis is the field used for redshift syncing
manual_correlationWhether the incident has been manually merged or splitString8
created_atSystem field - can be disregardedString9System field - can be disregarded

maintenance-events

A single event that matched a maintenance plan. Maintenance_plan was the old naming convention for maintenance plans V1.

Column NameDescriptionData TypeCSV Index #Notes
event_idThe unique identifier of the incoming eventString0
entity_idThe unique identifier of the entity to which this event belongsString1
plan_idThe unique identifier of the plan that this event matchesString2
timestampThe unix time of the eventFloat
_sys_updated_dateThe last time the database record was updated in unix timeFloat3
_sys_offsetSystem field - can be ignoredInteger4
_sys_import_timeThe time of the last import to redshiftStringThis is the field used for redshift syncing

maintenance-plans

A single maintenance plan in BigPanda. Maintenance_plan was the old naming convention for maintenance plans V1.

Column NameDescriptionData TypeCSV Index #Notes
maintenance_plan_idThe unique identifier of the maintenance planString0
plan_nameThe name of the planString1
mantainance_keyThe key of the maintenance planString2Displayed in the UI
start_timeThe unix time when the plan will startFloat3
end_timeThe unix time when the plan will endFloat4
plan_creation_dateThe unix time that the plan was createdFloat5
created_byThe unique identifier of the user who created the planString6
descriptionA brief summary of the planString7
last_update_dateThe unix time when the plan was last updatedInteger8
last_updated_byThe unique identifier of the user who last updated the planString9
_sys_updated_dateThe last time the database record was updated in unix timeFloat10
_sys_offsetSystem field - can be ignoredInteger11
_sys_import_timeThe time of the last import to redshiftFloatThis is the field used for redshift syncing

matchers_log

The correlation patterns that matched and applied to a single BP incident

Column NameDefinitionData TypeCSV Index #Notes
incident_idThe unique identifier of the incidentString0
matchers_idID of correlation pattern (can be input into a URL)String2
_sys_offsetSystem field - can be ignoredInteger6
_sys_import_timeThe time of the last import to redshiftFloatThis is the field used for redshift syncing
_sys_execution_idSystem field - can be ignoredString
matchers_log_offsetSystem field - can be ignoredInteger1
tagsAn array of tags applied by the correlation patternsArray of Strings3
matchers_windowMax time window to correlate alertsInteger4
source_systemThe system that sent this change to BigPandaString5
_sys_updated_dateSystem field - can be ignoredInteger7

rcc_related_changes

A single change suspected as potential root cause by the BigPanda algorithm for a single incident

Column NameDescriptionData TypeCSV Index #Notes
change_idThe unique identifier of the related changeString0
incident_idThe unique identifier of the related incidentString1
_sys_offsetSystem field - can be ignoredInteger6
_sys_import_timeThe time of the last import to redshiftFloatThis is the field used for redshift syncing
_sys_execution_idSystem field - can be ignoredString
match_certaintyThe confidence level of the matchString2None, Suspect, or Match
commentA comment included with the actionString3
scoreThe relatedness score assigned to the change relation by the BigPanda algorithmDecimal4
match_typeThe type of matchString5Always ALGO_RCC meaning the BigPanda algorithm
_sys_updated_dateSystem field - can be ignoredInteger7

tags

All tags of an entity with normalized fields. Tag values appear on their own row in columns for each entity.

After ETL, this data is condensed into the vw_tags table.

Column NameDescriptionData TypeCSV Index #Notes
tag_idThe unique identifier of the tagString0
entity_idThe unique identifier of the entityString1
incident_idThe unique identifier of the incidentString2
keyTag name in systemString3
valueTag valueString4
_sys_updated_dateThe last time the database record was updated in unix timeFloat5
_sys_offsetSystem field - can be ignoredInteger6

user

A single BigPanda user.

Column NameDescriptionData TypeCSV Index #Notes
user_idThe unique identifier of the userString0
nameThe in-system name of the userString1
emailThe email of the userString2
_sys_updated_dateThe last time the database record was updated in unix timeFloat3
_sys_offsetSystem field - can be ignoredInteger4
first_loginThe unix time of the user’s first login to BigPandaFloat5
last_loginThe unix time of the user’s last login to BigPandaFloat6
created_atThe unix time when the user was created in BigPandaFloat7
rolesThe list of roles assigned to this userString8
_sys_import_timeThe time of the last import to redshiftFloatThis is the field used for redshift syncing

vw_tags

All tags of an entity with normalized fields. In the raw CSV, this table appears as the tags table.

During ETL, this data is condensed. Each tag configured in the tagsWhiteList will have its own value column in the view tags or vw-tags table, with one row per entity. The system will also look through a configured list of host and check fields and create the _normalized_check and _normalized_host columns with the first value found. To adjust the list of host and check fields to be normalized, please contact support at [email protected]

Column NameDescriptionData TypeNotes
entity_idThe unique identifier of the entityString
incident_idThe unique identifier of the incidentString
updated_dateThe last time the database record was updated in unix timeFloat
_sys_offsetSystem field - can be ignoredInteger
_sys_import_timeThe time of the last import to redshiftFloatThis is the field used for redshift syncing
_abc1String value of tag abc1StringEach tag will have its own value column
_abc2String value of tag abc2StringEach tag will have its own value column
_normalized_checkAll "check" related fieldsStringFields are normalized for easier reporting
_normalized_hostAll "host" related fieldsStringFields are normalized for easier reporting

Next Steps

Learn how to set up the ETL Sync

Dig into additional reporting options with BigPanda Analytics

Release Notes


v2 (4/5/22)

  • Added new tables matchers_log and rcc_related_changes