Open Integration Manager

The Open Integration Manager enables you to create customizable inbound alert integrations through the configuration of a generic inbound integration rather than creating custom code. The integration manager sets parsing rules for incoming payloads, pre-processing the content to match incoming data requirements before it reaches the BigPanda enrichment engine.

Key Features

• Modify existing standard integrations
• Create a customized integration for any incoming tool leveraging the alerts API
• Tag mapping enables systems to send native payloads, reducing necessary tool configuration
• Editable fields include:
  • Timestamps
  • Status
  • Deduplication Logic
  • Additional Primary / Secondary Properties
  • Custom tags
  • Filter Logic

Users can map payload fields to BigPanda tags, select fields, and tags to populate key values such as Status and Primary tag, and preview sample alerts based on the configuration settings.

Ideal for monitoring tools and systems that do not support customizing REST API payloads, the Open Integration Manager creates integrations that normalize payload data to ready it for the BigPanda enrichment engine.

📘

Single Alert or Array of Alerts

The Open Integration Manager is able to read multiple alerts from a single payload using Tag Mapping.

For more information about the Open Integration Manager, see the BigPanda University Open Integration Manager and Email Parser Course.

Install the Integration

Administrators can install the integration by following the on-screen instructions in BigPanda.

The Open Integration Manager is available for these integrations:

• AppDynamics
• Azure Monitor
• Cisco Meraki
• CloudWatch
• Cribl
• Datadog
• Dynatrace APM
• Entuity
• Grafana v8+
• LogicMonitor
• LogScale
• Nagios
• New Relic
• OpsView v6+
• Prometheus
• ScienceLogic
• SCOM (Microsoft System Center Operations Manager)
• SiteScope
• SNMP
• SolarWinds
• Splunk
• SQL Sentry
• Sumo Logic
• ThousandEyes
• Tivoli EIF
• VMware Aria Operations
• Zabbix
• REST API

❗️

Initial Configuration Delay

After first creating an OIM-enabled integration in the BigPanda UI, ensure that the integration configuration has been saved and wait five minutes before sending alerts via the integration. Alerts sent immediately after saving an integration within the UI may not be successful.

Add OIM to existing Integrations

To use the Open Integration Manager (OIM) for a standard integration that is already set up in BigPanda, the endpoint/destination URL must first be updated. Depending on the integration, you will need to either reinstall the integration or adjust the configuration. Updating the endpoint will not change the tag keys and values that are sent to BigPanda until you update the configuration.

❗️

Caution while Editing

Changing property names within OIM can cause unintended consequences downstream in the BigPanda system.

❗️

Rest API Integrations

REST API integrations cannot have OIM enabled after initial configuration. To use OIM with a REST API integration create a new integration using the OIM REST API integration type.

Alternate Authentication

The Open Integration Manager (OIM) offers additional flexibility with authentication parameters. The following syntax can be used for the Token and App Key:

Auth Token:

query string: access_token=<token>

header: Authorization: Bearer <token>

header: x-auth-token: <token>

App Key:

query string: app_key=<app_key>

header: x-app-key: <app_key>

header: app_key: <app_key>

body: { "app_key": "<app_key>" }

This means that an event can be sent to BigPanda even if the integrated tool is not able to include any information in the API call's header, like in the example below:

https://integrations.bigpanda.io/oim/api/alerts?access_token=xxx&app_key=xxx

Integration Manager

You can customize how BigPanda processes payloads by adjusting the configuration defined in the Integration Manager.

Tag Mapping

Payload fields can be mapped to BigPanda alert tags. Each matched field will be processed into BigPanda tags used for enrichment, normalization, and deduplication.

Additional tag mapping can be added to accommodate the full list of enrichment values to add to the event.

Source tag names are sourced from the Test Payload. To add a new tag to the configuration, upload a new Test Payload with the tag name.

📘

Recent Payloads is Preview Only

The last 10 recent alerts can be viewed in the Preview window to visualize mapping choices. Recent Payloads cannot be used to populate new source tags.

Upload a test payload with the relevant tag to add it to the available source tags.

Destination tag names must:

• Start with a letter from a to z
• Max 64 characters
• Contain only lowercase letters (a-z), numbers (0-9), underscores ( _ ) and hyphens ( - )
• Cannot include spaces

See Tag Naming Requirements for more information.

📘

Note

Null values

OIM automatically removes null values, empty strings, empty objects, and empty arrays from payloads received.

See the Open Integration Manager Advanced Configuration documentation for details on how to use the Advanced Tag Mapping Options.

Payload Processing Options

Open Integration Manager offers flexibility in payload structure and content, giving you tools to adjust how the event data should appear in BigPanda.

Force Lowercasing

The Lowercase created tag name toggle lets you automatically convert all tag names to lowercase. This keeps tags consistent throughout BigPanda, regardless of the capitalization from ingested tags.

Multiple alerts per payload

The Multiple alerts per payload toggle enables you to select a field that is an array of alert JSON or XML objects. Toggling this feature on will present you with a dropdown list of available fields. Select the field name of the object array. The fields listed will be populated by the last 10 previous payloads received as well as the sample payload. Add a test payload with the desired array field to have a new tag array appear in the dropdown.

If a tag contains nested JSON or XML objects (one tag contains multiple pieces of relevant information), you can choose which properties to include depending on the configuration.

Exclude Tags

The Ignore any tag not mapped above toggle controls whether only the desired fields are included in alerts. When toggled on, only tags that match the mapped fields will be included in an alert.

You can choose to instead select specific tags to leave out in the Exclude the following tags field. All tags that are not ignored will be included in the alert, even if they do not map to an existing BigPanda tag.

❗️

Either/Or

If Ignore any tag not mapped above is toggled on, all non-mapped tags will be excluded, not just those defined in the Exclude the following tags field.

Primary and Secondary Properties

Two tags are identified as the Primary and Secondary properties within BigPanda. Primary and Secondary properties are key data fields that function as the main name for alerts and drive correlation, event normalization, and deduplication. Ensure that the fields that map to the BigPanda tags marked as primary or secondary are included in all payloads. You can identify any tag as either Primary or Secondary properties by clicking the three dots icon to the right of the BigPanda Tag Name field.

❗️

Destination Tag Names

OIM integration configuration defines Primary and Secondary properties based on Destination tags rather than Source tags.

❗️

Primary Property is Required

BigPanda cannot receive events without a primary_property. Ensure all event payloads will include the field you have mapped as primary property.

Add Multiple Source Tags

Multiple source tags can be used for a single BigPanda tag, with the first tag populated defining the value. When listing multiple source tags, the tags run in the order they appear in the editor. Drag and drop source tags in the tag manager fields to rearrange the run order.

For example:

If two source tags, host and device, are listed, the system will first check for the host field in the payload. If there is a value for host, this will set the BigPanda tag value. If host is empty, the system will then check for the device field.

If all source tag fields are empty, the system will use a default value if defined.

Array Flattening

❗️

Default Change

As of October 2024, new OIM integrations will not flatten arrays by default. Existing integrations will not be affected by this change. If you would like to turn off array flattening for an integration created before October 2024, please reach out to support to update the array flattening setting.

When flattened, each element within the array is sent as a separate tag in the event payload.

For example, an array like "annotation":["description1","summary1"] would send two tags when flattened: "annotation_0":"description1" and "annotation_1":"summary1".

By default, arrays are not flattened. Select a tag within Tag Mapping and enable the Array Flattening toggle to flatten the incoming payload into separate tags.

Status Mapping

BigPanda alert statuses are determined by specific incoming tag values. Alert status determines several system events and is necessary for closing resolved alerts. Read more about alert status in the Alert/Incident Status documentation.

In the Status Mapping fields, list payload values to map to each BigPanda event status. These must be an exact match, meaning that if ‘warn’ is in the value mapping, an alert message with ‘warned’ will not be matched. If there are multiple tags that can determine an alert's status (severity, statusExtended, etc.), tags will be evaluated against all mapping values in listed order until a match is found or all tags rules have run.

One status will be selected as Default. If an alert payload does not match any listed status values for any of the specified tags, this status will be applied.

Event Timestamp

Event time can be set automatically by BigPanda or by a payload field.

By default, all customized integrations created through the Open Integration Manager use the time the event entered BigPanda.

If the event instead uses a time tag, the source tag and format can be configured to match the event time field.

Human Readable Timestamp

You also have the option to include the human-readable timestamp tag bp_timestamp. This tag will display as: bp_timestamp: Tue Mar 19 2024 14:53:57 GMT+0000 (Coordinated Universal Time)

This tag is based on an alert payload's timestamp tag, as defined in the Event Timestamp step. If there’s no timestamp field in the payload, bp_timestamp will be the received time.

Event Deduplication Tags

To eliminate redundant data and reduce noise, BigPanda creates an incident identifier for each incoming event.

By default, this identifier is created using the primary and secondary property tags.

❗️

Adjust with Caution

We recommend keeping the default setting for the incident identifier. Changing this setting can create a mismatch between alerts reported through different integrations and may result in duplicate incidents or alerts that fail to resolve properly.

Event Filtering

The Event Filtering section uses the same BigPanda Query Language as the rest of the BigPanda platform. Any events that meet the defined criteria will be dropped upon ingestion and never visible in BigPanda.

This feature should only be used as a garbage filter for events that can never be actionable and would only add clutter in BigPanda. Below are some examples:

• Misconfiguration (necessary tags are missing or cannot be parsed)
• Non-actionable (lowest severity or contains no signal)
• Events from Dev/QA environments
• Non-events (info, logs, etc.)

Keep in mind that the syntax used here is based on source tags included in the event payload, not destination tags. If two different types of events from your tool use different tag names, be sure to include both in your filter criteria. The Query Assist feature will provide guidance based on your sample event payloads and recently delivered events.

❗️

Arrayed and Nested Alerts

If you are using an array of alerts by selecting Multiple alerts per payload, you will need to use the @alert.<tag> syntax. For example, your filter criteria could be something like: @alert.error_code="sdwan-datapath-down".

If the source system payload contains nested JSON, use dot walking notation: element.code=INFO, element.0.code=INFO.

Preprocessing Functions

Before being ingested into BigPanda, the Open Integration Manager performs preprocessing functions on some complex integrations. This allows the data to be mapped appropriately to tags in BigPanda. Visibility into these functions is available within OIM by clicking the Settings cog icon.

For more information and a full list of these functions, see the Open Integration Manager Preprocessing Functions documentation.

Preview Tag Settings

The Preview pane enables you to view sample BigPanda alerts as they would be generated using the configuration settings and an example event payload.

To preview a sample alert using the existing settings:

1. At the top of the Integration Manager, click Test Payload
2. In the window, paste or type to add a sample JSON payload
3. Click Create Sample Alert(s)

In the preview pane, sample alerts appear individually, with values populated based on the tag mapping configuration.

The sample alert values will update automatically whenever you make a change to the configuration in the left panel.

Recent Payloads View

BigPanda can generate event previews and configure tag mappings based on actual alerts. To generate a sample payload, use the integrated tool to send a test event to BigPanda after integration installation, but before configuration. The sample must include all payload fields to be referenced in the configuration settings.

To use this configuration option, go to the Event Preview section and click Recent Payloads View to switch from configuring based on the Test Payload to the Recent Payloads View. You will see No recent payloads available if no events have yet been delivered to an integration.

Once you are ready, send an alert to the endpoint specified in the installation instructions. Alerts sent to BigPanda through the OIM will time out if no response is received in 30 seconds. A failed event will attempt to retry up to 3 times unless there are payload errors that will prevent it from processing.

📘

Payload Size Limit

OIM can accept payloads up to 6 MB.

Edit the Integration Configuration

Administrators can change the configuration settings of integrations at any time within BigPanda.

To open the Integration Manager:

1. Navigate to the Integrations Tab
2. Select the integration from the list
3. Click Review Instructions
4. Select the Integration Manager tab
5. Upload an example payload or send events to preview the configuration
6. Make any desired changes

📘

Test Before Finalizing

We recommend testing configuration changes using test events and the preview panel before finalizing changes to an integration configuration.

Uninstall the Integration

Deleting an integration requires that you remove the integration in both the integrated system and BigPanda. We recommend that you first uninstall the integration on the integrated system to prevent traffic from being sent and rejected by BigPanda, since the app key will not exist once you delete the integration in BigPanda.

❗️

Caution During Replacement

When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.

Deactivate Inbound Integration

If you want to stop sending data to BigPanda but don’t want to delete your integration, you can temporarily deactivate it.

To deactivate an inbound integration:

1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list. This will open integration details on the right side of the window.
2. At the top of the integration details, click the Active/Inactive toggle next to the application name to change the status of the integration.

In the integrations list, inactive integrations will be marked with a gray bar.

❗️

Alert Resolution for Inactive Integrations

Any active alerts belonging to an inactive integration must be manually resolved or they will stay in the system until the auto-resolve window is reached.

Stop Sending Data to BigPanda

Within the integrated system, disable any settings that send data to BigPanda.

Each system requires specific changes to disable the integration with BigPanda. For example, you must delete the topic in CloudWatch, and you must disable the alert channel in New Relic. To determine the changes for your integrated system, reference the relevant documentation or contact BigPanda support.

Manually resolve any open alerts sent from the integration to remove the associated incidents from your incident feed. These incidents will not automatically resolve without an ok status from the original sending integration.

Delete the Integration in BigPanda

Take the following steps to delete the integration from BigPanda:

1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
2. In the integration details on the right of the page, click the trash icon, then confirm you want to delete the integration. The integration will be removed immediately.

👍

️Automatic Alert Resolution for Deleted Integrations

All active alerts from the integration will be resolved after deletion.

❗️

Data Removal

This procedure does not remove any data from the integrated system.