Splunk

BigPanda provides a native Splunk App to let you easily send Splunk alerts to BigPanda. The app provides a native Splunk alert action which will forward the Splunk alert to the BigPanda integration. The integration will take the Splunk alert and normalize it into one or more BigPanda alerts, one alert for each row in the search result.

Splunk Enterprise and Splunk Cloud have similar but different requirements when integrating with BigPanda.

• See the Install a Splunk Enterprise Integration documentation for details on configuring and installing Splunk Enterprise.
• See the Install a Splunk Cloud Integration documentation for details on configuring and installing Splunk Cloud.

👍

Open Integration Manager

The Open Integration Manager is available for use with Splunk. For more information, see the Open Integration Manager documentation.

❗️

Considerations before updating Splunk to OIM

The Splunk OIM integration uses Time Based Alert Resolution to resolve alerts. Do not switch from the legacy solution if you require variable resolve times based on the Splunk expire time.

The legacy endpoint for Splunk is https://integrations.bigpanda.io/splunk/alerts, and the new OIM endpoint is https://integrations.bigpanda.io/oim/splunk_v2/alerts.

❗️

Noisy Integration

Splunk can be a noisy monitoring system. For organizations using the BigPanda consumption based pricing model, it can consume large amounts of credits very quickly. For help configuring Splunk to be less noisy, see the Splunk FAQ.

Key Features

• Easily send Splunk alerts to BigPanda using the native BigPanda action in Splunk
• Keep your team focused on what matters with auto-resolution of Splunk alerts in BigPanda
• Simple & efficient management of all alerts going to BigPanda via custom alert management dashboard and search commands
• Customizable alert properties
• Proxy support for on-premise Splunk Enterprise deployments

Auto-Resolve

Splunk does not send notifications when alerts are resolved. Instead, BigPanda will auto-resolve the alerts in BigPanda based on the expiration time defined for the alert in Splunk.

Splunk Data Model

BigPanda normalizes the search alert results into multiple BigPanda alerts. Each of these results becomes a BigPanda alert where the data from Splunk is turned into tags. You can use tag values to filter the incident feed and to define filter conditions for Environments.

The primary and secondary properties are also used during the correlation process. The incident_identifier used for deduplication is generated from the _bkt and _cd internal fields. If these are not present, the primary and secondary properties or their overrides will be used.

Standard Tags

🚧

Primary Property

Each alert must have a primary property specified. If not all Splunk alerts have a host associated, you may need to override the Primary Property. BigPanda will default to use the server_host if a host is not defined, but it is recommended to set up an override if hosts will not be defined.

See the Override Alert Tags section to learn more about configuring an alternate Primary Property. If using the OIM framework, please use the configuration page in BigPanda to define this property.

Customize Splunk Alerts

For any Splunk alert that is sent to BigPanda, you can define a custom description, primary property, or secondary property. If you have more than one Splunk integration with BigPanda for testing or environment management, you can also specify which integration to use for a given alert.

Customizing the alerts is especially useful if your Splunk alerts do not always include the default information such as the host. BigPanda will default to using the server_host if a host is not defined, but it is recommended to set up an override if hosts will not be defined.

Override Alert Tags

This section is not required if you are using the Open Integration Manager framework with Splunk. If the OIM framework is available, please use the configuration page in BigPanda or Integration Configuration API to define these properties.

The default Description, Primary, and Secondary properties can be overridden by defining custom alert variables. All variables will continue to be sent with the Splunk payload, but these key correlation fields will use these alternate variables.

1. From the Alerts page in Splunk, select an alert to open its detail page, and then Click to edit actions.
2. (Optional) Alternatively, run a search, and go to Save As > Alert.
3. At the bottom of the window, click Add Actions, and then select BigPanda.
4. Enter the custom alert values you want to send to BigPanda when the alert is triggered.

📘

Custom tag fields are optional

The incident_identifier used for deduplication is generated from the _bkt and _cd internal fields. If these are not present, the primary and secondary properties or their overrides will be used. If using the OIM framework, please use the configuration page in BigPanda to define this property.

Use Splunk Variables in Custom Tag Values

When overriding Splunk alert tags in BigPanda, customers can leverage any search data that is available as a Splunk Token, including search metadata and values from the first row of the search results.

To override alert tag values, a variable name or value must exist in the payload to BigPanda. If the override variable is not found, BigPanda will use the default values for that field.

• To override with search metadata, use the name of the variable. For example, source or sourcetype
• When overriding Description, you are also able to override using field values, using the format $result.<fieldname>$
• For example, if search=hello : If you enter host in the description field it will display only host. If instead you enter $result.host$ the description will be the host for hello.
• Any of the Splunk alert fields can be added to the Splunk search.
• Adding alert fields to the override fields will make the data available as a variable

To override alert tag values, a variable name or value must exist in the payload to BigPanda. If the override variable is not found, BigPanda will use the default values for that field.

Uninstall the Integration

Deleting an integration requires that you remove the integration in both the integrated system and BigPanda. We recommend that you first uninstall the integration on the integrated system to prevent traffic from being sent and rejected by BigPanda, since the app key will not exist once you delete the integration in BigPanda.

❗️

Caution during replacement

When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.

Deactivate Inbound Integration

If you want to stop sending data to BigPanda but don’t want to delete your integration, you can temporarily deactivate it.

To deactivate an inbound integration:

1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list. This will open integration details on the right side of the window.
2. At the top of the integration details, click the Active/Inactive toggle next to the application name to change the status of the integration.

In the integrations list, inactive integrations will be marked with a gray bar.

❗️

Alert resolution for inactive integrations

Any active alerts belonging to an inactive integration must be manually resolved or they will stay in the system until the auto-resolve window is reached.

Stop Sending Data to BigPanda

Within the integrated system, disable any settings that send data to BigPanda.

Each system requires specific changes to disable the integration with BigPanda. For example, you must delete the topic in CloudWatch, and you must disable the alert channel in New Relic. To determine the changes for your integrated system, reference the relevant documentation or contact BigPanda support.

Manually resolve any open alerts sent from the integration to remove the associated incidents from your incident feed. These incidents will not automatically resolve without an ok status from the original sending integration.

Delete the Integration in BigPanda

Take the following steps to delete the integration from BigPanda:

1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
2. In the integration details on the right of the page, click the trash icon, then confirm you want to delete the integration. The integration will be removed immediately.

👍

️Automatic alert resolution for deleted integrations

All active alerts from the integration will be resolved after deletion.

❗️

Data removal

This procedure does not remove any data from the integrated system.

FAQ

I have applied the BigPanda add-on to all the saved searches, why am I not getting data?

There are a few reasons why you may not be getting an alert. One of the most common is that when you apply the add-on to your saved searches, it does not automatically enable the alert. You need to enable the saved search for it to start sending data to BigPanda.

Another reason is that a saved search can be an alert or a report, but only alerts trigger in BigPanda. If you want a report to work with BigPanda, you need to convert it to an alert.

If you are still not seeing data come through, it might be because the alert is returning too many results, taking too long to process, or it may not be making it to BigPanda. See the FAQs below on how to make a saved search actionable.

What are saved searches and how does BigPanda use them for alerts?

Saved searches save the search criteria, rather than the full search itself. You can use saved searches to create alerts that will trigger when certain conditions are met.

The two types of saved searches are alerts and reports. Alerts are triggered on intervals and can also trigger other actions, such as sending an email or logging a Splunk event. Reports don't trigger any other actions, but they can still be useful for alerting you to something important.

BigPanda focuses only on the alert saved search type. Alerts allow you to send searches using the add-on action in order to appear in the BigPanda console.

BigPanda also distinguishes between non-actionable alerts and actionable alerts. Non-actionable alerts return more than 10 results when triggered and can contain INFO logs. Actionable alerts return a max of 10 results per trigger and are intended to provide useful information about what's happening. Ideally, these would not return any search results since each returned search result is considered an alert of an issue.

What makes a saved search monitor and its alerts actionable?

Actionable alerts are those that a user can act on. The saved search alert should return fewer than 10 results in each trigger. When you have too many results, it's hard to focus on the ones that are most important and need your attention. Alerts with more than 10 results in each trigger are considered noise and are not actionable.

You need to be able to filter the search query to hone in on the results that provide information about your systems health and should only return if an alert should trigger. The saved search alert query is generally considered actionable if it indicates that something needs to be fixed.

How do I make my monitor saved searches actionable?

Making your saved search alerts actionable can be a challenge. Here are some easy things you can do to ensure that you are focusing on the most actionable saved search alerts.

Filter your results

Focus on the things you want by filtering your results to what matters to you.

Use the following filter:

YOUR SEARCH | search value="Value to filter on"

Filter out the INFO log_level

You can use filters to drop any result that contains the log_level INFO, allowing you to remove noise and make it easier to send data over.

Use the following filter:

YOUR SEARCH | search NOT [log_type="INFO"]

Dedup your results

You can dedupe results that contain the same value by using the following filter:

YOUR SEARCH | dedupe value 1 value 2 .... value N

Return only top results

If you are getting similar results and only want the latest one, you can use the following to reduce the returned results to a set of N:

YOUR SEARCH | head N

I have installed/upgraded the latest version of the BigPanda Splunk app, but it is not working

To ensure that your BigPanda Splunk app is properly installed, check the compatibility of the application and version of your Splunk instance. If you have confirmed that BigPanda works with your version of Splunk, you may have to restart for the changes to persist.

If that does not work, we recommend deleting the BigPanda app, restarting, and then reinstalling the application.

Why am I getting a CERTIFICATE_VERIFY_FAILED message when using Splunk behind a proxy?

If you are connecting to Splunk through a proxy, then the Splunk instance needs the proxy server's certificate installed. 

Install a Splunk app called CA Cert Manager

CA Cert Manager will add the necessary certificate to the cert managers cert directory, and then restart Splunk with the cert added into the correct python bundle.

Install CA Cert Manager from the Splunk app store.

Add the proxy's CA certification to python directly

You can add your proxy's CA Cert directly to python's certifi cert bundle. The cert bundle can be found at a path similar to: $SPLUNK_HOME/lib/python3.7/site-packages/certifi/cacert.pem. This URL will depend on your specific python configuration.

Once the cert is added to the bundle, restart Splunk.

Release Notes

v2.1.8 (Feb 2, 2024)

• Updated Splunk's Add-On Builder to 4.1.4
• Updated the configuration api_url placeholder to point to the oim endpoint

v2.1.6 (11/21/22)

• Added Basic Auth header support

v2.1.5 (7/25/22)

• Updated Add-On Builder to 4.1.1
• Fixed JQuery version issue for Splunk Cloud support
• Updated splunk-sdk for python to version 1.7.0

v2.1.4 (1/20/22)

• Fixed typo in action management dashboard

v2.1.2 (1/10/22)

• Support for replicating configuration in clustered search head environments
• Removed extraneous warning messages for Splunk cloud

v2.1.1 (12/1/21)

• Upgrade potentially unwanted HTTP calls to HTTPS when running search commands

v2.1.0 (10/26/21)

• Fixed issue with the alert assignment management feature
• Updated the default endpoint in the application

v2.0.0 (9/9/21)

• Support sourcing of API credentials via environment variables
• Update with support for Splunk Add-on Builder v4.0.0

v1.0.0

• Initial release