Splunk

Install this integration to correlate alerts from Splunk into high-level incidents in BigPanda, and see insights from Splunk alongside the problems detected by other tools in your monitoring stack.

BigPanda provides a native Splunk App to let you easily send Splunk alerts to BigPanda. The app provides a native Splunk alert action which will forward the Splunk alert to the BigPanda integration. The integration will take the Splunk alert and normalize it into one or more BigPanda alerts, one alert for each row in the search result

Key Features


  • Easily send Splunk alerts to BigPanda using the native BigPanda action in Splunk
  • Keep your team focused on what matters with auto-resolution of Splunk alerts in BigPanda
  • Simple & efficient management of all alerts going to BigPanda via custom alert management dashboard and search commands
  • Customizable alert properties
  • Proxy support for on-premises Splunk deployments

📘

Looking for our old Splunk docs?

This page has been updated with documentation for our new Splunk integration. If you are looking for docs for the previous Splunk integration please find them here.

Supported Versions

Type

Authentication Type

Splunk Enterprise, Splunk Cloud

Native App

Bearer Token

Auto-Resolve

Splunk does not send notifications when alerts are resolved. However, BigPanda will auto-resolve the alerts in BigPanda based on the expiration time defined for the alert in Splunk. This helps you keep your BigPanda incident feed clear and keep your team focused on solving incidents that matter.

Splunk Data Model


BigPanda normalizes the search alert results into multiple BigPanda alerts. Each of these results becomes a BigPanda alert where their data from Splunk is turned into tags. You can use tag values to filter the incident feed and to define filter conditions for Environments. The primary and secondary properties are also used during the correlation process.

Standard Tags

Tag

Description

Attributes

host

The host of the Splunk search result

Default Primary Property

search_name

Name of the search in Splunk.

Default Secondary Property

splunk_source

Source field value in the first result row of the Splunk search, if available

server_host

Host name of the Splunk server

Splunk uses server_host as the default Primary Property if it doesn’t find host

server_url

Management port of the Splunk server, identified by host, port, and protocol

app

Name of the Splunk app that contains the alert

results_link

Link to the search results in Splunk

results_file

Path to the raw results in a CSV file on your Splunk server

Additional Attributes (Varies)

If available, tag for each additional data field in the alert payload from Splunk

🚧

Primary Property

Each alert must have a primary property specified. If not all Splunk alerts have a host associated, you may need to override the Primary Property. BigPanda will default to use the server_host if a host is not defined, but it is recommended to set up an override if hosts will not be defined.

See the Overriding Alert Tags section to learn more about configuring an alternate Primary Property.

Installing the Integration

The Splunk integration can be installed in 3 ways:

  • From the Splunk Web Page
  • Within Splunk Enterprise using Splunkbase
  • Using CLI

Prerequisites

  • Create an integration in BigPanda for Splunk
  • Have Admin permissions in Splunk
  • Users who own searches which are sent to BigPanda must have the list_storage_passwords permission in Splunk
    * If users cannot be granted this permission, then Splunk can be configured with credentials provided via environment variables instead of the config page. See the Configuring Alternate Permissions section to learn how

Install the BigPanda Splunk Add-on from the Web Page

  1. From the Splunk Web home page, click the Apps gear icon.
  2. Click Install Apps.
  3. Select Install to install an app. If the app that you want is not listed, or if the app indicates self-service installation is not supported, contact Splunk Support.
  4. Follow the prompts to complete the installation.

Install BigPanda Splunk Add-ons from within Splunk Enterprise

  1. Download the BigPanda for Splunk add-on from Splunkbase
  • If you are using a distributed Splunk search environment with multiple instances please install the app on your search head instance*)
  1. Log into Splunk Enterprise
  2. On the Apps menu, click Manage Apps
  3. Click Install app from file
  4. In the Upload app window, click Choose File
  5. Locate the .tar.gz file you just downloaded, and then click Open or Choose
  6. Click Upload
  7. Click Restart Splunk, and then confirm that you want to restart

Install the Splunk Add-on directly into Splunk Enterprise through CLI:

  1. Download the BigPanda for Splunk add-on from Splunkbase
  • If you are using a distributed Splunk search environment with multiple instances please install the app on your search head instance*)
  1. Put the downloaded file in the $SPLUNK_HOME/etc/apps directory
  2. Untar and ungzip your app or add-on, using a tool like tar -xvf (on *nix) or WinZip (on Windows).
  3. Restart Splunk

Using Splunk Deployment Server for Distributed Search Environments

When installing the BigPanda app in a Splunk distributed-search environment, Splunk does not automatically propagate the app to all nodes in the cluster, and the app must be installed on each node in the cluster either manually or through the deployment server.

Search Head Clusters

To deploy apps to a search head cluster, you must use the deployer. The deployer is a Splunk Enterprise instance that distributes apps and configuration updates to search head cluster members. The deployer cannot be a search head cluster member and must exist outside the search head cluster. See Use the deployer to distribute apps and configuration updates in the Distributed Search manual to learn more about the role of a deployer instance.

🚧

You cannot push an application with predefined credentials to a clustered search head in Splunk. To get around this, you have 2 options:

  • Push application from deployer to search head cluster without credentials – then manually enter the credentials afterwards via the UI
  • Push application from deployer to search head cluster with credentials in plain text (not encrypted)

When pushing the credentials from deployer to the search head cluster, they are sent in plain text, and then splunk will automatically encrypt the values when the search heads reload with the new app/changes

Configuring the Integration

The Splunk integration can be configured in 2 ways:

  • Through the BigPanda App Configuration Tab
  • Using the CLI

Configuring Through the Configuration Tab

  1. Open the BigPanda app in Splunk and navigate to the Configuration > Global Settings tab
  2. Fill in your BigPanda App Key and API Token (BigPanda Bearer Token) inputs
  3. Click Save
Global Settings in the Configuration TabGlobal Settings in the Configuration Tab

Global Settings in the Configuration Tab

  1. (Optional) If you are using a Proxy, enter the Proxy information in the Proxy tab
Proxy Options in the Configuration TabProxy Options in the Configuration Tab

Proxy Options in the Configuration Tab

  1. Navigate to the Action Manager page and configure which alerts to send to BigPanda

🚧

Users who own searches which are sent to BigPanda must have the list_storage_passwords permission in Splunk

Configuring Alternate Permissions

Instead of using the Splunk search owner permissions, you can configure BigPanda to use environment variables instead. These environment variables can be set by the system administrator or configured in /etc/splunk-launch.conf

These variables will provide the BigPanda bearer token and bypass the search owner permission:
BIGPANDA_USE_ENV_PASSWORDS=true
BIGPANDA_BEARER_TOKEN=XXXXXXXXXXXXXXXXXXXXXX
BIGPANDA_PROXY_PASSWORD=XXXX

Configuring Through the CLI

  1. Ensure there is no $SPLUNK_HOME/etc/apps/TA-bigpanda/local/passwords.conf file
  2. Create $SPLUNK_HOME/etc/apps/TA-bigpanda/local/ta_bigpanda_settings.conf with the following:
[additional_parameters]
api_url = https://inbound.bigpanda.io/splunk/alerts
app_key = app_key_here_in_plain_text
token = bearer_token_here_in_plain_text
  1. (Optional) Configure a Proxy through the CLI
    You can also set the proxy through the CLI. All you need to do is create ta_bigpanda_settings.conf at $SPLUNK_HOME/etc/apps/TA-bigpanda/local/ta_bigpanda_settings.conf. If it already exists then all you need to add is the following stanza:
[proxy] 
proxy_password = XXXXXXXXXXXXXX 
proxy_port = PORT_NUMBER_HERE 
proxy_rdns = 1 | 0 (1 = enabled, 0 = disabled)
proxy_type = http | socks4 | socks5 
proxy_url = URL_HERE 
proxy_username = USERNAME_HERE 
proxy_enabled = 1 | 0 (1 = enabled, 0 = disabled)
  1. Navigate to the Action Manager page and configure which alerts to send to BigPanda

Sending Alerts to BigPanda

Once BigPanda is configured in Splunk, and the BigPanda action has been configured for alerts, you will need to enable the alerts to send to Bigpanda.

🚧

When defining Trigger Conditions for Alerts to send to Bigpanda, make sure to select the Trigger as Once. BigPanda will extract all individual events from the search results as unique alerts, and does not need to send for each result.

Enable alerts using either the Action Manager, or Search Commands.

Action Manager

The Splunk Action Manager lists all alerts that have been created from previously saved searches. Each of these alerts can be enabled or disabled to send to BigPanda based on the configuration of the alert.

Basic Action ManagerBasic Action Manager

Basic Action Manager

  1. Navigate to BigPanda > Action Manager
  2. The top left panel lists alerts that are currently enabled to send to BigPanda. The top right panel lists any alerts that are not enabled.
  3. Click Stop Sending All Alerts to BigPanda to disable all alerts from sending to BigPanda
  4. Click Send All Alerts to BigPanda to enable the alerts listed in the left pane to send to BigPanda

📘

Click any of the alerts on the Action Manager to open up the alert and see configuration settings and the original search query.

Advanced Action Manager

The advanced action manager allows you to enable or disable a subset of alerts using a query filter.

Advanced Action ManagerAdvanced Action Manager

Advanced Action Manager

  1. Enter a search value into the Filter text box to narrow the list below to only alerts that fit that value
  2. Click Send Filtered Alerts to BigPanda to enable all alerts listed to send to BigPanda.
  3. Click Stop Sending Filtered Alerts to BigPanda to disable all alerts listed from sending to BigPanda

Search Commands

BigPanda also provides search commands to stop or start sending alerts to BigPanda. The two available search commands are addbigpanda and removebigpanda.

BigPanda Search CommandsBigPanda Search Commands

BigPanda Search Commands

  1. In Splunk, Navigate to Search & Reporting
  2. In the search bar do a query following this search structure: | rest /services/saved/searches | YOUR FILTER | SEARCH COMMAND

🚧

Search Commands in a Distributed Cluster

When using a distributed cluster, you must be logged into the captain node to be able to successfully use the BigPanda Search Commands. If you are logged into a different node you will receive authentication errors when attempting the BigPanda Search Commands

The action manager relies in the backend on the BigPanda search commands addbigpanda and removebigpanda

Customizing Splunk Alerts

For any Splunk alert that is sent to BigPanda, you can define a custom description, primary property, or secondary property. If you have more than one Splunk integration with BigPanda for testing or environment management, you can also specify which integration to use for a given alert.

👍

Customizing the alerts is especially useful if your Splunk alerts do not always include the default information such as the host. BigPanda will default to using the server_host if a host is not defined, but it is recommended to set up an override if hosts will not be defined.

Overriding Alert Tags

The default Description, and Primary and Secondary properties can be overridden by defining custom alert variables. All variables will continue to be sent with the Splunk payload, but these key correlation fields will use these alternate variables.

  1. From the Alerts page in Splunk, select an alert to open its detail page, and then use the Click to edit actions link.
  2. (Optional) Alternatively, run a search, and go to Save As > Alert.
  3. At the bottom of the window, click Add Actions, and then select BigPanda.
  4. Enter the custom alert values you want to send to BigPanda when the alert is triggered.

Field

Description

Primary

Adds a custom tag with the name custom_primary and the value specified. You can use Splunk variable names in the alert value. If defined, this tag is treated as the primary property for alert correlation and in the incident title.

Secondary

Adds a custom tag with the name custom_secondary and the value specified. You can use Splunk variable names in the alert value. If defined, this tag is treated as the secondary property for alert correlation and in the incident subtitle.

Description

Overrides the alert description. You can use Splunk variables in the description value. The default description is the search name. In addition to the alert variable names, the description field can be populated with alert variable values.

📘

Custom tag fields are optional.

Using Splunk Variables in Custom Tag Values

When overriding Splunk alert tags in BigPanda, customers can leverage any search data that is available as a Splunk Token, including search metadata and values from the first row of the search results.

To override alert tag values, a variable name or value must exist in the payload to BigPanda. If the override variable is not found, BigPanda will use the default values for that field.

  • To override with search metadata, use the name of the variable. For example, source or sourcetype
  • When overriding Description, you are also able to override using field values, using the format $result.<fieldname>$
  • For example, if search=hello : If you enter host in the description field it will display only host. If instead you enter $result.host$ the description will be the host for hello.
  • Any of the Splunk alert fields can be added to the Splunk search.
  • Adding alert fields to the override fields will make the data available as a variable

🚧

To override alert tag values, a variable name or value must exist in the payload to BigPanda. If the override variable is not found, BigPanda will use the default values for that field.

Custom Field ValuesCustom Field Values

Custom Field Values

Splunk Variables as OverridesSplunk Variables as Overrides

Splunk Variables as Overrides