Splunk

Splunk allows you to search, monitor, and analyze machine data that is generated by your infrastructure. Install this integration to correlate alerts from Splunk into high-level incidents in BigPanda, and see insights from Splunk alongside the problems detected by other tools in your monitoring stack.

Supported Versions:
On-premise Deployments, versions 4.0+

Type:
Action Script

Key Features

  • Supports the Splunk 6.3 release.

  • Allows you to trigger BigPanda incidents from Saved-Search Alerts.

  • Adds an option to the Add Actions list in the Splunk user interface.

  • Allows you to define custom alert values using Splunk search data.

  • Provides a separate version of the integration to support earlier versions of Splunk.

How It Works

For Splunk 6.3 and up, the integration uses the Custom Alert Actions capability to add an action for Saved-Search Alerts in Splunk.

For Splunk versions earlier than 6.3, the BigPanda Splunk action script sends all Splunk alerts to BigPanda. You must install the BigPanda Python package on the Splunk server before completing the integration.

How and When Alerts Are Closed

Splunk incidents are not closed automatically because Splunk does not send notifications when alerts are resolved. You must manually resolve Splunk incidents in BigPanda to remove them from the incident feed.

Splunk Clustering

Splunk has a three-tier architecture—search heads, indexers, and forwarders. The BigPanda custom alert action is a function that runs on the search heads, so clustering of the search heads can affect how BigPanda receives alerts from Splunk. To ensure BigPanda receives alerts from all nodes:

  • For Splunk 6.2 and up with search head clustering enabled, ensure the Splunk deployer pushes the BigPanda action script to all search heads.

  • For versions prior to Splunk 6.2 with search head pooling enabled, ensure the BigPanda action script is located in the shared storage and used by all search heads.

Installing the Integration

Administrators can install the integration by following the on-screen instructions in BigPanda. For more information, see Installing an Integration.

Splunk Data Model

BigPanda normalizes alert data from Splunk into tags. You can use tag values to filter the incident feed and to define filter conditions for Environments. The primary and secondary properties are also used during the correlation process.

Standard Tags

Tag
Description
Attributes

search_name

Name of the search in Splunk.

splunk_source

Source field value in the first result row of the Splunk search, if available.

Secondary Property. (Default)

custom_primary
custom_secondary

Custom tag values, if defined for the alert in Splunk. If they exist, these fields are treated as the primary and secondary properties for the alert.

Primary & Secondary Properties. (Custom, if defined)

server_host

Host name of the Splunk server.

server_url

Management port of the Splunk server, identified by host, port, and protocol.

app

Name of the Splunk app that contains the alert.

results_link

Link to the search results in Splunk.

results_file

Path to the raw results in a CSV file on your Splunk server.

Additional Tags (Varies)

If available, tag for each additional data field in the alert payload from Splunk.

Customizing Splunk

For any Splunk alert that is sent to BigPanda, you can define a custom description, primary property, or secondary property. You can also specify which integration to use for a given alert, if you have more than one Splunk integration with BigPanda.

Prerequisites

  • Obtain administrator access to BigPanda.
  • Install the Splunk integration with BigPanda.

Defining Custom Alert Tags

You can enrich Splunk alerts with additional information by defining custom alert tags.

  1. From the Alerts page in Splunk, select an alert to open its detail page, and then use the Click to edit actions link.Alternatively, run a search, and go to Save As > Alert.

  2. At the bottom of the window, click Add Actions, and then select BigPanda.

  3. Enter the custom alert values you want to send to BigPanda when the alert is triggered.

Field
Description

Primary

Adds a custom tag with the name custom_primary and the value specified. You can use Splunk variables in the tag value. If defined, this tag is treated as the primary property for alert correlation and in the incident title.

Secondary

Adds a custom tag with the name custom_secondary and the value specified. You can use Splunk variables in the alert value. If defined, this tag is treated as the secondary property for alert correlation and in the incident subtitle.

Description

Overrides the alert description. You can use Splunk variables in the description value. The default description is the search name.

Custom tag fields are optional.

Using Splunk Variables in Custom Tag Values

To enrich Splunk alerts in BigPanda, you can leverage any search data that is available as a Splunk Token, including search metadata and values from the first row the search results.

  • To access search metadata, use the format $<fieldname>$. For example, $name$ for the search name.

  • To access field values from the first result row that a search returns, use the format $result.<fieldname>$. For example, $result.host$ for the host value and $result.sourcetype$ for the source type.

    • You can leverage any of the Selected Fields in the Splunk search.

    • Add any of the Interesting Fields to the Selected Fields to make the data available as a variable

The following example uses $result.host$ in the primary property value and $result.sourcetype$ for the secondary property value.

In BigPanda, the alert is enriched with data from the first row of search results—a host value of localhost.localdomain in the custom_primary tag and a source type of vendor_sales in the custom_secondary tag.

Selecting a Splunk Integration

You can configure multiple Splunk integrations in BigPanda by creating a separate app key to identify each instance as a unique source. Then, you can specify which integration to use for a given alert.

  1. From the Alerts page in Splunk, select an alert to open its detail page, and then use the Click to edit actions link. Alternatively, run a search, and go to Save As > Alert.
  2. At the bottom of the window, click Add Actions, and then select BigPanda.
  3. In the App Key field, specify which Splunk integration to use as the source.
    By default, the field is populated with the app key used when installing the BigPanda app for Splunk.

Post-Requisites

Repeat the procedure for any Splunk alerts that you want to customize.

Upgrading To Splunk 6.3

If you are using a previous version of the Splunk integration, you must upgrade the integration to use Splunk 6.3 with BigPanda.

Prerequisites

Administrator access to Splunk and BigPanda.

Procedure

  1. Uninstall the previous Integration.
  2. In BigPanda, click Integrations.
  3. In your list of integrations, click the Splunk integration. The integration details open in the right pane.
  4. Click Review Instructions.
  5. Click Version 6.3+.
  6. Follow the instructions to install the integration for Splunk 6.3.

Use the same credentials as you did for the previous Splunk integration.

Uninstalling Splunk Versions Prior to 6.3

The BigPanda integration creates or updates a Splunk configuration file to register its action script for all alerts. To stop sending alerts to BigPanda, you must modify the configuration file, which is located at $SPLUNK_HOME/etc/system/local/savedsearches.conf.

The location of $SPLUNK_HOMEmay vary depending on your setup. Consult with your system administrator to get the value for your system.

Prerequisites

  • Manually resolve any open incidents related to the integration to remove them from your incident feed. Optionally, remove any additional data in BigPanda. For example, modify or remove any Environments or AutoShare rules that reference the integration.

  • Determine whether the configuration file:

    • Was created by BigPanda.

    • Already existed and was only updated by BigPanda. This scenario occurs when you have customized your Splunk installation.

Uninstalling Splunk (BigPanda Created The File)

  1. SSH to your Splunk server.
  2. Delete $SPLUNK_HOME/etc/system/local/savedsearches.conf.
  3. Restart Splunk. Data from Splunk no longer flows into BigPanda.

Uninstalling Splunk (File Existed, BigPanda Updated)

  1. SSH to your Splunk server.
  2. Edit $SPLUNK_HOME/etc/system/local/savedsearches.conf.
  3. Set action.script to 0.
  4. Restart Splunk. Data from Splunk no longer flows into BigPanda.

Post-Requisites

Delete the integration in BigPanda to remove the Opsview integration from your UI.

Splunk


Splunk allows you to search, monitor, and analyze machine data that is generated by your infrastructure. Install this integration to correlate alerts from Splunk into high-level incidents in BigPanda, and see insights from Splunk alongside the problems detected by other tools in your monitoring stack.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.