Install a Splunk Enterprise Integration

BigPanda provides a native Splunk App to let you easily send Splunk alerts to BigPanda. The app provides a native Splunk alert action which will forward the Splunk alert to the BigPanda integration. The integration will take the Splunk alert and normalize it into one or more BigPanda alerts, one alert for each row in the search result.

Splunk Enterprise and Splunk Cloud have similar but different requirements when integrating with BigPanda.

• See the Install a Splunk Enterprise Integration documentation for details on configuring and installing Splunk Enterprise.
• See the Install a Splunk Cloud Integration documentation for details on configuring and installing Splunk Cloud.

👍

Open Integration Manager

The Open Integration Manager is available for use with Splunk Enterprise. For more information, see the Open Integration Manager documentation.

❗️

Considerations before updating Splunk to OIM

The Splunk OIM integration uses Time Based Alert Resolution to resolve alerts. Do not switch from the legacy solution if you require variable resolve times based on the Splunk expire time.

The legacy endpoint for Splunk is https://integrations.bigpanda.io/splunk/alerts, and the new OIM endpoint is https://integrations.bigpanda.io/oim/splunk_v2/alerts.

❗️

Noisy Integration

Splunk can be a noisy monitoring system. For organizations using the BigPanda consumption based pricing model, it can consume large amounts of credits very quickly. For help configuring Splunk to be less noisy, see the Splunk FAQ.

Key Features

• Easily send Splunk alerts to BigPanda using the native BigPanda action in Splunk
• Keep your team focused on what matters with auto-resolution of Splunk alerts in BigPanda
• Simple & efficient management of all alerts going to BigPanda via custom alert management dashboard and search commands
• Customizable alert properties
• Proxy support for on-premise Splunk Enterprise deployments

Install the Integration

The Splunk integration can be installed in 3 ways:

• From the Splunk Web Page
• Within Splunk Enterprise using Splunkbase
• Using CLI

Before You Start

• Splunk admin permissions are required to install the integration
• Users who own searches sent to BigPanda must have the list_storage_passwords permission in Splunk If users cannot be granted this permission, then Splunk can be configured with credentials provided via environment variables instead of the config page. See the Configure Alternate Permissions section to learn how.

Create an App Key

Create an app key in BigPanda.

👍

Integration specific

You'll need a separate app key for each integrated system.

Install the BigPanda Splunk Add-on from the Web Page

1. From the Splunk Web home page, click the Apps gear icon.
2. Click Install Apps.
3. Select Install to install an app. If the app that you want is not listed, or if the app indicates self-service installation is not supported, contact Splunk Support.
4. Follow the prompts to complete the installation.

Install BigPanda Splunk Add-ons from within Splunk Enterprise

1. Download the BigPanda for Splunk add-on from Splunkbase

   🚧

   Distributed environments use search head instance

   If you are using a distributed Splunk search environment with multiple instances please install the app on your search head instance*

2. Log into Splunk Enterprise

3. On the Apps menu, click Manage Apps

4. Click Install app from file

5. In the Upload app window, click Choose File

6. Locate the .tar.gz file you just downloaded, and then click Open or Choose

7. Click Upload

8. Click Restart Splunk, and then confirm that you want to restart

Install the Splunk Add-on directly into Splunk Enterprise through CLI:

1. Download the BigPanda for Splunk add-on from Splunkbase

   🚧

   Distributed environments use search head instance

   If you are using a distributed Splunk search environment with multiple instances please install the app on your search head instance*

2. Put the downloaded file in the $SPLUNK_HOME/etc/apps directory

3. Untar and ungzip your app or add-on, using a tool like tar -xvf (on *nix) or WinZip (on Windows).

4. Restart Splunk

Use the Splunk Deployment Server for Distributed Search Environments

When installing the BigPanda app in a Splunk distributed-search environment, Splunk does not automatically propagate the app to all nodes in the cluster, and the app must be installed on each node in the cluster either manually or through the deployment server.

Search Head Clusters

To deploy apps to a search head cluster, you must use the deployer. The deployer is a Splunk Enterprise instance that distributes apps and configuration updates to search head cluster members. The deployer cannot be a search head cluster member and must exist outside the search head cluster. See Use the deployer to distribute apps and configuration updates in the Distributed Search manual to learn more about the role of a deployer instance.

You cannot push an application with predefined credentials to a clustered search head in Splunk. To get around this, you have 2 options:

• Push application from deployer to search head cluster without credentials – then manually enter the credentials afterwards via the UI
• Push application from deployer to search head cluster with credentials in plain text (not encrypted)

When pushing the credentials from deployer to the search head cluster, they are sent in plain text, and then Splunk will automatically encrypt the values when the search heads reload with the new app/changes.

Configure the Integration

Users who own searches which are sent to BigPanda must have the list_storage_passwords permission in Splunk. Alternatively you can set permissions through environment variables. See Configure Alternate Permissions

The Splunk integration can be configured in 2 ways:

• Through the BigPanda App Configuration Tab
• Using the CLI

Configure Through the Configuration Tab

1. Open the BigPanda app in Splunk and navigate to the Configuration > Global Settings tab

   

2. Fill in your BigPanda App Key and API Token (BigPanda Bearer Token) inputs

3. (Optional) Add a Basic Auth header in the Authorization Override field. This may be required by a third-party server when routing traffic through it.

4. Click Save

5. (Optional) If you are using a Proxy, enter the Proxy information in the Proxy tab

   

6. Navigate to the Action Manager page and configure which alerts to send to BigPanda

Configure Through the CLI

1. Ensure there is no $SPLUNK_HOME/etc/apps/TA-bigpanda/local/passwords.conf file

2. Create $SPLUNK_HOME/etc/apps/TA-bigpanda/local/ta_bigpanda_settings.conf with the following:

   $SPLUNK_HOME/etc/apps/TA-bigpanda/local/ta_bigpanda_settings.conf

   
   [additional_parameters]
   
   api_url = https://integrations.bigpanda.io/oim/splunk_v2/alerts
   
   app_key = app_key_here_in_plain_text
   
   token = bearer_token_here_in_plain_text
   

3. (Optional) Configure a Proxy through the CLI. To set the proxy through the CLI create ta_bigpanda_settings.conf at $SPLUNK_HOME/etc/apps/TA-bigpanda/local/ta_bigpanda_settings.conf. If it already exists then all you need to add is the following stanza:

   $SPLUNK_HOME/etc/apps/TA-bigpanda/local/ta_bigpanda_settings.conf

   [proxy] 
   proxy_password = XXXXXXXXXXXXXX 
   proxy_port = PORT_NUMBER_HERE 
   proxy_rdns = 1 | 0 (1 = enabled, 0 = disabled)
   proxy_type = http | socks4 | socks5 
   proxy_url = URL_HERE 
   proxy_username = USERNAME_HERE 
   proxy_enabled = 1 | 0 (1 = enabled, 0 = disabled)
   

4. Navigate to the Action Manager page and configure which alerts to send to BigPanda

(Optional) Configure Alternate Permissions

Instead of using the Splunk search owner permissions, you can configure BigPanda to use environment variables instead. These environment variables can be set by the system administrator or configured in /etc/splunk-launch.conf.

❗️

Manual Updates Required

These variables will override any values entered in the Splunk App UI page. Adding permissions in the Splunk App while alternate permissions are configured may cause issues.

When authentication is configured in the UI, Authentication and API Token are encrypted in the bigpanda_settings.conf file. You must manually enter values in plain text for these fields in the ta_bigpanda_settings.conf file to leverage alternate permissions.

🚧

Enterprise Only

Alternate permissions will only work with Splunk Enterprise and are not supported for Splunk Cloud.

Remove existing permissions

Alternate permissions can only be configured through CLI. If you have already set permissions through the UI, these permissions must first be removed.

You do not need to take this step if permissions were not previously configured through the UI.

1. Delete the passwords file
2. Remove all configs from ta_bigpanda_settings.conf
3. Restart Splunk

Add alternate permissions

1. Update the ta_bigpanda_settings.conf file with the url, bearer token, and app key for this Splunk integration.

2. Set the following environment variables using a shell startup script for splunk:

   BIGPANDA_USE_ENV_PASSWORDS=true
   BIGPANDA_BEARER_TOKEN=<Your token>
   

3. If you are using a proxy then you also need to setup the following ENV variable:

   BIGPANDA_PROXY_PASSWORD=XXXX
   

4. Restart Splunk

Send Alerts to BigPanda

Once BigPanda is configured in Splunk, and the BigPanda action has been configured for alerts, you will need to enable the alerts to send to BigPanda.

When defining Trigger Conditions for Alerts to send to Bigpanda, make sure to select the Trigger as Once. BigPanda will extract all individual events from the search results as unique alerts, and does not need to send for each result.

❗️

Triggered Alerts action required

If the Triggered Alerts action has not been added to your Splunk configuration, you may need to first add this in addition to the BigPanda add on.

There are two types of saved searches in Splunk: alerts and reports. You can only use the add-on with alerts that you created, or that were shared with you by the owner.

The alert does not automatically start triggering when the add-on is used. The user must manually enable the alert for it to begin working.

Enable alerts using either the Action Manager, or Search Commands.

Action Manager

The Splunk Action Manager lists all alerts that have been created from previously saved searches. Each of these alerts can be enabled or disabled to send to BigPanda based on the configuration of the alert.

1. Navigate to BigPanda > Action Manager
2. The top left panel lists alerts that are currently enabled to send to BigPanda. The top right panel lists any alerts that are not enabled.
3. Click Stop Sending All Alerts to BigPanda to disable all alerts from sending to BigPanda
4. Click Send All Alerts to BigPanda to enable the alerts listed in the left pane to send to BigPanda

🚧

Potential Noise

The Send All Alerts to BigPanda option will attempt to send every search return for each saved search. For Splunk instances with a high volume of saved searches, this may result in oversized payloads that fail to be processed by BigPanda. Any payload over 6 MB will fail to process with BigPanda.

We recommend reviewing your saved searches to ensure that only actionable, useful information is being sent to BigPanda. See the FAQ How do I make my monitor saved searches actionable? for more information.

👍

View search query at any time

Click any of the alerts on the Action Manager to open up the alert and see configuration settings and the original search query.

Advanced Action Manager

The advanced action manager allows you to enable or disable a subset of alerts using a query filter.

1. Enter a search value into the Filter text box to narrow the list below to only alerts that fit that value
2. Click Send Filtered Alerts to BigPanda to enable all alerts listed to send to BigPanda.
3. Click Stop Sending Filtered Alerts to BigPanda to disable all alerts listed from sending to BigPanda

Search Commands

BigPanda also provides search commands to stop or start sending alerts to BigPanda. The two available search commands are addbigpanda and removebigpanda.

1. In Splunk, Navigate to Search & Reporting
2. In the search bar do a query following this search structure: | rest /services/saved/searches | YOUR FILTER | SEARCH COMMAND
3. This may return saved Report entities in addition to saved Alerts. To ensure that only Alerts are displayed, add | search alert_threshold!="" to your search command.

📘

Search commands in a distributed cluster

When using a distributed cluster, you must be logged into the captain node to be able to successfully use the BigPanda Search Commands. If you are logged into a different node you will receive authentication errors when attempting the BigPanda Search Commands.

The action manager relies in the backend on the BigPanda search commands addbigpanda and removebigpanda.

Configure Proxy Settings for the BigPanda Add-On (Optional)

1. From the BigPanda Add-On navigate to Configuration > Proxy

2. Check the box next to Enable

3. Select the Proxy Type you would like to use. The BigPanda Add-On supports HTTP, SOCKS4 and SOCKS5

4. Fill in your proxy settings. Username and password are optional

5. Click Save to apply the changes