Skip to main content

Install a Splunk Enterprise Integration

Send monitoring events from Splunk Enterprise to BigPanda.

Install the Integration

The Splunk integration can be installed in 3 ways:

Before You Start

  • Splunk admin permissions are required to install the integration

  • Users who own searches sent to BigPanda must have the list_storage_passwords permission in Splunk

    If users cannot be granted this permission, then Splunk can be configured with credentials provided via environment variables instead of the config page. See the Configure Alternate Permissions section to learn how.

Create an App Key

Create an app key in BigPanda.

Integration specific

You'll need a separate app key for each integrated system.

https://files.readme.io/9865f2f-Integrations_App_Key_Creation.jpg

App Key Configuration in BigPanda

Install the BigPanda Splunk Add-on from the Web Page

  1. From the Splunk Web home page, click the Apps gear icon.

  2. Click Install Apps.

  3. Select Install to install an app. If the app that you want is not listed, or if the app indicates self-service installation is not supported, contact Splunk Support.

  4. Follow the prompts to complete the installation.

Install BigPanda Splunk Add-ons from within Splunk Enterprise

  1. Download the BigPanda for Splunk add-on from Splunkbase

    Distributed environments use search head instance

    If you are using a distributed Splunk search environment with multiple instances please install the app on your search head instance*

  2. Log into Splunk Enterprise

  3. On the Apps menu, click Manage Apps

  4. Click Install app from file

  5. In the Upload app window, click Choose File

  6. Locate the .tar.gz file you just downloaded, and then click Open or Choose

  7. Click Upload

  8. Click Restart Splunk, and then confirm that you want to restart

Install the Splunk Add-on directly into Splunk Enterprise through CLI

  1. Download the BigPanda for Splunk add-on from Splunkbase

    Distributed environments use search head instance

    If you are using a distributed Splunk search environment with multiple instances please install the app on your search head instance*

  2. Put the downloaded file in the $SPLUNK_HOME/etc/apps directory

  3. Untar and ungzip your app or add-on, using a tool like tar -xvf (on *nix) or WinZip (on Windows).

  4. Restart Splunk

Use the Splunk Deployment Server for Distributed Search Environments

When installing the BigPanda app in a Splunk distributed-search environment, Splunk does not automatically propagate the app to all nodes in the cluster, and the app must be installed on each node in the cluster either manually or through the deployment server.

Search Head Clusters

To deploy apps to a search head cluster, you must use the deployer. The deployer is a Splunk Enterprise instance that distributes apps and configuration updates to search head cluster members. The deployer cannot be a search head cluster member and must exist outside the search head cluster. See Use the deployer to distribute apps and configuration updates in the Distributed Search manual to learn more about the role of a deployer instance.

You cannot push an application with predefined credentials to a clustered search head in Splunk. To get around this, you have 2 options:

  • Push application from deployer to search head cluster without credentials – then manually enter the credentials afterwards via the UI

  • Push application from deployer to search head cluster with credentials in plain text (not encrypted)

When pushing the credentials from deployer to the search head cluster, they are sent in plain text, and then Splunk will automatically encrypt the values when the search heads reload with the new app/changes.

Configure the Splunk Integration

Users who own searches which are sent to BigPanda must have the list_storage_passwords permission in Splunk. Alternatively you can set permissions through environment variables. See Configure Alternate Permissions

The Splunk integration can be configured in 2 ways:

  • Through the BigPanda App Configuration Tab

  • Using the CLI

Configure Through the Configuration Tab

  1. Open the BigPanda app in Splunk and navigate to the Configuration > Global Settings tab

    splunk_bigpandaaddon

    Global Settings in the Configuration Tab

  2. Fill in your BigPanda App Key and API Token (BigPanda Bearer Token) inputs

  3. (Optional) Add a Basic Auth header in the Authorization Override field. This may be required by a third-party server when routing traffic through it.

  4. Click Save

  5. (Optional) If you are using a Proxy, enter the Proxy information in the Proxy tab

    splunk_bigpandaaddonconfig

    Proxy Options in the Configuration Tab

  6. Navigate to the Action Manager page and configure which alerts to send to BigPanda

Configure Through the CLI

  1. Ensure there is no $SPLUNK_HOME/etc/apps/TA-bigpanda/local/passwords.conf file

  2. Create $SPLUNK_HOME/etc/apps/TA-bigpanda/local/ta_bigpanda_settings.conf with the following:

    $SPLUNK_HOME/etc/apps/TA-bigpanda/local/ta_bigpanda_settings.conf

[additional_parameters]

api_url = https://integrations.bigpanda.io/oim/splunk_v2/alerts

app_key = app_key_here_in_plain_text

token = bearer_token_here_in_plain_text

  3. (Optional) Configure a Proxy through the CLI. To set the proxy through the CLI create ta_bigpanda_settings.conf at $SPLUNK_HOME/etc/apps/TA-bigpanda/local/ta_bigpanda_settings.conf. If it already exists then all you need to add is the following stanza:

    $SPLUNK_HOME/etc/apps/TA-bigpanda/local/ta_bigpanda_settings.conf
[proxy] 
proxy_password = XXXXXXXXXXXXXX 
proxy_port = PORT_NUMBER_HERE 
proxy_rdns = 1 | 0 (1 = enabled, 0 = disabled)
proxy_type = http | socks4 | socks5 
proxy_url = URL_HERE 
proxy_username = USERNAME_HERE 
proxy_enabled = 1 | 0 (1 = enabled, 0 = disabled)

  4. Navigate to the Action Manager page and configure which alerts to send to BigPanda

(Optional) Configure Alternate Permissions

Instead of using the Splunk search owner permissions, you can configure BigPanda to use environment variables instead. These environment variables can be set by the system administrator or configured in /etc/splunk-launch.conf.

Manual Updates Required

These variables will override any values entered in the Splunk App UI page. Adding permissions in the Splunk App while alternate permissions are configured may cause issues.

When authentication is configured in the UI, Authentication and API Token are encrypted in the bigpanda_settings.conf file. You must manually enter values in plain text for these fields in the ta_bigpanda_settings.conf file to leverage alternate permissions.

Enterprise Only

Alternate permissions will only work with Splunk Enterprise and are not supported for Splunk Cloud.

Remove existing permissions

Alternate permissions can only be configured through CLI. If you have already set permissions through the UI, these permissions must first be removed.

You do not need to take this step if permissions were not previously configured through the UI.

  1. Delete the passwords file

  2. Remove all configs from ta_bigpanda_settings.conf

  3. Restart Splunk

Add alternate permissions

  1. Update the ta_bigpanda_settings.conf file with the url, bearer token, and app key for this Splunk integration.

  2. Set the following environment variables using a shell startup script for splunk:

    ```
BIGPANDA_USE_ENV_PASSWORDS=true
BIGPANDA_BEARER_TOKEN=<Your token>
```

  3. If you are using a proxy then you also need to setup the following ENV variable:

    ```
BIGPANDA_PROXY_PASSWORD=XXXX
```

  4. Restart Splunk

Send Alerts to BigPanda

Once BigPanda is configured in Splunk, and the BigPanda action has been configured for alerts, you will need to enable the alerts to send to BigPanda.

When defining Trigger Conditions for Alerts to send to Bigpanda, make sure to select the Trigger as Once. BigPanda will extract all individual events from the search results as unique alerts, and does not need to send for each result.

Triggered Alerts action required

If the Triggered Alerts action has not been added to your Splunk configuration, you may need to first add this in addition to the BigPanda add on.

There are two types of saved searches in Splunk: alerts and reports. You can only use the add-on with alerts that you created, or that were shared with you by the owner.

The alert does not automatically start triggering when the add-on is used. The user must manually enable the alert for it to begin working.

Enable alerts using either the Action Manager, or Search Commands.

Action Manager

The Splunk Action Manager lists all alerts that have been created from previously saved searches. Each of these alerts can be enabled or disabled to send to BigPanda based on the configuration of the alert.

Splunk_ActionManager

Basic Action Manager

  1. Navigate to BigPanda > Action Manager

  2. The top left panel lists alerts that are currently enabled to send to BigPanda. The top right panel lists any alerts that are not enabled.

  3. Click Stop Sending All Alerts to BigPanda to disable all alerts from sending to BigPanda

  4. Click Send All Alerts to BigPanda to enable the alerts listed in the left pane to send to BigPanda

Potential Noise

The Send All Alerts to BigPanda option will attempt to send every search return for each saved search. For Splunk instances with a high volume of saved searches, this may result in oversized payloads that fail to be processed by BigPanda. Any payload over 6 MB will fail to process with BigPanda.

We recommend reviewing your saved searches to ensure that only actionable, useful information is being sent to BigPanda. See the FAQ How do I make my monitor saved searches actionable? for more information.

View search query at any time

Click any of the alerts on the Action Manager to open up the alert and see configuration settings and the original search query.

Advanced Action Manager

The advanced action manager allows you to enable or disable a subset of alerts using a query filter.

Splunk_AdvancedAssignmentManagement

Advanced Action Manager

  1. Enter a search value into the Filter text box to narrow the list below to only alerts that fit that value

  2. Click Send Filtered Alerts to BigPanda to enable all alerts listed to send to BigPanda.

  3. Click Stop Sending Filtered Alerts to BigPanda to disable all alerts listed from sending to BigPanda

Search Commands

BigPanda also provides search commands to stop or start sending alerts to BigPanda. The two available search commands are addbigpanda and removebigpanda.

Splunk_SearchCommand

BigPanda Search Commands

  1. In Splunk, Navigate to Search & Reporting

  2. In the search bar do a query following this search structure: | rest /services/saved/searches | YOUR FILTER | SEARCH COMMAND

  3. This may return saved Report entities in addition to saved Alerts. To ensure that only Alerts are displayed, add | search alert_threshold!="" to your search command.

Search commands in a distributed cluster

When using a distributed cluster, you must be logged into the captain node to be able to successfully use the BigPanda Search Commands. If you are logged into a different node you will receive authentication errors when attempting the BigPanda Search Commands.

The action manager relies in the backend on the BigPanda search commands addbigpanda and removebigpanda.

Configure Proxy Settings for the BigPanda Add-On (Optional)

1. From the BigPanda Add-On navigate to Configuration > Proxy

2. Check the box next to Enable

3. Select the Proxy Type you would like to use. The BigPanda Add-On supports HTTP, SOCKS4 and SOCKS5

4. Fill in your proxy settings. Username and password are optional

5. Click Save to apply the changes

In this section: