BigPanda Daemon (SNMP)

Supported Versions	Type	Authentication Type
v1, v2, v2c, v3	Daemon and Agent	Org Bearer Token

Supported systems

BigPanda SNMP supports RPM and Debian-based distributions.

BigPanda SNMP requires Python >= 3.6.8

Open Integration Manager

The Open Integration Manager is available for use with SNMP. For more information, see the Open Integration Manager documentation. To use OIM with SNMP, v10.2 of the BigPanda Agent must be installed.

BigPanda features the ability to ingest all SNMP Traps and convert them into BigPanda Alerts for correlation and aggregation in the platform. This is accomplished via a 2-pronged approach by utilizing the BigPanda SNMP Daemon and BigPanda Agent.

https://files.readme.io/5974aae-SNMP_08.jpg

Trap messages

Note that only SNMP TRAP messages are consumed. Other types of SNMP messages like GET and SET are not in scope.

Payload size

Alert payloads must be 6MB or smaller. Larger payloads will fail to process with BigPanda. We recommend reviewing your configurations to ensure that only actionable, useful information is being sent to BigPanda.

Consume SNMP Traps

The SNMP Daemon listens to traps on a port specified in the configuration. To safeguard against data loss or congestion, its consumption module does not share threads with the event-saving module.

SNMP communities up to v2 and SNMP v3 modes (noAuthNoPriv, authNoPriv, authPriv) are supported.

SNMP v3 requirement

Network discovery is required to identify the engine_id for each device sending SNMP v3 traps prior to configuring SNMP v3.

Both UDP and TCP are supported.

Translate SNMP Object ID names

SNMP traps in their original form are similar to key-value pairs. Field names are represented using number tuples called Object IDs, for example, 1.3.6.1.6.3.1.1.4.3.0. These tuples are then translated into human-readable file names via configuration files called Management Information Base (MIB) files.

MIB files

MIB files are typically provided by the vendors of systems and devices emitting the SNMP traps.

SNMP Daemon Configuration File

The snmp-daemon.json configuration file is the primary configuration file for the BigPanda SNMP agent.

By default, its location is /etc/bigpanda/snmpd/snmp-daemon.json

The following are a few important properties to note:

Property	Type	Description
`snmp.protocol`	string	The SNMP transport protocol.
`snmp.port`	int	The SNMP trap receiver port.
`snmp.community`	list	A list of community strings that are used by your SNMP trap emitters. (Default: `public`)
`processing.mibs-dir`	string	The location where the compiled Python MIB files reside. You likely won't need to change this.
`processing.event-configs`	list	A list of locations to find the event configuration files. The location path is relative to the directory in which the snmp-daemon.json file resides (by default `/etc/bigpanda/snmpd`).
`processing.target-dir`	string	The directory to write BigPanda events to. (Note, you’ll need to first ensure the directory exists.)
`processing.alert_on_failure`	boolean	Generate a BigPanda alert for traps that have a missing event config (default: `true`).
`processing.generate_event_configs`	boolean	The agent can auto generate a skeleton config file for traps that are not found. (default: `true`)
`processing.stress_test`	boolean	Enables the stress test feature.
`health_check.port`	int	The health check port to listen on (HTTP).
`health_check.file_name`	string	The file name to save the last event processed timestamp. This is stored relative to the `logs-dir` configuration in the logging section of the config.
`logging.log_format`	string	The daemon logger format - typically does not need to change.
`logging.log_level`	string	The daemon logging level. Possible values are `INFO` and `DEBUG` (default: `INFO`). `DEBUG` will debug level logs of the bp-snmpd agent.
`logging.verbose`	boolean	Logs internal pysnmp logs for debugging purposes (default: `false`).
`logging.raw_capture_level`	string	The BigPanda SNMP agent can log received SNMP alerts. It can log ones that failed to be sent to BigPanda, all received alerts, or none of them. Possible values are none, error, all (default: `error`).
`logging.logs-dir`	string	The directory to write daemon logs to. (Note, you’ll need to first ensure the directory exists.)
`logging.raw`	boolean	Logs raw incoming traps (default: `false`).
`logging.translated`	boolean	Logs trap payload after translation phase (default: `true`).
`logging.manipulated`	boolean	Logs trap payload after manipulation phase (default: `true`).
`security.local_engine_id`	string	The SNMP engine id of the local agent. This isn’t required for SNMP Traps, but will create a persistent boot file if used (typically in `/tmp/__pysnmp`).
`security.users`	list	User objects required for SNMP v3.

The following are a few important properties of a user object in the security.users list:

Property	Type	Description
`username`	string	Username of the sending client device.
`level`	string	`noAuthNoPriv` `authNoPriv` `authPriv`
`auth_protocol`	string	`usmHMACSHAAuthProtocol` `usmHMACMD5AuthProtocol`
`privacy_protocol`	string	`usmAesCfb128Protocol` `usmAesCfb256Protocol` `usmDESPrivProtocol`
`auth_key`	string	The user Authentication key.
`privacy_key`	string	The user Privacy key.
`engine_id`	string	Engine ID of the remote client. The ID must be unique to the device sending the trap.

Log location

All logs can be found in /var/log/bigpanda/

Community property

To prevent unauthorized traps, the SNMP daemon will only process traps when the captured community string matches one of the strings defined in the snmp.community property list. Traps that do not match will be dropped.

The snmp.community property is managed in the snmp-daemon.json config file.

️

Security object

The object is required for SNMP v3 support. The SNMP daemon will only process user objects with a matching engine_id for the device sending the trap. The engine_id is unique and is used to identify the device sending the trap.

The security.users list is managed in the the snmp-daemon.json config file.

Event Configuration Files

Event Configuration files are how the BigPanda SNMP agent knows how to process each trap. Each trap will have a configuration object. Multiple trap configurations can reside in one event configuration file, and multiple event configuration files can be used at the same time via the snmp-daemon.json configuration file.

In order for an incoming trap to be processed, it needs to have a corresponding trap configuration defined in an event configuration file.

Below are the properties used to capture the trap:

Setting	Type	Description
mib	string	(optional) The name of the MIB file to look for traps. The MIB should reside in the directory defined in the `snmp-daemon.json` configuration file. File extensions should be omitted. Required when using a MIB.
type	string	(optional) Declare and set this value to "`NO_MIB`" when using the no MIB feature. Required when using no MIB.
trap	string	The name of the incoming trap.
trap-oid	string	(optional) The incoming trap's Object Identifier. Required when using no MIB.
trap-var-binds	object	(optional) An object of key / value pairs where the key is the trap property name and the value is its position received in the trap. Required when using no MIB.
primary	string	(required) Sets the property that the BigPanda console would leverage for incident identification.
secondary	string	(required) Sets the property that the BigPanda console would leverage for incident identification.

Validation

If there are any integrity issues with an event config file, the agent will not load it at runtime.

The event config file will be skipped with a log entry stating as much:

ERROR: failed to load event config, skipping. path: /etc/bigpanda/snmpd/event_configs/file_name.json

Auto-generated event config

If no matching trap is found within the loaded event configs, the agent is designed to auto-generate a config file for the trap. The alert will not be sent to the BigPanda console.

More details on the auto-generate feature.

No MIB Support

The BigPanda SNMP agent can consume traps for which there is not a corresponding MIB file. As described in Event Configuration Files, the following properties in the Event Configuration file are required when using this feature:

"type": "NO_MIB" - Use this declaration when using the no MIB feature
trap - A friendly name for the trap. This will be used to easily identify the trap used.
trap-oid - The trap's Object Identifier
trap-var-binds - An object of key / value pairs where the key is the trap property name and the value is its position received in the trap

For example, here is an example Event Configuration snippet showing how to declare no MIB for a MIBless trap:

[
  {
    "type": "NO_MIB",
    "trap": "eatBamboo",
    "trap-oid": "2.9.7.16.1.14.4.1",
    "trap-var-binds": {
      "host": 2,
      "check": 4,
      "status": 7,
      "description": 5,
      "cluster": 9
    }
    ...
  }
]

Notice that in the trap-var-binds object, the host property would be the second property received in the trap, check is in the fourth position, status is in the seventh spot, and so on.

Auto Generate Event Config

The BigPanda SNMP agent can generate a basic event config file for a trap that is not found. This will work both for MIB and NO_MIB types.

There are two templates that are leveraged when creating an event config file. Both can be found in /etc/bigpanda/snmpd/event_configs/.

mib_template.json - Used when generating a MIB type config
no_mib_template.json - Used when generating a NO_MIB type config

Auto-generated event config location

New auto-generated event configs can be found in /etc/bigpanda/snmpd/auto-generated-event-configs/

Here is an example of an auto-generated event config file:

[
  {
    "type": "NO_MIB",
    "trap": "2.9.7.16.1.14.4.1",
    "trap-oid": "2.9.7.16.1.14.4.1",
    "trap-var-binds": {
      "varbind1": 1,
      "varbind2": 2,
      "varbind3": 3
    },
    "copy": {
      "severity": "status"
    },
    "rename": {
    },
    "set": {
    },
    "map-status": {
      "Fatal": "critical",
      "Critical": "critical"
    },
    "primary": "<PRIMARY_VARBIND>",
    "secondary": "<SECONDARY_VARBIND>"
  }
]

New files

In new auto-generated event config files, you'll want to be sure to:

Change the trap value to something more descriptive
Map the primary and secondary properties to the proper trap-var-bind keys
Ensure that the correct status-based field is mapped to the "status" property
Ensure that the incoming status values are properly mapped to expected BigPanda values.

Once you've made the necessary changes to the file, rename it to something more descriptive and move it to /etc/bigpanda/snmpd/event_configs/.

Then add the event config file name to the list of event configs in the snmp-daemon.json file.

Finally, restart the bigpanda-snmpd service to put the changes into effect.

Primary and secondary properties

There are certain tags within the payload that are reserved words for BigPanda and cannot be used as the primary or secondary properties.

Please reference this list of reserved words before naming your properties.

Manipulate the Payload

The SNMP trap payload is manipulated in order to prepare it for consumption by the BigPanda Agent, and ultimately BigPanda. There are several supported types of manipulation actions. All manipulations are optional.

Manipulation	Type	Description
`copy`	object	For each 'from' / 'to' pair, copy the value from key 'from' to a new key with the name 'to'. (See more details.)
`rename`	object	For each key / value pair, rename the key to the corresponding value. (See more details.)
`set`	object	For each key / value pair, set the key to the corresponding value in the trap payload Value could be a template (ex. "`my name is {name}`"). (See more details.)
`drop`	list	List of fields to drop from the trap payload. (See more details.)
`map-status`	object	Translates a trap's statuses to corresponding BigPanda supported event severities (`critical`, `warning`, `unknown`, and `ok`)
`resolve-mapping`	object	Change the status field value to "ok" if one of the key value pairs are found in the trap payload.
`set-time`	boolean	Whether or not to set unix time in the trap payload.
`add_system_fields`	boolean	Whether or not to add system fields on each trap (ex. community, daemon port). (default `true`)
`primary`	string	Set the primary property to the specified value. (See more details.)
`secondary`	string	Set the secondary property to the specified value. (See more details.)
`conditions`	object array	An array of objects that can perform conditional set or suppress manipulations. (See more details.)
`custom-actions`	object	An object containing the key / value pairs to add to the alert if a condition is met. (See more details.)

Processing order

Please note that the copy, rename, set, drop, map-status, and resolve-mapping manipulations are processed in this order.

Copy, Rename, Set, & Drop Fields

The SNMP trap key-value pairs employ key names defined by the vendor of the system or device emitting the SNMP trap. Some of these keys might need to be copied, renamed or dropped before they can be sent to BigPanda. Other keys might need to be set.

Case sensitivity is not enforced on string matches. Below is an excerpt of drop, rename, and set examples.

{
  "mib": "xyz-platform-mib",
  "trap": "generic-problem-trap",
  ...
  "copy": {
    "eventOccurences": "event_occurrences"  
  },
  "rename": {
    "notification": "description",
    "clientName": "host",
    "checkName": "check",
    "checkSeverity": "status"
  },
  "set": {
    "team": "billing",
    "runbook": "http://docs.acme.com/af432"
  },
  "drop": [
    "eventID", 
    "eventOccurences", 
  ]
  ...
}

The Set field can also accept variable substitution as the value. Available variables are fields returned by the MIB, or if renaming fields, use the new name.

For example:

{
  "mib": "xyz-platform-mib",
  "trap": "generic-problem-trap",
  ...
  "set": {
    "team": "billing",
    "runbook": "http://docs.acme.com/af432",
    "interesting_field": "${host}_${check}" 
  }
  ...
}

Conditions

The BigPanda SNMP agent can perform actions against trap data if certain conditions are met. If the conditions are met, the daemon can either suppress the trap - meaning that the trap is processed no further and will not be sent to BigPanda - or the daemon can set additional fields that will be included in the trap data sent to BigPanda.

In the example Event Configuration below, you can see both the suppress and set conditions in action.

{
  "mib": "xyz-platform-mib",
  "trap": "generic-problem-trap",
  ...
  "conditions": [
    {
      "bp-name": "suppress-these",
      "bp-action": "suppress",
      "host": ".*moo.*"
    },
    {
      "bp-name": "set-these",
      "bp-action": "set-the-region",
      "host": ".*-mtnview-.*"
    }
  ],
  "custom-actions": {
    "set-the-region": {
      "type": "set",
      "region": "mountain_view"
    }
  }
  ...
}

Suppress

When "bp-action": "suppress" is declared, it will suppress any alerts that match the key / value trap properties.

In the example above, any trap with a host property containing moo will be suppressed.

Suppressed traps

Suppressed traps are written to the bp-suppress log

Set

When bp-action is not suppress, it is treated as an alias for a custom action. The value must therefore exist as an object within the custom-actions object.

Within the custom-actions object is a list of aliased action objects. The aliased action object is structured as such:

{
  ...
  "custom-actions": {
    "<ALIASED_ACTION_NAME>": {
      "type": "set",
      "key1": "value1",
      "key2": "value2",
      ...
    }
  }
  ...
}

"<ALIASED_ACTION_NAME>" is the value set from the bp-action value in the conditions object.
"type": "set" tells the BigPanda SNMP agent which action to perform
Following the set declaration, all other key / value pairs are the properties that are dynamically added to the alert.

From the example above:

{
  "mib": "xyz-platform-mib",
  "trap": "generic-problem-trap",
  ...
  "conditions": [
    {
      "bp-name": "suppress-these",
      "bp-action": "suppress",
      "host": ".*moo.*"
    },
    {
      "bp-name": "set-these",
      "bp-action": "set-the-region",
      "host": ".*-mtnview-.*"
    }
  ],
  "custom-actions": {
    "set-the-region": {
      "type": "set",
      "region": "mountain_view"
    }
  }
  ...
}

Any trap with a host property with a value containing -mtnview- will have a new tag added to the alert called region with a value of mountain_view.

Regex Capture Capability

The agent allows for a more flexible approach to manipulating your trap using Python Regex to match an expression and extract any relevant information.

Take, for example, the following event-config file

[
   {
       "type": "NO_MIB",
       "trap": "SAMPLE_TRAP",
       "trap-oid": "1.3.6.1.4.1.1234",
       "trap-var-binds": {
           "host": 1,
           "check": 2,
           "description": 3
       },
       "set": {
         ...
       },
       "conditions": [
           {
               // Tag to use a python regex on
               "description": ".*(?P<animal>chicken).*",
               "bp-action": "set-message",
               "bp-name": "Setting Message"
           }
       ],
       "custom-actions": {
           "set-message": {
               "type": "set"
               // Using the defined capture group names
               // a custom value can be created
               "message": "Yes, the {cap_group1} did cross!"
           }
       },
       "primary": "host",
       "secondary": "check"
   }
]

The conditions array has an object which will look at the description tag and perform the set-message action that is defined in the custom-actions property.

The above sample is performing the following actions:

Creating a capture group called animal and looking for the string literal "chicken" to match in the description tag.
Next, the set-message action is of type set which in this case is creating a new tag message.
The value for this new tag is a string using the defined capture group name as a variable.

The python syntax to create a capture group and label it is as follows: (?P<capture_name>{{regex expression}})

Escape characters

If a more complex Python Regex is required that includes the need to escape characters.

For Example: (?P<name>._) $ ._ $ (?P<check>._)._

You need to account that this is a string within your JSON file and the escape character needs its own escape character.

A properly formatted string to the above example would be "(?P<name>._) \$ ._ \$ (?P<check>._)._"

Map Severities

SNMP traps may contain a wide variety of status codes or severities. These may be represented as a string or an integer. The map-status function translates a trap's statuses to corresponding BigPanda supported event severities (critical, warning, unknown, and ok).

Status property

map-status requires a status property. If the trap does not natively send one, then you will need to either rename the trap's status field or copy its status field to a new status field

{
  "mib": "xyz-platform-mib",
  "trap": "generic-problem-trap",
  ...
  "map-status": {
    "Major": "critical",
    "Minor": "warning",
    "Resolved": "Ok"
  }
  ...
}

Map severities logic

Once the map severities logic has been executed, future events without a valid status will be logged and then dropped.

Define Primary & Secondary Properties

Events propagated through the BigPanda platform contain two optional attributed called primary property and secondary property. These attributes are used to determine which fields should be used as a title and subtitle, respectively, in the UI.

As this is a proprietary concept to BigPanda and nonexistent in SNMP traps, users may configure these attributes on a per trap-type basis.

{
  "mib": "xyz-platform-mib",
  "trap": "generic-problem-trap",
  ...
  "primary": "host",
  "secondary": "check"
  ...
}

Capture Events to Local Disk

By this point, the SNMP traps have been consumed, translated, manipulated, and readied for saving to a local file storage. The target directory for saving is specified in the configuration file. The generated files should have Read/Write permissions assigned only to the bigpanda group for security reasons. All other users and groups should have denied read access.

Below are two examples of event files as they appear on disk.

{
  "timestamp": 1512923088,
  "mib": "xyz-platform-mib",
  "trap": "generic-problem-trap",
  "switch": "ny-switch-83",	
  "problem": "firmware update required",
  "status": "critical",
  "primary_property": "switch",
  "secondary_property": "problem"
}
```
```Example File2.JSON
{
  "timestamp": 1512923138,
  "mib": "bmc-mib",
  "trap": "system-resources-trap",
  "host": "server8",
  "check": "cpu load",
  "description": "RESOLVED - CPU Load < 80%",
  "status": "ok"
}

Once saved to disk, the event files are collected by the BigPanda Agent and forwarded to the BigPanda platform in the cloud.

Troubleshooting

The following Shell commands will help you perform various checks on the SNMP daemon.

Check pid file:

cat /var/run/bigpanda-snmpd.pid

Check if application is running:

ps -ef | grep bigpanda-snmpd

Check if application is listening

sudo netstat -nltup | grep <PORT>

All logs can be found at /var/log/bigpanda

Simulate a Trap

Simulating an SNMP trap to the daemon may be useful for testing purposes. First, install snmptrap with these commands in your Shell.

RedHat/CentOS

sudo yum install net-snmp-utils

Debian/Ubuntu

sudo apt-get install snmp

The SNMP Trap command structure up to SNMP v2c follows this format: snmptrap -v ${SNMP_VERSION} -c ${COMMUNITY} ${DESTINATION_HOST}:${DESTINATION_PORT} ${UPTIME} ${OID or MIB} ${OBJECT} ${VALUE_TYPE} ${VALUE}

Below is an example of this structure in use.

sudo snmptrap -v 2c -c public 127.0.0.1:5000 1 1.3.6.1.2.1.1 1.3.6.1.2.1.1.5 s "production-switch-1" 1.3.6.1.2.1.1.1 s "SNMP trap test"

The SNMP Trap command structure for SNMP v3 follows this format: snmptrap -v3 -e ${ENGINE_ID} -u ${USERNAME} -l ${LEVEL} -a ${AUTH_TYPE} -A ${AUTH_KEY} -x ${PRIV_TYPE} -X ${PRIVACY_KEY} ${DESTINATION_HOST}:${DESTINATION_PORT} ${UPTIME} ${OID or MIB} ${OBJECT} ${VALUE_TYPE} ${VALUE}

Below is an example of this structure in use.

$ sudo snmptrap -v3 -e 0x8000000008675309 -u panda -l authPriv -a SHA -A snmpauthpass123 -x AES -X snmpkeypass123 127.0.0.1:5000 "" 1.3.6.1.2.1.1 1.3.6.1.2.1.1.5 s "production-switch-v3" 1.3.6.1.2.1.1.1 s "SNMP v3 trap test"

Replay Traps

The BigPanda SNMP package includes a python utility that can replay raw traps that were captured to the bp-snmp-raw-captures or bp-snmp-raw-captures-errors log files.

Within the capture log files, you will find raw trap data that looks like this:

SNMP_VERSION 1.3.6.1.2.1.1.1234 = v2
pysnmp.proto.rfc1902.TimeTicks 1.3.6.1.2.1.1.3.0 = 1
pyasn1.type.univ.ObjectIdentifier 1.3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.2.1.1.1234
pysnmp.proto.rfc1902.OctetString 1.3.6.1.2.1.1.1234.1 = acme-mtvnview-01
pysnmp.proto.rfc1902.OctetString 1.3.6.1.2.1.1.1234.5 = some_check
pysnmp.proto.rfc1902.Integer 1.3.6.1.2.1.1.1234.6 = 1
--------------------------------------------------

Each raw trap entry begins with SNMP_VERSION and ends with --------------------------------------------------.

When replaying the raw traps, we recommend to copy the the raw trap(s) from the log to a dedicated replay file.

To replay the traps, run this command:

python3 bp-snmp-replay.py --capture_log_file_path CAPTURE_LOG_FILE_PATH

Location

By default, bp-snmp-replay.py is located in /opt/bigpanda/bigpanda-snmpd/bp_snmpd

Uninstall the Integration

Deleting an integration requires changes to both the integrated system and BigPanda. You must uninstall the integration on the integrated system and then delete the integration from BigPanda.

Replacing an existing integration

When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.

Stop Sending Data to BigPanda

On the integrated system, disable any settings that send data to BigPanda.

Manually resolve any open alerts sent from the integration to remove the associated incidents from your incident feed. These incidents will not automatically resolve without an ok status from the original sending integration.

Delete the Integration from BigPanda

In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
In the integration details on the right, click Delete Integration. A support message opens, pre-populated with a request to delete the selected integration.
Press Enter to send the request.
The BigPanda support team will remove the integration from the UI.

Data removal

This procedure does not remove any data from BigPanda or the integrated system. As needed, remove data from each system before deleting the integration.

In this section: