Zabbix

Zabbix provides an open-source solution for network and application monitoring of millions of metrics.

Supported VersionsTypeAuthentication Type
2.2, 3.0, 3.2, 4.x, 5.xAgentBearer Token

Zabbix is an open-source solution for network and application monitoring. The BigPanda Agent queries for all new Zabbix trigger events in real-time via the Zabbix API. To ensure a full sync of all current alerts, the BigPanda Agent also queries for all active triggers on each monitored host. The query intervals are configurable. See Customizing Zabbix for more information.

Key Features

  • Keeps you up-to-date on the state of triggers by correlating the Zabbix events for each trigger into easy-to-track alerts.
  • Correlates alerts for triggers in the PROBLEM state into actionable incidents in BigPanda.

Before You Start

To ensure a successful integration of Zabbix with BigPanda, the following requirements need to be established first.

  • A Zabbix user with Read access to all of the host groups that you want to monitor in BigPanda. Please see the Configuring Zabbix Permissions section for more information.
  • Root or sudo permissions to install the BigPanda agent on a Linux server that is on the same network or can access the Zabbix API.
  • The Zabbix server must be able to communicate with the BigPanda server via https (port 443).

Install the Integration

Administrators can install the integration by following the on-screen instructions in BigPanda.

Disable Zabbix Triggers

Triggers that are disabled in Zabbix are not monitored by BigPanda. If a trigger is disabled when an alert is open, the alert will not close automatically and the BigPanda incident will remain open. In this case, you must manually resolve the BigPanda incident.

We recommend that Zabbix administrators check that all alerts are resolved in BigPanda before disabling the related trigger.

Limitations

  • The agent creates a dedicated user and user group in Zabbix for querying the Zabbix data. The user type affects how BigPanda is granted access to your monitoring data. For more information about how to grant the desired access, see Configuring Zabbix Permissions..
  • To stay within the memory limit of the Zabbix API, the agent uses batching during the full sync. For a successful full sync, your Zabbix configuration must remain within these guidelines:
    • Less than 100,000 host groups.
    • Less than 100,000 monitored hosts per host group.
    • Less than 100,000 active triggers per monitored host.

📘

The success of the full sync does not affect the continuous sync, which ensures that all new trigger events are received and correlated accurately in BigPanda.

##Installing The Integration
Administrators can install the integration by following the on-screen instructions in BigPanda. For more information, see Installing an Integration.

Zabbix Data Model

BigPanda normalizes alert data from Zabbix into tags. You can use tag values to filter the incident feed and to define filter conditions for Environments.

During alert correlation, BigPanda assigns correlated events an incident identifier or incident_key. By default, the incident_identifier for Zabbix alerts is the triggerid.

The incident_identifier is critical during the alert correlation process and defines deduplication and resolution rules.

Standard Tags

Tag

Description

Attributes

hosts

Names of the Zabbix hosts that the trigger belongs to.

Primary Property.

items

Zabbix items that are contained in the trigger. Zabbix items are individual metrics defined for Zabbix hosts. For example:

  • net.tcp.service[smtp,,25].last

  • ntp.client.offset.max

  • disk.io.status[sda].sum

Secondary Property.

item_values

Items and their corresponding values. The items match the list in the items tag. For example:

  • net.tcp.service[smtp,,25].last: 1

  • ntp.client.offset.max: 0.5258

  • disk.io.status[sda].sum: 86.0000%

triggerid

Name of the trigger in Zabbix.

Incident_Identifier

trigger_description

Description of the trigger in Zabbix.

maintenance

Flag indicating if the host is under maintenance. This tag is included only when the value is true.

expression

Trigger expression configured in Zabbix to calculate the trigger state.

acknowledged

Flag indicating if the latest event for the trigger has been acknowledged in Zabbix.

rum book_url

URL configured in Zabbix as a link associated with the trigger.

Configure Zabbix Permissions

Connecting BigPanda with Zabbix requires a new Zabbix user with read access to all the host groups that you want to monitor in BigPanda. During configuration, you can choose the level of access for this user—Zabbix Super Admin or Zabbix Admin—and the user is created automatically with the user name Bigpanda in a new user group called BigPanda Service.

Granting the user Zabbix Super Admin permissions is the easiest way to ensure BigPanda has the required access. If granting this level of access is not an option, you can choose Zabbix Admin. The user will be given permissions to all existing host groups at the time of the configuration and you must manually grant permissions to any host groups added after the initial configuration. You can view and change the type of user in Zabbix at any time.

Before You Start

Obtain access to the Zabbix UI with a Zabbix Super Admin user.

Grant BigPanda Access To Host Groups In Zabbix

If the Bigpanda user has Zabbix Super Admin permissions, then BigPanda has access to relevant data for all host groups in Zabbix without additional configuration. If the Bigpanda user has Zabbix Admin permissions, then any access to host groups must be explicitly granted in Zabbix by configuring permissions for the BigPanda Service user group.
To manually grant BigPanda access to host groups in Zabbix:

  1. In Zabbix, go to Administration > Users > BigPanda Service > Permissions. The Zabbix permissions for the BigPanda Service user group are displayed.
  2. Under Composing Permissions > Read only, click Add and then select the host groups.

📘

Host groups that appear under Calculated Permissions > Deny are not accessible to the Bigpanda user, so events for these host groups will not be sent to BigPanda. If you want to send events for these host groups to BigPanda, add them to the Read-Only permissions, as described in step 2, and they will be automatically removed from the Deny list.

  1. At the bottom of the screen, click Update.

Change the BigPanda Access Level in Zabbix

1.In Zabbix, go to Administration > Users.

  1. Under Members, click the Bigpanda user.
  2. Click Permissions.
  3. In the User type field, select Zabbix Super Admin or Zabbix Admin to change the access level.
  4. At the bottom of the screen, click Update.

Customize Zabbix

By editing the config file for the agent, you can configure settings for how the BigPanda agent queries Zabbix for events. You can configure the batch size for the incremental sync and the query interval and batch size for the full sync. You can also disable the full sync.

Before You Start

Obtain access to the server where the BigPanda agent is installed.

Configure The Incremental Sync

The agent queries for all new Zabbix trigger events in real-time via the Zabbix API. You may want to configure this incremental sync to reduce the load on your Zabbix server, depending on the volume of events Zabbix generates.

  1. Open the config file, which is located at /etc/bigpanda/bigpanda.conf.
  2. Locate the settings for the Zabbix incremental sync, named zabbix/state.
  3. Change the value of the max_event_batch_size to your preferred batch size.
    The batch size is the maximum number of events that the Zabbix API can return at a time during an incremental sync.
    The default value is 100.

Configure the Full Sync

To ensure a full sync of all current alerts, the BigPanda agent queries for all Zabbix events at regular intervals. You may want to configure the full sync to reduce the load on your Zabbix server, depending on the number of monitored triggers you have configured.

  1. Open the config file, which is located at /etc/bigpanda/bigpanda.conf.
  2. Locate the settings for the Zabbix full sync, named zabbix/fullsync.
  3. Change the following settings to your preferred values.
SettingDescriptionDefault Value
full_sync_intervalInterval between full syncs, in seconds.900 (15 minutes)
max_trigger_batch_sizeMaximum number of triggers that the Zabbix API can return at a time during a full sync.50

Disable the Full Sync

📘

The full sync acts as a backup to the ongoing incremental sync to ensure accuracy. Disable this feature only if you need to reduce the load on your Zabbix server.

  1. Open the config file, which is located at /etc/bigpanda/bigpanda.conf.
  2. Locate the settings for the Zabbix full sync, named zabbix/fullsync
  3. Add the following key and value: "enabled": false.

❗️

Make sure you add the pair outside of the config property.

"name": "zabbix/fullsync", 
"enabled": false, 
"config": {
  "user": "BigPanda",...
  },

Troubleshooting

Follow these steps to diagnose and solve common problems with the Zabbix integration. When troubleshooting, you can use the agent log files, which are located at /var/log/bigpanda/agent.log by default.

Problem: BigPanda agent cannot connect to Zabbix API

Symptoms & Diagnosis
This error appears in the BigPanda agent log:
Error while logging into Zabbix API

Possible Causes
The log may contain further details about the possible cause for this error. Possible causes include:

  • The agent does not have the correct credentials for the BigPanda user in Zabbix (Log file contains: Invalid params., Login name or password is incorrect.)
  • The agent does not have the correct Zabbix host credentials (Log file contains: 404 Client Error: Not Found).

Solutions
Verify that the BigPanda user credentials and the Zabbix host credentials are correct.

  1. Open the agent configuration file, which is located at etc/bigpanda/bigpanda.conf.
  2. Locate the configuration settings for the Zabbix plugin.
  3. Verify that the credentials for the BigPanda user in Zabbix are correct.
  • The user and password values should match the credentials for the BigPanda user in Zabbix. To check the credentials in Zabbix, go to Administration > Users.
  • The default values are user: bigpanda, password: <App Key>.
  1. Verify that the credentials for the Zabbix host are correct.
  • The host should match the Zabbix host address.
  • The default value is is host: <http://localhost/zabbix.>

Problem: Specific Zabbix alerts are not showing up in BigPanda

Symptoms & Diagnosis
Zabbix is displaying an event for a trigger in PROBLEM state, and the alert is not showing up in BigPanda.

Possible Causes
Several causes may lead to the issue. Please follow the solutions section below.

Solutions

  1. Check the agent log for errors.
  2. If the log shows the error Error while logging into Zabbix API, try the solutions in Problem: BigPanda agent cannot connect to Zabbix API. If the error cannot be resolved, contact BigPanda support with the recommended troubleshooting information.
  3. If the agent log does not show errors, verify that the BigPanda user in Zabbix has permissions to the host group of the host that is generating the alert. See Configuring Zabbix Permissions.
  4. If the problem is still not resolved, contact BigPanda support with the recommended troubleshooting information.

Contact BigPanda Support About the Zabbix Integration

When contacting BigPanda support, please provide the following information:

  1. BigPanda agent.log The default location is: /var/log/bigpanda/agent.log.
  2. Events for the Zabbix triggers: If the issue involves specific Zabbix triggers, please provide all of the events for those Zabbix triggers. Obtain the events by running the following Zabbix API calls.

❗️

Run the first call to obtain an authentication token. In the second call, use the token from the first call to replace <YOUR AUTH>. Also, replace the with the relevant Zabbix trigger.

curl -i -X POST -H 'Content-Type: application/json' -d '{"params": {"password": "zabbix", "user": "Admin"}, "jsonrpc": "2.0", "method": "user.login", "id": 1, "auth": null}' http://localhost/zabbix/api_jsonrpc.php

curl -i -X POST -H 'Content-Type: application/json' -d '{"jsonrpc": "2.0", "params": {"objectids": “<TRIGGER ID>", "output": "extend"},"method": "event.get", "id": 1, "auth": "<YOUR AUTH>"}' http://localhost/zabbix/api_jsonrpc.php
  1. (Recommended) Versions of the following:
  • Zabbix
  • OS and OS Arch (32-bit or 64-bit)
  • Python >= 3.6.8

Uninstall the Integration

To stop sending Zabbix events to BigPanda, you can uninstall the agent from the server or remove the Zabbix integration from the agent. Then, delete the BigPanda user and user group from Zabbix.

Before You Start

  • Manually resolve any open incidents related to the integration to remove them from your incident feed. Optionally, remove any additional data in BigPanda. For example, modify or remove any Environments or AutoShare rules that reference the integration.
  • Obtain access to the server where the BigPanda agent is installed.
  • Determine if the agent on the server supports only the Zabbix integration or supports multiple BigPanda integrations.
  • Determine the OS on the server.
  • Obtain access to the Zabbix UI with a Zabbix Super Admin user.

🚧

When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.

Uninstall the Agent

If the agent supports only the Zabbix integration, you can uninstall the agent from the server.

📘

Do not uninstall the BigPanda agent if it is supporting other BigPanda integrations. In this case, remove the Zabbix integration from the agent.

Remove the Zabbix Integration from the Agent

If the agent is supporting multiple integrations, you can remove only the specific integration without disabling the other BigPanda integrations that the agent supports.

  1. Stop the BigPanda agent by running the following command.

sudo service bigpanda stop

  1. Remove the integration from the agent configuration file by running the following command.

sudo bigpanda-config -r <INTEGRATION APP KEY>

❗️

Replace the app key with the corresponding value in BigPanda.

  1. Start the BigPanda agent by running the following command.

sudo service bigpanda start

Delete the BigPanda User and User Group from Zabbix

  1. In the Zabbix UI, go to Administration > Users.
  2. Locate the BigPanda Service user group and the Bigpanda user.
  3. Delete the user group and user.

Delete the Integration from BigPanda

  1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
  2. In the integration details on the right, click Delete Integration. A support message opens, pre-populated with a request to delete the selected integration.
  3. Press Enter to send the request.
  4. The BigPanda support team will remove the integration from the UI.

This procedure does not remove any data from BigPanda or the integrated system. As needed, remove data from each system before deleting the integration.