Agent-based Zabbix Integration

Zabbix is an open-source solution for network and application monitoring. Install this integration to correlate noisy trigger alerts from Zabbix into actionable incidents in BigPanda.

The Agent-based integration is no longer the desired configuration approach but will still be supported for customers who have yet to migrate to the webhook version.

When using the agent-based integration, BigPanda Agent queries for all new Zabbix trigger events in real-time via the Zabbix API. To ensure a full sync of all current alerts, the BigPanda Agent also queries for all active triggers on each monitored host. The query intervals are configurable.

📘

Webhook Version Available

The webhook Zabbix integration brings the normalization and customization options of the Open Integration Manager to your Zabbix alerts.

If you previously installed the agent-based BigPanda Zabbix integration, you can upgrade at any time to the webhook version.

See the Zabbix documentation for details on creating a webhook integration.

❗️

Payload size

Alert payloads must be 6MB or smaller. Larger payloads will fail to process with BigPanda. We recommend reviewing your configurations to ensure that only actionable, useful information is being sent to BigPanda.

Limitations

• The agent creates a dedicated user and user group in Zabbix for querying the Zabbix data. The user type affects how BigPanda is granted access to your monitoring data. For more information about how to grant the desired access, see Configure Zabbix Permissions..
• To stay within the memory limit of the Zabbix API, the agent uses batching during the full sync. For a successful full sync, your Zabbix configuration must remain within these guidelines:
  • Less than 100,000 host groups.
  • Less than 100,000 monitored hosts per host group.
  • Less than 100,000 active triggers per monitored host.

The success of the full sync does not affect the continuous sync, which ensures that all new trigger events are received and correlated accurately in BigPanda.

Configure the Integration

By editing the config file for the agent, you can configure settings for how the BigPanda agent queries Zabbix for events. You can configure the batch size for the incremental sync and the query interval and batch size for the full sync. You can also disable the full sync.

Before You Start

Obtain access to the server where the BigPanda agent is installed.

Configure The Incremental Sync

The agent queries for all new Zabbix trigger events in real-time via the Zabbix API. You may want to configure this incremental sync to reduce the load on your Zabbix server, depending on the volume of events Zabbix generates.

1. Open the config file, which is located at /etc/bigpanda/bigpanda.conf.

2. Locate the settings for the Zabbix incremental sync, named zabbix/state.

3. Change the value of the max_event_batch_size to your preferred batch size.

   The batch size is the maximum number of events that the Zabbix API can return at a time during an incremental sync. The default value is 100.

Configure the Full Sync

To ensure a full sync of all current alerts, the BigPanda agent queries for all Zabbix events at regular intervals. You may want to configure the full sync to reduce the load on your Zabbix server, depending on the number of monitored triggers you have configured.

1. Open the config file, which is located at /etc/bigpanda/bigpanda.conf.
2. Locate the settings for the Zabbix full sync, named zabbix/fullsync.
3. Change the following settings to your preferred values.

Disable the Full Sync

📘

Only necessary to reduce Zabbix server load

The full sync acts as a backup to the ongoing incremental sync to ensure accuracy. Disable this feature only if you need to reduce the load on your Zabbix server.

1. Open the config file, which is located at /etc/bigpanda/bigpanda.conf.
2. Locate the settings for the Zabbix full sync, named zabbix/fullsync
3. Add the following key and value: "enabled": false.

️

❗️

Not part of config

Make sure you add the pair outside of the config property.

"name": "zabbix/fullsync",

"enabled": false,

"config": {

"user": "BigPanda",...

},

Troubleshoot the Agent-Based Zabbix Integration

Follow these steps to diagnose and solve common problems with the Zabbix integration. When troubleshooting, you can use the agent log files, which are located at /var/log/bigpanda/agent.log by default.

Problem: BigPanda agent cannot connect to Zabbix API

Symptoms & Diagnosis

This error appears in the BigPanda agent log:

Error while logging into Zabbix API

Possible Causes

The log may contain further details about the possible cause for this error. Possible causes include:

• The agent does not have the correct credentials for the BigPanda user in Zabbix (Log file contains: Invalid params., Login name or password is incorrect.)
• The agent does not have the correct Zabbix host credentials (Log file contains: 404 Client Error: Not Found).

Solutions

Verify that the BigPanda user credentials and the Zabbix host credentials are correct.

1. Open the agent configuration file, which is located at etc/bigpanda/bigpanda.conf.

2. Locate the configuration settings for the Zabbix plugin.

3. Verify that the credentials for the BigPanda user in Zabbix are correct:

   The user and password values should match the credentials for the BigPanda user in Zabbix. To check the credentials in Zabbix, go to Administration > Users.

   The default values are user: bigpanda, password: <App Key>.

4. Verify that the credentials for the Zabbix host are correct.

   The host should match the Zabbix host address.

   The default value is host: <http://localhost/zabbix>

Problem: Specific Zabbix alerts are not showing up in BigPanda

Symptoms & Diagnosis

Zabbix is displaying an event for a trigger in PROBLEM state, and the alert is not showing up in BigPanda.

Possible Causes

Several causes may lead to the issue. Please follow the solutions section below.

Solutions

1. Check the agent log for errors.
2. If the log shows the error Error while logging into Zabbix API, try the solutions in Problem: BigPanda agent cannot connect to Zabbix API. If the error cannot be resolved, contact BigPanda support with the recommended troubleshooting information.
3. If the agent log does not show errors, verify that the BigPanda user in Zabbix has permissions to the host group of the host that is generating the alert. See Configure Zabbix Permissions.
4. If the problem is still not resolved, contact BigPanda support with the recommended troubleshooting information.

Contact BigPanda Support About the Zabbix Integration

When contacting BigPanda support, please provide the following information:

1. BigPanda agent.log: The default location is: /var/log/bigpanda/agent.log.

2. Events for the Zabbix triggers: If the issue involves specific Zabbix triggers, please provide all of the events for those Zabbix triggers. Obtain the events by running the following Zabbix API calls:

   Call to Obtain Authentication TokenObtain Events

   curl -i -X POST -H 'Content-Type: application/json' -d '{"params": {"password": "zabbix", "user": "Admin"}, "jsonrpc": "2.0", "method": "user.login", "id": 1, "auth": null}' http://localhost/zabbix/api_jsonrpc.php
   

   ### In the second call, use the token from the first call to replace <YOUR AUTH>. Also, replace <TRIGGER ID> with the relevant Zabbix trigger.
   curl -i -X POST -H 'Content-Type: application/json' -d '{"jsonrpc": "2.0", "params": {"objectids": “<TRIGGER ID>", "output": "extend"},"method": "event.get", "id": 1, "auth": "<YOUR AUTH>"}' http://localhost/zabbix/api_jsonrpc.php
   

3. (Recommended) Versions of the following:

   • Zabbix
   • OS and OS Arch (32-bit or 64-bit)
   • Python >= 3.6.8

️

Uninstall the Agent-Based Zabbix Integration

To stop sending Zabbix events to BigPanda, you can uninstall the agent from the server or remove the Zabbix integration from the agent. Then, delete the BigPanda user and user group from Zabbix.

Before You Start

• Modify or remove any Environments or AutoShare rules that reference the integration. All active alerts from the integration will be resolved after deletion.
• Obtain access to the server where the BigPanda agent is installed.
• Determine if the agent on the server supports only the Zabbix integration or supports multiple BigPanda integrations.
• Determine the OS on the server.
• Obtain access to the Zabbix UI with a Zabbix Super Admin user.

When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.

Uninstall the Agent

If the agent supports only the Zabbix integration, you can uninstall the agent from the server.

Do not uninstall the BigPanda agent if it is supporting other BigPanda integrations. In this case, remove the Zabbix integration from the agent.

Remove the Zabbix Integration from the Agent

If the agent is supporting multiple integrations, you can remove only the specific integration without disabling the other BigPanda integrations that the agent supports.

1. Stop the BigPanda agent by running the following command.

   sudo service bigpanda stop
   

2. Remove the integration from the agent configuration file by running the following command.

   sudo bigpanda-config -r <INTEGRATION APP KEY>
   

   Replace the app key with the corresponding value in BigPanda.

3. Start the BigPanda agent by running the following command.

   sudo service bigpanda start
   

Delete the BigPanda User and User Group from Zabbix

1. In the Zabbix UI, go to Administration > Users.
2. Locate the BigPanda Service user group and the Bigpanda user.
3. Delete the user group and user.

Delete the Integration from BigPanda

1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
2. In the integration details on the right of the page, click the trash icon, then confirm you want to delete the integration. The integration will then be removed immediately.