Machine Learning Engine

The BigPanda Machine Learning Engine processes your company's alert stream to identify and suggest patterns that may improve correlation.

How It Works

At its core, BigPanda's Algorithmic Correlation relies on pattern recognition. A pre-configured list of patterns is matched against alerts to identify alert clusters in real-time. To classify alerts into incidents, BigPanda looks at information in 4 dimensions:

  • Time
  • Topology (e.g., datacenter, rack, cluster)
  • Context (e.g., criticality, team, customer impact)
  • Alert types (e.g., network, storage, application)

Each correlation pattern defines general properties to correlate: source, timespan, common alert attributes, and a filter.

Common patterns include:

Pattern

Properties

Connectivity alerts

Alerts triggered by devices attached to a single network in a 15-minute timespan

Load-related alerts

Alerts triggered by multiple servers supporting a single database in a 2-hour timespan

Common application alerts

Alerts triggered by tools like Splunk and AppDynamics in a 30-minute timespan

To learn more about how BigPanda merges events into alerts and clusters alerts into incidents, see the Alert Correlation Logic documentation.

Suggested Pattern Generation

In general, correlation patterns are managed by Administrators and the BigPanda Customer Success team. The Machine Learning Engine is able to supplement the human designed patterns by autonomously searching for new correlations and suggesting new patterns.

BigPanda's Machine Learning Engine will automatically generate correlation pattern suggestions based on historical user data. Upon the integration of a monitoring tool, the review process begins and an automatically generated pattern will be suggested in the Correlation Patterns settings page in a few days.

📘

The rate at which the first pattern is generated is dependent upon the richness and size of the available data. Over time, as more data flows through the system, additional patterns will be recommended at an increased and variable rate.

Pattern Recognition EnginePattern Recognition Engine

Pattern Recognition Engine

Once the Machine Learning Engine suggests a pattern, administrators can decide to activate it, reject it, or customize it within the editor. The Real-Time Preview in the patterns editor gives you instant visibility on the impact a suggested pattern would have on correlated alerts in your system.

Suggested correlation patterns are made to stand on their own, but new patterns can also be modified to complement an existing set of in-use patterns.

The end result is better correlation reach with higher quality incidents to help your team resolve issues faster.

Unsupervised Autonomy

BigPanda's Machine Learning Engine is unsupervised in function and does not require training. It works by clustering the alert stream into high-quality Incidents. It will run autonomously in the background as soon as relevant data is present. Unlike supervised machine learning, human interaction and consistent input are not required for its upkeep and efficacy.

📘

Our unsupervised approach uniquely maintains transparency and consistency in forming Incidents but it does not enact changes. The suggestion model for generated patterns is utilized to grant administrators full discretion and control of code changes in a Production environment.

To learn more about defining and managing correlation patterns, see our Working with Correlation Patterns guide.