Enrichment V2 - Migration Plan

This guide describes the differences between Enrichment V1 and V2, and provides a step-by-step plan for a successful migration.

📘

Alert Enrichment V2

Alert Enrichment V2 offers several important new features, including improved execution plan logic and Mapping Enrichment. For more information, see Alert Enrichment V2.

Key differences between V1 and V2

1. Execution Plan Logic

In V1, the execution plan runs according to last match logic and has a pre-defined order that can’t be changed by the user.
The custom tag's rules run one after another in the following order: Maps rules → extraction rules → composition rules, according to their time of creation from oldest to latest. The last matching rule is the determining rule and overrides all previous values.

For example:
There are 2 tags: Cluster (3 rules) & Host (1 extraction rule), ordered by creation time:
C1 - Composition rule with creation time = Jan 01 2020
H1 - Extraction rule with creation time = Jun 01 2020
C2 - Mapping rule with creation time = Dec 01 2020
C3 - Extraction rule with creation time = Jan 01 2021

The execution plan is as follows:
C2 (mapping) → H1 & C3 (extractions by creation time) → C1 (composition)

  • In the Cluster tag the last rule that returns a value is the decisive rule.
  • If Host depends on Cluster (e.g., it is referenced in the condition field), in some cases Cluster may be executed after Host, when C1 or C3 returns a value. This may lead to unexpected outcomes.

In V2, the execution plan runs according to first match logic.
Each alert enrichment tag is an aggregation of its rules. You can manually change the order in which the alert enrichment tags and rules run. The rules run in the defined order and the first matching rule to return a value is the determining rule. Once the system returns a value, it does not run the remaining rules and skips to the next tag.

Returning to the previous example:
There are 2 tags: Cluster (3 rules) & Host (1 extraction rule), ordered by creation time:
C1 - Composition rule with creation time = Jan 01 2020
H1 - Extraction rule with creation time = Jun 01 2020
C2 - Mapping rule with creation time = Dec 01 2020
C3 - Extraction rule with creation time = Jan 01 2021

The V2 execution plan is now as follows:
Cluster [C1, C2, C3] → Host [H1]

  • The tags run as a complete unit (all its rules as a batch).
  • Cluster runs before Host because its creation time is earlier.
  • Inside Cluster, the first rule that returns a value is the decisive rule (e.g., if C1 returns a value the next tag to be run is Host and C2 & C3 are not executed).
Tag presentationTag presentation

Tag presentation

2. Mapping order

In V1, maps run in parallel and we can not control which map runs first. Therefore, if there are dependencies between maps, the results might be different each run. For instance, if one map is dependent on another (e.g., Map A’s result tag is Map B’s query tag), the outcome is likely to be different each time the map runs.

In V2, maps run according to a set order.
The difference in mapping order may generate different results when migrating maps from V1 to V2.

🚧

Enrichment Preview

The Enrichment Preview uses your own real-time data to demonstrate how alert enrichment tags would be applied. The preview shows the calculated tag value for a random set of matching alerts. As new alerts and data enter your system, the preview will show different results. To see a different set of matching alerts or to update the preview after changing the definition, click Refresh.

Migration Plan

Due to the differences in the execution plan and map order, the results generated in V2 may differ from those generated in V1. To preserve the enrichment results after migrating to V2 , the migration reverses the execution plan order in each tag and runs the rules from latest to oldest. This reversal preserves that the last match returned in V1 becomes the first match in V2. Nonetheless, in some cases, the results received in V1 cannot be preserved to V2 (see Mapping order above).

The following step-by-step migration plan helps to ensure a successful migration from Enrichment V1 to V2:

Step 1: Data Migration

Tags and Maps are migrated from V1 to V2.

Tags run in the following order: mapping, extraction tags, and then composition tags.

The rules of each group of tags run from the latest to the oldest in order to support the first match logic. This preserves the result of the values returned in V1.

Step 2: Validation - Open V2 alongside V1 feature toggle

The V2 UI is displayed and controlled by a feature toggle without affecting your data in V1.

You can preview tags and enrichment items and their order.

You can preview the enriched alerts for testing.

The V2 UI and API comply with the same permissions defined in V1.

Creating, deleting, or updating enrichment items in V2 does not affect your data or settings in V1.

The V2 APIs for enrichment maps are disabled and this is the returned error:

  • "This method is temporarily not allowed for your organization. Please contact BigPanda for additional info."

These V2 APIs will be enabled when turning off V1 and enabled V2 - step 4.

V1 APIs for enrichment maps are available and any data updates will be migrated again when completing step 4.

Previewing the tagsPreviewing the tags

Previewing the tags

Step 3: Analyze data and make adjustments

The BigPanda team uses the dependency tool for analyzing data and determining the correct execution plan before activating V2. You’ll receive a dependency report from BigPanda with a full list of tags and potential issues.

The Enrichment Analysis doc will have 2 tabs - Ordered List and Warnings

The Ordered List tab includes the full list of all current enrichment items in the order they ran in V1. This tab includes details about the enrichment tags including the enrichment type, tag query, tag creation time, and more.

The Warnings tab lists tags that might have a dependency issue. There are three types of dependency issues:

  • Normal - The tag cannot be created before the dependent tag, but is currently triggering before the dependent tag. To resolve normal dependency issues, rearrange the enrichment order so that the tag listed in the Dependency column will trigger first.
  • Circular - The tag is dependent on a series of tags that create a closed loop (for example: region depends on the datacenter tag, which itself is dependent on the region tag.) To resolve a circular tag error, adjust at least one of the enrichment tag queries or conditions so that they no longer refer to each other.
  • Self Circular - The tag is created to be dependent on itself (for example: a region tag with the query: region != "*EMEA*") To resolve a self circular tag error, adjust the enrichment tag queries or conditions so that they refer only to alert data or tags earlier in the enrichment order.

If any issues are listed in the Enrichment Analysis document, find the tags in the V2 UI to ensure that the dependency needs of each tag is met. You may need to manually adjust the enrichment items and execution order in Enrichment V2 through the UI or API.

🚧

The order of tags in V2 may not always match the order listed in the migration doc. It is best practice to review all tags with dependencies in the V2 UI to ensure that new dependency issues weren’t created during migration.
When enabled, enrichment will follow the order exactly as found in the UI, regardless of the order listed in the migration doc.

Once all dependencies have been repaired, continue to preview and adjust enrichment in the UI to make sure BigPanda is enriching and correlating alerts correctly.

Step 4: Enable the Enrichment V2 toggle

Once the enrichment rules are working as expected, BigPanada will turn on the Enrichment V2 toggle to complete the migration process (including migrating maps updates).

When Enrichment V2 is on, all data flows through the Enrichment V2 pipeline and affects the Customer's data.

During the migration process, you can expect up to an hour of downtime.

Here is another example showing the difference between the V1 and V2 execution plan:
There are 6 tags and each one includes several different types of rules.


Recommended Reading