Best Practices
Find best practices for getting the most out of BigPanda.
Data normalization in BigPanda is the process of standardizing and structuring event data from various monitoring and observability tools into a consistent format.
This process involves:
Standardization: Identify a standard set of tags to map tags and data fields from different sources, ensuring uniformity across all integrations. Consider downstream impacts as you identify the tags.
Integration: Integrate your monitoring tools with these standardized tags in mind. BigPanda has 50+ standard integrations , or you can integrate using the Open Integration Manager (OIM) , Alerts API, Email Parser, or BigPanda Daemon (SNMP).
Cleansing: Ensure data within tags from your monitoring tools are cleansed where possible, such as removing domain names from the host tag names to align with how data is stored in enrichment sources.
Consistency: Adjust Primary and Secondary properties to drive event marshaling, deduplication, and compression
Enrichment: Enhance alert payloads by adding contextual information and consolidating similar tags into a single, consistent tag to ensure uniformity across all alerts.
Integrating observability tools from different vendors can be challenging due to differences in how they tag and report data. Without standardization, the downstream components of the BigPanda pipeline can be negatively impacted. When tags aren’t consistent or normalized, the entire system can be less effective and accurate, leading to slower response times and missed critical alerts.
Normalizing tags prepares you to introduce additional monitoring sources, allowing you to scale and evolve with your organization’s needs. Normalized tags must define and meet minimal criteria for downstream actions, such as ticketing or automation, ensuring that all necessary information is available for these processes to function smoothly.
Key Benefits of Data Normalization
Normalization establishes data presentation consistency across events from all sources within BigPanda.
Different alert sources will look similar from a tag perspective.
The data unification process simplifies downstream configuration and improves the maintainability of BigPanda while also driving improved alert quality.
Best Practices and Tips
Start small and scale
Your data doesn’t have to be perfect to begin with. Start with a handful of data sources, gain visibility into the quality of alerts, and slowly ingest more sources.
Ideally, you should select your top services that impact business. Ask yourself, “What keeps you up at night? Tackle the most critical services and applications. Don’t prioritize low-hanging fruit if the important services aren't done.
Make sure each monitoring tool is passing along all of the data that your team needs to detect, understand, and remediate incidents. Standardized tags drive everything downstream from event compression to auto-remediation. Collaborate across teams when defining your organization's standard tags to ensure they meet everyone's needs.
Identify opportunities to bring in and streamline the data that will power remediation and automation such as runbook URLs. As tools are added or changed, make sure to configure them to leverage the existing standard tags to prevent unnecessary reconfiguration.
Tag standardization
Tag standardization is a key step towards data normalization. Standardizing tags helps enhance the performance and reliability of BigPanda.
Tag standardization involves mapping tags from your observability tools to a common set of core tags, ensuring uniformity across all tools. This consistency gives you a cohesive view of the data, making it easier to manage and interpret.
Key principles for tag standardization
Less is more. Having too many standard tags in an organization can make it difficult for operators to manage alerts.
Be consistent with tags across all monitoring tools - different alert sources then will look similar from a tag perspective.
The data unification process improves alert quality and simplifies downstream configurations such as correlation, reporting, and incident enrichment, making the platform easier to maintain.
Tag classification
Tags can be classified as functional or contextual.
Functional tags are crucial for BigPanda’s core operations. These tags facilitate tasks such as correlation, reporting, CMDB lookups, and integration with downstream tools like ticketing and notification platforms.
Contextual tags provide valuable insights to operators about issues but are not essential for BigPanda’s functionality.
Sometimes less is more; having fewer tags can streamline the console for operators. However, your administrator must determine which contextual tags should be included based on your organization’s needs. This involves working with the teams using the console or receiving the relevant data to action an incident. We generally recommend avoiding redundant tags, but you should also ensure that operators have all of the information they need to troubleshoot incidents.
To improve data normalization and consistency, we recommend using specific tag names for common monitoring elements. Standard tag names ensure that your data performs consistently and that you can leverage the full automation capabilities of BigPanda.
We understand that every organization has unique requirements and needs. BigPanda allows for flexibility based on your use cases, but we recommend leveraging as much of our standard as possible. Since Unified Analytics is designed to use standard tags, additional configuration may be required if these standards are not implemented.
If necessary, your options to change the data model include:
Renaming tags - suggested when you have terms specific to your organization that you’d like to maintain
Adding new tags - when there are specific use cases you’d like to account for in your standard set of alert tags
Removing tags - for tags not pertinent to your organization’s use cases
The processes for renaming, adding, and removing tags will be addressed later in this guide.
Normalization can happen at the source monitoring tool, during the OIM integration, or through the enrichment process. BigPanda's standard integrations do not automatically use standard tags, so implementation teams and administrators must configure them within the platform.
The following are the standard tag names recommended by BigPanda:
Standard Tag Name | Tag Description | Source Tag Names |
|---|---|---|
host | Name of the target system that generated the alert. This tag is the default | hostname, instance, device, machine, server, hosts |
object | The object impacted that generated the alert. | object, resource, resource_name, target, instance, job, url, etc. |
impacted_entity | The entity impacted that generated the alert. | impacted_entity, resource, resource_name, target, instance, job, url, etc. |
service | The impacted service. | impact_service, srvc |
application | The impacted application. | app, app_name, impacted_app |
check | Name or title of the alert. This tag is the default | alert, alertgroup, title, monitor, monitorname, testname,event_id, rule, rulename |
description | Full text or description of the alert. This tag should not be used as the primary or secondary property. | summary, ldesc, long_desc, problem |
assignment_group | Assignee group in the outbound tool. | support_group, routing_group, escalation_group, workgroup |
support_group | The support group associated with the CI. | support_group, routing_group, escalation_group, workgroup |
app_support_group | The support group associated with the impacted application. This could be a service. | support_group, routing_group, escalation_group, workgroup |
business_group | The line of business impacted by the alert. | business_unit, logical_group, lob |
priority | Importance or severity of the alert. | severity, inc_priority, alert_severity |
environment | The instance or environment that triggered the alert. For example, prod, dev, etc. | env, tier, stage |
location | The physical location where the alert triggered. | physical_location, device_location, address |
runbook_url | Link or location of the related runbook. | runbook_link, knowledge_base, ki_article, support_link, kb_article, wiki_url |
Special tags
Some tags have functionality that is essential to BigPanda’s operation. These tags play a vital role in data visualization, deduplication, and alert resolution.
BigPanda special tags are:
primary_property
secondary_property
incident_identifier
description
timestamp
status
Reserved Words
Some words are reserved and cannot be used as alert tag names in BigPanda. Tags with these names may be saved, but they will not enrich alerts and can cause issues when filtering alerts.
Reserved words during OIM configuration:
incident_identifier
primary_property
secondary_property
Reserved words during Alert Enrichment tag configuration
alert_updates
description
namespace
severity
Some tag names may also have downstream limitations. For more information on naming requirements, see the Tag Naming documentation.
Primary and secondary properties are key fields used for normalization, deduplication, and event correlation in BigPanda.
Each event must have a primary property associated with it. The secondary property is optional, but highly recommended as it improves event deduplication and normalization. The value of each property should be the name of another tag in the payload. BigPanda allows you to explicitly set the primary property as part of the integration and/or event payload for granular control.
When the primary property is not explicitly set, BigPanda will leverage the default hierarchy:
Primary - host, service, application, impacted_entity, object
Secondary - check
Troubleshooting primary and secondary properties
For each standard integration, BigPanda provides visibility into events that failed to process in the pipeline so you can troubleshoot your integration.
This troubleshooting happens at the ingest point prior to deduplication. This does not account for custom integration monitoring.
Keep in mind that some errors you see could be due to:
Misaligned primary/secondary properties
Malformed JSON
Invalid characters
Missing required fields
If the primary property is set to a tag that doesn’t exist in the payload or if none of the hierarchy fields are found in the payload, the event will fail to process and display an error in the integration troubleshooting screen.
Best Practice
The primary property should align to the physical or virtual entity that is experiencing the issue (host, object, application, service, impacted_entity tags), and the secondary property align to the name or title of the issue (the secondary should always align to check tag).
Primary and secondary property in the UI
The primary property and secondary properties are one of the display fields on the alert when:
An incident does not have any correlation applied. The primary property will display as the incident title in the incident panel and the secondary will display as the supplemental field when focused on an incident.
Correlation is applied. The primary property will display as the supplemental field when focused on an incident. The secondary will not display as a highlighted attribute. If the primary is the same as the correlation, the secondary properties will still display as the supplemental field.
To maintain a consistent look in the UI, we recommend limiting primary tags to a select few, specifically host, object, application, and service. The secondary property should always be set to check to ensure clarity and coherence in tag presentation.
The incident identifier is the event attribute that determines uniqueness. For granular control, you can set it as part of the integration and/or event payload.
A common scenario is if a monitoring tool contains a unique identifier (something like alert_id as a random character string), but does not function as a tag an operator would leverage in triage. Unlike primary and secondary, this value should be the explicit value and not the tag name to set the configuration. You can also use ${field1}${field2} to substitute the values in from the tag names.
When the incident identifier is not set, BigPanda will concatenate the tag values associated with the primary and secondary properties together. If the secondary is not set, only the primary tag value is considered.
Best Practice
By default, the incident identifier is created using the primary and secondary properties.
We recommend setting the incident_identifier based on the primary and secondary properties, provided that these properties ensure uniqueness, to avoid deduplicating unrelated alerts.
If the primary and secondary properties include timestamps or metric numbers, other methods for deduplication using the incident_identifier tag should be considered.
The Description tag of an event sent by a monitoring tool describes the problem alert in detail. Description is a specialized tag that allows for up to 2048 characters, which is larger than the standard size of 512 characters.
Description is a reserved word, which means it cannot be changed or redefined for custom enrichment. See Tag Naming Limitations for more information.
The timestamp tag shows the time when a monitoring tool triggered an event or when an action happened in BigPanda. Timestamp is shown in Unix epoch format in seconds.
BigPanda supports timestamps in seconds only. If the time is sent in milliseconds, BigPanda will convert it to seconds. This can cause issues with flapping alerts that flap within milliseconds.
Best Practice
If your monitoring source sends a timestamp in epoch format, we recommend you map this value to the timestamp field. If no timestamp is provided, the time BigPanda received the alerts will be used as the time the alert was triggered.
Timestamps set for future times
BigPanda cannot accept timestamps later than the time the event is processed. Timestamps set in the future will instead use the system timestamp when the event is processed.
Tag standardization for automated workflows
The first step in tag standardization for automated workflows is to identify existing automation tools and how they will interact with BigPanda.
Identify the required tags for the outbound tool BigPanda will communicate to. Since most of these will end up being an autoshare, we recommend adding autoshare_[category] as a standard.
The following are common tools or categories implemented with BigPanda:
autoshare_itsm
autoshare_automation or autoshare_remediation
autoshare_email
autoshare_sms
autoshare_aia
autoshare_paging (call, on-call, etc.)
autoshare_chat (Slack, Teams, etc.)
Best Practice
We recommend the values for these tags be boolean [true, false, or #] to indicate whether alerts need to be routed to the proper workflow. If an autoshare delay is needed, you could expand the tag value to include numeric delay timers.
For example, if you have 2 auto-share delays and a 3 and 5-minute timer, you could set your autoshare tag as [0, 3, 5] to indicate which delay timer to use.
Tag Normalization
Tag normalization transforms and cleanses data from disparate tools into a standardized format. This includes tasks like converting fully qualified domain names to short names and ensuring that all data adheres to the same naming conventions.
Key principles of tag normalization
Start with a baseline of standard tags (host, check, application, service, assignment_group)
Adapt to your company tools and terminology
Normalized alert tags should represent the primary focus for downstream teams
Normalized tags allow for the introduction of additional monitoring sources
Normalized tags should always define and meet minimal criteria for downstream actions (ticketing, automation, reporting)
Downstream setup can be done at various points (integration level, Open Integration Manager (OIM) level, enrichment sources, or custom tags)
Use Alert Views to bring standardized tags to the top of an alert’s list of tags
The 5 Ws of tag normalization
Normalized tags should always answer one of the “5 Ws”:
Who is generating event (host, application, service)
What do I need to solve this issue (runbook_url)
Where do I need to escalate to (assignment_group)
Why is this event happening (check)
When do I need to have this addressed by (priority, impact, urgency)
These 5 Ws align with the Alert Tag Categories that define Alert Quality.
Mapping to standard tags
Tags can be thought of as building blocks that can be mapped to a logic definition that impacts downstream processes or helps with incident triage.
You may find yourself having “tags of tags” - where a normalized tag is built upon the input from multiple normalized tags. For instance, the cmdb_ci tag could be sourced from both the host and application tags based on the nature of the monitoring.
Once the proper set of standard tags is defined, the next step is to map each of your monitoring integrations to a set of these tags. This can include a tag renaming exercise and applying data transformation techniques to set the correct value to the right tag name.
Event normalization in BigPanda occurs at several points including the alert source, the Open Integration Manager (OIM) configuration, and during alert enrichment. We recommend configuring normalization as early in the event pipeline as possible.
The following options are available for transforming tags:
Adjust values at the monitoring source layer
Leverage the OIM configuration to set appropriate tag names and values
Use Alert Enrichment to take advantage of mapping enrichment along with extraction and composition alert tags
You can also proactively review downstream configurations to adjust your normalized tag set. Examples include:
Normalization in the Monitoring Source
Some monitoring tools allow you to edit key/value pairs in the event configuration or free forms to define payloads. Normalizing early in the pipeline simplifies downstream configuration as events enter with the proper set of tags.
Best Practice
We recommend that Tool Admins determine early on if the monitoring tool allows these types of edits.
Normalization in the Open Integration Manager (OIM)
The Open Integration Manager (OIM) contains a tag mapping component that allows you to normalize the original monitoring tool payloads, often referred to as source tags, to your destination tag, which aligns with your standardized model.
You can also map multiple source tags. OIM can scan a CSV, take the first tag it finds in the alert payload, and align that tag to the destination tag.
Advanced tag mapping allows for extraction directly within the integration configuration using regular expressions. This functionality is similar to event enrichment for alert extraction tags.
Common use cases include:
Integration sources that don’t send the same types of values. For example, Source A sends host, and Source B sends FQDN. Extraction can align these to the same values.
Nested values within the alert tag details. For example, monitoring tools may concatenate multiple values into one tag which need to be pulled out to be set as their own individual tag.
If the value cannot be referenced or requires rewriting in a different format, then this must be done in Alert Enrichment via Composition.
Normalization in Alert Enrichment
Monitoring tools often do not provide all the detailed information required for operators to efficiently manage and respond to alerts. Key details, such as support_group, assignment_group, runbook_url, and environment, are sometimes missing from the initial alert payloads. To ensure these critical tags are available, many of them need to be added through the Alert Enrichment processes.
The Open Integration Manager (OIM) configuration has some limitations in its ability to extract and normalize data to create standardized tags. For example, OIM may struggle with complex regex extraction, such as extracting specific components from an alert payload or transforming the data into a more usable format. Alert enrichment can handle these tasks more effectively. Complex regular expressions can be employed within the alert enrichment process to extract specific values, such as a hostname, from an alert payload.
Alert enrichment can also be used to manipulate extracted data, such as by removing the domain name from a fully qualified hostname, making the alert information more relevant and actionable. By leveraging alert enrichment, you can ensure that alerts contain all the necessary information for efficient and effective incident management.
Alert Views
Normalized tags drive consistency, allowing you to understand which tags to focus on for triage and remediation. These tags should be displayed first to quickly summarize an issue, with additional information available in the alert details. You can use Alert Views to define the order in which tags appear within an alert.
Measure Success with Unified Analytics
You can use Unified Analytics to track and measure the progress and success of tag standardization and normalization across your monitoring environment.
The Alert Quality, Enrichment, and Tag Standardization Dashboards allow you to monitor the quality of your alerts, assess how effectively they are being standardized and normalized, and determine how well they align with your enrichment sources. These dashboards provide a comprehensive view of consistency and quality, helping ensure that alerts are uniformly processed and enriched across all your monitoring tools.
The Alert Quality Dashboard provides visibility into the quality of alerts sent to BigPanda and their impact on Mean Time to Resolve (MTTR). You can use this dashboard to quickly evaluate your alert quality level and identify key areas for improvement. Alert quality uses the host, CI, assignment_group, runbook_url, impact, and enrichment tags to determine quality levels.
The Enrichment Dashboard displays the effectiveness of your enrichment configuration and highlights alerts that are slipping through enrichment rules. By visualizing the enrichment flow, the dashboard ensures that alerts are sufficiently detailed when they reach downstream systems, making it easier for incident responders to understand the context and take appropriate actions.
The Enrichment Dashboard can also help you track the effectiveness of normalizing key tags such as host and application. If alerts are not being enriched, it's likely that your host or application tags are missing or not normalized correctly.
The Tag Standardization Dashboard helps you track how well you are standardizing across all your alert sources. It tracks the implementation and success of standard tags that are applied during the normalization process. You can customize this dashboard to help you track the tags that are important in your organization.
The Tag Standardization Dashboard isn’t available as a standard dashboard and will need to be created custom for each organization. Contact your account team for more information.
Correlation allows your team to easily cut through unnecessary clutter and visual noise with event aggregation and alert groupings. By combining related alerts across tools and systems, correlation helps you understand the scope and impact of issues faster.
Correlation patterns are high-level definitions that determine how alerts are clustered into BigPanda incidents. To increase the effectiveness of your alert correlation, you can customize the correlation pattern definitions in BigPanda based on the structure and processes of your company's production infrastructure.
As new alerts are received, BigPanda evaluates all matching patterns and determines whether to update an existing incident or create a new one. With this powerful algorithm, BigPanda can effectively and accurately correlate alerts to reduce your monitoring noise by as much as 90 – 99%. Correlation occurs in under 100ms so you see updates in real time.
For example, you can create patterns that correlate:
Network-related connectivity issues within the same data center.
Application-specific checks on the same host.
Load-related alerts from multiple servers in the same database cluster.
Low memory alerts on a distributed cache.
This guide uses the functional and contextual tags from the Data Normalization Best Practices and utilizes the newly standard metadata into proper groupings based on the most critical services and applications.
Correlation basics
Looking for just the basics? Check out the Correlation one page document in BigPanda University!
Benefits of Correlation
Alert correlation provides the following key benefits:
Reduction in alert and incident noise allowing for a consolidated view of related impacts
Reduction in tickets as a result of reducing one-to-one alert to ticket volume
Better understanding of cross-source relationships and workflows
Key Principles of Correlation
Good data normalization and enrichment are prerequisites for successful correlation. If you have not completed those steps, we recommend you start there first. For more information, check out the Data Normalization Best Practices Best Practices.
Your business work streams, operational process, and operational maturity sets the requirements for correlation. Understanding how teams operate today and building that information in tags is vital to proper correlation.
Understanding alerting behavior from your monitoring sources helps set the proper configuration for correlation including source system, tags, time window, and filter condition through BigPanda Query Language .
Consider infrastructure, network, and application relationships, then identify common alerting metadata to correlate on.
Primary Alert Criteria
When multiple alerts are correlated into a single incident, a single alert is chosen as a primary alert. This alert is used in downstream automations and collaboration tools using Dynamic Variables, and it can be added as an incident tag. Every incident must have a primary alert.
Primary alert criteria is made up of selection criteria and filter conditions.
Selection criteria selects a single alert from all correlated alerts in the incident. Selection criteria will apply the first criteria, then the fallback. Only alerts that match all filter conditions will be selected as potential primary alerts.
For example: Latest start time AND Highest severity AND (filter)prod = test will first filter out any alerts that do not have the prod tag = test. Next, it will identify the alert with the latest start time. If there are multiple alerts with the same start time, it will then select the alert with the highest severity from that subset.
By default, the primary alert is the alert with the:
Highest severity
Earliest time correlated into the incident
Not in maintenance
Best Practices and Tips
Start small and scale
Your data doesn’t have to be perfect to begin with. Start with a handful of data sources, gain visibility into the quality of alerts, and slowly ingest more sources.
Ideally, you should select your top services that impact business. Ask yourself, “What keeps you up at night? Tackle the most critical services and applications. Don’t prioritize low-hanging fruit if the important services aren't done.
Make sure each monitoring tool is passing along all of the data that your team needs to detect, understand, and remediate incidents. Standardized tags drive everything downstream from event compression to auto-remediation. Collaborate across teams when defining your organization's standard tags to ensure they meet everyone's needs.
Identify opportunities to bring in and streamline the data that will power remediation and automation such as runbook URLs. As tools are added or changed, make sure to configure them to leverage the existing standard tags to prevent unnecessary reconfiguration.
Actionability
An important concept within monitoring and observability is actionability. Consider what operators or automation should do to meet the service level indicator/objective (SLI/SLO). Your tags will tell the story.
If we use a tag as an actionability utility (make a ticket, chat with someone, page someone in PagerDuty/OpsGenie, send an email) we can then use it to guide the work. This is called a Workflow Tag.
Actionability can be remediated through automation, or by a user (L1, L2, SRE, SME, or any other resolver). Knowing how many times an alert was actioned is a useful metric.
Ask yourself what functional and contextual information you need to create a unit of work. Be sure to qualify what is truly actionable. Creating a ticket doesn’t necessarily deem an alert actionable. If the ticket is automatically closed or ignored without any meaningful resolution, either the alert needs a modification with its SLI/SLO, or the alert should be deemed as noise.
Noise is telemetry without adequate metadata. To reduce noise, we can enrich or eliminate low-quality alerts from our ecosystem and focus only on alerts that support monitoring the top applications.
Correlation can be configured in many ways. This section will focus on implementing simple patterns and then identifying advanced-level correlations.
For more information about how correlation works, see the BigPanda Alert Correlation Logic documentation.
The BigPanda alert correlation engine clusters high-quality alerts into actionable incidents by looking at 4 properties:
Source system
Tags
Time window
Filter (optional)
Source System
The integrated monitoring systems for which the pattern applies. For example, you can set up a pattern to show alerts that come from Nagios, Datadog, etc.
Best practice for source system
Single-source correlation can be a good way to segregate alerts that need a specific workflow.
Cross-source correlation is helpful if two different sources monitor components of the same service or host. This should be selected if the alerts will support the same workstream. Common tags could be a shared host, check, or assignment_group.
Tags
Tag names correlate alerts with matching values. For example, enter ‘cluster’ and ‘check’ to correlate all alerts from the same cluster with the same check.
You can enter up to five tags in any order.
Time Window
The time window is the amount of time between when the alerts started. For example, network-related alerts may start within a short time from one another, while load issues may develop over a longer period of time.
Best practice
Understanding alert trigger behaviors from various monitoring sources can guide time windows. BigPanda will set the default window as 120 min (2 hours).
Consider the following scenarios when setting the time window:
Known failure intervals: Some alerting will require very specific groupings. If a known subset of alerts should be grouped, set the appropriate time window.
Noisy alerts: For services with many alerts and high volume, you could reduce the default to a 30-minute or even 5-minute time window based on how often the alert triggers.
Long interval alerts: If an alert triggers once every 24 hours, a specific correlation pattern can be set to a 1440 minute (24 hour) time window.
Filter (optional)
The filter is the conditions that further refine which alerts the relationship applies to. For example, you could correlate only network-related alerts by data center.
Functional tags can be used in simple correlation immediately. However, we suggest keeping these patterns minimal and avoiding over-correlating.
Generally, you shouldn’t need to set up a blanket correlation pattern. There are times where correlating on one functional tag will be more confusing than helpful.
We recommend starting simple. However, these patterns are not “set and forget.” Focus on continuing conversations with your team and making updates as your processes change.
Examples of simple correlations:
Single tag correlation: host, site or location, ip_class, or application/service
Integration correlation: Alerts from a specific integration could be correlated, especially if it only monitors one thing. However, you will still need a single tag to correlate on.
For example, you could do an integration correlation with a timestamp tag based on set time intervals.
Assignment group correlation: This is based only on groups who want to see their alerts grouped into an ITSM ticket. This can be paired with a functional tag such as host, ci, location, or an application the team owns. Using this can narrow the scope of the correlation grouping, so we recommend you use this for smaller subsets of alerting that require a very accurate assignment group.
When creating patterns, you are building rules based on your functional and contextual tags that describe what is happening and what actions need to be taken. We recommend using your business goals and high-level services to guide groupings of related alert subsets.
Reading the grouped data can quickly tell a story about what is happening. You should be able to easily understand what the alert means and how to begin solving the issue.
Tribal knowledge from operators and artifacts like runbooks and metadata within contextual tags can also be transferred into correlation patterns. Accurately using this information marks successful contextual correlation.
Here are some examples of advanced correlations to group workflows:
location_type
location_type is a regional-based correlation that can be useful for tracking geographic impacts.
Example tags include location, datacenter, site, and interconnect.
category_type
category_type can loosely follow an Open Systems Interconnection (OSI) model to segregate layers tags.
Example tags include network, infrastructure, db_instance, etc.
alert_type / check_type
alert_type or check_type can be useful for rule-based system checks. A common example of this is a specific type of alerting within a category tag such as disk, cpu, memory, firewall, loadbalancer, BGP, OSPF, etc.
For example, if you have CPU utilization that is important for a specific set of servers, you can use the query condition to filter on specific server types. Or perhaps storage disk-all is more critical to watch. A composition tag under check_type can help keep a library of important check types.
We recommend using check_type in an extraction/composition mapping to account for various use cases and then use check-type as an option for correlation.
parent/child
parent/child is a relational correlation. Complete enrichment is required for this to work well.
A common use case for this correlation type is when a network device goes down, and downstream impacts occur. In this situation, you need to have a tag that shows commonality to the device that affects the downstream alerting. Knowing root cause information helps guide strategy around this type of correlation when determining which tag to use.
For example, you could use router = device01 for the network router. Adding the router tag to enrich alerts from downstream objects ties together all servers or dependencies and provides a way to group all related devices or objects.
Validating correlation patterns ensures that the groupings created in BigPanda are accurate. This step is short, but should happen on a continuous basis.
Be sure to get the right stakeholders to review this, such as management, user groups, technical teams, and leadership. Service group owners need to provide feedback on how the correlation is working. Determine what needs to be changed and iterate upon the existing configurations.
Best practices
Use the preview pane and adjust the time window, tags, and filter queries to understand how the correlation will work with the pattern in place
Use Split/Merge as a direct feedback mechanism in the tool for immediate feedback - review this in a dashboard
Configuration Drift
As you change monitoring tools, alert/cmdb enrichments, and naming conventions, you must ensure you account for correlation patterns that rely on tagging that comes from any of those.
Best practice
Please ensure you audit and keep your patterns up to date. Ensure the tags and alerts that are removed from your environment are also removed from your correlation pattern
Unified Analytics
BigPanda provides dashboards to help measure the effectiveness of your correlation patterns. See the Correlation Patterns Dashboard Dashboard documentation to learn how to measure your correlation effectiveness in Unified Analytics.
Learn how to share dashboards, collaborate with your team, and manage permissions in BigPanda Unified Analytics.
Key Concepts
Shared dashboards are the official version everyone sees—make all production changes here
Private dashboards are your personal workspace for testing—changes here won't affect others
Co-Owners have the same permissions as the original owner (edit, share, delete)
Filter settings are personal—each user controls their own filters independently
Always republish after making permission changes to apply them
Understanding User Types
Every user in Unified Analytics has one of two permission levels:
Designers can create, edit, and manage dashboards. Organizations have a limited number of Designer seats.
Viewers can only view dashboards and adjust their own filter settings. They cannot make changes to dashboards. There's no limit on Viewer seats.
Co-owners have full control over a dashboard, just like the original owner. They can edit, republish, share, and delete the dashboard.
Best practice
Add co-owners to mission-critical dashboards to ensure continuity if someone leaves the team or goes on vacation.
Add a Co-Owner
To add a co-owner to a dashboard:
Open the dashboard in Shared mode (look for the label in the top-right corner).
Click Share > Manage Access.
Add the user or group as a Co-Owner.
Click Republish to apply the changes.
License types
A user's license type always determines what they can do. Even if you give a Viewer "Co-Owner" access, they still can't edit dashboards. Only Designers can make changes.
View ownership
Hover over any dashboard to see who owns it (including all co-owners), when it was created, and when it was last modified.
To make navigation and management easier, Dashboards can be grouped into folders and subfolders.
The following folder concepts are important to understand:
You can see a folder if you created it or if any dashboard inside it is shared with you
Only the folder creator can add new dashboards to that folder
To move a dashboard into a folder, you must own both the dashboard and the folder
You can create sub-folders to organize dashboards hierarchically
Folder ownership
Hover over any folder to see who owns it, when it was created, and when it was last modified.
Folders cannot be directly transferred. To give someone control:
The original owner must share all dashboards in the folder with the new owner.
The new owner creates a new folder.
The new owner moves the shared dashboards into their new folder.
When a folder owner leaves an organization, the following occurs:
Shared dashboards remain accessible to users who had access
The folder becomes read-only—no one can add new dashboards
Dashboard owners can still move their dashboards to other folders they control
Contact BigPanda Support if you need help transferring ownership.
Users must visit the Analytics tab at least once to be registered in Unified Analytics. They are not automatically created when added to BigPanda. Once registered, each user receives their own unique username, which appears in the sharing dropdowns.
Each organization has one built-in group: ua_all_users_group_<org-name>
This group automatically includes all Designers and Viewers in your organization. You can assign this group View, Edit, or Co-Owner access to share dashboards with everyone at once.
Sharing with a group
To share a dashboard with everyone in your organization, grant access to ua_all_users_group_<org-name>
Filter settings are always personal, regardless of Shared or Private mode:
Each user controls their own filter selections
Publishing a dashboard does not change anyone else's filter settings
Your filter preferences are saved independently from other users
Use the table below to find solutions for common issues related to dashboards:
Issue | Likely Cause | Solution |
|---|---|---|
Co-owner option is greyed out | User doesn't have a Designer license | Request a Designer license upgrade for the user |
Changes disappeared after refresh | Changes were made in Private mode instead of Shared | Switch to Shared mode and re-apply your changes, then republish |
Filters aren't saving | Filters were set in Private mode | Switch to Shared mode and republish with your desired filters |
Teams can't see a shared dashboard | Dashboard was shared with the wrong user or group | Share with ua_all_users_group_<org> for org-wide access |
User list is empty when sharing | Browser cache or session issue | Refresh the page or clear your browser cache |
Can't find a dashboard you previously had access to | Dashboard was moved to a folder you can’t access | Contact the dashboard owner to request access or ask them to move it |
Changes not taking effect | Forgot to republish after making changes | Click the Republish button to apply your changes |
Mapping enrichment allows you to upload a CSV to map tag values to an enrichment table. Incoming alert data will be compared to the values in query_tag columns. If a match is found, the related tag will be updated with a value from the result_tag column.
Mapping enrichment items are added automatically to existing tags upon the upload of a mapping enrichment table with matching result_tag column names. If a map contains a result_tag column name that does not match an existing tag, a new tag is automatically created for the enrichment item.
You can define a mapping enrichment with the Alert Enrichment API using these resources:
Enrichments object—defines the schema for your enrichment table. The schema determines how the information in each column of the table will be used in the enrichment process—as a query field, to match a value in the incoming alert, or as a result field, to enrich the alert.
Map endpoint—allows you to upload and maintain a mapping table that follows a given enrichments schema.
Mapping Schema and File
Each enrichment map is made up of a mapping schema, and a mapping file.
The mapping schema defines rules about how the system should interpret the mapping file. Mapping schemas determine:
The name and description of the enrichment map
A condition for when mapping rules should apply
Which columns function as lookup items
Result tag names
Override settings for result item values
The mapping file is the actual data that should populate alert item values. Mapping files are uploaded to BigPanda as a .csv. The map does not have to be fully populated on initial upload, but it must include headers for all columns. Map reuploads must match the existing schema, with the same column headers that were included in the initial upload.
Dependencies
In certain cases of enrichment, you need to ensure that a specific tag or enrichment item runs before another because the enrichment items are dependent on each other.
Tags and enrichment items within BigPanda are executed in the order they appear in the BigPanda UI. Dependent tags or enrichment items should be below the source tag or item in the enrichment order.
In certain cases when creating dependencies the results may be different in each run. For instance, if the tag value is received in the original payload then the dependent tag uses this original value and not the value that would have been created during the enrichment process.
Map overrides ensure that tag values are normalized even across systems with disparate data models.
Step 1: Define The Mapping Structure
For a mapping enrichment, the Enrichments object defines how the information in each column of the table will be used in the enrichment process:
Query field—used to match a value in the incoming alert.
Result field—used to enrich the alert.
To define the mapping structure for a new enrichment:
Send a
POSTrequest to/enrichments. Specify thetypeasmapping.For each column in your data table, specify the name and whether to use the values as aquery_tagor aresult_tag. The request creates a newEnrichmentsobject for this mapping enrichment schema.Token value
Be sure to replace
${token}with the corresponding value for your organization.Copy the
idvalue from the response body. You will use it to identify this schema when you upload and update the mapping data.
Step 2: Upload The Mapping Data
After you create an Enrichments object that defines its structure, you can upload the table of the mapping data. For example, suppose that the Billing and Sales applications both have owners and remediation procedures. You can upload a table with three rows—one row for the column names and one row for each application.
To upload the mapping data for an enrichment:
Send a
POSTrequest to/enrichments/${id}/map. Replace${id}with the value you copied from the response body in Step 1 and${token}with the corresponding value for your organization.Mapping data
The mapping data must be sent as comma-separated values (CSV) in the body of the request or in a CSV file. Ensure the data follows the structure you defined in Step 1 and has the same column names.
Enriched tag for analytics
If your organization would like to accurately visualize enrichment rates in Unified Analytics, you must include 2 enrichment flag columns in the CSV:
An
enrichedorenrichmentcolumn with true/false rows, where true means alerts that match on that row should be considered enriched.A
<map name>column withenrichedorenrichmentin each row (whichever was the name of the previous column), to add the enriched quality to any alert matched on the map.
The request initiates an asynchronous job to upload the table and creates a Job object to track its progress.
The payload must end with an empty new line. Make sure there are no spaces before the closing quotation marks on the last line.
Copy the
trigger_urivalue from the response body. You will use it to check the status of the upload.Send a
GETrequest to the${trigger_uri}to verify that the upload has completed successfully. Replace${trigger_uri}with the value you copied from the response body and${token}with the corresponding value for your organization.(Recommended) Send test alerts to ensure that the mapping enrichment works as expected. For example, you can run the following command to send matching test alerts via the Alert API.
Resolve any test alerts, as necessary.
Step 3. Maintain the Mapping Data
As your team and infrastructure changes, you may need to update the values for your mapping enrichment. Depending on what changes you want to make, you can either completely replace the existing table, change only specific rows in the table, or create a new enrichment to change the structure.
To completely replace the existing table but keep the same structure, repeat Step 2. Upload the Mapping Data.
To change only specific rows in the existing table:
Send a
PATCHrequest to/enrichments/${id}/map. Replace${id}with the value you copied from the response body in Step 1 and${token}with the corresponding value for your organization. For each row, specify the operation to perform and the data required to perform the operation. The request initiates an asynchronous job to upload the table and creates a Job object to track its progress.Copy the
trigger_urivalue from the response body. You will use it to check the status of the update.Send a
GETrequest to the${trigger_uri}to verify that the update has completed successfully. Replace${trigger_uri}with the value you copied from the response body and${token}with the corresponding value for your organization.(Recommended) Send test alerts to ensure that the mapping enrichment works as expected.
Resolve any test alerts, as necessary.
To change the structure of your mapping data:
Deactivate the existing enrichment by sending a
POSTrequest to/enrichments/${id}. Replace${id}with the value you copied from the response body in Step 1 and${token}with the corresponding value for your organization. Setactivetofalse.Create a new mapping enrichment, starting from Step 1: Define The Mapping Structure.
Synchronizing your CMDB data from ServiceNow with BigPanda allows you to enrich your inbound alerts with contextual information. This helps improve correlation, MTTR, reporting accuracy, and increases consistency within the platform.
ServiceNow CMDB basics
Looking for just the basics? Check out the ServiceNow CMDB Enrichment one page document in BigPanda University!
Before You Start
The information in this document is for reference purposes only and is not intended to take the place of architectural decision-making and engineering advice from a BigPanda expert. Considerations must be made before configuration should be attempted. Further configuration information is available in the ServiceNow CMDB documentation .
CMDB enrichment should be done after or in conjunction with Data Normalization .
Considerations Before Making Changes
Before making any changes to the BigPanda ServiceNow integration:
You must switch the Application Scope to BigPanda.
Ensure you have only one browser tab open on the BigPanda ServiceNow integration Configuration page to prevent issues. This prevents:
out-of-sync errors caused by browser window timeouts
unsaved configuration changes due to non-BigPanda Application Scopes interfering with your configuration tab or window. Often, errors happen when multiple browser windows with different application scopes selected are open.
We recommend making any changes in a lower environment (non-prod, sandbox, development BigPanda org) first to familiarize yourself with the process before executing them in your production BigPanda instance.
If you have questions about the configuration, please contact BigPanda support ([email protected]).
Enrichment Map Example
Data synchronized from your ServiceNow CMDB will appear in Enrichment Maps within BigPanda. These enrichment maps begin with “snow_cmdb_” and end with your “BigPanda Map Name”, such as “_network_gear”.
When finished your Enrichment tables might look something like this:
Synchronization may take several minutes to hours to complete depending on the size of each CI Class table.
Trigger it once and watch the synchronization happen in the ServiceNow Application Logs.
Best Practices
Incomplete or inaccurate information in your CMDB can cause inconsistencies in BigPanda. Remember the old saying, “Garbage in = Garbage out.” If there are discrepancies in your CMDB, we recommend resolving them before integrating with BigPanda.
When implementing this integration, we recommend you only export columns that are useful for improving correlation or enhancing the detail for manual triage processes. Be sure to review other enrichment sources that provide details before implementing any changes.
The exported column headers and synchronization update times are visible within BigPanda at Settings > Enrichment Maps .
Manual CMDB Synchronization
You can manually trigger a CMDB synchronization to verify newly configured tables.
To set up a manual trigger:
In ServiceNow, go to All > BigPanda > CMDB > CMDB Processing Schedule.
Set the run to On Demand and Save.
In the upper right corner, click the Execute Now button. This manually starts the sync.
After the sync has completed, set the Run state to Periodically and set your Preferred Run Time.
Trigger CMDB Synchronization
To trigger CMDB synchronization, you need to make a change to a table’s data. If no changes have occurred, the synchronization step will not be triggered, and this will be noted in the ServiceNow Application Logs.
If you change the configuration without performing the manual synchronization steps described above, you will need to wait until the next time the table has changed before you can see and verify the table in BigPanda. This could take hours or up to several days. Manually changing a record in the table will ensure the sync runs at the next scheduled interval.
Synchronization Settings
We recommend using the following synchronization settings:
Name | Recommended Setting | Notes |
|---|---|---|
Page Size | 1000 | This is the default setting and recommended maximum. |
Max Retries | 1 | If the CMDB integration has failed, it will try again once. If that fails, it will stop until the next execution interval. Setting this value low reduces unnecessary overhead on both BigPanda and ServiceNow. For example, if there are network communication issues between BigPanda and ServiceNow, a disruption from AWS, or a similar issue, you won’t want it to keep retrying continuously. The system will synchronize on the next execution interval, so there will only be a short period of time where enrichment data might be partially out of sync. |
Max Pages | 2500 | This is the recommended maximum page count. The maximum size is not the continually expected sync size, but the maximum that it will attempt to process at once. Typically, your sync will be between 200 and 500 pages and will take less than 20 minutes to complete. |
Retry Interval | 120 (seconds) | This value is how quickly the system will attempt a retry if there is an issue with CMDB sync. We recommend setting this to a minimum of 2 minutes and a maximum of 10 minutes. Keep in mind that if the sync has failed, retrying immediately will also generally fail. |
Execution Interval | 24 (hours) | We generally recommend 24-hour sync intervals, as this meets the requirements of the majority of customers. |
Configuration Examples
 |
You can use the examples below to populate the CMDB table export configuration. Please note that this list is not meant to be exhaustive. We recommend you refer to the tables and CMDB Classes used in your CMDB before taking steps to configure.
Below is a list of recommended tables in ServiceNow for use with enrichment.
The tables you decide to use should depend on the data you need for enrichment in BigPanda and the quality of the data in those tables and fields.
The list contains the default table names from a standard ServiceNow deployment. Your table names and data alignment may vary.
Recommended
Table Name | Table Label | Table Description |
|---|---|---|
cmdb_ci_appl | Application | A collection of files and data that deliver a service and manage business processes. |
cmdb_ci_service | Service | IT service that directly supports a business process (ITIL). |
cmdb_ci_business_app | Business Application | All business applications. |
cmdb_ci_server | Server | Base class for all types of servers. |
cmdb_ci_netgear | Network Gear | Extension of the Hardware table, that captures network equipment such as router, switch, hub, gateway, and bridge. |
cmdb_ci_storage_device | Storage Device | Base table for block storage devices such as DAS, SAN, and NAS. |
cmdb_ci_database | Database | Organized collection of data such as the set of files where data is stored, the reason for a database, and the metadata. |
cmdb_ci_msd | Mass Storage Device | Physical storage device. |
Advanced
Table Name | Table Label | Table Description |
|---|---|---|
cmdb_ci_service_discovered | Mapped Application Service | Application services, created by the Manual service population method. For each application service, there is a container CI record that models the application service. |
cmdb_ci_cluster | Cluster | Logical group of computing resources bound together by software in order to function as one logical computing resource. |
cmdb_ci_vm | Virtual Machine HyperVisor | Hypervisor software. |
cmdb_ci_vm_instance | Virtual Machine Instance | Generic virtual machines information. |
cmdb_ci_lb | Load Balancer | Server functioning as a load balanacer. |
cmdb_ci_lb_pool | Load Balancer Pool | Collection of host-to-port mappings to be balanced. |
Special Cases
Table Name | Table Label | Table Description |
|---|---|---|
cmdb_ci_docker | Docker Container | Docker containers (a runtime instance of a Docker image). |
cmdb_ci_db_catalog | Database Catalog | Metadata which defines database objects such as base tables, views (virtual tables), synonyms, value ranges, indexes, users, and user groups, for a specific database instance. |
Base Fields
Each table will have its own unique set of fields that may or may not be valuable depending on the object type, the data quality of the table, and the enrichment needs in BigPanda. We typically see the following base fields used in most maps, but you can bring in any field that is needed.
Field | Tag map | Description |
|---|---|---|
name | host | Usually the query value for the map. |
sys_id | object_sys_id | The ServiceNow object sys_id, typically prefixed with the object type. (i.e., server) |
sys_class_name | object_sys_class_name | The class or type of object. |
support_group | object_assignment_group | The assignment group that triages incidents for this object. |
Other tags that are often exported include ip_address, model, location, criticality, used_for, sla, department, category, priority, etc
Be sure to choose the exported columns that are populated and used by the CI Class. This list isn’t exhaustive; please refer to the CI Class definition inside of ServiceNow.
You can see what is defined per class via the CMDB CI Manager view within ServiceNow.
Best practice
Mapping assignment, ownership, and support is important. Review your configuration and ensure these fields are set based on your use case. Your CMDB tables may be affected by plugin applications, feature adoption, and age of creation.
Query Filter
The query filter field defines filtering for including or excluding certain rows from an export. This field is optional.
Entering nothing in this field will include all entries from the system class table defined. We recommend excluding retired systems from the CMDB export, however, you might find it useful to keep those synchronized. If you need to filter which table rows are synchronized to BigPanda, you can choose the operational status and opt-in (“=”) to include only in service values. Alternatively, you can use opt-out (“!=6”) to only exclude retired status values.
Column Mapping
The column mapping section is used to define how the exported columns will be mapped to tags when they are created in BigPanda.
Typically we see these tags defined to align with the standard tags set in the Data Normalization process, but some customers choose to differentiate these tags with prefixes denoting their source and/or object type. Below are examples of different types of column mappings:
Normalized Tag | ServiceNow Prefix Tag | Object Prefix Tag | ServiceNow and Object Prefix Tag |
|---|---|---|---|
assignment_group | sn_assignment_group | app_assignment_group | sn_app_assignment_group |
sys_id | sn_sys_id | app_sys_id | sn_app_sys_id |
(Optional) ServiceNow Update Set with Base Tables and Default Column Mapping
A BigPanda update set is available that will set the CMDB configuration section with the base tables and the recommended defaults for exported tables, map name, query filters, and column mapping. Contact your account team for access.
(Optional) Additional Table Exports
Not every potential CI Class is included here, only examples. Be sure to consider all of the different CI Class types that might be defined in your ServiceNow CMDB.
Additional table exports are often required for systems that don’t fall under the reference categories defined in this document. A good example is “Mass Storage Devices.”
Using the above formatting, you should be able to quickly gather the necessary details to export additional tables in the same manner. If you’re unsure, you can consult the BigPanda support team, the related documentation, or your BigPanda Account Team. Further guidance may be available and BigPanda Professional Services might also be able to assist you with your configuration.
Measuring Success
You can use the Unified Analytics Enrichment dashboard to track the percentage of alerts being enriched and the enrichment rate change over time. It also provides you with a top list of alerting objects that are not being enriched. This list can be used as a starting point for improving CMDB enrichment by identifying objects missing from the CMDB or additional enrichment maps that need to be configured for use in BigPanda enrichment.
 |