Data Normalization Best Practices
Learn best practices for standardizing and structuring event data from various tools into a consistent format.
Data normalization in BigPanda is the process of standardizing and structuring event data from various monitoring and observability tools into a consistent format.
This process involves:
- Standardization: Identify a standard set of tags to map tags and data fields from different sources, ensuring uniformity across all integrations. Consider downstream impacts as you identify the tags.
- Integration: Integrate your monitoring tools with these standardized tags in mind. BigPanda has 50+ standard integrations, or you can integrate using the Open Integration Manager (OIM), Alerts API, Email Parser, or BigPanda Daemon (SNMP).
- Cleansing: Ensure data within tags from your monitoring tools are cleansed where possible, such as removing domain names from the host tag names to align with how data is stored in enrichment sources.
- Consistency: Adjust primary and secondary properties to drive event marshaling, deduplication, and compression
- Enrichment: Enhance alert payloads by adding contextual information and consolidating similar tags into a single, consistent tag to ensure uniformity across all alerts.
Integrating observability tools from different vendors can be challenging due to differences in how they tag and report data. Without standardization, the downstream components of the BigPanda pipeline can be negatively impacted. When tags aren’t consistent or normalized, the entire system can be less effective and accurate, leading to slower response times and missed critical alerts.
Normalizing tags prepares you to introduce additional monitoring sources, allowing you to scale and evolve with your organization’s needs. Normalized tags must define and meet minimal criteria for downstream actions, such as ticketing or automation, ensuring that all necessary information is available for these processes to function smoothly.
Key Benefits of Data Normalization
- Normalization establishes data presentation consistency across events from all sources within BigPanda.
- Different alert sources will look similar from a tag perspective.
- The data unification process simplifies downstream configuration and improves the maintainability of BigPanda while also driving improved alert quality.
Start Small and Scale
Your data doesn’t have to be perfect to begin with. Start with a handful of data sources, gain visibility into the quality of alerts, and slowly ingest more sources.
Ideally, you should select your top services that impact business. Ask yourself, “What keeps you up at night? Tackle the most critical services and applications. Don’t prioritize low-hanging fruit if the important services aren't done.
Make sure each monitoring tool is passing along all of the data that your team needs to detect, understand, and remediate incidents. Standardized tags drive everything downstream from event compression to auto-remediation. Collaborate across teams when defining your organization's standard tags to ensure they meet everyone's needs.
Identify opportunities to bring in and streamline the data that will power remediation and automation such as runbook URLs. As tools are added or changed, make sure to configure them to leverage the existing standard tags to prevent unnecessary reconfiguration.
Tag Standardization
Tag standardization is a key step towards data normalization. Standardizing tags helps enhance the performance and reliability of BigPanda.
Tag standardization involves mapping tags from your observability tools to a common set of core tags, ensuring uniformity across all tools. This consistency gives you a cohesive view of the data, making it easier to manage and interpret.
Key Principles for Tag Standardization
- Less is more. Having too many standard tags in an organization can make it difficult for operators to manage alerts.
- Be consistent with tags across all monitoring tools - different alert sources then will look similar from a tag perspective.
- The data unification process improves alert quality and simplifies downstream configurations such as correlation, reporting, and incident enrichment, making the platform easier to maintain.
Tag Classification
Tags can be classified as functional or contextual.
Functional tags are crucial for BigPanda’s core operations. These tags facilitate tasks such as correlation, reporting, CMDB lookups, and integration with downstream tools like ticketing and notification platforms.
Contextual tags provide valuable insights to operators about issues but are not essential for BigPanda’s functionality.
Sometimes less is more; having fewer tags can streamline the console for operators. However, your administrator must determine which contextual tags should be included based on your organization’s needs. This involves working with the teams using the console or receiving the relevant data to action an incident. We generally recommend avoiding redundant tags, but you should also ensure that operators have all of the information they need to troubleshoot incidents.
To improve data normalization and consistency, we recommend using specific tag names for common monitoring elements. Standard tag names ensure that your data performs consistently and that you can leverage the full automation capabilities of BigPanda.
We understand that every organization has unique requirements and needs. BigPanda allows for flexibility based on your use cases, but we recommend leveraging as much of our standard as possible. Since Unified Analytics is designed to use standard tags, additional configuration may be required if these standards are not implemented.
If necessary, your options to change the data model include:
- Renaming tags - suggested when you have terms specific to your organization that you’d like to maintain
- Adding new tags - when there are specific use cases you’d like to account for in your standard set of alert tags
- Removing tags - for tags not pertinent to your organization’s use cases
The processes for renaming, adding, and removing tags will be addressed later in this guide.
Normalization can happen at the source monitoring tool, during the OIM integration, or through the enrichment process. BigPanda's standard integrations do not automatically use standard tags, so implementation teams and administrators must configure them within the platform.
Standard Tags
The following are the standard tag names recommended by BigPanda:
| Standard Tag Name | Tag Description | Source Tag Names |
|---|---|---|
| host | Name of the target system that generated the alert. This tag is the default primary property for many integrations | hostname, instance, device, machine, server, hosts |
| object | The object impacted that generated the alert. | object, resource, resource_name, target, instance, job, url, etc. |
| impacted_entity | The entity impacted that generated the alert. | impacted_entity, resource, resource_name, target, instance, job, url, etc. |
| service | The impacted service. | impact_service, srvc |
| application | The impacted application. | app, app_name, impacted_app |
| check | Name or title of the alert. This tag is the default secondary property for many integrations. | alert, alertgroup, title, monitor, monitorname, testname,event_id, rule, rulename |
| description | Full text or description of the alert. This tag should not be used as the primary or secondary property. | summary, ldesc, long_desc, problem |
| assignment_group | Assignee group in the outbound tool. | support_group, routing_group, escalation_group, workgroup |
| support_group | The support group associated with the CI. | support_group, routing_group, escalation_group, workgroup |
| app_support_group | The support group associated with the impacted application. This could be a service. | support_group, routing_group, escalation_group, workgroup |
| business_group | The line of business impacted by the alert. | business_unit, logical_group, lob |
| priority | Importance or severity of the alert. | severity, inc_priority, alert_severity |
| environment | The instance or environment that triggered the alert. For example, prod, dev, etc. | env, tier, stage |
| location | The physical location where the alert triggered. | physical_location, device_location, address |
| runbook_url | Link or location of the related runbook. | runbook_link, knowledge_base, ki_article, support_link, kb_article, wiki_url |
Special Tags
Some tags have functionality that is essential to BigPanda’s operation. These tags play a vital role in data visualization, deduplication, and alert resolution.
BigPanda special tags are:
- primary_property
- secondary_property
- incident_identifier
- description
- timestamp
- status
Reserved Words
Some words are reserved and cannot be used as alert tag names in BigPanda. Tags with these names may be saved, but they will not enrich alerts and can cause issues when filtering alerts.
Reserved words during OIM configuration:
- incident_identifier
- primary_property
- secondary_property
Reserved words during Alert Enrichment tag configuration
- alert_updates
- description
- namespace
- severity
Some tag names may also have downstream limitations. For more information on naming requirements, see the Tag Naming documentation.
Primary and Secondary Properties
Primary and secondary properties are key fields used for normalization, deduplication, and event correlation in BigPanda.
Each event must have a primary property associated with it. The secondary property is optional, but highly recommended as it improves event deduplication and normalization. The value of each property should be the name of another tag in the payload. BigPanda allows you to explicitly set the primary property as part of the integration and/or event payload for granular control.
When the primary property is not explicitly set, BigPanda will leverage the default hierarchy:
- Primary - host, service, application, impacted_entity, object
- Secondary - check
Troubleshooting Primary and Secondary Properties
For each standard integration, BigPanda provides a Troubleshooting view that shows events that failed to process into the pipeline.
This troubleshooting happens at the ingest point prior to deduplication. This does not account for custom integration monitoring.
Keep in mind that some errors you see could be due to
- Misaligned primary/secondary properties
- Malformed JSON
- Invalid characters
- Missing required fields
If the primary property is set to a tag that doesn’t exist in the payload or if none of the hierarchy fields are found in the payload, the event will fail to process and display an error in the integration troubleshooting screen.
Best Practice
The primary property should align to the physical or virtual entity that is experiencing the issue (host, object, application, service, impacted_entity tags), and the secondary property align to the name or title of the issue (the secondary should always align to check tag).
Primary and Secondary Properties in the UI
The primary property and secondary properties are one of the display fields on the alert when:
- an incident does not have any correlation applied. The primary property will display as the incident title in the incident panel and the secondary will display as the supplemental field when focused on an incident.
- correlation is applied. The primary property will display as the supplemental field when focused on an incident. The secondary will not display as a highlighted attribute. If the primary is the same as the correlation, the secondary properties will still display as the supplemental field.
To maintain a consistent look in the UI, we recommend limiting primary tags to a select few, specifically host, object, application, and service. The secondary property should always be set to check to ensure clarity and coherence in tag presentation.
Incident Identifier
The incident identifier is the event attribute that determines uniqueness. For granular control, you can set it as part of the integration and/or event payload.
A common scenario is if a monitoring tool contains a unique identifier (something like alert_id as a random character string), but does not function as a tag an operator would leverage in triage. Unlike primary and secondary, this value should be the explicit value and not the tag name to set the configuration. You can also use ${field1}${field2} to substitute the values in from the tag names.
When the incident identifier is not set, BigPanda will concatenate the tag values associated with the primary and secondary properties together. If the secondary is not set, only the primary tag value is considered.
Best Practice
By default, the incident identifier is created using the primary and secondary properties.
We recommend setting the incident_identifier based on the primary and secondary properties, provided that these properties ensure uniqueness, to avoid deduplicating unrelated alerts.
If the primary and secondary properties include timestamps or metric numbers, other methods for deduplication using the incident_identifier tag should be considered.
Description
The Description tag of an event sent by a monitoring tool describes the problem alert in detail. Description is a specialized tag that allows for up to 2048 characters, which is larger than the standard size of 512 characters.
Description is a reserved word, which means it cannot be changed or redefined for custom enrichment.
Timestamp
The timestamp tag shows the time when a monitoring tool triggered an event or when an action happened in BigPanda. Timestamp is shown in Unix epoch format in seconds.
Best Practice
If your monitoring source sends a timestamp in epoch format, we recommend you map this value to the timestamp field. If no timestamp is provided, the time BigPanda received the alerts will be used as the time the alert was triggered.
BigPanda supports timestamps in seconds only. If the time is sent in milliseconds, BigPanda will convert it to seconds. This can cause issues with flapping alerts that flap within milliseconds.
Tag Standardization for Automated Workflows
The first step in tag standardization for automated workflows is to identify existing automation tools and how they will interact with BigPanda.
Identify the required tags for the outbound tool BigPanda will communicate to. Since most of these will end up being an autoshare, we recommend adding autoshare_[category] as a standard.
The following are common tools or categories implemented with BigPanda:
- autoshare_itsm
- autoshare_automation or autoshare_remediation
- autoshare_email
- autoshare_sms
- autoshare_aia
- autoshare_paging (call, on-call, etc.)
- autoshare_chat (Slack, Teams, etc.)
Best Practice
We recommend the values for these tags be boolean [true, false, or #] to indicate whether alerts need to be routed to the proper workflow. If an autoshare delay is needed, you could expand the tag value to include numeric delay timers.
For example, if you have 2 auto-share delays and a 3 and 5-minute timer, you could set your autoshare tag as [0, 3, 5] to indicate which delay timer to use.
Tag Normalization
Tag normalization transforms and cleanses data from disparate tools into a standardized format. This includes tasks like converting fully qualified domain names to short names and ensuring that all data adheres to the same naming conventions.
Key Principles of Tag Normalization
- Start with a baseline of standard tags (host, check, application, service, assignment_group)
- Adapt to your company tools and terminology
- Normalized alert tags should represent the primary focus for downstream teams
- Normalized tags allow for the introduction of additional monitoring sources
- Normalized tags should always define and meet minimal criteria for downstream actions (ticketing, automation, reporting)
- Downstream setup can be done at various points (integration level, Open Integration Manager (OIM) level, enrichment sources, or custom tags)
- Use Alert Views to bring standardized tags to the top of an alert’s list of tags
The 5 Ws of Tag Normalization
Normalized tags should always answer one of the “5 Ws”:
- Who is generating event (host, application, service)
- What do I need to solve this issue (runbook_url)
- Where do I need to escalate to (assignment_group)
- Why is this event happening (check)
- When do I need to have this addressed by (priority, impact, urgency)
Mapping to Standard Tags
Tags can be thought of as building blocks that can be mapped to a logic definition that impacts downstream processes or helps with incident triage.
You may find yourself having “tags of tags” - where a normalized tag is built upon the input from multiple normalized tags. For instance, the cmdb_ci tag could be sourced from both the host and application tags based on the nature of the monitoring.
Once the proper set of standard tags is defined, the next step is to map each of your monitoring integrations to a set of these tags. This can include a tag renaming exercise and applying data transformation techniques to set the correct value to the right tag name.
Event normalization in BigPanda occurs at several points including the alert source, the Open Integration Manager (OIM) configuration, and during alert enrichment. We recommend configuring normalization as early in the event pipeline as possible.
The following options are available for transforming tags:
- Adjust values at the monitoring source layer
- Leverage the OIM configuration to set appropriate tag names and values
- Use Alert Enrichment to take advantage of mapping enrichment along with extraction and composition alert tags
You can also proactively review downstream configurations to adjust your normalized tag set. Examples include:
Normalization in the Monitoring Source
Some monitoring tools allow you to edit key/value pairs in the event configuration or free forms to define payloads. Normalizing early in the pipeline simplifies downstream configuration as events enter with the proper set of tags.
Best Practice
We recommend that Tool Admins determine early on if the monitoring tool allows these types of edits.
Normalization in the Open Integration Manager (OIM)
The Open Integration Manager (OIM) contains a tag mapping component that allows you to normalize the original monitoring tool payloads, often referred to as source tags, to your destination tag, which aligns with your standardized model.
You can also map multiple source tags. OIM can scan a CSV, take the first tag it finds in the alert payload, and align that tag to the destination tag.
Advanced tag mapping allows for extraction directly within the integration configuration using regular expressions. This functionality is similar to event enrichment for alert extraction tags.
Common use cases include:
- Integration sources that don’t send the same types of values. For example, Source A sends host, and Source B sends FQDN. Extraction can align these to the same values.
- Nested values within the alert tag details. For example, monitoring tools may concatenate multiple values into one tag which need to be pulled out to be set as their own individual tag.
If the value cannot be referenced or requires rewriting in a different format, then this must be done in Alert Enrichment via Composition.
Normalization in Alert Enrichment
Monitoring tools often do not provide all the detailed information required for operators to efficiently manage and respond to alerts. Key details, such as support_group, assignment_group, runbook_url, and environment, are sometimes missing from the initial alert payloads. To ensure these critical tags are available, many of them need to be added through the Alert Enrichment processes.
The Open Integration Manager (OIM) configuration has some limitations in its ability to extract and normalize data to create standardized tags. For example, OIM may struggle with complex regex extraction, such as extracting specific components from an alert payload or transforming the data into a more usable format. Alert enrichment can handle these tasks more effectively. Complex regular expressions can be employed within the alert enrichment process to extract specific values, such as a hostname, from an alert payload.
Alert enrichment can also be used to manipulate extracted data, such as by removing the domain name from a fully qualified hostname, making the alert information more relevant and actionable. By leveraging alert enrichment, you can ensure that alerts contain all the necessary information for efficient and effective incident management.
Alert Views
Normalized tags drive consistency, allowing you to understand which tags to focus on for triage and remediation. These tags should be displayed first to quickly summarize an issue, with additional information available in the alert details. You can use Alert Views to define the order in which tags appear within an alert.
Measure Success with Unified Analytics
You can use Unified Analytics to track and measure the progress and success of tag standardization and normalization across your monitoring environment.
The Alert Quality, Enrichment, and Tag Standardization Dashboards allow you to monitor the quality of your alerts, assess how effectively they are being standardized and normalized, and determine how well they align with your enrichment sources. These dashboards provide a comprehensive view of consistency and quality, helping ensure that alerts are uniformly processed and enriched across all your monitoring tools.
The Alert Quality Dashboard provides visibility into the quality of alerts sent to BigPanda and their impact on Mean Time to Resolve (MTTR). You can use this dashboard to quickly evaluate your alert quality level and identify key areas for improvement. Alert quality uses the host, CI, assignment_group, runbook_url, impact, and enrichment tags to determine quality levels.
The Enrichment Dashboard displays the effectiveness of your enrichment configuration and highlights alerts that are slipping through enrichment rules. By visualizing the enrichment flow, the dashboard ensures that alerts are sufficiently detailed when they reach downstream systems, making it easier for incident responders to understand the context and take appropriate actions.
The Enrichment Dashboard can also help you track the effectiveness of normalizing key tags such as host and application. If alerts are not being enriched, it's likely that your host or application tags are missing or not normalized correctly.
The Tag Standardization Dashboard helps you track how well you are standardizing across all your alert sources. It tracks the implementation and success of standard tags that are applied during the normalization process. You can customize this dashboard to help you track the tags that are important in your organization.
The Tag Standardization Dashboard isn’t available as a standard dashboard and will need to be created custom for each organization. Contact your account team for more information.
Resources
Updated 3 months ago