Features

Collecting data for centralized storage and analysis

In IBM Cloud Logs, you can collect telemetry data that is generated in IBM Cloud, on-premises, and by other cloud providers.

IBM Cloud Logs integrates with popular logging frameworks and libraries, letting you easily transfer your data to IBM Cloud Logs for centralized storage and analysis.

Optimizing value and controlling your observability budget

Not all data is valued equally. IBM Cloud Logs helps optimize the value of the data that you keep by using the TCO optimizer.

When you review your observability needs and budget, you can select from three tiers of log and event processing:

• Store and search: Data that is retained primarily for compliance obligations can be stored and searched as necessary at a lower cost/GB.

• Analyze and alert: Log and event data with analysis and alert value is processed at a mid-tier cost/GB. The mid-tier includes adding the definition of metrics from logs, allowing the visualization of trends and preparation for quickly handling future incidents.

• Priority insights: Select and configure most critical and highest value data to your operations for priority query results. Data in this tier is retained in hot storage.

In the TCO optimizer, you configure policies that define which data pipeline handles data after ingestion. Data prioritization before data is stored or indexed reduces monitoring costs while simultaneously improving system visibility.

For more information, see Controlling costs, TCO Optimizer, and Data Usage.

Adhering to regulations for compliance and security

Data security and compliance is an IBM Cloud Logs priority.

IBM Cloud Logs offers encryption at rest and in transit, access controls, and adheres to industry-standard security practices.

IBM Cloud Logs helps businesses meet standards, compliance requirements and regulations, such as GDPR and HIPAA, by providing features such as data anonymization and audit logs.

Predicting abnormal behavior to promptly detect abnormal situations

In IBM Cloud Logs, you can detect abnormal behavior by configuring anomaly detection alerts that use artificial intelligence algorithms to analyze incoming logs and predict their expected behavior for 24 hours.

For example, you might want to detect rising trends of errors that are related to bad API responses that cause a spike in your logs and indicate something unusual is happening. Discover when a transaction’s response time exceeds its usual duration, letting you pinpoint and address performance bottlenecks. Or, detect an unexpected spike of logs in your environment due to outgoing traffic of a host that exceeds its usual levels and might indicate that a potential security breach is occurring.

Generating metrics derived from logs to enhance observability

In IBM Cloud Logs, you can improve observability to identify performance issues, monitor system's reliability, or troubleshoot problems by using the Event2metrics feature.

For example, you might use a log-based metric to monitor the page load time, the duration of a process, or count the number of log entries that contain a specific error code.

As data is ingested, metric data is derived from your logs and converted into Prometheus metrics. You can use dashboards to visualize your metrics. Alerts can be configured to notify of unexpected behavior.

Using metrics that are generated from log data is a great way to look at vast amounts of data quickly when you search on different data sources.

For more information, see Configuring collection of metrics from logs.

Restructuring data for analysis and troubleshooting

You can restructure and parse log data to aid processing and increase the value of your data by using parsing rules.

IBM Cloud Logs parsing tools help you evaluate if data is essential or redundant. Restructuring data can help you aggregate dissimilar information that teams need to quickly find to address incidents.

IBM Cloud Logs is designed to convert log data to summarize what is happening.

For more information, see Parsing rules.

Enriching telemetry data with context information for enhanced analysis

In IBM Cloud Logs, you can enrich your log data with more context. You can automatically add fields to your JSON logs based on specific matches in your log data by using a pre-defined custom data source of your own. This way, you can enhance your log data with business, operations, or security information that is not available at run time. Enhanced telemetry data is more meaningful and actionable for effective troubleshooting, root cause analysis, and performance optimization.

For more information, see Enriching data.

Archiving logs for long-term storage

In IBM Cloud Logs, you can configure a data bucket and a metrics bucket in IBM Cloud Object Storage for long-term storage and search of your logs and metrics. Reindexing of logs from the archive is not needed to explore and query archived data. You can use the same queries and dashboards that you use to initially monitor the data.

In the TCO optimizer, you configure policies that define which data pipeline handles data after ingestion. For data handled through the Analyze and alert or the Store and search pipelines, data is sent only to the bucket. All data that you choose to store for search across all the TCO data pipelines is stored in the data and metrics buckets.

Data is stored after enrichment policies and parsing rules are applied. The data in long-term storage includes the context information that you choose to add and any additional restructuring of log data that is applied to enhance your monitoring, analysis, and troubleshooting.

For more information, see Configuring buckets for long-term storage and search.

Querying data to gain insights for troubleshooting and analysis

With IBM Cloud Logs, you can search and easily query all data that is retained by IBM Cloud Logs.

Data is stored in IBM Cloud Object Storage in a search-friendly format. When rapid-search results are needed, data can also be stored in hot storage for priority insights into the data. Query results for both hot storage and buckets can be displayed in the same dashboards and views.

IBM Cloud Logs offers multiple tools to effectively query data:

• Simple boolean search.
• Regular expressions (RegEx queries) to match patterns in your log.
• Queries based on Apache Lucene for key value querying.
• Queries based on the IBM Cloud Logs DataPrime language. Dataprime is a query syntax for describing event transformations and aggregations. You can use DataPrime to explore data, do schema on read transformations, group and aggregate fields, extract data, and much more.
• Prometheus Query Language (PromQL) to select and aggregate time series data that is derived from your logs.

Prompted help is available to construct queries for complex analysis.

For more information, see Querying data.

Improving operational visibility to gain insights for better analysis

In IBM Cloud Logs, you can improve operational visibility by configuring dashboards to gain insights for better analysis, troubleshooting, and decision-making of your organization's environments and applications.

You can create unlimited, personalized custom dashboards catered to your specific observability needs, or take advantage of pre-built dashboards to help you analyze and visualize log data.

Dashboard insights, which are paired with IBM Cloud Logs machine learning analytics, give SREs the ability to quickly identify the start of an incident before it becomes a significant issue.

For more information, see Dashboards.

Preconfigured alerts and dashboards are available as extensions for common application environments and can be tailored to your specific environment needs. For more information, see Extensions.

Notifying about issues to raise awareness and act promptly

You can configure alerts to detect and address issues before users notice them by proactively notifying you.

Sophisticated alert rules can be configured to reduce triage time. Examples include:

• Notifying when a combination of alert events happens within a defined set of criteria.

• Receiving alerts when new errors or log types are detected, or anomalous values occur on established data.

For more information, see Alerts.

Incident management control

IBM Cloud Logs provides alert incident management control. This control helps manage the operation of workloads and comprehensive environments with maintenance windows that can be managed within the tool. When complex incidents occur triggering multiple alarms, users can see the situation quickly within IBM Cloud Logs. Configured alert management within IBM Cloud Logs can suppress unnecessary alerts to other alert management solutions.

For more information, see Managing triggered alerts in IBM Cloud Logs.

Integrating with other applications

IBM Cloud Logs is designed to integrate with most common application and systems management tools and fits within most toolchains. Sharing data with other operational tools is built in by design:

• Integrate with alert management tools by using webhook values within alert messages so that information is included in the alert. This information can be used to quickly identify the source that triggered the alert.

• Share alert data with the IBM Cloud Event Notifications service for comprehensive IBM Cloud alert management visibility and control.

• Share alert data with PagerDuty and other specialized alert management tools.

• Integrate with other observability, SIEM, and data analysis tools. IBM Cloud Logs can send data to IBM® Event Streams for IBM Cloud®, a Kafka service implementation, where data can be shared with a wide variety of tools and applications.

For more information about integrating using IBM Cloud Event Notifications, see Working with alerts. For more information about streaming using IBM® Event Streams for IBM Cloud®, see Streaming data