Monitoring Docker - Metrics and Log Forwarding
Overview
Details
Use cases
- Application Monitoring
- Log Aggregation
- Cluster Health Monitoring
- Security and Audit
- Reduce complexity and improve productivity
With 10 minutes setup, you will get a monitoring solution, that includes log aggregation, performance and system metrics, metrics from the control plane and application metrics, a dashboard for reviewing network activity, and alerts to notify you about cluster or application performance issues.
An application requires Collectord built by Outcold Solutions, see https://www.outcoldsolutions.com/docs/monitoring-docker/
Overview
Outcold Solutions provide solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer Splunk applications, which give you insights across all containers environments. We are helping businesses to reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications to help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer a unique solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance and cluster health.
Description
We provide solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. With 10 minutes setup, you will get a monitoring solution, that includes log aggregation, performance and system metrics, metrics from the control plane and application metrics, a dashboard for reviewing network activity, and alerts to notify you about cluster or application performance issues.
All our solutions are powered by the Collectord, a container-native software built by Outcold Solutions that provides capabilities for discovering, transforming and forwarding logs, collecting system metrics, collecting metrics from the control plane of the orchestration frameworks and forwarding network activity. Collectord provides flexible and powerful tools for transforming logs. With our software you can hide sensitive information from the loglines before forwarding them. With Collectord you can reduce the licensing costs associated with logging aggregation by choosing which data you want to forward from the log streams. Collectord forwards container logs, host logs and can discover logs written by the containerized applications.
Use cases
Application Monitoring
See detailed metrics from containers and processes, including performance metrics, utilization metrics and security insights. Forward application-specific metrics, exported in Prometheus format. Use prebuilt Splunk dashboards for a comprehensive overview.
Log Aggregation
Aggregate logs from containers, applications, and servers. Use flexible mappings to filter logs enriched with container metadata, correlate logs with metrics, and leverage Splunk capabilities for analyzing logs. Use Collectord to transform logs before they reach Splunk, remove sensitive information, remove PII data to help keep your logs GDPR compliant. With Collectord you can reduce licensing and storage costs by choosing which loglines you want to forward.
Cluster Health Monitoring
Diagnose cluster issues by looking at historical events, monitoring allocations, and regulating cluster capacity. Leverage pre-built alerts for monitoring the health of the clusters out of the box.
Security and Audit
Define access to the data by clusters, namespaces and even pods or containers. Review network activities, happening inside your cluster, and outside connections. Verify containers running with elevated security permissions. Use audit logs for monitoring changes in deployments.
Reduce complexity and improve productivity
Use one tool to collect and forward logs and metrics required by developers for reviewing performance and health of their applications. With the annotations developers can define how they want to see the data in log aggregation tool, specify multiline log patterns, removing terminal escape codes, override types, sources and indexes.
Links
Release Notes
Version 5.15.300
5.15.300 - 2020-06-01
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.15.300 or above (see https://www.outcoldsolutions.com for latest configuration)
Collectord updates:
- Support for annotations to add custom user fields to data
- Support for blacklisting and whitelisting Prometheus metrics (significally reducing the indexing cost of data)
- Verify command improvements - verify proper configurations for cgroup (memory/memory.use_hierarchy is 1)
- Bug fix: fix bug in prometheus metrics parser, empty fields can be filled with previous fields
- Bug fix: occasionally addon can report warnings about trying to delete expired keys from ack db
- Bug fix: better handle of connections to metrics endpoints exported in Prometheus format
- Bug fix: http connections improvements for when Splunk is unresponsive
- Bug fix: broken diag
Version 5.14.280
- Logs dashboard: filters depend on selection
Collectord updates:
- Support templates in the index, source and sourcetype
- Allow to exclude indexed fields when forwarding to Splunk
- Support annotation for stats interval for containers
- Bug fix: verify command can show incorrect error about verifying journald input
- Bug fix: index on namespace should set index for application logs
Version 5.12.273
5.12.273 - 2019-12-16
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.12.272 or above (see https://www.outcoldsolutions.com for latest configuration)
- Bug fix: Swarm Services dashboard compatibility with the new format of metrics
Version 5.12.271
5.12.271 - 2019-11-07
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.12.271 or above (see https://www.outcoldsolutions.com for latest configuration)
- Improvements for the macros for backward compatibility
- Bug fix: Swarm Services dashboard could not filter based on the stack name
Collectord updates:
- Bug fix: when event pattern is used for joining multi-line events, the error can not be showed if raised by the input in pipeline.
- Bug fix: reduce warnings failed to get the new event in pipeline - submitted
- Stability improvements
Version 5.12.270
- Compact metrics (pre-calculated on Collectord side)
- Switched stats for host and cgroup in different macros
- Containers count for hosts on Hosts and Host dashboards
- Improve performance of the search in Docker Services and ECS Services dashboards
- Use base macro for alerts
- Add cluster name in the alert results
Collectord updates:
- Describe command to see applied annotations for containers
- Bug fix: panic when pipe join configuration is removed
- Bug fix: panic when proc stats is enabled and cgroup stats is disabled
- Bug fix: support ProxyBasicAuthorization for license server checks
- Bug fix: Fix for collecting first sample (can show high CPU usage for first sample)
- Beta: dynamic index, source and sourcetype names based on the metafields
- Beta: cluster diagnostics with one ru
Version 5.11.260
5.11.260 - 2019-09-09
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.11.260 or above (see https://www.outcoldsolutions.com for latest configuration)
- Bug fix: changing source type does not allow to use Swarm and ECS services dashboards
- Bug fix: improving usability of Swarm and ECS services dashboards for large deployments
Collectord updates:
- Bug fix: duplicate events then pipeline is getting throttled
- Bug fix: don't use throttling for devnull output
- Bug fix: better recovery for ack db corruption
- Bug fix: crash on journald input initialization when ack db is corrupted
- Bug fix: annotations joinmultiline requires joinpartial
- Bug fix: configurations for stdout only with annotations can crash collectord
- Set events = 50 by default for Splunk output batches
Version 5.10.250
- Cluster field filters
- Base macro for overriding macros for other macros
Collectord updates:
- Support for volatile and persistent journald storage with default configuration
- Updated YAML configuration to include most common resources
- Better support for overriding sourcetype, that does not require to update the Splunk macros
- Bug fix: rarely when collectord fails to post to HEC it can panic
- Bug fix: better support for OpenShift 4.x and CRI-O storage
- Bug fix: space characters in index annotations can break the pipeline
Version 5.9.240
5.9.240 - 2019-05-14
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.9.240 or above (see https://www.outcoldsolutions.com for latest configuration)
- Visual improvements on the graphs for the number of logs and events
Collectord updates:
- Support for multiple Splunk destinations (outputs)
- Support subdomains for annotations (to deploy multiple collectord instances)
- Bug fix: journald input keeps fd open to the rotated files
- Bug fix: fix in the annotation parser for the interval annotations
- Bug fix: fix splunk url selection configuration for multiple splunk URLs
Version 5.8.231
5.8.231 - 2019-04-25
--------------------------------------------------------------------------------
- Bug fix: Collectord usage report shows trial licenses for all instances
Version 5.8.230
5.8.230 - 2019-04-22
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.8.230 or above (see https://www.outcoldsolutions.com for latest configuration)
- Bug fix: Swarm dashboard does not render containers, when namespace field is not available.
- Use multiselect filters for most dashboards and filters with possibility to input custom filters.
- Reduce dedup usage to improve performance on dashboards.
Collectord updates:
- Bug fix: clogging collectord output with errors when incorrect index is used.
- Bug fix: short lived containers can results in duplicating logs.
- Bug fix: clogging collectord output with warnings when kernel reports incorrect VmRss size.
- Bug fix: annotations cannot override timestamp location for fields extraction.
- Bug fix: verify command reports Journald input in incorrect place.
- Better support for cgroup symlinks, automatically discover correct location.
Version 5.7.220
Requires collectorfordocker version 5.7.220 or above (see https://www.outcoldsolutions.com for latest configuration)
- Review savedsearches/alerts to support indexing delay (start searches from 2 minutes behind) and run them in more random time.
- Fixed single value memory panel on host dashboard (missed span)
- Use SEGMENTATION=none for stats events to use less disk space (needs to me moved to indexers)
Collectord updates:
- Support hostname formatting with environment variables in configuration
- New rotated file logic uses less file descriptors and frees rotated files quicker
- Allow to specify a default sampling value for container logs
- Reimplemented shutdown sequence to stop collectord faster
- Allow to override sampling percent with annotations
- New Input: journald
Version 5.6.212
5.6.212 - 2019-02-19
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.6.212 or above (see https://www.outcoldsolutions.com for latest configuration)
- New: Alert: high CPU usage on the host.
- Fixed: Splunk usage dashboard - charts do not show the data, when the used indexed aren't searchable by default.
- New: Support Dark theme.
- New: Free text search in Logs dashboard.
- New: Add auto-refresh options to the dashboard.
- Fixed: Revisited CPU limits and requests for Pods and Containers.
Read more https://www.outcoldsolutions.com/docs/monitoring-docker/release-history/
Version 5.5.202
5.5.202 - 2019-01-24
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.5.202 or above (see https://www.outcoldsolutions.com for latest configuration)
- New: Dashboard Services -> AWS ECS. Review containers running as an ECS Service.
- New: Dashboard Services -> Swarm Services. Review containers running as a Swarm Service.
Collectord updates:
- Fixed: Interval 0 in Prometheus input can crash the collectord.
- Fixed: When both glob and match are set for the application logs, the glob pattern can block the match pattern from
finding the files in the volume.
Version 5.4.201
5.4.201 - 2018-12-19
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.4.201 or above (see https://www.outcoldsolutions.com for latest configuration)
- Fixed: Alerts for licenses issued with AWS Subscriptions
Collectord updates:
- Fixed: Better handling rotated files (less open fd)
- Fixed: Events input can hang in the err loop.
Version 5.4.200
5.4 - 2018-12-17
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.4 or above (see https://www.outcoldsolutions.com for latest configuration)
- Compatibility update for collectord 5.4.
Collectord updates:
- New: Attach EC2 metadata fields
- New: Basic Auth for Proxy (License Server and Splunk)
- Fixed: Collectord verifies reports CRI-O as unsupported runtime.
- Fixed: Rare crash on Prometheus metrics definition.
- Fixed: Better handling of acknowledgment database corruption.
- Fixed: When handling incorrect indexes, collectord can send index with an empty string, that Splunk recognize as an incorrect index
Version 5.3.190
5.3 - 2018-11-19
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.3 or above (see https://www.outcoldsolutions.com for latest configuration)
- New: Alert for showing when Collectord reports errors in Processing pipelines (as an example if it failed to extract fields).
- New: Alert for showing when Collectord reports warnings.
- New: Alert if lag in the indexing of the data.
- New: Splunk Usage (License usage, number of events) report under Setup.
Read more https://www.outcoldsolutions.com/docs/monitoring-docker/release-history/
Version 5.2.180
5.2.180 - 2018-10-28
- Fixed: lookup with alerts causing very often replication activities on SHC
5.2 - 2018-10-15
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)
- New: Review/Storage dashboard based on storage metrics.
- New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
- Fixed: Performance improvements
...
For details https://www.outcoldsolutions.com/docs/monitoring-docker/release-history/
Version 5.2.176
5.2 - 2018-10-15
--------------------------------------------------------------------------------
Requires collectorfordocker version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)
- New: Review/Storage dashboard based on storage metrics.
- New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
- Fixed: Performance improvements
...
For details https://www.outcoldsolutions.com/docs/monitoring-docker/release-history/
Version 5.1.175
- New: Network metrics (MB, Packets, Drops, and Errors) for host and containers.
- New: Network socket tables (list of the port that containers and hosts are listening on, connections to external resources).
- New: Network review dashboard to see the list of connection to public services and in private network.
- Improvement: Replace python-based lookup with a macro written with eval.
- Improvement: Visual improvement for showing when the object was Last Seen (highlighting and showing minutes ago).
...
For details:
https://www.outcoldsolutions.com/docs/monitoring-docker/release-history/
Version 5.0.174
Highlights:
- Support for Application logs
- Show Memory and CPU limits for container lists.
- Visual updates for the panels, highlighting high CPU and Memory usages
For more details
https://www.outcoldsolutions.com/docs/monitoring-docker/release-history/
Version 3.0.22
New security dashboard, CPU Shares, Quotas and Memory Limits monitoring.
A lot of of bug fixes and performance improvements.
Relese Notes: https://www.outcoldsolutions.com/docs/monitoring-docker/release-history/#30-2018-02-07
Upgrade instructions: https://www.outcoldsolutions.com/docs/monitoring-docker/upgrade-2-to-3/
Requires collectorfordocker version 3.0 or above (see https://www.outcoldsolutions.com for latest configuration)
Version 2.1.21
2.1.21 - 2018-01-02
--------------------------------------------------------------------------------
Requires collectorfordocker version 2.1.59.171209 or above
- Updated author and description
2.1.18 - 2017-12-09
--------------------------------------------------------------------------------
- Implemented collectors dashboard to track number of collectors, their versions
and used licenses.
- Fallback to the process IO statistics when blkio is not available.
- Fix IO statistic graphs, showed average, when sum should be used.
- [collector] Improved resistance for storage failures.
- [collector] License checks reporting.
Version 2.1.18
2.1.18 - 2017-12-09
--------------------------------------------------------------------------------
Requires collectorfordocker version 2.1.59.171209 or above
- Implemented collectors dashboard to track number of collectors, their versions
and used licenses.
- Fallback to the process IO statistics when blkio is not available.
- Fix IO statistic graphs, showed average, when sum should be used.
- [collector] Improved resistance for storage failures.
- [collector] License checks reporting.
Version 2.0.17
2.0 - 2017-10-22
--------------------------------------------------------------------------------
Requires collectorfordocker version 2.0.37.171023 or above
- Better labels support in Dashboards.
Collector has a breaking feature, replacing format for labels from
`docker_labels_LABEL1=VALUE1` to `docker_labels=[LABEL1=VALUE1,LABEL2=VALUE2]`.
- Process level metrics.
- Uptime for hosts and processes.
- Fields extraction and support in dashboards for docker daemon (setup
host logs collection with collector).
- New top-like dashboards allow to monitor Hosts/Containers/Processes in realtime.
- Improved dashboards navigation.
- Other bugs and improvements based on user feedback.
Version 1.0.3
Updated links to official documentation on how to install collector.
Version 1.0.1
- App Certification
- Fix layout, time/period synchronization between stat graphs
Version 1.0
Initial release
Docker logs, metrics and events in one place