Monitoring Kubernetes - Metrics and Log Forwarding

Overview

Details

Focus on your applications, we will take care of infrastructure monitoring and logs forwarding.

Use cases

- Application Monitoring
- Log Aggregation
- Cluster Health Monitoring
- Security and Audit
- Reduce complexity and improve productivity

With 10 minutes setup, you will get a monitoring solution, that includes log aggregation, performance and system metrics, metrics from the control plane and application metrics, a dashboard for reviewing network activity, and alerts to notify you about cluster or application performance issues.

An application requires Collectord built by Outcold Solutions, see https://www.outcoldsolutions.com/docs/monitoring-kubernetes/

Overview

Outcold Solutions provide solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer Splunk applications, which give you insights across all containers environments. We are helping businesses to reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications to help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer a unique solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance and cluster health.

Description

We provide solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. With 10 minutes setup, you will get a monitoring solution, that includes log aggregation, performance and system metrics, metrics from the control plane and application metrics, a dashboard for reviewing network activity, and alerts to notify you about cluster or application performance issues.
All our solutions are powered by the Collectord, a container-native software built by Outcold Solutions that provides capabilities for discovering, transforming and forwarding logs, collecting system metrics, collecting metrics from the control plane of the orchestration frameworks and forwarding network activity. Collectord provides flexible and powerful tools for transforming logs. With our software you can hide sensitive information from the loglines before forwarding them. With Collectord you can reduce the licensing costs associated with logging aggregation by choosing which data you want to forward from the log streams. Collectord forwards container logs, host logs and can discover logs written by the containerized applications.

Use cases

Application Monitoring

See detailed metrics from containers and processes, including performance metrics, utilization metrics and security insights. Forward application-specific metrics, exported in Prometheus format. Use prebuilt Splunk dashboards for a comprehensive overview.

Log Aggregation

Aggregate logs from containers, applications, and servers. Use flexible mappings to filter logs enriched with container metadata, correlate logs with metrics, and leverage Splunk capabilities for analyzing logs. Use Collectord to transform logs before they reach Splunk, remove sensitive information, remove PII data to help keep your logs GDPR compliant. With Collectord you can reduce licensing and storage costs by choosing which loglines you want to forward.

Cluster Health Monitoring

Diagnose cluster issues by looking at historical events, monitoring allocations, and regulating cluster capacity. Leverage pre-built alerts for monitoring the health of the clusters out of the box.

Security and Audit

Define access to the data by clusters, namespaces and even pods or containers. Review network activities, happening inside your cluster, and outside connections. Verify containers running with elevated security permissions. Use audit logs for monitoring changes in deployments.

Reduce complexity and improve productivity

Use one tool to collect and forward logs and metrics required by developers for reviewing performance and health of their applications. With the annotations developers can define how they want to see the data in log aggregation tool, specify multiline log patterns, removing terminal escape codes, override types, sources and indexes.

5.17.370 - 2021-10-20

Show milicores/cores CPU usage instead of percents
New dashboard: Review - Resource Quotas
Review - Projects: filter by project name
Review - Clusters: filter by node label
Review - Clusters: include max and avg usage
Bug fix: storage dashboard might not render in some Splunk versions
Bug fix: Namespaces dashboard shows only one namespace label
Collectord updates:
Upgrade to Go 1.17.2
Support query in Prometheus URLs for metrics
Collectord now reports source and source type for the events with incorrect index
Support for licensing server
Support for CPU-based licenses
Allow to specify multiple values for blacklist and whitelist for host logs
Bug fix: Collectord clogs the output with WARN messages for stopped containers running with Containerd
...

Version 5.16.363

May 27, 2021

5.16.363 - 2021-05-26

Requires collectorforkubernetes version 5.16.363 or above (see https://www.outcoldsolutions.com for latest configuration)

Bug fix: Put in parentheses source selection in macro_openshift_prometheus_metrics

Collectord updates:

Upgrade go runtime to 1.16.3
Bug fix: fix verbose logging for docker watcher with messages "failed to get next event"
Bug fix: NetworkPolicy cannot be watched, as Collectord does not convert it in plural form properly
Bug fix: Verify command fails on Containerd runtime
Bug fix: DefaultIdleConnTimeout is ignored for HTTP clients

Version 5.16.361

March 17, 2021

Overview dashboard filters respect filters (show only namespaces from selected cluster)
Bug fix: use correct units for Memory and Storage (MiB, MB, Mb)
Bug fix: compatibility with new format of Events from API server (FirstSeen, LastSeen, Source could be shown as null)
Bug fix: Collectord metrics request time shows the summary on the period, not the individual request times

Collectord updates:

ARM64 image
Allow removing managed fields from events (enabled with new configurations by default)
Upgrade to Go 1.16.2
Bug fix: precise time to Splunk HEC, sending with milliseconds instead of nanoseconds (which are incorrectly ronded by HEC)
Bug fix: first sample of the container can record above 100% of the CPU usage, as the values are pretty low
Bug fix: verify command does not respect glob patterns for Prometheus inputs (certs, tokens)
Bug fix: trim spaces in token value for Prometheus inputs
Bug fix: better recovery from corrupted journald

Version 5.16.360

March 17, 2021

5.16.360 - 2021-03-16

Overview dashboard filters respect filters (show only namespaces from selected cluster)
Bug fix: use correct units for Memory and Storage (MiB, MB, Mb)
Bug fix: compatibility with new format of Events from API server (FirstSeen, LastSeen, Source could be shown as null)

Collectord updates:

ARM64 image
Allow removing managed fields from events (enabled with new configurations by default)
Upgrade to Go 1.16.2
Bug fix: precise time to Splunk HEC, sending with milliseconds instead of nanoseconds (which are incorrectly ronded by HEC)
Bug fix: first sample of the container can record above 100% of the CPU usage, as the values are pretty low
Bug fix: verify command does not respect glob patterns for Prometheus inputs (certs, tokens)
Bug fix: trim spaces in token value for Prome

Version 5.16.352

Feb. 22, 2021

Bug fix: events dashboard does not filter by the namespace name

Version 5.16.351

Dec. 15, 2020

New dashboard: Collectord metrics
Compatibility for Kubernetes 1.20
Bug fix: broken link in Allocatable Resources dashboard

Collectord updates:

Annotations for collecting prometheus metrics: authorization keys and CAName for SSL certificates
Improvement for DNS resolutions of Splunk output FQDN
Export internal collectord metrics in Prometheus format
Forwarding internal collectord metrics to Splunk
For the watch objects inputs being able to hide management fields
In the diag include all open file descriptors
Upgrade go runtime to 1.14.13
Remove \0 symbol from the labels values in the prometheus metrics
Allow to filter host logs with blacklist and whitelist
Bug fix: less verbose warnings about not being able to load resources from API server
Bug fix: performance improvements for Ack DB
Bug fix: custom prometheus metrics forwarded by Collectord do not include cluster field or custom user fields
Bug fix: addon pod terminates faster
....

Version 5.15.300

June 1, 2020

5.15.300 - 2020-06-01

Requires collectorforkubernetes version 5.15.300 or above (see https://www.outcoldsolutions.com for latest configuration)

Events dashboard: filters depend on selection of cluster and node labels
Improvements for supporting Kubernetes 1.14 and higher (OpenShift 4.2+)
Improvement for alert "Cluster Warning: high number of errors to Kubernetes API" (only alert on 5xx errors)
Bug fix: node events aren't visible in Events tab

Collectord updates:

Support for annotations to add custom user fields to data
Support for blacklisting and whitelisting Prometheus metrics (significally reducing the indexing cost of data)
Verify command improvements - verify proper configurations for cgroup (memory/memory.use_hierarchy is 1)
Bug fix: fix bug in prometheus metrics parser, empty fields can be filled with previous fields
Bug fix: occasionally addon can report warnings about trying to delete expired keys
..

Version 5.14.280

Jan. 24, 2020

Logs dashboard: filters depend on selection
Overview dashboard: namespace counter for list of projects

Collectord updates:

Support templates in the index, source and sourcetype
Allow to exclude indexed fields when forwarding to Splunk
Support annotation for stats interval for containers
Support containerd runtime
Bug fix: verify command can show incorrect error about verifying journald input
Bug fix: index on namespace should set index for application logs
Bug fix: warning about not being able to retrieve node information

Version 5.12.272

Nov. 12, 2019

Improvement for the alerts verifying that control components or nodes are down.

Version 5.12.271

Nov. 7, 2019

5.12.271 - 2019-11-07

Requires collectorforkubernetes version 5.12.271 or above (see https://www.outcoldsolutions.com for latest configuration)

Improvements for the macros for backward compatibility

Collectord updates:
- Bug fix: when event pattern is used for joining multi-line events, the error can not be showed if raised by the input in pipeline.
- Bug fix: reduce warnings failed to get the new event in pipeline - submitted
- Stability improvements

Version 5.12.270

Oct. 18, 2019

Compact metrics (pre-calculated on Collectord side)
Switched stats for host and cgroup in different macros
Use base macro for alerts
Improved command extraction for exec in Audit Logs
Add cluster name in the alert results

Collectord updates:

Watch namespaces and workloads for changes
Global configurations with Custom Resources and selectors
Describe command to see applied annotations for pods
Bug fix: panic when pipe join configuration is removed
Bug fix: panic when proc stats is enabled and cgroup stats is disabled
Bug fix: support ProxyBasicAuthorization for license server checks
Bug fix: Fix for collecting first sample (can show high CPU usage for first sample)
Bug fix: if list of URLs is used for Splunk output, the empty URL is still required
Beta: dynamic index, source and sourcetype names based on the metafields
Beta: cluster diagnostics with one rule: node entropy

Version 5.11.260

Sept. 9, 2019

5.11.260 - 2019-09-09

Requires collectorforkubernetes version 5.11.260 or above (see https://www.outcoldsolutions.com for latest configuration)

GPU Monitoring (NVIDIA)

Collectord updates:

Support for PVC volumes for application logs
Bug fix: small memory leak in addon
Bug fix: duplicate events then pipeline is getting throttled
Bug fix: don't use throttling for devnull output
Bug fix: better recovery for ack db corruption
Bug fix: crash on journald input initialization when ack db is corrupted
Bug fix: annotations joinmultiline requires joinpartial
Bug fix: configurations for stdout only with annotations can crash collectord
Set events = 50 by default for Splunk output batches

Version 5.10.250

June 15, 2019

Security dashboard: Access: access to host via ssh, sudo, exec commands, failed access
Security dashboard: Audit (users and namespaces)
Security dashboard: Network (traffic)
Security dashboard: Network (connections)
Security dashboard: Objects (pods) - review pods with host network, age of pods, image pull policy, attached host paths, security context and restart policies
Review dashboard: Clusters (allocations and usage)
Cluster field filters
Base macro for overriding macros for other macros

Collectord updates:

Support for volatile and persistent journald storage with default configuration
Updated YAML configuration to include most common resources
Better support for overriding sourcetype, that does not require to update the Splunk macros
Bug fix: rarely when collectord fails to post to HEC it can panic
Bug fix: better support for Kubernetes 1.14 and CRI-O storage
Bug fix: space characters in index annotations can break the pipeline

Version 5.9.240

May 13, 2019

5.9.240 - 2019-05-14

Requires collectorforkubernetes version 5.9.240 or above (see https://www.outcoldsolutions.com for latest configuration)

Visual improvements on the graphs for the number of logs and events
New alerts for the CPU and Memory reservation

Collectord updates:

Support for multiple Splunk destinations (outputs)
Support subdomains for annotations (to deploy multiple collectord instances)
Support for streaming objects from Kubernetes API to Splunk
Bug fix: journald input keeps fd open to the rotated files
Bug fix: fix in the annotation parser for the interval annotations
Bug fix: fix splunk url selection configuration for multiple splunk URLs

Version 5.8.231

April 25, 2019

5.8.231 - 2019-04-25

Bug fix: Collectord usage report shows trial licenses for all instances

Version 5.8.230

April 20, 2019

5.8.230 - 2019-04-22

Requires collectorforkubernetes version 5.8.230 or above (see https://www.outcoldsolutions.com for latest configuration)

Use multiselect filters for most dashboards and filters with possibility to input custom filters.
Reduce dedup usage to improve performance on dashboards.
Add critical pod annotations for Kubernetes ...1.13, and priority class for Kubernetes 1.14...
Fix: statefulset dashboard does not show data with filters.
Add graph of number of pods per namespace on Overview dashboard.

Collectord updates:

Bug fix: clogging collectord output with errors when incorrect index is used.
Bug fix: short lived containers can results in duplicating logs.
Bug fix: clogging collectord output with warnings when kernel reports incorrect VmRss size.
Bug fix: annotations cannot override timestamp location for fields extraction.
Bug fix: verify command reports Journald input in incorrect place.

Version 5.7.220

March 16, 2019

Requires collectorforkubernetes version 5.7.220 or above (see https://www.outcoldsolutions.com for latest configuration)

Review savedsearches/alerts to support indexing delay (start searches from 2 minutes behind) and run them in more random time.
Workload dashboard - change CPU (of host) in table to real CPU
Fixed single value memory panel on host dashboard (missed span)
Use SEGMENTATION=none for stats events to use less disk space (needs to me moved to indexers)

Collectord updates:

Support hostname formatting with environment variables in configuration
New rotated file logic uses less file descriptors and frees rotated files quicker
Allow to specify a default sampling value for container logs
Reimplemented shutdown sequence to stop collectord faster
Allow to override sampling percent with annotations
New Input: journald

Version 5.6.212

Feb. 19, 2019

5.6.212 - 2019-02-19

Requires collectorforkubernetes version 5.6.212 or above (see https://www.outcoldsolutions.com for latest configuration)

New: Alert: high CPU usage on the host.
Fixed: Splunk usage dashboard - charts do not show the data, when the used indexed aren't searchable by default.
New: Support Dark theme.
New: Free text search in Logs dashboard.
New: Add auto-refresh options to the dashboard.
Fixed: Revisited CPU limits and requests for Pods and Containers.
New: add CPU Max, Memory Max and Project/Namespace labels to the Review-Namespaces dashboard.
Fixed: Show deleted events

Version 5.5.202

Jan. 23, 2019

5.5.202 - 2019-01-24

Requires collectorforkubernetes version 5.5.202 or above (see https://www.outcoldsolutions.com for latest configuration)

New: Dashboard Review -> namespaces. Review allocations and requests for namespaces and pods.
Fixed: kubernetes_stats_cpu_request_percent - is divided by the number of CPU.

Collectord updates:
- Fixed: Interval 0 in prometheus input can crash the collectord.
- Fixed: When both glob and match are set for the application logs, the glob pattern can block the match pattern from
finding the files in the volume.

Version 5.4.201

Dec. 20, 2018

5.4.201 - 2018-12-19

Requires collectorforkubernetes version 5.4.201 or above (see https://www.outcoldsolutions.com for latest configuration)

Fixed: Alerts for licenses issued with AWS Subscriptions

Collectord updates:
- Fixed: Better handling rotated files (less open fd)
- Fixed: Events input can hang in the err loop.

Version 5.4.200

Dec. 16, 2018

5.4 - 2018-12-17

Requires collectorforopenshift version 5.4 or above (see https://www.outcoldsolutions.com for latest configuration)

Improved: etcd metrics representation for bucket values.
Fixed: API latency alert - exclude imagestreamimports.
Compatibility update for collectord 5.4.

Collectord updates:
- New: Attach EC2 metadata fields
- New: Basic Auth for Proxy (License Server and Splunk)
- Fixed: Collectord verifies reports CRI-O as unsupported runtime.
- Fixed: Rare crash on Prometheus metrics definition.
- Fixed: Better handling of acknowledgment database corruption.
- Fixed: When handling incorrect indexes, collectord can send index with an empty string, that Splunk recognize as an incorrect index

Version 5.3.190

Nov. 17, 2018

5.3 - 2018-11-19

Requires collectorforkubernetes version 5.3 or above (see https://www.outcoldsolutions.com for latest configuration)

Fixed: Improved Workload dashboard. Allows to filter by namespace, see all Pods in a specific namespace, filter by workload label.
New: Alert for showing when Collectord reports errors in Processing pipelines (as an example if it failed to extract fields).
New: Alert for showing when Collectord reports warnings.
Fixed: Add node labels filter to Storage Dashboard and Control Plane Dashboards.
New: Alert if lag in the indexing of the data.
New: Splunk Usage (License usage, number of events) report under Setup.
Fixed: adjusted high amount of errors to Kubernetes API dashboard to make it less verbose.

https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.2.181

Oct. 29, 2018

5.2.180 - 2018-10-29
- Fixed: misprint in the search for showing alerts

5.2.180 - 2018-10-28
- Fixed: lookup with alerts causing very often replication activities on SHC

5.2.179 - 2018-10-17
- Fixed: changed search time for few alerts that cause false positives with indexing lag on large installations

5.2 - 2018-10-15

Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)

New: Review/Storage dashboard based on storage metrics and PVC metrics.
New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
Fixed: Performance improvements
...

For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.2.180

Oct. 28, 2018

5.2.180 - 2018-10-28
- Fixed: lookup with alerts causing very often replication activities on SHC

5.2.179 - 2018-10-17
- Fixed: changed search time for few alerts that cause false positives with indexing lag on large installations

5.2 - 2018-10-15

Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)

New: Review/Storage dashboard based on storage metrics and PVC metrics.
New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
Fixed: Performance improvements
...

For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.2.179

Oct. 17, 2018

5.2.179 - 2018-10-17
- Fixed: changed search time for few alerts that cause false positives with indexing lag on large installations

5.2 - 2018-10-15

Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)

New: Review/Storage dashboard based on storage metrics and PVC metrics.
New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
Fixed: Performance improvements
...

For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.2.176

Oct. 15, 2018

5.2 - 2018-10-15

Requires collectorforkubernetes version 5.2 or above (see https://www.outcoldsolutions.com for latest configuration)

New: Review/Storage dashboard based on storage metrics and PVC metrics.
New: predefined alerts to help you monitor the health of the clusters and performance of the applications.
Fixed: Performance improvements
...

For details https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.1.175

Sept. 17, 2018

New: Network metrics (MB, Packets, Drops, and Errors) for host and containers.
New: Network socket tables (list of the port that containers and hosts are listening on, connections to external resources).
New: Network review dashboard to see the list of connection to public services and in private network.
Improvement: Replace python-based lookup with a macro written with eval.
Improvement: Visual improvement for showing when the object was Last Seen (highlighting and showing minutes ago).
... and more

https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 5.0.174

Sept. 4, 2018

Highlights:
- Application logs
- Annotations for fields extraction, hiding sensitive information, time extraction, redirecting to /dev/null, stripping terminal colors and more

For more details:
https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/

Version 4.0.24

May 16, 2018

New dashboard: Cluster/Audit
New dashboard: Cluster/Kubernetes API Server
New dashboard: Cluster/Kubelet
New dashboard: Cluster/etcd
New dashboard: Cluster/Scheduler
New dashboard: Cluster/Controller Manager.
Include image name, when list containers.
Added syslog component to the list of host logs.
Fixed: Include Daemon Set on Overview dashboard, list of namespaces.

Version 3.0.23

Feb. 17, 2018

3.0.23 - bug fixes release

3.0.22

New overview, security and capacity dashboards. Workload aggregation dashboard.
A lot of of bug fixes and performance improvements.

Relese Notes: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/release-history/#30-2018-02-07
Upgrade instructions: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/upgrade-2-to-3/
Requires collectorforkubernetes version 3.0 or above (see https://www.outcoldsolutions.com for latest configuration)

Version 3.0.22

Feb. 8, 2018

New overview, security and capacity dashboards. Workload aggregation dashboard.
A lot of of bug fixes and performance improvements.

Version 2.1.21

Jan. 2, 2018

2.1.21 - 2018-01-02

Requires collectorforkubernetes version 2.1.59.171209 or above
- Updated author and description

2.1.20 - 2017-12-09

Fixed link to setup / installation instructions.

2.1.18 - 2017-12-09

Implemented collectors dashboard to track number of collectors, their versions
and used licenses.
Fallback to the process IO statistics when blkio is not available.
Fix IO statistic graphs, showed average, when sum should be used.
Fields extraction support for nginx ingress 0.9 and above.
[collector] Improved resistance for storage failures.
[collector] License checks reporting.
[collector] Better support for openshift environment (default configuration).

Version 2.1.20

Dec. 17, 2017

2.1.20 - 2017-12-09

Requires collectorforkubernetes version 2.1.59.171209 or above

Fixed link to setup / installation instructions.

2.1.18 - 2017-12-09

Requires collectorforkubernetes version 2.1.59.171209 or above

Implemented collectors dashboard to track number of collectors, their versions
and used licenses.
Fallback to the process IO statistics when blkio is not available.
Fix IO statistic graphs, showed average, when sum should be used.
Fields extraction support for nginx ingress 0.9 and above.
[collector] Improved resistance for storage failures.
[collector] License checks reporting.
[collector] Better support for openshift environment (default configuration).

Version 2.0.17

Oct. 24, 2017

2.0 - 2017-10-22

Requires collectorforkubernetes version 2.0.37.171023 or above

Better labels support in Dashboards.
Collector has a breaking feature, replacing format for labels from
kubernetes_node_labels_LABEL1=VALUE1 to kubernetes_node_labels=[LABEL1=VALUE1,LABEL2=VALUE2].
Process level metrics.
Uptime for hosts and processes.
Fields extraction for kubernetes controller manager and scheduler.
Fields extraction and support in dashboards for main kubernetes components (setup
host logs collection with collector).
New top-like dashboards allow to monitor Hosts/Pods/Containers/Processes in realtime.
Rewritten Kubernetes Objects Dashboards with support of Events and Labels.
Improved dashboards navigation.
Support for host logs.
Other bugs and improvements based on user feedback.

Version 1.0.3

Oct. 13, 2017

Updated links to official documentation for installation instructions.

Version 1.0.2

Oct. 5, 2017

Fix labels on Kubernetes Dashboards (Most of the filters has incorrect label Daemon Sets)

Version 1.0.1

Oct. 4, 2017

Monitoring Kubernetes

628

Installs

7,883

Downloads

Share Subscribe

Version

Built by

Outcold Solutions LLC

Support

Developer Supported

Contact Developer Questions on Splunk Answers Flag as inappropriate

Compatibility

This version of the app (5.17.370) is not available for Splunk Cloud. However, version 5.16.363 of this app is available for Splunk Cloud.

Products: Splunk Enterprise

Products: Splunk Enterprise, Splunk Cloud

Splunk Versions: 7.2, 7.3, 8.0, 8.1, 8.2