Monitoring Kubernetes - Metrics and Log Forwarding

Monitoring Kubernetes brings your Kubernetes clusters into Splunk, for the platform team that runs them, the application teams that deploy to them, and security. When something breaks in a cluster, the evidence is spread out and short-lived: container logs sit in files on each node until rotation overwrites them, and the API server holds Kubernetes events for about an hour. Dashboards that only show state leave you to work out what is wrong. And when one cluster serves many teams, one fixed set of alerts fires the same way for everyone, while each team's data has to stay in its own indexes. Monitoring Kubernetes shows what Collectord forwards from each cluster. Collectord runs as a DaemonSet on the nodes, control-plane nodes included, and a single addon Deployment, and enriches what it sends with Kubernetes metadata. Container logs, application log files, host logs, and Kubernetes events and objects go to an events index, and so does the API server audit log once you turn it on. The node, pod, container and process stats, and the Prometheus metrics of the control plane and your own pods, go to the same index or to a Splunk metrics index, your choice. Annotations on a namespace or pod send its data to its own index and HEC token, join multi-line stack traces and mask values, and can send repeated lines as one event that counts them, for fewer events and less noise. The app opens on an Overview with one verdict for all your clusters, the numbers behind it and a row per cluster, and its tiles drill down to the page that shows the cause. From there it covers nodes, pods, containers and processes; the Kubernetes events, where a click on a crash loop or an OOM kill opens that pod's logs; capacity, requests and limits; the control plane from the API server to etcd and CoreDNS; pod security posture and network connections; and an Audit menu that reads the API audit log, from detections to the timeline of one object. Every stats dashboard comes in two sets that look the same, one for each kind of index. Alerts come from an Alert Catalog of 118 templates, written for Splunk admins, the platform team, security and application teams. Each team picks the alerts it needs, scopes them to its clusters and namespaces, and creates them as its own saved searches, such as Kubernetes: Pod crash looping [cluster prod; namespace payments]. Install the app on your search heads, create a HEC token, and deploy Collectord with the Helm chart or one manifest. It works with containerd, CRI-O and Docker on Kubernetes 1.24 and later, managed clusters such as EKS, GKE and AKS included. The app also maps your nodes and the API audit to the Common Information Model for Splunk Enterprise Security. The app requires Collectord and a license key from Outcold Solutions. Evaluation and developer keys are free. Documentation: https://www.outcoldsolutions.com/docs/monitoring-kubernetes/