Monitoring OpenShift - Metrics and Log Forwarding

Monitoring OpenShift brings your OpenShift clusters into Splunk: for the platform team that runs them, the application teams that deploy to them, and the security team that audits them. When a release goes wrong on OpenShift, the evidence is spread out: container logs on every node, the journal, events the API server keeps for about three hours, the audit logs of four API and OAuth servers, and the control plane's Prometheus endpoints. On a cluster many teams share, each team also needs its own data and its own alerts, and the platform team needs rules that hold for every project. Monitoring OpenShift shows what Collectord forwards from each cluster. Collectord runs as a DaemonSet on every node, masters included, and one addon pod. It sends container and application logs, the journal, the audit logs, and OpenShift events and objects to an events index, and node, pod, container, process, network and disk stats and the control plane's Prometheus metrics to an events index or a Splunk metrics index. Annotations change what it does for one project, workload or pod: an index or a HEC token per team, field extraction, multi-line events, masking and hashing, application log files and Prometheus scrapes. A Configuration resource sets the platform team's rules for the whole cluster. New features, off until you turn them on, group repeated lines and fold storms of alike lines, for fewer events and less noise. The dashboards open on an Overview with one verdict for the fleet, Healthy, Warning, Critical or Unknown, the numbers behind it and a row per cluster, and its tiles and rows drill down to the page that shows the cause. From there they cover nodes down to a single process; crash loops, OOM kills and failing probes, each a click from the pod's logs; capacity, quotas and what the Pod specs declare; the API server, etcd and the cluster DNS; pod security posture; and an Audit menu from detections to the timeline of one object, OAuth logins included. The stats dashboards come in two sets that look the same, one for each kind of index. Alerts come from an Alert Catalog of 117 templates, written for Splunk admins, the platform team, security and application teams. Each team picks the alerts it needs and creates them as its own saved searches, scoped to its clusters and projects. Install the app on your search heads, then Collectord with one Helm chart or the manifest rendered from it, on OpenShift 4.11 or later. It sends to Splunk Enterprise or Splunk Cloud over the HTTP Event Collector. A Red Hat certified image is available from registry.connect.redhat.com, and FIPS 140 images are published beside the standard ones. The app requires Collectord and a license key from Outcold Solutions. Evaluation and developer keys are free. Documentation: https://www.outcoldsolutions.com/docs/monitoring-openshift/