Monitoring Docker and Podman - Metrics and Log Forwarding

Monitoring Docker brings the hosts that run your containers into Splunk: Docker hosts and Compose projects, Docker Swarm clusters, Podman hosts, and Amazon ECS container instances. Containers fail in ways their logs alone don't explain. A container restarts in a loop, is OOM killed, or fails its health check. A Swarm service runs fewer tasks than it should, and the reason is in the Swarm's services, tasks and nodes, which only a manager can read. The Docker data root fills up with images, writable layers and json-file logs. Some containers run their processes as root or hold Linux capabilities. The answers are spread over every host, in the logs, the cgroups, /proc and the Docker API. Monitoring Docker shows what Collectord forwards from each host, where it runs as one container. Collectord sends container and host logs, the Docker daemon's events, and the containers, images, volumes and Swarm objects it reads from the API to an events index. Host, container, process, network, filesystem and disk stats go to the same events index, to a Splunk metrics index, or to both while you compare. The containers of rootless Podman users are monitored too. The app opens on an Overview that says whether anything needs attention and where: hosts whose stats went silent, Collectord problems, and containers OOM killed, in a restart loop, unhealthy or exited with a non-zero code. From there it drills down to hosts, containers and processes, Compose projects, Swarm services and nodes, ECS services, privileged containers, the network, filesystems, disk I/O and CPU throttling. Every stats dashboard has a twin that reads the metrics index, and the Configuration page chooses which set the menu shows. Name each cluster, such as prod or staging, and the dashboards and alerts filter by it. Alerts come from an Alert Catalog of 61 templates, written for Splunk admins, the platform team and application teams. Each team picks the alerts it needs and creates them as its own saved searches, scoped to its own clusters. Container labels configure one container at a time: its index and Splunk output, field extraction, multi-line events, masking, log files from its volumes, and Prometheus endpoints. Three features, off by default, send fewer events: repeated lines as one counted event, storms of alike lines folded into a summary a minute, and a report of each stream's heaviest log patterns. Install the app on your search heads, create an HTTP Event Collector token, and start Collectord on each host with docker run, Docker Compose, a Swarm stack, Podman, or an Amazon ECS daemon service. The app requires Collectord and a license key from Outcold Solutions. Evaluation and developer keys are free. Documentation: https://www.outcoldsolutions.com/docs/monitoring-docker/