Kubernetes CronJobs

A CronJob can fail without anyone noticing: the job errors, the pod never schedules, the image cannot be pulled, or the CronJob is suspended and forgotten. A heartbeat monitor catches all of them, because a missing ping is an alert.

You create monitors in the dashboard under Monitors → New monitor. The manifests below only send pings, so they need no API key.

1. Store the ping URL in a Secret

Create one heartbeat monitor per CronJob and store each ping URL in a Secret:

kubectl create secret generic asniher-heartbeats \
  --from-literal=nightly-backup=https://app.asniher.com/api/heartbeat/<token>

2. Wrap the job command

Add asniher-run to the job's image. It pings /start, runs the command, then reports its exit status with the last 10 KB of output, and exits with the command's exit status so Kubernetes sees the same result as before.

FROM alpine:3.20
RUN apk add --no-cache curl \
 && curl -fsSL https://app.asniher.com/asniher-run.sh -o /usr/local/bin/asniher-run \
 && chmod +x /usr/local/bin/asniher-run
COPY backup.sh /app/backup.sh

Then put it in front of the command in the CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "30 2 * * *"
  timeZone: "Europe/Berlin"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 0
      activeDeadlineSeconds: 3600
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: ghcr.io/acme/backup:1.4.0
              command: ["asniher-run", "--", "/app/backup.sh"]
              env:
                - name: ASNIHER_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: asniher-heartbeats
                      key: nightly-backup

On the monitor, use the same cron expression 30 2 * * * and time zone Europe/Berlin, and a grace period a little longer than the job's longest normal run. timeZone needs Kubernetes 1.27 or later; without it, CronJobs run in the controller's time zone, usually UTC.

If the image already has sh and curl and you would rather not add a file, replace the container's command with the same logic inline:

command:
  - sh
  - -c
  - |
    curl -fsS -m 10 --retry 3 -o /dev/null -X POST "$ASNIHER_HEARTBEAT_URL/start" || true
    /app/backup.sh
    status=$?
    curl -fsS -m 10 --retry 3 -o /dev/null -X POST "$ASNIHER_HEARTBEAT_URL/$status" || true
    exit $status

For distroless images without a shell, ping from the application code instead; see Node.js, Python or ASP.NET Core.

What gets caught

Problem	What Asniher sees
The job exits non-zero	A failure ping with the exit status and output. Alert right away.
The job hangs until activeDeadlineSeconds kills it	A start ping and nothing after it. Alert when the grace period runs out.
The pod cannot be scheduled or the image cannot be pulled	No ping. Alert when the run is overdue.
The CronJob is suspended or deleted	No ping. Alert when the run is overdue.

With backoffLimit above 0, a failed attempt sends a failure ping and opens an incident, and a successful retry closes it again. Keep backoffLimit: 0 if you only want to hear about runs that fail for good, and let the next scheduled run retry.

HTTP monitor: services behind an Ingress

Liveness and readiness probes run inside the cluster. An HTTP monitor checks the path your users take: DNS, load balancer, Ingress controller, TLS certificate and the app.

Setting	Value
URL	https://api.example.com/healthz
Method	GET
Expected status	200
Interval	Your plan's minimum

Use an endpoint that checks the app's critical dependencies. Keep the liveness probe on a cheaper endpoint, so a database outage does not make Kubernetes restart every pod.

Check that it works

Start one run by hand:

kubectl create job --from=cronjob/nightly-backup nightly-backup-manual

The monitor's ping log shows a start ping, then the exit status and output of the run.