The task reported one Job it would delete. Then I looked at the cluster, and the Job was gone.

It was a Job I meant to delete anyway, so the damage was zero. The question was not. If --check can delete things, every rehearsal I had run against production was a gamble.

What check mode actually promises

Ansible itself does nothing in check mode. It sets a flag and each module decides what that means. For most modules that is “compute the change, skip the write”. The kubernetes.core.k8s module does something better: it sends the real request to the Kubernetes API with server-side dry run. The API server runs admission, validation and defaulting, and returns what would happen, without persisting anything.

For a delete, the dry run is a query parameter:

DELETE /apis/batch/v1/namespaces/kube-system/jobs/helm-delete-vault?dryRun=All

My task had one more line, because a Job should take its pods with it:

- name: Delete orphaned helm-controller Jobs
  kubernetes.core.k8s:
    state: absent
    kind: Job
    namespace: kube-system
    name: "{{ item }}"
    delete_options:
      propagationPolicy: Background

delete_options travels as a DeleteOptions object in the request body. So the request that went out was the URL above plus this body:

{"apiVersion": "v1", "kind": "DeleteOptions", "propagationPolicy": "Background"}

The API server reads one or the other

The DELETE handler in the Kubernetes API server (handlers/delete.go) gets its DeleteOptions from exactly one place. In outline:

if len(body) > 0 {
    // decode DeleteOptions from the body
} else {
    // decode DeleteOptions from the URL query parameters
}

dryRun is a field of DeleteOptions. With a body present, the query string is never read, dryRun=All is ignored, and the body says nothing about a dry run. The server did exactly what it was asked: a real delete with background propagation.

Neither side is wrong on its own. The API accepts the options in either place; the client put half of them in each.

A bug that was already fixed

The module’s side is a known issue: kubernetes.core#892, “kubernetes.core.k8s deletes objects in check mode if delete_options are set”, reported in March 2025. Release 5.3.0 fixed it by leaving the body out in check mode. Every 5.x release before it, and everything older, sends the body.

My infrastructure repository pins kubernetes.core 6.5.0, and CI installs exactly that. In CI, the same task in check mode would have been a real dry run.

But I had not rehearsed in CI. I had run the task locally, from a scratch directory, in a Python environment with the ansible 9.10 package. That package bundles its own copy of every collection. CI installs the pinned versions into a collection path that takes precedence; my scratch directory had no ansible.cfg pointing there, so Ansible quietly used the bundled copy:

$ ansible-galaxy collection list kubernetes.core
kubernetes.core 2.4.2

The rehearsal was faithful to a stack I do not deploy with.

The fix

The task now puts the dry run in the body as well, which is correct for every version of the collection. Old versions send the body and the server honours dryRun from it; new versions drop the body in check mode and the server reads dryRun from the query.

    delete_options: >-
      {{ {'propagationPolicy': 'Background'}
         | combine({'dryRun': ['All']} if ansible_check_mode else {}) }}

The more important fixes are about the environment, not the task:

• Rehearse with the versions you deploy with. The project pins its collections; a local run that cannot find them should fail, not fall back to whatever the ansible package bundles. That silent fallback is the real bug here.
• Print the versions. ansible --version and ansible-galaxy collection list at the start of a rehearsal cost nothing and would have shown 2.4.2 next to the 6.5.0 in requirements.yml.
• Do not rehearse destructive tasks against production. A delete path deserves a disposable cluster (kind, k3d) with a fixture object, where a “dry run” that is not dry costs nothing. Check mode against production is for tasks that create and update.

Check your playbooks

Any kubernetes.core.k8s task with state: absent and delete_options is affected on collection versions before 5.3.0:

# Which kubernetes.core will your ansible actually use?
ansible-galaxy collection list kubernetes.core

# Delete tasks with options: these are the ones to look at.
grep -rn -B8 'delete_options' --include='*.yml' . | grep -E 'state: absent|delete_options'

If the version is older than 5.3.0, upgrade, or put dryRun: ['All'] into delete_options when ansible_check_mode is true.

Lessons

• Check mode is a promise each module makes for itself, not one Ansible makes. Read how the module implements it before you trust it with deletes.
• An API that accepts options in two places will eventually receive them in both. Know which one wins.
• A rehearsal is only as good as its resemblance to the real run. Same versions, same collections, same flags.
• The cheapest dry run is one where being wrong deletes nothing that matters.

It took four incidents before I saw that they were one incident. This post is the mechanism, the trap I stepped into while cleaning up, and the changes that make it boring again.

How k3s installs Helm charts

k3s ships a small controller, the helm-controller, that turns a HelmChart custom resource into a Helm release. You drop a manifest into /var/lib/rancher/k3s/server/manifests/, k3s applies it as an Addon, and the helm-controller creates a Job that runs helm upgrade --install inside a klipper-helm container:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: loki
  namespace: kube-system
spec:
  repo: https://grafana.github.io/helm-charts
  chart: loki
  version: 7.3.0
  targetNamespace: monitoring
  valuesContent: |-
    deploymentMode: SingleBinary

It is convenient for bootstrap: networking and storage are installed before anything else in the cluster can run. Three properties matter later.

• A HelmChart is state in etcd, not a file. Deleting its YAML file from the manifests directory leaves the Addon and the HelmChart in place, and rebuilding a server node keeps them, because etcd survives.
• The install Job runs once, but it is not the end of the story. The helm-controller keeps comparing the Job it would create today with the one that exists. The Job’s definition includes the HelmChart’s values and the klipper-helm image, which is tied to the k3s version. When anything in it differs, the controller deletes the finished Job and creates a new one, and the new Job runs helm upgrade again.
• spec.failurePolicy defaults to reinstall.

The mechanism

The trigger. My servers follow the k3s v1.35 channel through the system-upgrade-controller, so patch releases arrive unattended. Each one ships a new klipper-helm image. The helm-controller now wants a different install Job for every HelmChart, replaces each finished Job with a new one, and every chart in the cluster is upgraded again, with nobody touching it. The three events in my logs are 26 June, 5 August and 1 October, each followed within minutes by a burst of helm-install-* pods.

The fuel. A re-run is harmless when the HelmChart matches what you deploy. Mine did not always:

• HelmCharts left behind after I moved kube-prometheus-stack and Loki to Ansible-managed Helm releases. They still pointed at chart versions from 2024.
• Rook HelmCharts without a version, so the Job installed whatever was newest.
• Values that only worked as a three-way patch, and CRDs whose fields a second manager also wrote.

The amplifier. With failurePolicy: reinstall, a failed upgrade leaves the release in failed. The Job’s own retry, seconds later, finds a failed release and uninstalls it before installing again. The release did not need to be broken beforehand; any upgrade failure is enough. From the Calico CRD Job:

21:15:53  Error: UPGRADE FAILED: conflict occurred while applying object
          /bgpconfigurations.crd.projectcalico.org ...
          Apply failed with 1 conflict: conflict with "OpenAPI-Generator" ... .spec.versions
21:16:06  Uninstalling failed helm chart

Thirteen seconds from a field-ownership conflict to deleting every Calico CRD.

Four incidents, one cause

The August one hurts most in hindsight. The logging stack was swapped for an older one for eight days and nothing alerted, because the thing that should have told me was the thing being replaced.

The cleanup that uninstalled my monitoring

On 1 October I knew the stale HelmCharts were the problem, so I wrote a playbook to remove them. A helm-delete Job runs on deletion because of a finalizer, so the plan was: remove the finalizers, then delete the objects.

The helm-controller put the finalizer back before the delete arrived, ran its helm-delete Jobs, and uninstalled the very releases I was trying to protect. Prometheus, Alertmanager and Loki were gone.

The controller has an explicit opt-out that I had not read about. In helm-controller v0.17, shouldManage skips any HelmChart with the helmcharts.helm.cattle.io/unmanaged annotation. The safe order to retire a HelmChart whose release you want to keep:

# 1. Remove the manifest file, so nothing re-applies the object.
# 2. Tell the controller to stop acting on it.
kubectl -n kube-system annotate helmchart loki helmcharts.helm.cattle.io/unmanaged=true
# 3. Now the finalizer can go, and the object with it.
kubectl -n kube-system patch helmchart loki --type=merge -p '{"metadata":{"finalizers":[]}}'
kubectl -n kube-system delete helmchart loki

The lesson I keep relearning: when you change a controller’s objects, read how the controller reacts in its source, not in your head.

Values that only work as a patch

Getting Prometheus back took another surprise. Its values had worked for every deploy for years, but every one of those deploys had been an upgrade: Helm sends a three-way patch, and in a patch null means “delete this field”. A fresh install creates the objects instead, and creation rejects the same values.

The klipper-helm Job installs with --server-side=true, and fresh installs with Helm 4 use server-side apply too, so every reinstall applies this strictness. A chart that cannot be installed from scratch is one bad upgrade away from an outage.

What changed

• failurePolicy: abort on every HelmChart. A failed upgrade now stays failed and waits for a human. A test in the repository fails the build if a HelmChart template lacks it.
• Stale HelmCharts and Addons removed, with the unmanaged annotation first, thirteen Addons in total, including ones from install methods I had abandoned years ago.
• Every chart version pinned. Renovate proposes upgrades; nothing floats.
• Values valid for creation. A test rejects null and numeric label values in the inventory. Before an upgrade I render each chart with its live version and values and run kubectl apply --server-side --dry-run=server on the result under temporary names.
• A read-only drift check that fails when a HelmChart has no template in the repository, lacks abort, or a Helm release’s latest revision is not deployed. Its first run found nothing, which is the point: it runs before the next patch upgrade does.
• One owner per object. The Calico operator no longer writes its own CRDs; the CRD chart does.

Next is moving the remaining HelmCharts that bootstrap does not need into Ansible-managed releases, so the controller that upgrades k3s stops redeploying applications.

Check your cluster

If you run k3s or RKE2 with HelmCharts, two commands show most of the risk:

# Every HelmChart with its pinned version and failure policy.
# An empty POLICY means reinstall.
kubectl get helmcharts -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,CHART:.spec.chart,\
VERSION:.spec.version,POLICY:.spec.failurePolicy

# Releases whose latest revision is not deployed.
helm list -A --failed --pending

An empty VERSION means the next re-run installs whatever is newest. An empty POLICY means the next failed upgrade is an uninstall. A HelmChart that no file in your repository describes is the stale kind.

Lessons

• Unattended upgrades re-run everything. Anything stale surfaces there, at a time you did not choose.
• reinstall is not a retry. For releases that own data or APIs, only abort is safe.
• A manifest directory is not the source of truth; etcd is. Retiring a file needs a cleanup step.
• “It deploys fine” may only mean “it upgrades fine”. Test fresh installs.
• An outage that nobody notices for eight days is a monitoring bug, not luck.