Roman Danko / Writing

When --check deletes for real

An Ansible check-mode rehearsal deleted a Kubernetes Job. The API server ignored the dry-run flag because the request also had a body, and the rehearsal ran a different collection version than production.

I was adding a cleanup task to a playbook: delete leftover Kubernetes Jobs that belong to nothing. Before letting it loose on the cluster I did what Ansible teaches you to do and ran it with --check. Check mode is the rehearsal; it reports what would change and changes nothing.

The task reported one Job it would delete. Then I looked at the cluster, and the Job was gone.

It was a Job I meant to delete anyway, so the damage was zero. The question was not. If --check can delete things, every rehearsal I had run against production was a gamble.

What check mode actually promises#

Ansible itself does nothing in check mode. It sets a flag and each module decides what that means. For most modules that is “compute the change, skip the write”. The kubernetes.core.k8s module does something better: it sends the real request to the Kubernetes API with server-side dry run. The API server runs admission, validation and defaulting, and returns what would happen, without persisting anything.

For a delete, the dry run is a query parameter:

DELETE /apis/batch/v1/namespaces/kube-system/jobs/helm-delete-vault?dryRun=All

My task had one more line, because a Job should take its pods with it:

- name: Delete orphaned helm-controller Jobs
  kubernetes.core.k8s:
    state: absent
    kind: Job
    namespace: kube-system
    name: "{{ item }}"
    delete_options:
      propagationPolicy: Background

delete_options travels as a DeleteOptions object in the request body. So the request that went out was the URL above plus this body:

{"apiVersion": "v1", "kind": "DeleteOptions", "propagationPolicy": "Background"}

The API server reads one or the other#

The DELETE handler in the Kubernetes API server (handlers/delete.go) gets its DeleteOptions from exactly one place. In outline:

if len(body) > 0 {
    // decode DeleteOptions from the body
} else {
    // decode DeleteOptions from the URL query parameters
}

dryRun is a field of DeleteOptions. With a body present, the query string is never read, dryRun=All is ignored, and the body says nothing about a dry run. The server did exactly what it was asked: a real delete with background propagation.

Neither side is wrong on its own. The API accepts the options in either place; the client put half of them in each.

A bug that was already fixed#

The module’s side is a known issue: kubernetes.core#892, “kubernetes.core.k8s deletes objects in check mode if delete_options are set”, reported in March 2025. Release 5.3.0 fixed it by leaving the body out in check mode. Every 5.x release before it, and everything older, sends the body.

My infrastructure repository pins kubernetes.core 6.5.0, and CI installs exactly that. In CI, the same task in check mode would have been a real dry run.

But I had not rehearsed in CI. I had run the task locally, from a scratch directory, in a Python environment with the ansible 9.10 package. That package bundles its own copy of every collection. CI installs the pinned versions into a collection path that takes precedence; my scratch directory had no ansible.cfg pointing there, so Ansible quietly used the bundled copy:

$ ansible-galaxy collection list kubernetes.core
kubernetes.core 2.4.2

The rehearsal was faithful to a stack I do not deploy with.

The fix#

The task now puts the dry run in the body as well, which is correct for every version of the collection. Old versions send the body and the server honours dryRun from it; new versions drop the body in check mode and the server reads dryRun from the query.

    delete_options: >-
      {{ {'propagationPolicy': 'Background'}
         | combine({'dryRun': ['All']} if ansible_check_mode else {}) }}

The more important fixes are about the environment, not the task:

  • Rehearse with the versions you deploy with. The project pins its collections; a local run that cannot find them should fail, not fall back to whatever the ansible package bundles. That silent fallback is the real bug here.
  • Print the versions. ansible --version and ansible-galaxy collection list at the start of a rehearsal cost nothing and would have shown 2.4.2 next to the 6.5.0 in requirements.yml.
  • Do not rehearse destructive tasks against production. A delete path deserves a disposable cluster (kind, k3d) with a fixture object, where a “dry run” that is not dry costs nothing. Check mode against production is for tasks that create and update.

Check your playbooks#

Any kubernetes.core.k8s task with state: absent and delete_options is affected on collection versions before 5.3.0:

# Which kubernetes.core will your ansible actually use?
ansible-galaxy collection list kubernetes.core

# Delete tasks with options: these are the ones to look at.
grep -rn -B8 'delete_options' --include='*.yml' . | grep -E 'state: absent|delete_options'

If the version is older than 5.3.0, upgrade, or put dryRun: ['All'] into delete_options when ansible_check_mode is true.

Lessons#

  • Check mode is a promise each module makes for itself, not one Ansible makes. Read how the module implements it before you trust it with deletes.
  • An API that accepts options in two places will eventually receive them in both. Know which one wins.
  • A rehearsal is only as good as its resemblance to the real run. Same versions, same collections, same flags.
  • The cheapest dry run is one where being wrong deletes nothing that matters.

← All posts