Three silent uninstalls: what k3s does to your HelmCharts on every patch upgrade

Between June and October 2026 my k3s cluster quietly upgraded its storage operator, lost eight days of logs, lost its pod networking, and lost its monitoring for about fifteen hours. Nobody ran helm. Nobody merged anything risky. Each time, the cluster did it to itself, a few minutes after a routine Kubernetes patch upgrade.

It took four incidents before I saw that they were one incident. This post is the mechanism, the trap I stepped into while cleaning up, and the changes that make it boring again.

How k3s installs Helm charts#

k3s ships a small controller, the helm-controller, that turns a HelmChart custom resource into a Helm release. You drop a manifest into /var/lib/rancher/k3s/server/manifests/, k3s applies it as an Addon, and the helm-controller creates a Job that runs helm upgrade --install inside a klipper-helm container:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: loki
  namespace: kube-system
spec:
  repo: https://grafana.github.io/helm-charts
  chart: loki
  version: 7.3.0
  targetNamespace: monitoring
  valuesContent: |-
    deploymentMode: SingleBinary

It is convenient for bootstrap: networking and storage are installed before anything else in the cluster can run. Three properties matter later.

• A HelmChart is state in etcd, not a file. Deleting its YAML file from the manifests directory leaves the Addon and the HelmChart in place, and rebuilding a server node keeps them, because etcd survives.
• The install Job runs once, but it is not the end of the story. The helm-controller keeps comparing the Job it would create today with the one that exists. The Job’s definition includes the HelmChart’s values and the klipper-helm image, which is tied to the k3s version. When anything in it differs, the controller deletes the finished Job and creates a new one, and the new Job runs helm upgrade again.
• spec.failurePolicy defaults to reinstall.

The mechanism#

The trigger. My servers follow the k3s v1.35 channel through the system-upgrade-controller, so patch releases arrive unattended. Each one ships a new klipper-helm image. The helm-controller now wants a different install Job for every HelmChart, replaces each finished Job with a new one, and every chart in the cluster is upgraded again, with nobody touching it. The three events in my logs are 26 June, 5 August and 1 October, each followed within minutes by a burst of helm-install-* pods.

The fuel. A re-run is harmless when the HelmChart matches what you deploy. Mine did not always:

• HelmCharts left behind after I moved kube-prometheus-stack and Loki to Ansible-managed Helm releases. They still pointed at chart versions from 2024.
• Rook HelmCharts without a version, so the Job installed whatever was newest.
• Values that only worked as a three-way patch, and CRDs whose fields a second manager also wrote.

The amplifier. With failurePolicy: reinstall, a failed upgrade leaves the release in failed. The Job’s own retry, seconds later, finds a failed release and uninstalls it before installing again. The release did not need to be broken beforehand; any upgrade failure is enough. From the Calico CRD Job:

21:15:53  Error: UPGRADE FAILED: conflict occurred while applying object
          /bgpconfigurations.crd.projectcalico.org ...
          Apply failed with 1 conflict: conflict with "OpenAPI-Generator" ... .spec.versions
21:16:06  Uninstalling failed helm chart

Thirteen seconds from a field-ownership conflict to deleting every Calico CRD.

Four incidents, one cause#

The August one hurts most in hindsight. The logging stack was swapped for an older one for eight days and nothing alerted, because the thing that should have told me was the thing being replaced.

The cleanup that uninstalled my monitoring#

On 1 October I knew the stale HelmCharts were the problem, so I wrote a playbook to remove them. A helm-delete Job runs on deletion because of a finalizer, so the plan was: remove the finalizers, then delete the objects.

The helm-controller put the finalizer back before the delete arrived, ran its helm-delete Jobs, and uninstalled the very releases I was trying to protect. Prometheus, Alertmanager and Loki were gone.

The controller has an explicit opt-out that I had not read about. In helm-controller v0.17, shouldManage skips any HelmChart with the helmcharts.helm.cattle.io/unmanaged annotation. The safe order to retire a HelmChart whose release you want to keep:

# 1. Remove the manifest file, so nothing re-applies the object.
# 2. Tell the controller to stop acting on it.
kubectl -n kube-system annotate helmchart loki helmcharts.helm.cattle.io/unmanaged=true
# 3. Now the finalizer can go, and the object with it.
kubectl -n kube-system patch helmchart loki --type=merge -p '{"metadata":{"finalizers":[]}}'
kubectl -n kube-system delete helmchart loki

The lesson I keep relearning: when you change a controller’s objects, read how the controller reacts in its source, not in your head.

Values that only work as a patch#

Getting Prometheus back took another surprise. Its values had worked for every deploy for years, but every one of those deploys had been an upgrade: Helm sends a three-way patch, and in a patch null means “delete this field”. A fresh install creates the objects instead, and creation rejects the same values.

The klipper-helm Job installs with --server-side=true, and fresh installs with Helm 4 use server-side apply too, so every reinstall applies this strictness. A chart that cannot be installed from scratch is one bad upgrade away from an outage.

What changed#

• failurePolicy: abort on every HelmChart. A failed upgrade now stays failed and waits for a human. A test in the repository fails the build if a HelmChart template lacks it.
• Stale HelmCharts and Addons removed, with the unmanaged annotation first, thirteen Addons in total, including ones from install methods I had abandoned years ago.
• Every chart version pinned. Renovate proposes upgrades; nothing floats.
• Values valid for creation. A test rejects null and numeric label values in the inventory. Before an upgrade I render each chart with its live version and values and run kubectl apply --server-side --dry-run=server on the result under temporary names.
• A read-only drift check that fails when a HelmChart has no template in the repository, lacks abort, or a Helm release’s latest revision is not deployed. Its first run found nothing, which is the point: it runs before the next patch upgrade does.
• One owner per object. The Calico operator no longer writes its own CRDs; the CRD chart does.

Next is moving the remaining HelmCharts that bootstrap does not need into Ansible-managed releases, so the controller that upgrades k3s stops redeploying applications.

Check your cluster#

If you run k3s or RKE2 with HelmCharts, two commands show most of the risk:

# Every HelmChart with its pinned version and failure policy.
# An empty POLICY means reinstall.
kubectl get helmcharts -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,CHART:.spec.chart,\
VERSION:.spec.version,POLICY:.spec.failurePolicy

# Releases whose latest revision is not deployed.
helm list -A --failed --pending

An empty VERSION means the next re-run installs whatever is newest. An empty POLICY means the next failed upgrade is an uninstall. A HelmChart that no file in your repository describes is the stale kind.

Lessons#

• Unattended upgrades re-run everything. Anything stale surfaces there, at a time you did not choose.
• reinstall is not a retry. For releases that own data or APIs, only abort is safe.
• A manifest directory is not the source of truth; etcd is. Retiring a file needs a cleanup step.
• “It deploys fine” may only mean “it upgrades fine”. Test fresh installs.
• An outage that nobody notices for eight days is a monitoring bug, not luck.