A Runner Self-Update Broke npm on All Five of My CI Runners
A GitHub Actions runner updated itself overnight and turned npm into a 54-byte file that could not find its own code. It hit all five self-hosted runners on epsilon at once. How Claude Code and I found it, fixed it and set up a 5-minute timer to repair it after the next update.
I started this morning with a handoff note from a Claude Code session the night before. Four items were left. The first said to repair the pnpm install under ~/setup-pnpm on epsilon and re-run the CI for an 8bitoracle-next pull request. The second said to merge that pull request (which deploys to production).
The note was wrong about the cause. Claude read the failed CI log before touching anything. The job never got as far as ~/setup-pnpm. It died in the step that installs pnpm:
Error: Cannot find module '../lib/cli.js'
Require stack:
- /home/gh-runner/actions-runner-8bitoracle/externals.2.338.0/node24/bin/npm
That path is not pnpm. It is the copy of Node that the GitHub Actions runner ships inside itself, under externals/. pnpm/action-setup uses that Node's npm to install pnpm, and that npm was broken.
Why the runners are on epsilon at all
My main VPS, epsilon, has hosted self-hosted GitHub Actions runners for several repos since September. I moved CI there because self-hosted minutes are not metered, and my Actions budget was frozen. Each runner lives in its own directory under a gh-runner user, with no sudo. The runner keeps itself up to date: when GitHub releases a new version, it downloads it, swaps itself out and restarts.
The runner's log shows the update ran at 04:21 Hong Kong time on 2026-10-23, from 2.337.0 to 2.338.0. The self-update log ended with Update succeed. The pull request's CI failed twice, just after midnight the next day. The session that hit it read the failure as a broken pnpm install and put that in its handoff.
Checking every runner
Claude could not look inside /home/gh-runner, and sudo on epsilon needs my password. So it wrote a script for me to run with sudo. In check mode it runs npm --version with each runner's bundled Node, as the runner user, which is the same call that failed in CI.
The first run reported 30 broken installs out of 40. That was too many, and some of it was the script's fault. Each externals folder also carries node20_alpine and node24_alpine builds. Those are linked against musl for Alpine containers and can never run on an Ubuntu host, so they always fail the test. After Claude dropped them from the check, the count was 10 out of 20: node20 and node24 in every externals.2.338.0, on all five runners. The 2.337.0 copies all worked.
So it was not one runner. The 2.338.0 update had broken every runner on epsilon.
The repair
The script's repair mode stops each broken runner and moves its externals.2.338.0 aside (it never deletes it). Then it extracts externals from the official 2.338.0 tarball and restarts the runner. Before I ran it, Claude checked with the GitHub API that none of the runners had a job in progress.
All five came back with npm 10.8.2 on Node 20 and npm 11.19.0 on Node 24. The pull request's CI passed in 3 minutes 23 seconds, and I merged it.
What was wrong
I asked Claude whether we were protected against this happening again. "The update broke it" does not answer that. Claude wrote a second script to compare the broken folder with the restored one.
They were the same size, 601 MB each. The broken one had three more files per Node folder: 4,324 against 4,321 for Node 20. No file was missing.
The listing of node24/bin showed the reason. In a normal Node install, npm, npx and corepack are symlinks into lib/node_modules/. In the broken folder they were plain files. npm was 54 bytes, which is the size of npm-cli.js. That file is one line: it requires ../lib/cli.js. Run from lib/node_modules/npm/bin/, that path finds npm's code. Copied into node24/bin/, the same path points at node24/lib/cli.js, which does not exist.
Something had copied the files the links pointed to, instead of the links. (The three extra files were the three links, now counted as files.)
A search of the runner's issue tracker found it straight away: actions/runner#2094, "Runner self-updates don't preserve symlinks in externals." It was opened in September 2022 and is still open. The last comment, from 2026-09-04, reports the same 54-byte npm on the update to 2.337.0.
So the fresh install would probably last only until the next update, which is likely to do the same thing to every runner.
Making it repair itself
The damage is three links per Node folder. Putting them back needs no download and no restart, so Claude wrote a fixer that does only that. It turns a plain npm, npx or corepack back into a link. It does this only when the file is byte for byte identical to the file the link should point to. If the file is anything else, the fixer leaves it alone and reports it.
A systemd timer runs every 5 minutes, as root:
- The fixer puts back any links an update has flattened.
- The npm check runs on every runner.
- If something is still wrong, it opens a GitHub issue with both outputs and the repair command. The issue closes itself on the next healthy run.
If the check finds nothing to examine, for example because the runners moved, it counts that as a failure and opens an issue too.
Claude tested the fixer without root by building the fault on purpose. cp -rL copies a directory with symlinks followed, which is what the runner's updater does. Copying a Node install that way gave the same Cannot find module '../lib/cli.js'. After one run of the fixer, npm and npx worked again. A second run changed nothing. When npm was replaced with a different file, the fixer refused to touch it.
The first real run checked 60 links and 20 Node installs and found nothing to do.
That run also filled the journal. The check switched to the runner user with sudo -u, and each call opened and closed a login session. That was about 500 pam_unix lines an hour. It now uses setpriv, which switches user without a session.
The next self-update is the real test. When it comes, journalctl -u runner-node-check -g relinked should show the fix within 5 minutes, with no issue filed.
Small things from the day
Two commands Claude gave me failed along the way. sudo grep … _diag/SelfUpdate-*.log returned "No such file," because my shell expanded the * as me before sudo ran, and my user cannot read that folder. It needs sudo sh -c '…'. Then a multi-line command broke into separate commands when I pasted it. After that, Claude put anything longer than one line into a script file and gave me one sudo bash line to run.
The scripts now live in epsilon-infra, a new private repo. Anything installed on epsilon should have its source there. Until today, the closest thing was a memory note about how the runners were set up.
Written with Claude Code, from the session where we fixed this.