fix(disk-monitor): stop the disk check from purging live build workspaces #1

Merged

zachhandley merged 2 commits from fix/disk-monitor-safe-cleanup into main 2026-09-02 20:55:45 +02:00

Conversation 0 Commits 2 Files changed 2 +20 -9

zachhandley commented 2026-09-02 08:49:11 +02:00

Owner

Copy link

Problem

The forgejo-runner-disk-monitor unit runs forgejo-runner cache clean --disk-check-only, whose "disk critical" branch (patches/forgejo-runner-fork.patch) purges the entire ~/.cache. On a host-executor runner that deletes the active act job workspace mid-build — the checkout/CWD, CARGO_HOME (registry crate sources), and git worktrees — so long compiles fail with could not parse/generate dep info … No such file or directory and failed to stat working directory …/hostexecutor.

Observed on a macOS host runner: a ~10-min Rust build was killed when the 5-min disk-monitor read 85% and logged Disk usage 85% is critical - clearing entire cache → Purged /Users/…/.cache (1.6 GB freed) at minute ~6.

Fix (Linux)

Point the disk-monitor at the same safe orchestrator the daily cleanup service already uses — forgejo-runner-cleanup --once: age-guarded cache (>CACHE_MAX_AGE_DAYS) and workspace (>WORKSPACE_MAX_AGE_DAYS, skips currently-running jobs) pruning plus threshold-based docker cleanup. No entire-cache purge. Unit timeout raised for the scan.

macOS (separate follow-up)

scripts/install-macos.sh generates an equivalent disk-monitor plist with the same destructive command. The mac cleanup path differs (inline bash -c daemon in com.forgejo.runner.cleanup.plist; no forgejo-runner-cleanup binary is shipped), so it needs its own change. The one live mac runner has been remediated directly in the meantime (disk-monitor repointed to an age-guarded pruner; ~/.cache/act restored to the external-volume symlink).

Root source (separate follow-up)

The destructive escalation itself lives in patches/forgejo-runner-fork.patch ("… is critical - clearing entire cache" → Purged %s). Making that branch age-guarded would fix every invoker, but the fork build is currently red (patches don't apply cleanly to upstream v12.13.x), so this unit-level change is what protects the deployed fleet now.

## Problem The `forgejo-runner-disk-monitor` unit runs `forgejo-runner cache clean --disk-check-only`, whose **"disk critical" branch** (`patches/forgejo-runner-fork.patch`) purges the **entire `~/.cache`**. On a host-executor runner that deletes the **active** act job workspace *mid-build* — the checkout/CWD, `CARGO_HOME` (registry crate sources), and git worktrees — so long compiles fail with `could not parse/generate dep info … No such file or directory` and `failed to stat working directory …/hostexecutor`. Observed on a macOS host runner: a ~10-min Rust build was killed when the 5-min disk-monitor read 85% and logged `Disk usage 85% is critical - clearing entire cache` → `Purged /Users/…/.cache (1.6 GB freed)` at minute ~6. ## Fix (Linux) Point the disk-monitor at the same safe orchestrator the daily cleanup service already uses — `forgejo-runner-cleanup --once`: age-guarded cache (>`CACHE_MAX_AGE_DAYS`) and workspace (>`WORKSPACE_MAX_AGE_DAYS`, **skips currently-running jobs**) pruning plus threshold-based docker cleanup. No entire-cache purge. Unit timeout raised for the scan. ## macOS (separate follow-up) `scripts/install-macos.sh` generates an equivalent disk-monitor plist with the same destructive command. The mac cleanup path differs (inline `bash -c` daemon in `com.forgejo.runner.cleanup.plist`; no `forgejo-runner-cleanup` binary is shipped), so it needs its own change. The one live mac runner has been remediated directly in the meantime (disk-monitor repointed to an age-guarded pruner; `~/.cache/act` restored to the external-volume symlink). ## Root source (separate follow-up) The destructive escalation itself lives in `patches/forgejo-runner-fork.patch` (`"… is critical - clearing entire cache"` → `Purged %s`). Making that branch age-guarded would fix every invoker, but the fork build is currently red (patches don't apply cleanly to upstream v12.13.x), so this unit-level change is what protects the deployed fleet now.

zachhandley added 1 commit 2026-09-02 08:49:12 +02:00

fix(disk-monitor): stop the disk check from purging live build workspaces ... b0f3b5f793

The disk-monitor unit ran `forgejo-runner cache clean --disk-check-only`, whose
"disk critical" branch (forgejo-runner-fork.patch) purges the entire ~/.cache.
On a host-executor runner that deletes the ACTIVE act job workspace mid-build —
the checkout/CWD, CARGO_HOME (registry crate sources), and git worktrees — so
long compiles fail with "could not parse/generate dep info ... No such file or
directory".

Point the disk-monitor at the same safe orchestrator the daily cleanup service
uses (`forgejo-runner-cleanup --once`): age-guarded cache/workspace pruning that
skips currently-running jobs, plus threshold-based docker cleanup. Raise the
unit timeout for the scan.

macOS install-macos.sh generates an equivalent disk-monitor plist with the same
destructive command; the mac cleanup path differs (inline daemon, no
forgejo-runner-cleanup binary) and needs a separate change — the one live mac
runner was remediated directly in the meantime.

zachhandley added 1 commit 2026-09-02 10:52:00 +02:00

fix(cleanup): don't let set -e abort the cycle on a benign no-op ... 7d8a74a16d

forgejo-runner-cleanup's tasks (remove_stale_locks, clean_cache, clean_workspaces,
clean_docker) end on — or early-return after — a false test (`[ $count -gt 0 ]`,
`[ $VERBOSE -eq 1 ]`, `[ usage -lt threshold ]`) when there is nothing to do, so
under `set -e` the function returns non-zero and the whole run aborts with exit 1.
This bricked both the disk-monitor and the daily cleanup service on any runner
where a cleanup pass had nothing to remove (the common case) — the daily service
sat in activating/status=1/FAILURE.

Wrap the task calls in run_cleanup with `|| true` so a benign non-zero from a
cleanup task never aborts the cycle; real mid-task failures still surface via the
task's own logging.

zachhandley merged commit f6ddb9deb3 into main 2026-09-02 20:55:45 +02:00

zachhandley referenced this pull request from a commit 2026-09-02 20:55:48 +02:00

Merge pull request 'fix(disk-monitor): stop the disk check from purging live build workspaces' (#1) from fix/disk-monitor-safe-cleanup into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Public/ForgejoRunner!1

No description provided.