Forgejo Runner Autoscaler

Go controller for ephemeral Forgejo Actions runners backed by Incus VMs.

The controller keeps local durable state in SQLite, polls Forgejo runner jobs and runner records, polls managed Incus VMs, and only treats a runner as usable capacity once Forgejo reports it online, idle, and label-eligible.

Current Status

This is the first production-oriented implementation. It includes:

• Explicit runner lifecycle state persisted in SQLite.
• Forgejo API and Incus CLI adapters behind interfaces.
• Reconciliation loop with context timeouts.
• Warm-pool and queued-job scaling decisions.
• Safe managed-VM cleanup using a name prefix plus Incus metadata.
• Forgejo runner deregistration during teardown.
• Structured JSON logs.
• Prometheus-style /metrics and /healthz.
• Auto-updating protected web dashboard for runners, managed VMs, and waiting jobs.
• Dry-run mode.
• Unit tests for lifecycle transitions and scaling decisions.

Lifecycle

Tracked runners move through:

requested -> launching -> booting -> configuring -> registered -> idle -> assigned -> running -> finished -> draining -> destroying -> destroyed

Failure and timeout paths move through failed or stale_timeout, then drain and destroy.

Capacity is based on tracked Forgejo runner records that are online, idle, have the required labels, and still have an active runner service in the VM. A VM that is merely RUNNING in Incus is not counted as usable.

Build

go test ./...
go build ./cmd/forgejo-runner-autoscaler

Configuration

Start from configs/example.yaml.

Important settings:

• controller.max_runners: hard cap for tracked active runners.
• controller.max_running_jobs: cap for job-driven runner starts. Use this with min_idle_runners to keep a hot pool without letting backlog scaling consume the whole host.
• controller.min_idle_runners: warm pool size.
• controller.start_parallelism: maximum concurrent launching/configuring runners.
• controller.cleanup_parallelism: reserved for bounded cleanup concurrency.
• incus.name_prefix: every managed VM must use this prefix.
• incus.vm_image: Incus VM image alias.
• incus.configure_command: command executed inside the VM after Incus reports it running.
• forgejo.token_path: API token used to list jobs/runners and deregister records.
• forgejo.registration_token_path: optional runner registration token. If omitted, the controller calls the Forgejo registration-token endpoint.
• forgejo.jobs_endpoint: defaults to /api/v1/orgs/{org}/actions/runners/jobs; keep this configurable because Actions job endpoints vary across Forgejo versions.
• dashboard.enabled: serves the web UI when true.
• dashboard.addr: listen address for the dashboard, default :8080.
• dashboard.env_path: .env file containing DASHBOARD_USERNAME and DASHBOARD_PASSWORD.

The dashboard is protected by default. Either create the .env file or set DASHBOARD_ENABLED=false.

sudo install -m 0640 configs/dashboard.env.example /etc/forgejo-runner-autoscaler/.env
sudoedit /etc/forgejo-runner-autoscaler/.env

Environment variables override file config. The old-system values map directly:

MAX_RUNNERS=14
MAX_RUNNING_JOBS=10
MIN_IDLE_RUNNERS=4
START_PARALLELISM=6
CLEANUP_PARALLELISM=18
MIN_AVAILABLE_MEM_MIB=24576
RUNNER_MEMORY_MIB=6144
VM_IMAGE=images:debian/12
FORGEJO_URL=https://forgejo.example.com
FORGEJO_ORG=carrtech

Incus VM Contract

The Incus adapter initially uses the CLI:

• incus launch <image> <name> --vm
• incus list --format=json
• incus exec <name> -- <configure_command...>
• incus delete <name> --force

Managed VMs are identified by both:

• name prefix, for example fj-runner-
• Incus config key user.forgejo-autoscaler.managed=true

The controller never deletes VMs that do not match both.

The VM image should boot quickly and contain the Forgejo runner binary plus any build dependencies. The configure command receives template values such as {RUNNER_NAME}, {FORGEJO_URL}, {FORGEJO_ORG}, {LABELS}, and {REGISTRATION_TOKEN}.

For warm-pool runners, use forgejo-runner one-job --wait in the guest service. Plain one-job exits immediately when no task is already available, so it cannot provide warm capacity. The controller checks incus.service_check_command and destroys the VM once the one-job service exits.

systemd Deployment

1. Build and install the binary:

go build -o forgejo-runner-autoscaler ./cmd/forgejo-runner-autoscaler
sudo install -m 0755 forgejo-runner-autoscaler /usr/local/bin/forgejo-runner-autoscaler

2. Create runtime directories and config:

sudo useradd --system --home /var/lib/forgejo-runner-autoscaler --shell /usr/sbin/nologin forgejo-runner-autoscaler
sudo install -d -m 0750 -o forgejo-runner-autoscaler -g forgejo-runner-autoscaler /var/lib/forgejo-runner-autoscaler
sudo install -d -m 0750 /etc/forgejo-runner-autoscaler
sudo install -m 0640 configs/example.yaml /etc/forgejo-runner-autoscaler/config.yaml
sudo install -m 0640 forgejo-token /etc/forgejo-runner-autoscaler/forgejo-token

3. Install the unit:

sudo install -m 0644 systemd/forgejo-runner-autoscaler.service /etc/systemd/system/forgejo-runner-autoscaler.service
sudo systemctl daemon-reload
sudo systemctl enable --now forgejo-runner-autoscaler

4. Check health and metrics:

curl http://127.0.0.1:9091/healthz
curl http://127.0.0.1:9090/metrics
open http://127.0.0.1:8080/

Rollout And Rollback

Recommended rollout:

1. Keep the old bash autoscaler installed but stopped.
2. Run this controller with dry_run: true and verify logs/metrics.
3. Start with max_runners: 2, min_idle_runners: 1, and a low-risk label set.
4. Confirm Forgejo sees online idle runners before raising max_runners.
5. Increase toward the live values once cleanup and registration are stable.

Rollback:

1. Stop this service.
2. Start the old autoscaler.
3. Remove any remaining managed VMs with the configured prefix only after confirming they are idle or stale.

Development

go test ./...
DASHBOARD_ENABLED=false go run ./cmd/forgejo-runner-autoscaler -config configs/example.yaml -check-config

The controller package uses fakes for Forgejo and Incus so scaling and lifecycle behavior can be tested without live infrastructure.

Forgejo Actions Release Workflow

The workflow in .forgejo/workflows/release.yml runs tests, builds Linux and macOS binaries for amd64 and arm64, and creates a Forgejo release when a merge commit is pushed to main.

Create a repository secret named RELEASE_TOKEN with permission to push tags and create releases. Release tags are generated as vYYYYMMDDHHMMSS-<short-sha>.

The workflow avoids actions/upload-artifact and actions/download-artifact so it works on Forgejo instances where the v4 artifact actions are not supported.

The release job intentionally checks that HEAD is a merge commit. If the repository uses squash or rebase merges into main, adjust that gate or the release will be skipped.