Skip to content

Release Process

This page documents the CI/CD pipeline: what each workflow does, how releases are built, and how to run the process manually.

CI workflows overview

The project uses 4 GitHub Actions workflows. All use only official GitHub actions (actions/checkout, actions/setup-python, actions/upload-pages-artifact, actions/deploy-pages). No third-party actions.

Budget target: ~175 minutes/month on the GitHub free tier.

Workflow File Trigger
Build & Release build.yml Manual dispatch only
Deploy Site deploy-site.yml Push to main (platforms, emulators, provenance, wiki, scripts, database.json, mkdocs.yml), manual
PR Validation validate.yml PR touching bios/**, platforms/** or emulators/**
Weekly Sync watch.yml Cron Monday 06:00 UTC, manual dispatch

build.yml - Build & Release

Releasing is deliberate. Pushing never cuts one: somebody decides the collection is worth publishing and dispatches the workflow.

Trigger. workflow_dispatch only, with an optional force_release flag to bypass the rate limit.

Concurrency. Group build, queued rather than cancelled: a run that is already uploading assets must finish.

Steps:

  1. Checkout, Python 3.12, install pyyaml
  2. Run test_e2e
  3. Rate limit check: skip if last release was less than 7 days ago (unless force_release is set)
  4. Restore large files from the large-files release into .cache/large/
  5. Refresh data directories (refresh_data_dirs.py)
  6. Build packs (generate_pack.py --all --output-dir dist/)
  7. Split any pack over 2 GB into .zip.001, .zip.002, ... volumes. GitHub caps release assets at 2 GB, and the .001 convention is what 7-Zip and PeaZip open directly (they reject .partNN names as corrupt)
  8. Create GitHub release with tag v{YYYY.MM.DD} (appends .N suffix if a same-day release already exists)
  9. Clean up old releases, keeping the 3 most recent plus large-files

Release notes include file count, total size, per-pack sizes, the extract path per platform, and the last 15 non-merge commits touching bios/ or platforms/.

Pack variants. The workflow builds full packs only. Releases that also carry *_Platform_BIOS_Pack.zip had a second --source platform run added by hand, as in the manual process below. See advanced usage.

deploy-site.yml - Deploy Documentation Site

Trigger. Push to main when any of these paths change: platforms/, emulators/, provenance/, wiki/, scripts/generate_site.py, scripts/generate_readme.py, scripts/verify.py, scripts/common.py, database.json, mkdocs.yml. Also manual dispatch.

The list is the set of inputs the site is generated from. Adding a new input to generate_site.py means adding its path here, or the site silently goes stale.

Steps:

  1. Checkout, Python 3.12
  2. Install pyyaml, mkdocs-material>=9.7.5,<10, pymdown-extensions>=10.14
  3. Restore large files from the large-files release, refresh data directories
  4. Run generate_site.py (converts YAML data into MkDocs pages and rewrites mkdocs.yml)
  5. Run generate_readme.py (rebuilds README.md and CONTRIBUTING.md)
  6. mkdocs build --strict to produce the static site
  7. Run validate_site.py on the rendered HTML (metadata, headings, image alternatives, duplicate ids, local links and fragments)
  8. Require the committed README and CONTRIBUTING to match what the generator just produced. write_if_changed() compares content with the timestamp line stripped, so a run that only moves the clock leaves the files untouched and the check stays meaningful
  9. Upload artifact, deploy to GitHub Pages

Data contracts are validated with scripts/validate_schemas.py before the site is generated: database.json, the install and target manifests, the site API envelopes and the stats file, plus the semantic invariants those schemas cannot express (declared totals matching their lists, no destination both installed and omitted).

The site is deployed via the github-pages environment using the official actions/deploy-pages action. Pages deployments are queued rather than cancelled (cancel-in-progress: false): cancelling one mid-flight leaves the deployment stuck and the next runs time out waiting on it.

--strict turns MkDocs warnings into failures, so a broken internal link or a dangling anchor fails the build instead of shipping. The validation: block in mkdocs.yml is what promotes unrecognized links and missing anchors to warnings in the first place.

The theme version is pinned on both sides: >=9.7.5 because that is the release which caps mkdocs < 2 (MkDocs 2.0 ships without a license), <10 so a major theme release cannot change the site without a deliberate bump.

validate.yml - PR Validation

Trigger. Pull requests that modify bios/**, platforms/**, emulators/**, schemas/**, scripts/**, tests/** or install.py.

Concurrency. Per-PR group, cancel in-progress.

Four parallel jobs:

validate-bios. Diffs the PR to find changed BIOS files, runs validate_pr.py --markdown on each, and posts the validation report as a PR comment (hash verification, database match status).

validate-configs. Runs python scripts/validate_schemas.py --source-only, which validates every platform YAML against schemas/platform.schema.json and every emulator profile against schemas/emulator.schema.json. Both schemas set additionalProperties: false, so a typo in a field name fails the job instead of being silently ignored.

run-tests. Runs python -m unittest discover tests -v. Must pass before merge.

label-pr. Auto-labels the PR based on changed paths:

Path pattern Label
bios/ bios
bios/{Manufacturer}/ system:{manufacturer}
platforms/ platform-config
scripts/ automation

watch.yml - Weekly Platform Sync

Trigger. Cron schedule every Monday at 06:00 UTC, or manual dispatch.

Flow:

  1. Scrape live upstream sources (System.dat, batocera-systems, es_bios.xml, etc.) and regenerate platform YAML configs
  2. Auto-fetch missing BIOS files
  3. Refresh data directories
  4. Run dedup
  5. Regenerate database.json
  6. Create or update a PR with labels automated and platform-update

The PR contains all changes from the scrape cycle. A maintainer reviews and merges.

Large files management

Files larger than 50 MB are stored as assets on a permanent GitHub release named large-files (to keep the git repository lightweight).

Examples: PS3UPDAT.PUP, PSVUPDAT.PUP, PSP2UPDAT.PUP, the DSi NAND images, maclc3.zip, Firmware.19.0.0.zip (Switch), the QEMU EDK2 firmware, the ScummVM data bundle, the EasyRPG soundfont, the Dolphin/Ishiiruka SD card images, and the arcade sets over 100 MB. .gitignore is the authoritative list: every bios/ path listed there is a release asset.

Storage. Listed in .gitignore so they stay out of git history. The large-files release is excluded from cleanup (the build workflow only deletes version-tagged releases).

Build-time restore. The build workflow downloads all assets from large-files into .cache/large/ and copies them to their expected paths before pack generation.

Upload. To add or update a large file:

gh release upload large-files "bios/Sony/PS3/PS3UPDAT.PUP#PS3UPDAT.PUP"

Local cache. generate_pack.py calls fetch_large_file() which downloads from the release and caches in .cache/large/ for subsequent runs.

Manual release process

When build.yml is disabled, build and release manually:

# Run the full pipeline (DB + verify + packs + manifests + integrity + docs)
python scripts/pipeline.py

# Or step by step:
python scripts/generate_db.py --force --bios-dir bios --output database.json
python scripts/verify.py --all
python scripts/generate_pack.py --all --output-dir dist/                    # full packs
python scripts/generate_pack.py --all --source platform --output-dir dist/  # platform packs
python scripts/generate_pack.py --all --verify-packs --output-dir dist/

# Split anything over 2 GB (GitHub asset cap)
for f in dist/*.zip; do
  [ "$(stat -c%s "$f")" -gt 2000000000 ] || continue
  split --bytes=1900M --numeric-suffixes=1 --suffix-length=3 "$f" "$f." && rm "$f"
done

# Create the release
DATE=$(date +%Y.%m.%d)
gh release create "v${DATE}" dist/*.zip* \
  --title "BIOS Pack v${DATE}" \
  --notes "Release notes here" \
  --latest

The two generate_pack.py runs are what puts both *_BIOS_Pack.zip and *_Platform_BIOS_Pack.zip on the release. --all-variants builds all six combinations instead, which is more than a release needs.

Run the pipeline online for a release: --offline skips the data directory refresh and the MAME/FBNeo hash refresh, so the packs would ship stale data directories.

The workflow carries no push trigger, so there is nothing to disable and no guard to remove. Cutting a release means dispatching build.yml from the Actions tab, or:

gh workflow run build.yml
gh workflow run build.yml -f force_release=true   # within 7 days of the last

The rate limit refuses a second release inside seven days unless force_release is set, which is what keeps a stray dispatch from publishing twice in a day.