Skip to main content

Release Process

Flo releases follow a Gitflow close from develop to master, then deploy the resulting image to tenants with the blue-green CLI. This page covers the version scheme, the release-notes runbook, the deploy paths, maintenance mode, rollback, and post-deploy verification.

Version scheme​

Release versions are semver X.Y.Z; flo release create rejects any other format.

flo release create 1.38.0            # release branch -> master + tag v1.38.0 -> back to develop
flo release create 1.38.0 --no-push  # same, without pushing to the remote

The command requires a clean working tree, the current branch to be develop, and the vX.Y.Z tag to not exist yet. It then:

  1. creates release/X.Y.Z from develop;
  2. bumps the version in every manifest;
  3. merges the release branch into master (--no-ff) and tags vX.Y.Z;
  4. merges back into develop and deletes the release branch;
  5. pushes master, develop, and the tags (skipped with --no-push).

The synced version files are Flo.FE/package.json, cli/package.json, cli/package-lock.json, Flo.BE/Flo.BE.csproj, and the three Flo.FE/src/environments/environment*.ts files.

Between releases, develop carries a build-suffixed version: <major>.<minor>.<patch>-<build> (for example 1.37.0-501). The suffix is a build counter that never goes backwards; a running container exposes it as APP_VERSION together with the baked GIT_COMMIT_HASH.

flo release check (run on release/* branches) validates the process controls: the release-notes structure, a - Release X.Y.Z: ... bullet under # vNext, and the absence of SDD artifacts on the release branch.

Release notes​

ReleaseNotes.md in the repository root is mandatory before every commit or release.

  • Unreleased work lands under # vNext at the top of the file, as one-line bullets (what changed plus the touched surface).
  • Every # vNext must carry a ## Pre-deploy and a ## Post-deploy checkbox checklist: the runbook the operator follows around the deploy.
  • Hotfixes add a # hotfix/<name> (YYYY-MM-DD) block under # vNext.
  • Closing a release turns the accumulated # vNext content into a dated # vX.Y.Z (YYYY-MM-DD) block; newest releases stay on top.

Write the checklists as real steps, not changelog: environment variables to set before the deploy, one-off migrations that do not run on startup, disk and quota checks, canary order, health and version verification, smoke checks, and the rollback command. If a release needs nothing manual, say so explicitly (for example "no manual steps; EF migrations run on startup").

Deploying​

Single-tenant (Docker Compose and Nginx)​

Follow Single-Tenant Deployment. An update is a git pull followed by the deploy script:

cd ~/Flo
git pull
./deploy.sh --lite        # rebuild with cache
./deploy.sh --rebuild     # full rebuild without cache

--force recreates containers to refresh configs, --restart restarts the app container (re-running seeders), and --all starts every service after a full stop.

Multi-tenant (blue-green CLI)​

Follow Multi-Tenant CLI. A production deploy is:

flo --vps production deploy <tenant> --branch master -y

The blue-green sequence is:

  1. pull the image, waiting for the CI build of the branch HEAD commit (default timeout 20 minutes, override with --ci-timeout <min>);
  2. start the idle-color container;
  3. wait for its health check;
  4. swap Traefik routing to it;
  5. keep the old container running as an HA standby (no stop).

If the health check fails, the tenant registry is not flipped and the previously active container keeps serving traffic. A failed start of a single-container test instance leaves that instance down until a known-good image is redeployed.

Rules for live tenants:

  • --branch master is required. A development branch on a live tenant is refused unless the operator passes --force for that specific operation.
  • The remote resource preflight blocks the deploy when the host lacks memory headroom or disk space; there is no bypass flag.
  • --no-wait-ci is emergency-only: it pulls whatever image is already in the registry and may ship a stale commit.
  • Set new environment variables before the deploy; a release that reads a missing variable can crash-loop the new container.
  • DR-enrolled tenants also sync the immutable image to the standby after cutover. A failed sync reports "primary active, DR not ready" and can be retried with flo failover sync-image <tenant>.

Fleet release​

flo release deploy 1.38.0          # every production tenant, sequentially
flo release deploy --no-backup     # skip the per-tenant pre-deploy backup

flo release deploy resolves every tenant with location: production from the tenant index, then deploys master to them one at a time with a pre-deploy database backup (on by default) and per-tenant health checks. It stops at the first failure. It is a local orchestration command, not a --vps remote form.

For a manual fleet rollout, lead with a low-risk canary and watch health between tenants.

Maintenance mode​

Maintenance mode can be flipped live, without a container restart:

flo --vps production instance maintenance <tenant> status
flo --vps production instance maintenance <tenant> on -y
flo --vps production instance maintenance <tenant> off

The command calls the in-container maintenance API (/api/v1/maintenance/mode) on every backend of the tenant, authenticated with the tenant's maintenance key. The MAINTENANCE_MODE environment variable stays the boot-time default; the API overrides it for the running process. While maintenance is on, guarded APIs answer 503 with errors.general.maintenanceMode, and the maintenance endpoints remain reachable. Turning maintenance ON asks for confirmation unless -y is passed. When MON controls the tenant, activation is blocked until flo failover auto disarm completes.

The command verifies the resulting state on every backend. If verification fails it restores the previous state; if that also fails, it stops the backends to block traffic, and that situation must be treated as an operational incident.

On a single-tenant Compose install, call the same endpoint from inside the container. The CLI does it with:

docker exec <container> wget -q -O- \
  --header='X-Maintenance-Key: <key>' \
  'http://localhost:10001/api/v1/maintenance/mode'

Rollback​

flo deploy rollback <tenant> -y

Rollback swaps Traefik routing back to the previous color: no rebuild, nearly instant. Both containers keep running, and the rollback is recorded in the deploy history. It requires a previousColor (at least one prior blue-green deploy). Single-container test instances have no standby and refuse rollback; redeploy the previous image instead (flo deploy <tenant> --tag <previous-image>).

flo deploy history <tenant>       # deploys and rollbacks: branch, commit, image, color
flo deploy upgrade <tenant>       # pull the latest default image and redeploy

Post-deploy verification​

  1. Work through the ## Post-deploy checklist in ReleaseNotes.md.
  2. Health: flo instance health <tenant> (database, API, disk, containers), or probe /health externally.
  3. Version: confirm the running container reports the expected APP_VERSION and GIT_COMMIT_HASH (flo instance ls, flo instance inspect <tenant>).
  4. History: flo deploy history <tenant> shows the new entry; deploy notifications report success or failure.
  5. DR tenants: verify standby image parity and retry with flo failover sync-image <tenant> if the post-cutover sync failed. See Failover and Disaster Recovery.
  6. Keep the rollback command at hand until the canary window is closed.