Promotion and rollback

Manual buttons in the pipeline UI. They live in the promote stage, after deploy, so a button only becomes clickable once that exact commit has built and deployed successfully at the level below.

dev  pipeline -> promote_to_beta   fast-forwards beta to this commit
beta pipeline -> promote_to_prod   fast-forwards prod to this commit
prod pipeline -> rollback_prod     new revert commit on prod + MR back to dev

One-time setup per project

Promotion pushes a protected branch, which the CI job token cannot do. So:

1. Settings -> Access tokens -> create a Project Access Token

• role: Maintainer
• scope: write_repository

1. Settings -> CI/CD -> Variables -> add PROMOTE_TOKEN = that token

• Masked. Not protected, or it will not exist on dev pipelines, which

is exactly where promote_to_beta runs.

1. Settings -> Repository -> Protected branches -> for beta and prod,

add the token's user to "Allowed to push and merge"

Step 2 is the one people get wrong. Marking the variable protected feels safer and breaks promotion from dev.

The shared template fails fast when it is missing:

.promote:
  stage: promote
  image: docker:28-cli
  before_script:
    - test -n "$PROMOTE_TOKEN" || { echo "PROMOTE_TOKEN is not set"; exit 1; }

promote_to_beta

promote_to_beta:
  extends: .promote
  script:
    - echo "Promoting $CI_COMMIT_SHORT_SHA (dev) to beta"
    - git push "https://oauth2:${PROMOTE_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" \
        "${CI_COMMIT_SHA}:refs/heads/beta"
  rules:
    - if: '$CI_COMMIT_BRANCH == "dev"'
      when: manual
      allow_failure: true

A fast-forward of one specific commit. Not a merge, so beta is always exactly some commit that passed on dev, with no merge commit inventing a tree nobody tested.

allow_failure: true keeps the pipeline green when the button is never pressed.

promote_to_prod

Same, plus a marker so rollback has somewhere to return to:

promote_to_prod:
  extends: .promote
  script:
    - REPO="https://oauth2:${PROMOTE_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
    # Record the CURRENT prod HEAD as prod-prev, THEN fast-forward prod.
    - |
      PREV=$(git ls-remote "$REPO" refs/heads/prod | cut -f1)
      if [ -n "$PREV" ]; then
        git push "$REPO" ":refs/tags/prod-prev" 2>/dev/null || true
        git push "$REPO" "$PREV:refs/tags/prod-prev"
      fi
    - git push "$REPO" "${CI_COMMIT_SHA}:refs/heads/prod"
  rules:
    - if: '$CI_COMMIT_BRANCH == "beta"'
      when: manual
      allow_failure: true

The prod-prev tag is deleted and recreated on every promotion, so it always means "the release before the current one". One step back, which is the only step anyone takes under pressure.

rollback_prod

The design decision worth understanding: rollback does not force-push or reset. It adds a new forward commit whose tree equals prod-prev.

Why that matters:

• Branch protection stays on. No force-push exception to grant, no window where

prod is unprotected.

• The pipeline re-runs, so Dokploy gets a freshly built image that matches

the compose file and env it is deployed with. A reset would leave a running image that no commit describes.

• History is honest. The rollback is visible as a commit, not as a branch that

quietly moved backwards.

Two refinements in the script:

.gitlab-ci.yml is kept at the current version, not reverted. The pipeline definition must only ever move forward. Rolling it back could reintroduce a broken pipeline and make recovery harder than the incident.

TMP_IDX="$(mktemp)"
CIBLOB="$(git rev-parse "$PROD:.gitlab-ci.yml")"
export GIT_INDEX_FILE="$TMP_IDX"
git read-tree refs/tags/prod-prev
git update-index --add --cacheinfo 100644 "$CIBLOB" .gitlab-ci.yml
TREE="$(git write-tree)"
unset GIT_INDEX_FILE
NEW=$(git commit-tree "$TREE" -p "$PROD" -m "revert: roll prod back to $SHORT")
git push "$REPO" "$NEW:refs/heads/prod"

An MR carries the revert back to dev. Without it, the next dev -> beta -> prod promotion silently re-applies the change you just rolled back:

git push "$REPO" "$NEW:refs/heads/rollback/${CI_PIPELINE_ID}-to-dev" \
  -o merge_request.create \
  -o merge_request.target=dev \
  -o merge_request.title="Propagate prod rollback to dev" \
  -o merge_request.remove_source_branch

An MR rather than a direct push, so conflicts surface for review instead of being resolved silently. beta self-heals on the next promotion from dev, so it needs no MR of its own.

Guard the no-op case:

if [ "$PREV" = "$PROD" ]; then
  echo "prod is already at prod-prev - nothing to roll back."
  exit 0
fi

And fail with an explanation when the marker does not exist yet:

git fetch "$REPO" "+refs/tags/prod-prev:refs/tags/prod-prev" 2>/dev/null \
  || { echo "No prod-prev marker yet - promote to prod at least once first."; exit 1; }

What this model does not give you

• Rollback is one step. Two releases back means promoting an older commit

from beta, or reverting by hand. Deliberate: multi-step rollback tooling gets used once a year and is wrong when it matters.

• No automatic rollback on failed health checks. A human presses the button.
• Database migrations are not reverted. Rolling back application code does

not roll back schema, which is why migrations must stay backward compatible for one release. The application skill covers this.