Build the production image where the Erlang VM can actually run

ab5395c7642d · AtlantisPleb · · parent 87837ed082e3

Build the production image where the Erlang VM can actually run

`ops/deploy/build-image.sh` builds the amd64 image and then runs it to check
that its packaged revision equals the Git SHA. On an Apple Silicon machine that
run happens under amd64 emulation, and the Erlang VM cannot start there — it
dies at kernel start with `failed_to_start_child,user,nouser`, reproducible on
the bare `hexpm/elixir` base image and unaffected by `-noinput` or `TERM=dumb`.
So the local path cannot produce a production image on such a machine at all.
Earlier builds only appeared to work because the layer was cached; a Docker
restart cleared the cache and the wall appeared.

Cloud Build workers are native amd64, so the build and the check run for real.
The check is kept rather than dropped to get past the wall.

The three refusals before the build exist because skipping them cost a deploy.
The first attempt ran `gcloud builds submit .` after a `cd` that had silently
failed, so it uploaded a checkout one commit behind with untracked files, and
tagged the result with the SHA that was asked for. The packaged-revision check
did not catch it: the revision is passed in as a build argument, so it reported
the SHA it was told regardless of the source. A build argument can only confirm
what it was handed — what the source actually is has to be checked before a
build is spent on it.

This registry has tag immutability, so the wrong image could not be replaced
and that SHA is unusable forever. Hence the third check: refuse when the tag
already exists, rather than discovering it at push time.

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

pushed
by user · WAL seq 443 · 2026-08-26T09:12:11.405000Z
built
8 modules in 39.3 s
deployed
live · 8 modules on 3 nodes · push→live —
deployed
needs_rolling_replace · 8 modules on 0 nodes · push→live —

Changed files

  • added ops/deploy/build-image-cloud.sh

Diff

1 file changed, +157 -0

ops/deploy/build-image-cloud.sh added +157

@@ -0,0 +1,157 @@

1
#!/bin/sh
2
# Build the production image on Cloud Build, for a machine that cannot build it
3
# locally.
4
#
5
# `ops/deploy/build-image.sh` builds and then *runs* the amd64 image to check
6
# that its packaged revision equals the Git SHA. On an Apple Silicon machine
7
# that run happens under amd64 emulation, and the Erlang VM cannot start there:
8
# it dies at kernel start with
9
#
10
#     failed_to_start_child,user,nouser
11
#
12
# reproducible on the bare `hexpm/elixir` base image, and unaffected by
13
# `-noinput` or `TERM=dumb`. So the local path cannot produce a production image
14
# on such a machine at all. Cloud Build workers are native amd64, so the same
15
# build and the same check run for real rather than emulated.
16
#
17
# The revision check is kept rather than dropped to get past the wall. An image
18
# whose packaged BuildInfo does not equal the exact Git SHA must not reach the
19
# registry, because the release path identifies what the fleet runs by that SHA.
20
#
21
# Usage: ops/deploy/build-image-cloud.sh <git-sha>
22
set -eu
23
24
sha=${1:-}
25
[ -n "$sha" ] || { echo "usage: $0 <git-sha>" >&2; exit 2; }
26
27
script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
28
repo_root=$(CDPATH= cd -- "$script_dir/../.." && pwd)
29
30
project=${OPENAGENTS_BUILD_PROJECT:-openagentsgemini}
31
service_account=${OPENAGENTS_BUILD_SERVICE_ACCOUNT:-projects/openagentsgemini/serviceAccounts/oa-mvp-automation@openagentsgemini.iam.gserviceaccount.com}
32
image=${OPENAGENTS_IMAGE_REPOSITORY:-us-central1-docker.pkg.dev/openagents-staging-20260820/openagents-staging/openagents}
33
34
# The checks below exist because skipping them cost a real deploy.
35
#
36
# The first attempt at this ran `gcloud builds submit .` after a `cd` that had
37
# silently failed, so it uploaded a *different* checkout — one commit behind,
38
# with untracked files — and tagged the result with the SHA that was asked for.
39
# The packaged-revision check did not catch it: the revision is passed in as a
40
# build argument, so it reported the SHA it was told regardless of the source.
41
# The registry has tag immutability, so that tag could not be corrected and the
42
# SHA is unusable forever.
43
#
44
# A build argument can only ever confirm what it was handed. What the source
45
# actually is has to be checked here, before a build is spent on it.
46
47
head=$(git -C "$repo_root" rev-parse HEAD)
48
if [ "$head" != "$sha" ]; then
49
    echo "refusing: $repo_root is at $head, not $sha" >&2
50
    echo "  Build from a worktree checked out at the exact revision:" >&2
51
    echo "    git worktree add --detach <path> $sha" >&2
52
    exit 1
53
fi
54
55
dirty=$(git -C "$repo_root" status --porcelain --untracked-files=all |
56
    grep -v '^?? \.gcloudignore$' || true)
57
if [ -n "$dirty" ]; then
58
    echo "refusing: the worktree is not clean, so the image would not be $sha" >&2
59
    echo "$dirty" >&2
60
    exit 1
61
fi
62
63
# A tag in this repository cannot be moved, so a wrong image is permanent.
64
# Refuse before building rather than discovering it at push time.
65
existing=$(gcloud artifacts docker images describe "$image:$sha" \
66
    --format='value(image_summary.digest)' 2>/dev/null || true)
67
if [ -n "$existing" ]; then
68
    echo "refusing: $image:$sha already exists ($existing)" >&2
69
    echo "  Tags here are immutable. If that image is wrong, it cannot be" >&2
70
    echo "  replaced -- land another commit and build that SHA instead." >&2
71
    exit 1
72
fi
73
74
release_version=$(tr -d '\n' <"$repo_root/VERSION")
75
source_date_epoch=$(git -C "$repo_root" show -s --format=%ct "$sha")
76
77
# Keep the upload to the sources the Dockerfile copies.
78
cat >"$repo_root/.gcloudignore" <<'IGNORE'
79
.git
80
.gcloudignore
81
deps
82
_build
83
node_modules
84
assets/node_modules
85
erl_crash.dump
86
IGNORE
87
88
config=$(mktemp /tmp/openagents-cloudbuild.XXXXXX.yaml)
89
trap 'rm -f "$config"' EXIT INT TERM
90
91
cat >"$config" <<'BUILD'
92
steps:
93
  - id: build
94
    name: gcr.io/cloud-builders/docker
95
    args:
96
      - build
97
      - --build-arg
98
      - OPENAGENTS_BUILD_REVISION=${_SHA}
99
      - --build-arg
100
      - OPENAGENTS_RELEASE_VSN=${_RELEASE_VSN}
101
      - --build-arg
102
      - SOURCE_DATE_EPOCH=${_SOURCE_DATE_EPOCH}
103
      - --label
104
      - org.opencontainers.image.revision=${_SHA}
105
      - --tag
106
      - ${_IMAGE}:${_SHA}
107
      - --target
108
      - final
109
      - .
110
111
  # The assertion ops/deploy/build-image.sh makes locally. The erl invocation
112
  # lives in a file so its quoting survives YAML and two shells.
113
  - id: verify-revision
114
    name: gcr.io/cloud-builders/docker
115
    entrypoint: bash
116
    args:
117
      - -c
118
      - |
119
        set -eu
120
        cat > /workspace/packaged-revision.sh <<'CHECK'
121
        release_version=$$(awk '{print $$2}' /app/releases/start_erl.data)
122
        exec /app/erts-*/bin/erl \
123
          -boot_var RELEASE_LIB /app/lib \
124
          -boot "/app/releases/$$release_version/start_clean" \
125
          -noshell \
126
          -pa /app/lib/openagents-*/ebin \
127
          -eval "io:put_chars('Elixir.OpenAgents.BuildInfo':revision()), halt()."
128
        CHECK
129
        packaged=$$(docker run --rm \
130
          -v /workspace:/workspace \
131
          --entrypoint /bin/sh \
132
          "${_IMAGE}:${_SHA}" /workspace/packaged-revision.sh | tail -n 1)
133
        if [ "$$packaged" != "${_SHA}" ]; then
134
          echo "packaged BuildInfo revision '$$packaged' does not match the Git SHA '${_SHA}'" >&2
135
          exit 1
136
        fi
137
        echo "packaged revision matches: $$packaged"
138
139
images:
140
  - ${_IMAGE}:${_SHA}
141
142
options:
143
  machineType: E2_HIGHCPU_8
144
  logging: CLOUD_LOGGING_ONLY
145
146
timeout: 3600s
147
BUILD
148
149
echo "==> building $sha on Cloud Build (project $project)"
150
gcloud builds submit "$repo_root" \
151
    --project "$project" \
152
    --service-account "$service_account" \
153
    --config "$config" \
154
    --substitutions "_SHA=$sha,_RELEASE_VSN=$release_version,_SOURCE_DATE_EPOCH=$source_date_epoch,_IMAGE=$image"
155
156
rm -f "$repo_root/.gcloudignore"
157
echo "Built and pushed $image:$sha"

This page updates live while a promote is in flight · changelog