Define the integrated OpenAgents architecture

fcb27de52c36 · Christopher David · · parent 260174a50511

Define the integrated OpenAgents architecture

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/2026-08-20-integration-hardening-and-staging-readiness-recommendations.md
  • added docs/architecture.md
  • added docs/decisions/0001-integrate-the-complete-public-application.md
  • added docs/decisions/0002-model-sarah-as-an-openagents-persona.md
  • added docs/decisions/0003-keep-provider-credentials-behind-server-adapters.md
  • added docs/decisions/0004-retain-scoped-github-access-tokens.md
  • added docs/decisions/0005-use-basecoat-and-one-component-system.md
  • added docs/decisions/0006-isolate-web-and-distributed-fleet-staging.md
  • added docs/decisions/0007-cut-over-to-forge-canonical-source-control-after-proof.md

Diff

9 files changed, +442 -1

docs/2026-08-20-integration-hardening-and-staging-readiness-recommendations.md modified +19 -1

@@ -169,6 +169,24 @@ Add focused architecture decision records for:

169 169
- Forge-canonical source control and GitHub mirroring, if that remains the
170 170
  intended cutover.
171 171
172
### Gate 1 implementation status
173
174
Completed on 2026-08-20:
175
176
- Added `docs/architecture.md` as the source of truth for product ownership,
177
  durable authority, provider and trust boundaries, forge planes, deployment
178
  strategies, staging topology, and the source-control transition.
179
- Classified public, authenticated, operator, machine, internal-service, and
180
  Git principals. The architecture names Gate 6's exhaustive route ledger as
181
  the enforcement proof instead of claiming that route placement is enough.
182
- Added seven focused decision records for the complete public integration,
183
  Sarah's persona boundary, provider credentials, encrypted GitHub token
184
  retention, the Basecoat component system, isolated staging topology, and the
185
  proof-gated forge-canonical cutover.
186
- Recorded GitHub as the accurate temporary canonical remote during hardening.
187
  The forge cutover cannot occur until its Git, mirror, artifact, rollback, and
188
  recovery gates pass together with updated contributor automation.
189
172 190
**Exit criteria:** A contributor can explain the application and its trust
173 191
boundaries without reading a superseded plan or another repository.
174 192

@@ -1031,7 +1049,7 @@ each handoff.

1031 1049
1032 1050
## Final staging readiness checklist
1033 1051
1034
- [ ] The repository has one accurate architecture narrative.
1052
- [x] The repository has one accurate architecture narrative.
1035 1053
- [ ] Every remaining Sarah reference is intentional and specific.
1036 1054
- [ ] All documentation links and invariant evidence resolve.
1037 1055
- [ ] The application has one Markdown parser, component system, and documented
docs/architecture.md added +201

@@ -0,0 +1,201 @@

1
# OpenAgents architecture
2
3
Date: 2026-08-20
4
5
Status: Current product architecture and accepted hardening target
6
7
## Purpose
8
9
This document is the source of truth for the integrated `openagents.com`
10
application. Use it to understand product ownership, durable authority, trust
11
boundaries, and deployment strategies. Dated migration and implementation plans
12
remain useful as historical records, but they do not override this document.
13
14
OpenAgents is one public AGPL-3.0 application. It includes the source-code
15
forge, issues and projects, the Sarah conversational agent, provider
16
orchestration, memory, voice, delegated work, connected computers, and operator
17
surfaces. The application does not divide these features between a public shell
18
and a private product service.
19
20
## System map
21
22
```text
23
browser
24
  |
25
  | HTTPS, LiveView, WebSocket, or authenticated JSON
26
  v
27
OpenAgents Phoenix application
28
  |-- public and authenticated web surfaces
29
  |-- Sarah conversation, voice, memory, and work orchestration
30
  |-- issues, projects, machines, and data rights
31
  |-- Git HTTP and forge deployment control
32
  |-- server-side provider adapters
33
  |
34
  +--> PostgreSQL: durable product and deployment authority
35
  +--> object/artifact storage: immutable build artifacts
36
  +--> provider APIs: replaceable server-side dependencies
37
  +--> GitHub: identity and user-authorized repository access
38
  `--> BEAM cluster: ephemeral execution and fleet coordination
39
```
40
41
The `OpenAgents` namespace owns generic application infrastructure. Sarah is a
42
persona and behavior package within that application, not a second application
43
or service boundary.
44
45
## Product ownership
46
47
The repository owns these capabilities:
48
49
- Browser presentation and Phoenix LiveView state.
50
- Sarah persona artifacts, behavior, voice, and evaluation material.
51
- Conversation, message, turn, provider-step, and tool-step lifecycles.
52
- Provider selection, request construction, response handling, and usage
53
  accounting.
54
- Tools, memory systems, delegated work, and connected-computer orchestration.
55
- Accounts, data export, deletion, administrative policy, and incidents.
56
- Repositories, Git HTTP, issues, labels, milestones, projects, and public code
57
  browsing.
58
- Build, promotion, deployment, rollback, convergence, and deployment receipts.
59
60
Do not add a private Sarah service as an architectural dependency. A separate
61
provider or infrastructure service can exist behind a documented adapter, but
62
the public application remains responsible for its product contracts and data
63
rights.
64
65
## Durable authority
66
67
PostgreSQL is the durable authority for product state, authorization state,
68
repository metadata, work state, fleet targets, and receipts. A transaction
69
that does not commit cannot become product truth.
70
71
Treat these systems as projections or execution aids:
72
73
- LiveView socket assigns and browser state.
74
- Phoenix PubSub messages.
75
- BEAM registries, supervisors, tasks, and process mailboxes.
76
- In-memory caches and `:persistent_term` values.
77
- Network and deployment status pages.
78
- Local build and artifact caches.
79
80
Immutable artifacts can live in durable object storage, but PostgreSQL records
81
their identity, digest, lifecycle, and authorized target. Reconcile an
82
ephemeral projection from durable state after restarts or missed events.
83
84
## Trust boundaries
85
86
Every route belongs to one authority class. Gate 6 of the hardening plan owns
87
the exhaustive route ledger and enforcement tests.
88
89
| Class | Principal | Examples | Required enforcement |
90
| --- | --- | --- | --- |
91
| Public | Anonymous visitor | Home, docs, status, changelog, allowed public code | Read-only behavior, bounded output, and visibility policy |
92
| Authenticated | Active OpenAgents user | Chat, data rights, computers, issues, projects | Signed session, CSRF protection for browser mutations, and owner or repository scope |
93
| Operator | Configured active administrator | Administration, promotion, deployment, recording review | Authenticated session plus server-side operator authorization on every action |
94
| Machine | Paired machine credential | Controller socket, presence, and agent jobs | Hashed or encrypted scoped token, explicit machine ownership, rotation, and revocation |
95
| Internal service | Configured service identity or signed grant | Inference proxy and build/deployment adapters | Narrow audience and scope, expiry, replay defense, and no browser-held service secret |
96
| Git | Machine or operator Git credential | `/git` fetch and push | HTTP Basic transport with server-side token verification and repository authorization |
97
98
Route placement does not prove authorization. A controller, LiveView mount, or
99
socket must enforce the class and resource scope at the server boundary. Until
100
the Gate 6 route audit passes, treat the current router as implementation
101
evidence rather than a complete authorization policy.
102
103
## Provider boundary
104
105
Provider adapters implement replaceable server-side behavior. Direct OpenAI
106
integration is the current default adapter choice for text, embeddings, shadow
107
programs, and voice. It is not an application-wide contract.
108
109
The browser can receive bounded provider-derived events and media negotiation
110
results, but it must never receive an OpenAI API key, GitHub access token,
111
forge operator token, machine token, recording key, or internal-service signing
112
key. Adapters own transport details, timeouts, bounded retries, response
113
validation, error normalization, and secret redaction. Product contexts own
114
durable lifecycle and policy.
115
116
Tests replace network providers with explicit fakes. A provider outage must
117
produce a bounded durable failure outcome instead of abandoning an in-flight
118
turn, work item, or voice session.
119
120
## Forge planes
121
122
The forge contains two separate planes within the `OpenAgents.Forge`
123
application namespace:
124
125
- The Git plane accepts authenticated Git traffic, stores repositories and
126
  push receipts, controls visibility, serves public browsing, and mirrors refs.
127
- The deployment plane promotes an exact pushed SHA, builds immutable
128
  artifacts, selects a deployment strategy, changes fleet state, rolls back,
129
  converges restarted nodes, and records receipts.
130
131
A push never promotes itself. Git repository availability does not imply that
132
a build is safe, and a successful build does not imply that a candidate is
133
live. PostgreSQL transitions connect the planes through an operator-authorized
134
target.
135
136
## Deployment strategies
137
138
Direct BEAM load, relup, and rolling replacement solve different problems. The
139
classifier must select one strategy for the complete candidate and fail closed
140
when it cannot prove eligibility.
141
142
| Strategy | Eligible changes | Required safety proof |
143
| --- | --- | --- |
144
| Direct BEAM load | Allowlisted module additions or changes without structural effects | Immutable manifest and digest, fleet prepare/apply/verify transaction, exact binary rollback, and readiness verification |
145
| Relup | Compatible versioned application changes and explicit process-state migrations | Forward and reverse appup/relup chain, state migration tests, staged install, permanent-release verification, and rollback drill |
146
| Rolling replacement | ERTS, OTP, NIF, dependency graph, assets, configuration, migrations, module deletion, or unclassified changes | Immutable image digest, node drain, readiness-gated replacement, capacity limits, and last-known-good rollback |
147
148
Development code reloading is not a production deployment strategy. Keep every
149
deployment capability disabled by default until its local proof and isolated
150
staging drill pass.
151
152
## Runtime and staging topology
153
154
The accepted target has two isolated staging lanes:
155
156
- A web lane proves Phoenix, LiveView, PostgreSQL, authentication, chat, voice,
157
  data rights, and provider behavior without distributed deployment enabled.
158
- A three-node distributed lane proves Ra quorum, Git, immutable builds, direct
159
  loading, relups, rolling replacement, rollback, and boot convergence.
160
161
Both lanes use staging-only hosts, credentials, buckets, repositories, service
162
accounts, and a PostgreSQL instance that does not share a production failure
163
domain. Production is out of scope until all hardening gates, the complete
164
staging matrix, failure injection, and the 48-hour soak pass.
165
166
## Source control transition
167
168
GitHub remains the repository's temporary canonical remote during staging
169
hardening. The self-hosted forge becomes canonical only through an explicit
170
cutover after its Git, mirror, artifact, rollback, and recovery gates pass. The
171
cutover changes contributor instructions and push automation in the same
172
candidate. After cutover, the forge pushes a read-only GitHub mirror and direct
173
GitHub pushes become invalid.
174
175
Do not describe the cutover as complete while contributor clones and automated
176
pushes still target GitHub.
177
178
## Decision records
179
180
- [ADR 0001: Integrate the complete product in the public application](decisions/0001-integrate-the-complete-public-application.md)
181
- [ADR 0002: Model Sarah as an OpenAgents persona](decisions/0002-model-sarah-as-an-openagents-persona.md)
182
- [ADR 0003: Keep provider credentials behind server adapters](decisions/0003-keep-provider-credentials-behind-server-adapters.md)
183
- [ADR 0004: Retain scoped GitHub access tokens in the server vault](decisions/0004-retain-scoped-github-access-tokens.md)
184
- [ADR 0005: Use Basecoat and one OpenAgents component system](decisions/0005-use-basecoat-and-one-component-system.md)
185
- [ADR 0006: Isolate web and distributed fleet staging](decisions/0006-isolate-web-and-distributed-fleet-staging.md)
186
- [ADR 0007: Cut over to forge-canonical source control only after proof](decisions/0007-cut-over-to-forge-canonical-source-control-after-proof.md)
187
188
## Superseded narratives
189
190
The following documents record earlier plans and measurements. They do not
191
define the current architecture:
192
193
- `docs/chat-inference-plan.md` proposed a private Sarah service boundary that
194
  the integrated application does not use.
195
- `docs/sarah-integration-plan.md` records the migration into this repository.
196
- `docs/2026-08-19-gap-implementation-plan.md` records earlier gap work.
197
- `docs/beam-hot-deployment-plan.md` contains detailed deployment design, but
198
  its phase status does not authorize production use.
199
200
The [integration hardening and staging readiness plan](2026-08-20-integration-hardening-and-staging-readiness-recommendations.md)
201
tracks the work required to make this architecture safe and verifiable.
docs/decisions/0001-integrate-the-complete-public-application.md added +31

@@ -0,0 +1,31 @@

1
# ADR 0001: Integrate the complete product in the public application
2
3
Date: 2026-08-20
4
5
Status: Accepted
6
7
## Context
8
9
Earlier plans divided a public web shell from private Sarah-specific chat,
10
voice, persona, and provider behavior. The repository now contains and operates
11
those product capabilities. Maintaining a fictional service split obscures
12
data ownership, weakens contributor understanding, and leaves public contracts
13
dependent on undocumented code.
14
15
## Decision
16
17
Keep the complete OpenAgents product in this AGPL-3.0 repository. The
18
`openagents.com` application owns the web interface, Sarah behavior,
19
conversation lifecycle, providers, tools, memory, delegated work, voice,
20
machines, issues, projects, forge, data rights, and operator surfaces.
21
22
Use external services only through explicit infrastructure or provider
23
adapters. Do not move product policy or Sarah behavior behind an undocumented
24
private API.
25
26
## Consequences
27
28
- Contributors can inspect, test, and modify every product contract.
29
- One application owns data-rights behavior and durable lifecycle recovery.
30
- Runtime secrets remain private even though orchestration code is public.
31
- Earlier public-shell and private-Sarah plans are superseded.
docs/decisions/0002-model-sarah-as-an-openagents-persona.md added +31

@@ -0,0 +1,31 @@

1
# ADR 0002: Model Sarah as an OpenAgents persona
2
3
Date: 2026-08-20
4
5
Status: Accepted
6
7
## Context
8
9
Sarah has a distinct identity, voice, behavior contract, persona artifacts,
10
and evaluation material. Generic runtime infrastructure also inherited Sarah
11
names during integration. Treating Sarah as either a separate application or a
12
name for all infrastructure makes ownership unclear.
13
14
## Decision
15
16
Model Sarah as a persona and behavior package inside the `OpenAgents`
17
application. Keep Sarah names when identity is part of the contract, including
18
persona artifact IDs, behavior revisions, evaluations, visible identity, and
19
voice copy.
20
21
Use OpenAgents names for generic supervisors, web helpers, style packs,
22
runtime paths, build services, test cases, and configuration. Gate 2 performs
23
that semantic rename and adds an allowlist for intentional references.
24
25
## Consequences
26
27
- The product can add other personas without duplicating application
28
  infrastructure.
29
- Persona-specific history and provenance remain accurate.
30
- Generic code no longer implies that all OpenAgents behavior belongs to one
31
  persona.
docs/decisions/0003-keep-provider-credentials-behind-server-adapters.md added +30

@@ -0,0 +1,30 @@

1
# ADR 0003: Keep provider credentials behind server adapters
2
3
Date: 2026-08-20
4
5
Status: Accepted
6
7
## Context
8
9
Text inference, embeddings, shadow programs, and realtime voice currently use
10
OpenAI implementations. Product lifecycle code must not depend on one
11
provider's transport, and browsers must not hold provider credentials.
12
13
## Decision
14
15
Define provider behavior through server-side adapters. Keep provider request
16
construction, credentials, transport, timeouts, bounded retries, response
17
validation, and error normalization behind those adapters. Keep durable turn,
18
voice, memory, and work policy in provider-independent OpenAgents contexts.
19
20
Never send a provider API key or internal-service credential to the browser.
21
Use explicit fake adapters for tests, and require adapters to redact secrets
22
from logs, receipts, telemetry, and user-visible errors.
23
24
## Consequences
25
26
- Direct OpenAI use remains an adapter choice rather than a product boundary.
27
- The application can replace a provider without rewriting durable lifecycle
28
  code.
29
- Browser compromise does not expose server credentials.
30
- Adapter contracts need conformance, timeout, and failure-path tests.
docs/decisions/0004-retain-scoped-github-access-tokens.md added +32

@@ -0,0 +1,32 @@

1
# ADR 0004: Retain scoped GitHub access tokens in the server vault
2
3
Date: 2026-08-20
4
5
Status: Accepted; hardening required before staging
6
7
## Context
8
9
GitHub provides sign-in identity and user-authorized repository access. The
10
repository already encrypts OAuth access tokens so server-side repository tools
11
can act with the user's authority. Other documents incorrectly say that the
12
callback discards the token.
13
14
## Decision
15
16
Retain the minimum-scoped GitHub access token in the server-side token vault
17
for features that require GitHub repository access. Encrypt every token with an
18
operator-managed key, associate it with one active user, and never expose it to
19
LiveView assigns, browser payloads, logs, receipts, or telemetry.
20
21
Keep OpenAgents issue and project data in PostgreSQL; do not use a retained
22
GitHub token as authority for OpenAgents-owned records. Revoke or delete the
23
token when the user disconnects GitHub, deletes their data, or loses account
24
access. Gate 6 must verify scopes, rotation, failure behavior, redaction, and
25
the disclosures shown to users.
26
27
## Consequences
28
29
- Authenticated repository tools can use the user's delegated GitHub rights.
30
- Token retention becomes an explicit data-handling obligation.
31
- Sign-in identity and repository authorization remain separate decisions.
32
- Documentation and deletion paths must describe the retained credential.
docs/decisions/0005-use-basecoat-and-one-component-system.md added +31

@@ -0,0 +1,31 @@

1
# ADR 0005: Use Basecoat and one OpenAgents component system
2
3
Date: 2026-08-20
4
5
Status: Accepted
6
7
## Context
8
9
The application previously combined DaisyUI, hand-written classes, and
10
persona-named component wrappers. Conflicting cascade layers made variants
11
render incorrectly and created multiple ways to build the same control.
12
13
## Decision
14
15
Use the pinned, vendored Basecoat component CSS as the structural foundation
16
and one OpenAgents style pack for product identity. Import only the Basecoat
17
component files that a surface uses. Wrap the system in one Phoenix component
18
module and use those components before adding surface-specific markup.
19
20
Do not add DaisyUI, another component library, remote fonts, icon fonts, or a
21
second Markdown presentation stack. Rename the generic `SarahUI` module and
22
style pack during Gate 2 without changing Sarah-specific visual copy.
23
24
## Consequences
25
26
- Components share one variant, accessibility, token, and interaction model.
27
- Vendored Basecoat remains separately licensed and must not be patched in
28
  place.
29
- The application owns its identity in the style pack rather than in vendored
30
  source.
31
- Component and screenshot regression tests can target one catalog.
docs/decisions/0006-isolate-web-and-distributed-fleet-staging.md added +35

@@ -0,0 +1,35 @@

1
# ADR 0006: Isolate web and distributed fleet staging
2
3
Date: 2026-08-20
4
5
Status: Accepted; implementation pending
6
7
## Context
8
9
Application regressions and distributed deployment failures need different
10
proof environments. Failure injection, database migration rehearsal, and
11
rolling replacement are unsafe when staging shares production data,
12
credentials, infrastructure, or a failure domain.
13
14
## Decision
15
16
Provision two staging lanes:
17
18
- A web lane for Phoenix, LiveView, authentication, chat, voice, data rights,
19
  PostgreSQL, and provider integration.
20
- A three-node distributed lane for Ra quorum, Git, builds, direct BEAM load,
21
  relup, rolling replacement, rollback, and boot convergence.
22
23
Give both lanes staging-only hosts, secrets, service accounts, buckets,
24
repositories, and a PostgreSQL instance that is separate from production.
25
Disable production promotion until both lanes pass the complete matrix,
26
failure injection, and a 48-hour soak on one exact candidate.
27
28
## Consequences
29
30
- Web regressions do not require a distributed deployment experiment.
31
- Fleet tests can terminate nodes and corrupt disposable artifacts without
32
  risking production.
33
- Staging costs more than sharing production infrastructure.
34
- Gate receipts must name the lane, exact source SHA, artifact and image
35
  digests, migration version, and staging revision without recording secrets.
docs/decisions/0007-cut-over-to-forge-canonical-source-control-after-proof.md added +32

@@ -0,0 +1,32 @@

1
# ADR 0007: Cut over to forge-canonical source control after proof
2
3
Date: 2026-08-20
4
5
Status: Accepted; cutover pending
6
7
## Context
8
9
The product intends to host its canonical Git repositories on the OpenAgents
10
forge and keep GitHub as a discoverable read-only mirror. This repository still
11
uses GitHub as its configured canonical remote during staging hardening. A
12
documentation-only cutover would split contributor and deployment state.
13
14
## Decision
15
16
Keep GitHub canonical until the self-hosted Git plane passes authentication,
17
authorization, durability, mirror, restore, and operational recovery gates.
18
Perform the cutover as one controlled change that updates contributor
19
instructions, operator automation, build source, deployment promotion, and
20
mirror monitoring.
21
22
After cutover, accept pushes only through the forge for normal operation. Push
23
a one-way read-only mirror to GitHub, monitor mirror lag, and treat a direct
24
GitHub push as an incident. Do not let a mirror push promote a deployment.
25
26
## Consequences
27
28
- Current contributors keep one accurate remote during hardening.
29
- The future cutover has explicit prerequisites and rollback evidence.
30
- GitHub remains available for discovery without becoming a second writable
31
  authority.
32
- The application must report canonical-source and mirror state separately.

This page updates live while a promote is in flight · changelog