Make repository imports fast and observable

a40a799818a2 · AtlantisPleb · · parent 56b6a169b5fe

Make repository imports fast and observable

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified config/runtime.exs
  • modified docs/openagents-cli/command-reference.md
  • modified docs/openagents-cli/create-repository.md
  • modified docs/openagents-cli/import-github.md
  • modified docs/repository-creation-and-openagents-cli-spec.md
  • modified docs/repository-creation-cli-implementation-roadmap.md
  • modified lib/openagents/forge/sync.ex
  • modified lib/openagents/repositories/importer.ex
  • modified priv/docs/cli-command-reference.md
  • modified priv/docs/create-repository.md
  • modified priv/docs/import-github.md
  • modified test/openagents/repositories/provisioner_test.exs

Diff

12 files changed, +215 -78

config/runtime.exs modified +1 -1

@@ -360,7 +360,7 @@ if config_env() == :prod and runtime_role == :web do

360 360
    inference_proxy_url: optional_text.("OPENAGENTS_INFERENCE_PROXY_URL"),
361 361
    forge_enabled: forge_enabled,
362 362
    forge_deploy_lane_enabled: forge_deploy_enabled,
363
    repository_provisioner_enabled: forge_deploy_enabled,
363
    repository_provisioner_enabled: forge_enabled,
364 364
    forge_boot_converge_enabled: boot_convergence_enabled,
365 365
    forge_rolling_provider: forge_rolling_provider,
366 366
    forge_repos: forge_repos,
docs/openagents-cli/command-reference.md modified +4 -1

@@ -85,7 +85,10 @@ openagents repo import [flags] <github-owner/repository>

85 85
| `--private` | Create a private destination, which is the default. |
86 86
| `--wait-timeout SECONDS` | Wait for import. The default is `300`; `0` does not wait. |
87 87
88
This command performs one import. It does not start synchronization.
88
This command performs one depth-1 import of every accepted branch and tag. It
89
does not copy older history or start synchronization. While create and import
90
commands wait, they write state changes, elapsed time, and a five-second
91
heartbeat to standard error.
89 92
90 93
### `repo list`
91 94
docs/openagents-cli/create-repository.md modified +3 -1

@@ -42,7 +42,9 @@ openagents repo create my-project \

42 42
43 43
The CLI waits up to 300 seconds for provisioning by default. Pass
44 44
`--wait-timeout 0` to return after the server accepts the durable request. The
45
repository continues provisioning on the server.
45
repository continues provisioning on the server. While it waits, the CLI
46
writes the current lifecycle state, elapsed time, and a five-second heartbeat
47
to standard error.
46 48
47 49
Run one create command without a global installation:
48 50
docs/openagents-cli/import-github.md modified +26 -19

@@ -62,9 +62,10 @@ openagents repo import OpenAgentsInc/example --namespace OpenAgentsInc

62 62
The `--namespace` value must match the GitHub source owner in this release. You
63 63
cannot import `SOURCE/repository` directly into an unrelated namespace.
64 64
65
The CLI waits up to 300 seconds by default. Pass `--wait-timeout 0` to return
66
after the server accepts the durable import. A client timeout does not cancel
67
the server-side import.
65
The CLI waits up to 300 seconds by default. It writes state changes, elapsed
66
time, and a five-second heartbeat to standard error while it waits. Pass
67
`--wait-timeout 0` to return after the server accepts the durable import.
68
A client timeout does not cancel the server-side import.
68 69
69 70
Run one import without a global installation:
70 71

@@ -75,7 +76,7 @@ npx --yes @openagentsinc/cli@latest repo import OWNER/REPOSITORY

75 76
Pin the package version for a reproducible qualification run:
76 77
77 78
```sh
78
npx --yes @openagentsinc/cli@0.1.4 \
79
npx --yes @openagentsinc/cli@0.1.5 \
79 80
  --profile staging \
80 81
  repo import OWNER/REPOSITORY \
81 82
  --private \

@@ -96,10 +97,16 @@ Git helper cannot call the temporary executable after `npx` exits.

96 97
97 98
## Import a large repository
98 99
99
OpenAgents keeps large Git bundles on disk and streams them to and from the
100
durable forge WAL in 1 MiB chunks. The application does not read the complete
101
bundle into the BEAM heap. The default server limits allow a bundle up to 20
102
GiB and an import to run for up to six hours.
100
OpenAgents imports every accepted branch and tag at depth 1 by default. This
101
shallow snapshot preserves each current tip and its files without copying the
102
source repository's full history. It makes repositories with years of history
103
available much faster and bounds the first transfer by current content rather
104
than commit count.
105
106
OpenAgents keeps the resulting Git bundle on disk and streams it to and from
107
the durable forge WAL in 1 MiB chunks. The application does not read the
108
complete bundle into the BEAM heap. The default server limits allow a bundle
109
up to 20 GiB and an import to run for up to six hours.
103 110
104 111
The CLI's `--wait-timeout` controls only how long that client waits. It does
105 112
not change or cancel the server import. For a large repository, accept the

@@ -115,11 +122,8 @@ Server logs record every stage and the bundle byte count. A bundle over the

115 122
server limit fails with `import_too_large`; an operation over the server time
116 123
limit fails with `import_timeout`.
117 124
118
Large imports still need enough temporary disk for the fetched Git objects and
119
the bundle. Git LFS objects remain outside the import. Start an environment's
120
first import with a small repository so you can qualify authentication,
121
lifecycle reporting, storage, and clone behavior before spending the time and
122
bandwidth of a large transfer.
125
Large imports still need enough temporary disk for the shallow Git objects and
126
the bundle. Git LFS objects remain outside the import.
123 127
124 128
## Verify the first production import
125 129

@@ -131,7 +135,7 @@ pointer files without copying the LFS objects.

131 135
1. Install the qualified CLI version:
132 136
133 137
   ```sh
134
   npm install --global @openagentsinc/cli@0.1.4
138
   npm install --global @openagentsinc/cli@0.1.5
135 139
   ```
136 140
137 141
2. Sign in to production and confirm the selected account:

@@ -154,8 +158,9 @@ pointer files without copying the LFS objects.

154 158
   openagents --profile production repo clone OWNER/REPOSITORY
155 159
   ```
156 160
157
5. Compare the cloned branches and tags with the accepted GitHub snapshot.
158
   Confirm that a later GitHub commit does not appear in the OpenAgents copy.
161
5. Compare the cloned branch and tag tips with the accepted GitHub snapshot.
162
   Confirm that the clone contains one commit of history per imported tip and
163
   that a later GitHub commit does not appear in the OpenAgents copy.
159 164
160 165
The release process does not create a production repository automatically.
161 166
An authenticated operator starts the first production import explicitly.

@@ -164,15 +169,17 @@ An authenticated operator starts the first production import explicitly.

164 169
165 170
| Copied | Not copied |
166 171
| --- | --- |
167
| Git commit and object history reachable from accepted refs | GitHub Issues |
172
| Current commit and file tree at every accepted ref, with depth 1 | Full Git history before each accepted tip |
168 173
| `refs/heads/*` branches | Pull requests and reviews |
169 174
| `refs/tags/*` tags | Actions workflows, runs, and secrets |
170 175
| The source default branch | Releases and repository settings |
171 176
| Submodule pointer commits | Wikis and Git LFS objects |
172 177
173 178
OpenAgents freezes the accepted branch and tag map before copying data. It
174
verifies that same ref snapshot before marking the repository ready. A GitHub
175
commit created after acceptance is not part of the import.
179
verifies that same ref snapshot before marking the repository ready. The
180
destination records those commits as shallow boundaries, so normal cloning and
181
new commits work without the omitted ancestry. A GitHub commit created after
182
acceptance is not part of the import.
176 183
177 184
Git LFS pointer files remain in Git history, but OpenAgents does not copy the
178 185
referenced LFS objects. Download or migrate those objects separately before
docs/repository-creation-and-openagents-cli-spec.md modified +22 -19

@@ -99,8 +99,9 @@ The first release follows these decisions:

99 99
  commands but does not replace Git.
100 100
- The CLI uses the server-provided clone URL instead of constructing one from a
101 101
  hard-coded host.
102
- The first release imports Git repository history, branches, and tags from
103
  GitHub as a one-time copy. It does not maintain a mirror or two-way sync.
102
- The first release imports the current tip of every GitHub branch and tag at
103
  depth 1 as a one-time copy. It does not copy older ancestry or maintain a
104
  mirror or two-way sync.
104 105
- Pull requests, rulesets, SSH keys, apps, and OpenAgents-native team sharing
105 106
  remain future slices.
106 107

@@ -305,16 +306,17 @@ The record never stores a GitHub access token, authenticated clone URL, local

305 306
path, raw Git diagnostic, or repository content.
306 307
307 308
At import acceptance, resolve the source through the GitHub API and freeze the
308
advertised `refs/heads/*` and `refs/tags/*` map. The import copies the objects
309
reachable from that map and verifies the same ref digest before it marks the
310
destination ready. GitHub changes after the accepted snapshot are not part of
311
the import.
312
313
The first release imports standard Git history, branches, tags, the default
314
branch, and submodule pointer commits. It does not import GitHub Issues, pull
315
requests, reviews, Actions runs or secrets, releases, repository settings,
316
wikis, or Git LFS objects. Git LFS pointer files remain ordinary Git content;
317
the UI and CLI must warn when the source uses LFS.
309
advertised `refs/heads/*` and `refs/tags/*` map. The import fetches each tip at
310
depth 1, copies the tip's commit and file-tree objects, preserves the shallow
311
boundaries in the destination, and verifies the same ref digest before it marks
312
the destination ready. GitHub changes after the accepted snapshot are not part
313
of the import.
314
315
The first release imports branches, tags, the default branch, current file
316
trees, and submodule pointers at the accepted tips. It does not import older
317
Git ancestry, GitHub Issues, pull requests, reviews, Actions runs or secrets,
318
releases, repository settings, wikis, or Git LFS objects. Git LFS pointer files
319
remain ordinary Git content; the UI and CLI must warn when the source uses LFS.
318 320
319 321
### Authority split
320 322

@@ -352,11 +354,11 @@ a transactional outbox and an idempotent provisioner.

352 354
   outbox record.
353 355
4. Commit before any filesystem, Git, or object-store operation begins.
354 356
5. Initialize an empty durable WAL namespace using `repository.storage_key`.
355
6. For an import, fetch the accepted GitHub refs into an isolated temporary
356
   repository, verify the frozen ref digest, and ingest the accepted objects and
357
   refs into the destination WAL. Use a server-owned credential adapter that
358
   never places the GitHub token in a URL, argv, log, receipt, or repository
359
   configuration.
357
6. For an import, fetch the accepted GitHub refs at depth 1 into an isolated
358
   temporary repository, verify the frozen ref digest, and ingest the accepted
359
   objects, shallow boundaries, and refs into the destination WAL. Use a
360
   server-owned credential adapter that never places the GitHub token in a URL,
361
   argv, log, receipt, or repository configuration.
360 362
7. Materialize or initialize the bare-repository cache with the selected
361 363
   symbolic default branch.
362 364
8. Verify that upload-pack and receive-pack resolve the same repository UUID.

@@ -1250,7 +1252,7 @@ The first repository and import slice is complete when:

1250 1252
  as the repository row.
1251 1253
- A repeat request with the same idempotency key cannot create a duplicate.
1252 1254
- The browser and CLI can import an authorized GitHub repository's accepted
1253
  history, branches, and tags into the matching GitHub namespace.
1255
  branch and tag tips at depth 1 into the matching GitHub namespace.
1254 1256
- An imported repository records its source and exact accepted snapshot without
1255 1257
  retaining a GitHub credential.
1256 1258
- A GitHub commit created after import acceptance does not appear in OpenAgents

@@ -1258,7 +1260,8 @@ The first repository and import slice is complete when:

1258 1260
- A public repository clones anonymously.
1259 1261
- A private repository is hidden from a nonmember.
1260 1262
- A permitted user can push and a read-only or unrelated user cannot.
1261
- Deleting the node-local bare cache does not lose accepted Git history.
1263
- Deleting the node-local bare cache does not lose the accepted shallow
1264
  snapshot or later OpenAgents history.
1262 1265
- Creating a repository cannot make it deployable.
1263 1266
- The CLI works in attended and noninteractive modes without placing a token in
1264 1267
  argv, logs, JSON, or a clone URL.
docs/repository-creation-cli-implementation-roadmap.md modified +5 -5

@@ -54,7 +54,7 @@ All of these statements require direct evidence before this roadmap can move to

54 54
- The browser and CLI create private or public repositories through the same
55 55
  Phoenix context operation.
56 56
- The browser and CLI import an authorized GitHub repository once, including
57
  its accepted branches, tags, and reachable Git objects.
57
  its accepted branch and tag tips and depth-1 Git objects.
58 58
- A later GitHub update does not change the imported OpenAgents repository.
59 59
- The repository API returns stable JSON and error envelopes from a versioned
60 60
  Phoenix-owned contract artifact.

@@ -251,14 +251,14 @@ does not change deployment targets.

251 251
1. Resolve and persist the immutable GitHub source repository and owner IDs,
252 252
   default branch, permissions, branch and tag map, ref digest, and default head
253 253
   before the database transaction.
254
2. Fetch the accepted refs into a unique temporary bare repository.
254
2. Fetch the accepted refs at depth 1 into a unique temporary bare repository.
255 255
3. Supply the retained GitHub token through a server-owned Git credential
256 256
   callback or askpass boundary. Never place it in a URL, argv, environment
257 257
   dump, Git config, log, or import record.
258 258
4. Verify that the fetched refs match the accepted digest. Fail with a stable
259 259
   source-change code if the snapshot cannot be reproduced.
260
5. Convert the imported objects and refs into the destination WAL and
261
   materialize the destination cache.
260
5. Convert the imported objects, shallow boundaries, and refs into the
261
   destination WAL and materialize the destination cache.
262 262
6. Set the symbolic default branch, accepted head SHA, import completion time,
263 263
   and repository readiness atomically at the final database transition.
264 264
7. Remove temporary workspaces after success, expected failure, interruption,

@@ -460,7 +460,7 @@ result. Do not store tokens, local absolute paths, or repository content.

460 460
| 2. GitHub namespace projection | Complete | GitHub adapter, exact `repo` and `read:org` scopes, projection tests |
461 461
| 3. Repository REST API | Complete | Repository controllers, JSON contract, namespace quota, idempotency, and pagination tests |
462 462
| 4. Durable provisioning | Complete | Provisioning outbox, reclaimable worker, bounded import timeout, crash-workspace recovery, audit events, and cache reconstruction tests |
463
| 5. One-time GitHub import | Complete | Frozen refs, Git bundle WAL entry, cache-loss and no-later-sync tests |
463
| 5. One-time GitHub import | Complete | Frozen refs, depth-1 Git bundle WAL entry, preserved shallow boundaries, cache-loss, and no-later-sync tests |
464 464
| 6. Repository Git HTTP authorization | Complete | Public reads, PAT writes, token reauthentication, role refusal, explicit machine grants, audited writes, and legacy-route tests |
465 465
| 7. CLI device authorization | Complete | One-time device-code context, API, browser approval, and polling tests |
466 466
| 8. Repository browser interface | Complete | Paginated list, create, import, empty, failed, private, and code-route LiveView tests |
lib/openagents/forge/sync.ex modified +31 -3

@@ -57,7 +57,7 @@ defmodule OpenAgents.Forge.Sync do

57 57
        {_output, 0} = run_receive_pack(path, payload)
58 58
59 59
      "git_bundle" ->
60
        :ok = unbundle_entry(repo, path, object)
60
        :ok = unbundle_entry(repo, path, object, entry["shallow"] || [])
61 61
62 62
      "empty_import" ->
63 63
        :ok

@@ -77,7 +77,7 @@ defmodule OpenAgents.Forge.Sync do

77 77
    end
78 78
  end
79 79
80
  defp unbundle_entry(repo, path, object) do
80
  defp unbundle_entry(repo, path, object, shallow_boundaries) do
81 81
    temporary_path =
82 82
      Path.join(
83 83
        Application.get_env(:openagents, :repository_import_temp_dir, System.tmp_dir!()),

@@ -88,7 +88,7 @@ defmodule OpenAgents.Forge.Sync do

88 88
      with :ok <- WAL.get_entry_file(repo, object, temporary_path),
89 89
           :ok <- File.chmod(temporary_path, 0o600) do
90 90
        case Repos.git(path, ["bundle", "unbundle", temporary_path]) do
91
          {_output, 0} -> :ok
91
          {_output, 0} -> write_shallow_boundaries(path, shallow_boundaries)
92 92
          {_output, _status} -> raise "repository bundle could not be materialized"
93 93
        end
94 94
      end

@@ -97,6 +97,34 @@ defmodule OpenAgents.Forge.Sync do

97 97
    end
98 98
  end
99 99
100
  defp write_shallow_boundaries(path, []) do
101
    shallow_path = Path.join(path, "shallow")
102
103
    case File.rm(shallow_path) do
104
      :ok -> :ok
105
      {:error, :enoent} -> :ok
106
      {:error, reason} -> raise File.Error, reason: reason, action: "remove", path: shallow_path
107
    end
108
  end
109
110
  defp write_shallow_boundaries(path, shallow_boundaries) do
111
    valid? =
112
      Enum.all?(shallow_boundaries, fn boundary ->
113
        is_binary(boundary) and Regex.match?(~r/\A[0-9a-f]{40,64}\z/, boundary)
114
      end)
115
116
    if valid? do
117
      File.write!(
118
        Path.join(path, "shallow"),
119
        Enum.join(shallow_boundaries, "\n") <> "\n"
120
      )
121
122
      :ok
123
    else
124
      raise "repository bundle has invalid shallow boundaries"
125
    end
126
  end
127
100 128
  defp run_receive_pack(path, payload) do
101 129
    GitHTTP.run_git_service("receive-pack", [path], payload, nil)
102 130
  end
lib/openagents/repositories/importer.ex modified +78 -8

@@ -146,14 +146,22 @@ defmodule OpenAgents.Repositories.Importer do

146 146
           import_stage(repository, repository_import, "verify_snapshot", fn ->
147 147
             verify_snapshot(source_repository, repository_import)
148 148
           end),
149
         {:ok, payload, format, payload_bytes} <-
149
         {:ok, payload, format, payload_bytes, shallow_boundaries} <-
150 150
           import_stage(repository, repository_import, "create_payload", fn ->
151 151
             create_payload(source_repository, refs, temporary_directory)
152 152
           end),
153 153
         :ok <- log_payload_ready(repository, repository_import, payload_bytes),
154 154
         :ok <-
155 155
           import_stage(repository, repository_import, "append_wal", fn ->
156
             append_import(repository, repository_import, payload, format, refs, 0)
156
             append_import(
157
               repository,
158
               repository_import,
159
               payload,
160
               format,
161
               refs,
162
               shallow_boundaries,
163
               0
164
             )
157 165
           end),
158 166
         :ok <-
159 167
           import_stage(repository, repository_import, "materialize_cache", fn ->

@@ -203,6 +211,8 @@ defmodule OpenAgents.Repositories.Importer do

203 211
        "fetch",
204 212
        "--force",
205 213
        "--prune",
214
        "--depth=1",
215
        "--no-tags",
206 216
        "--no-recurse-submodules",
207 217
        source_url,
208 218
        "+refs/heads/*:refs/heads/*",

@@ -286,7 +296,7 @@ defmodule OpenAgents.Repositories.Importer do

286 296
  end
287 297
288 298
  defp create_payload(_source_repository, refs, _temporary_directory) when map_size(refs) == 0,
289
    do: {:ok, "", "empty_import", 0}
299
    do: {:ok, "", "empty_import", 0, []}
290 300
291 301
  defp create_payload(source_repository, _refs, temporary_directory) do
292 302
    bundle_path = Path.join(temporary_directory, "snapshot.bundle")

@@ -296,7 +306,8 @@ defmodule OpenAgents.Repositories.Importer do

296 306
        case File.stat(bundle_path) do
297 307
          {:ok, %File.Stat{type: :regular, size: size}} ->
298 308
            if size <= maximum_bundle_bytes() do
299
              {:ok, {:file, bundle_path}, "git_bundle", size}
309
              {:ok, {:file, bundle_path}, "git_bundle", size,
310
               shallow_boundaries(source_repository)}
300 311
            else
301 312
              {:error, :import_too_large}
302 313
            end

@@ -319,27 +330,58 @@ defmodule OpenAgents.Repositories.Importer do

319 330
         _payload,
320 331
         _format,
321 332
         _refs,
333
         _shallow_boundaries,
322 334
         attempt
323 335
       )
324 336
       when attempt >= @maximum_append_attempts,
325 337
       do: {:error, :wal_cas_conflict}
326 338
327
  defp append_import(repository, repository_import, payload, format, refs, attempt) do
339
  defp append_import(
340
         repository,
341
         repository_import,
342
         payload,
343
         format,
344
         refs,
345
         shallow_boundaries,
346
         attempt
347
       ) do
328 348
    result =
329 349
      :global.trans({{:repository_import, repository.storage_key}, self()}, fn ->
330
        append_import_once(repository, repository_import, payload, format, refs)
350
        append_import_once(
351
          repository,
352
          repository_import,
353
          payload,
354
          format,
355
          refs,
356
          shallow_boundaries
357
        )
331 358
      end)
332 359
333 360
    case result do
334 361
      {:error, :cas_conflict} ->
335
        append_import(repository, repository_import, payload, format, refs, attempt + 1)
362
        append_import(
363
          repository,
364
          repository_import,
365
          payload,
366
          format,
367
          refs,
368
          shallow_boundaries,
369
          attempt + 1
370
        )
336 371
337 372
      other ->
338 373
        other
339 374
    end
340 375
  end
341 376
342
  defp append_import_once(repository, repository_import, payload, format, refs) do
377
  defp append_import_once(
378
         repository,
379
         repository_import,
380
         payload,
381
         format,
382
         refs,
383
         shallow_boundaries
384
       ) do
343 385
    with {:ok, expected, index} <- read_or_create_index(repository.storage_key),
344 386
         :missing <- import_entry(index, repository_import.id),
345 387
         true <- WAL.refs(index) == %{} or {:error, :destination_not_empty},

@@ -351,6 +393,7 @@ defmodule OpenAgents.Repositories.Importer do

351 393
           "format" => format,
352 394
           "import_id" => repository_import.id,
353 395
           "refs" => refs,
396
           "shallow" => shallow_boundaries,
354 397
           "principal" => "github-import:#{repository_import.id}",
355 398
           "pushed_at" => DateTime.to_iso8601(DateTime.utc_now())
356 399
         },

@@ -511,6 +554,10 @@ defmodule OpenAgents.Repositories.Importer do

511 554
        log_stage(repository, repository_import, stage, "completed")
512 555
        result
513 556
557
      {:ok, _value, _metadata, _measurement, _details} = result ->
558
        log_stage(repository, repository_import, stage, "completed")
559
        result
560
514 561
      {:error, reason} = result ->
515 562
        log_stage(repository, repository_import, stage, "failed", reason)
516 563
        result

@@ -565,6 +612,29 @@ defmodule OpenAgents.Repositories.Importer do

565 612
    end
566 613
  end
567 614
615
  defp shallow_boundaries(source_repository) do
616
    source_repository
617
    |> Path.join("shallow")
618
    |> File.read()
619
    |> case do
620
      {:ok, contents} ->
621
        contents
622
        |> String.split("\n", trim: true)
623
        |> Enum.filter(&Regex.match?(~r/\A[0-9a-f]{40,64}\z/, &1))
624
        |> Enum.uniq()
625
        |> Enum.sort()
626
627
      {:error, :enoent} ->
628
        []
629
630
      {:error, reason} ->
631
        raise File.Error,
632
          reason: reason,
633
          action: "read shallow boundaries",
634
          path: source_repository
635
    end
636
  end
637
568 638
  defp temporary_directory(import_id) do
569 639
    root = Application.get_env(:openagents, :repository_import_temp_dir, System.tmp_dir!())
570 640
    Path.join(root, "openagents-import-#{import_id}-#{System.unique_integer([:positive])}")
priv/docs/cli-command-reference.md modified +5 -2

@@ -84,8 +84,11 @@ openagents repo import [flags] <github-owner/repository>

84 84
| `--private` | Create a private destination, which is the default. |
85 85
| `--wait-timeout SECONDS` | Wait for import. The default is `300`; `0` does not wait. |
86 86
87
This command performs one import. It does not start synchronization. A client
88
timeout does not cancel the accepted server-side import.
87
This command performs one depth-1 import of every accepted branch and tag. It
88
does not copy older history or start synchronization. A client timeout does not
89
cancel the accepted server-side import. While create and import commands wait,
90
they write state changes, elapsed time, and a five-second heartbeat to standard
91
error.
89 92
90 93
## List repositories
91 94
priv/docs/create-repository.md modified +3 -1

@@ -42,7 +42,9 @@ openagents repo create my-project \

42 42
43 43
The CLI waits up to 300 seconds for provisioning by default. Pass
44 44
`--wait-timeout 0` to return after the server accepts the durable request. The
45
repository continues provisioning on the server.
45
repository continues provisioning on the server. While it waits, the CLI
46
writes the current lifecycle state, elapsed time, and a five-second heartbeat
47
to standard error.
46 48
47 49
For one command without a global install, prefix the same arguments with the
48 50
package name:
priv/docs/import-github.md modified +24 -17

@@ -63,9 +63,10 @@ openagents repo import OpenAgentsInc/example --namespace OpenAgentsInc

63 63
The `--namespace` value must match the GitHub source owner in this release. You
64 64
cannot import `SOURCE/repository` directly into an unrelated namespace.
65 65
66
The CLI waits up to 300 seconds by default. Pass `--wait-timeout 0` to return
67
after the server accepts the durable import. A client timeout does not cancel
68
the server-side import.
66
The CLI waits up to 300 seconds by default. It writes state changes, elapsed
67
time, and a five-second heartbeat to standard error while it waits. Pass
68
`--wait-timeout 0` to return after the server accepts the durable import.
69
A client timeout does not cancel the server-side import.
69 70
70 71
## Import once with npx
71 72

@@ -78,7 +79,7 @@ npx --yes @openagentsinc/cli@latest repo import OWNER/REPOSITORY

78 79
Pin the package version for a reproducible qualification run:
79 80
80 81
```sh
81
npx --yes @openagentsinc/cli@0.1.4 \
82
npx --yes @openagentsinc/cli@0.1.5 \
82 83
  --profile staging \
83 84
  repo import OWNER/REPOSITORY \
84 85
  --private \

@@ -102,10 +103,16 @@ shell output.

102 103
103 104
## Import a large repository
104 105
105
OpenAgents keeps large Git bundles on disk and streams them to and from the
106
durable forge WAL in 1 MiB chunks. The application does not read the complete
107
bundle into the BEAM heap. The default server limits allow a bundle up to 20
108
GiB and an import to run for up to six hours.
106
OpenAgents imports every accepted branch and tag at depth 1 by default. This
107
shallow snapshot preserves each current tip and its files without copying the
108
source repository's full history. It makes repositories with years of history
109
available much faster and bounds the first transfer by current content rather
110
than commit count.
111
112
OpenAgents keeps the resulting Git bundle on disk and streams it to and from
113
the durable forge WAL in 1 MiB chunks. The application does not read the
114
complete bundle into the BEAM heap. The default server limits allow a bundle
115
up to 20 GiB and an import to run for up to six hours.
109 116
110 117
The CLI's `--wait-timeout` controls only how long that client waits. It does
111 118
not change or cancel the server import. For a large repository, accept the

@@ -121,11 +128,8 @@ Server logs record every stage and the bundle byte count. A bundle over the

121 128
server limit fails with `import_too_large`; an operation over the server time
122 129
limit fails with `import_timeout`.
123 130
124
Large imports still need enough temporary disk for the fetched Git objects and
125
the bundle. Git LFS objects remain outside the import. Start an environment's
126
first import with a small repository so you can qualify authentication,
127
lifecycle reporting, storage, and clone behavior before spending the time and
128
bandwidth of a large transfer.
131
Large imports still need enough temporary disk for the shallow Git objects and
132
the bundle. Git LFS objects remain outside the import.
129 133
130 134
## Verify an import
131 135

@@ -154,7 +158,8 @@ files without copying LFS objects.

154 158
   openagents repo clone OWNER/REPOSITORY
155 159
   ```
156 160
157
4. Compare the cloned branches and tags with the accepted GitHub snapshot.
161
4. Compare the cloned branch and tag tips with the accepted GitHub snapshot.
162
   Confirm that the clone contains one commit of history per imported tip.
158 163
5. Add a later commit on GitHub and confirm that it does not appear in the
159 164
   OpenAgents copy.
160 165

@@ -165,15 +170,17 @@ must start each import explicitly.

165 170
166 171
| Copied | Not copied |
167 172
| --- | --- |
168
| Git history reachable from accepted refs | GitHub Issues |
173
| Current commit and file tree at every accepted ref, with depth 1 | Full Git history before each accepted tip |
169 174
| `refs/heads/*` branches | Pull requests and reviews |
170 175
| `refs/tags/*` tags | Actions runs and secrets |
171 176
| The source default branch | Releases and repository settings |
172 177
| Submodule pointer commits | Wikis and Git LFS objects |
173 178
174 179
OpenAgents freezes the accepted branch and tag map before copying data. It
175
verifies the same ref snapshot before marking the repository ready. A GitHub
176
commit created after acceptance is not part of the import.
180
verifies that same ref snapshot before marking the repository ready. The
181
destination records those commits as shallow boundaries, so normal cloning and
182
new commits work without the omitted ancestry. A GitHub commit created after
183
acceptance is not part of the import.
177 184
178 185
Git LFS pointer files remain in Git history, but OpenAgents does not copy the
179 186
referenced LFS objects. Download or migrate those objects separately before
test/openagents/repositories/provisioner_test.exs modified +13 -1

@@ -152,6 +152,9 @@ defmodule OpenAgents.Repositories.ProvisionerTest do

152 152
    git!(source, ["init", "--initial-branch=main"])
153 153
    git!(source, ["config", "user.email", "test@example.com"])
154 154
    git!(source, ["config", "user.name", "Import test"])
155
    File.write!(Path.join(source, "HISTORY.md"), "older history\n")
156
    git!(source, ["add", "HISTORY.md"])
157
    git!(source, ["commit", "-m", "Older history"])
155 158
    File.write!(Path.join(source, "README.md"), "accepted snapshot\n")
156 159
    git!(source, ["add", "README.md"])
157 160
    git!(source, ["commit", "-m", "Accepted snapshot"])

@@ -208,13 +211,22 @@ defmodule OpenAgents.Repositories.ProvisionerTest do

208 211
    assert "repository.import.created" in audit_types(repository.id)
209 212
    assert "repository.import.running" in audit_types(repository.id)
210 213
    assert {:ok, _generation, index} = WAL.read_index(repository.storage_key)
211
    assert [%{"format" => "git_bundle", "import_id" => import_id}] = WAL.entries(index)
214
215
    assert [
216
             %{
217
               "format" => "git_bundle",
218
               "import_id" => import_id,
219
               "shallow" => [_boundary]
220
             }
221
           ] = WAL.entries(index)
222
212 223
    assert import_id == repository_import.id
213 224
    assert WAL.refs(index) == refs
214 225
215 226
    File.rm_rf!(Repos.bare_path(repository.storage_key))
216 227
    assert :ok = OpenAgents.Forge.Sync.ensure_fresh(repository.storage_key, "main")
217 228
    assert Repos.refs(repository.storage_key) == refs
229
    assert String.trim(bare_git!(repository.storage_key, ["rev-list", "--count", "main"])) == "1"
218 230
219 231
    assert String.trim(bare_git!(repository.storage_key, ["show", "main:README.md"])) ==
220 232
             "accepted snapshot"

This page updates live while a promote is in flight · changelog