Back off exponentially when the pylon orchestration SQLite is busy

Three or more pylon workers starting at the same moment can race on the
shared orchestration SQLite. withSqliteBusyRetry was doing 4 attempts with
25/50/75ms sleeps, which is not enough time for the writer to clear. It now
does 8 attempts with exponential backoff capped at 5s.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Co-Authored-By

Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Changed files

• modified apps/pylon/src/orchestration/supervisor-state.ts

Diff

1 file changed, +6 -3

const withSqliteBusyRetry = <T>(fn: () => T): T => {

  let lastError: unknown

  for (let attempt = 0; attempt < 4; attempt += 1) {

  for (let attempt = 0; attempt < 8; attempt += 1) {

    try {

      return fn()

    } catch (error) {

      if (!isSqliteBusy(error) || attempt === 3) throw error

      if (!isSqliteBusy(error) || attempt === 7) throw error

      lastError = error

      sleepSync(25 * (attempt + 1))

      // SQLite contention is transient, but 3 or more workers can start at

      // the same moment, so the backoff grows exponentially rather than

      // racing on the same short windows.

      sleepSync(Math.min(5000, 50 * 2 ** attempt))

    }

  }

  throw lastError