Build the cloud computer capacity and quota broker #4

Project

Cloud computer platform

Source: Cloud computer scale architecture audit

Outcome

Build a durable admission and scheduling layer that lets a chat own up to 30 logical computers while bounding active compute across the command, computer, conversation, user, tenant, provider, region, and global scopes.

Initial policy

• Allow 30 logical computers per chat.
• Admit four active standard runtimes per chat by default.
• Allow up to eight active standard runtimes only for an explicitly budgeted high-fan-out operation.
• Admit no more than two active strong runtimes per chat until stronger density evidence exists.
• Maintain a small global warm pool instead of a warm pool per chat.
• Reserve at least 25% of the observed provider or configured capacity ceiling for existing workloads, cleanup, replacements, and incident recovery.
• Apply a lower fixed private-preview ceiling even when the Google Cloud quota is larger.

Scope

• Store capacity reservations durably and create them in one serializable admission transaction.
• Enforce hierarchical concurrency, CPU, memory, scratch, duration, operation-rate, and cost budgets.
• Poll Google Cloud quota and provider allocatable capacity outside the admission transaction.
• Calculate the effective limit from the configured ceiling, fresh quota minus headroom, host or cluster allocatable capacity, incident drains, and budget.
• Fail closed for new starts when quota observations are stale while preserving cleanup and recovery capacity.
• Use weighted fair queuing across tenants and conversations.
• Bound start concurrency, add jitter, and release reservations after deadline-bound failures.
• Reconcile reservation state with GCE, GKE, Firecracker hosts, and Cloud Run without making cloud resource enumeration the admission authority.
• Detect leaked, missing, double-claimed, and quarantined capacity.

Deliverables

• Capacity and reservation schemas, storage, scheduler, and reconciler.
• Provider capacity adapters and freshness rules.
• Operator configuration for per-profile and per-scope limits.
• Queue-state and admission receipts suitable for chat and status projections.
• Load and race tests for concurrent admission, quota changes, start storms, fairness, and reconciliation.

Acceptance criteria

• Two concurrent admission transactions cannot allocate the same capacity.
• One chat cannot consume reserved cleanup capacity or starve another tenant indefinitely.
• A 15-computer request creates all logical records, admits only the allowed active leases, and leaves the remainder visibly queued or cold.
• Stale or exhausted quota returns a typed queued or refused result without a provider create call.
• Cleanup and host replacement still run when normal admission reaches its ceiling.
• Reconciliation repairs drift without deleting or reassigning a live lease from another generation.

Dependencies

Depends on the cloud_computer.v1 contract issue in this project.