Port grok-build streaming markdown rendering to coder-lite #104

Closed AtlantisPleb opened this 4d ago 1 comment

Context

coder-lite currently renders assistant output with ratatui-markdown. This gives basic bold/italic/lists/code blocks, but it is not the Grok Build engine.

What grok-build has that we do not

StreamingMarkdownRenderer with checkpoint freezing — only re-renders the tail as new tokens arrive, avoiding O(n²) re-parsing.
Source maps mapping terminal cells back to markdown bytes (click-to-edit / selection).
anstyle + supports-color color adaptation for 16/256-color terminals.
syntect-based syntax highlighting (vs optional tree-sitter).
URL scanning and OSC8 hyperlink emission.
LaTeX math to Unicode conversion ( $E=mc^2$ → E=mc²).
word_wrap_lines_with_joiners() with CJK-aware width and continuation markers (↳) so copied text can be reconstructed.
MarkdownStyle per-element padding/margins, vpad, block backgrounds, and the xai-grok-pager EntryRenderer accent column / ┃ chrome.
Open-code highlighting and Mermaid diagram rendering.
Cached re-wrap on resize keyed by (width, generation, theme).

Goal

Bring crates/coder-lite closer to the Grok rendering / line spacing behavior. The engine lives in xai-org/grok-build under crates/codegen/xai-grok-markdown and crates/codegen/xai-grok-pager/src/scrollback.

This is a future work item; do not start until it is pulled into a sprint.

AtlantisPleb opened this issue 4d ago
AtlantisPleb closed this as completed in 3b3a7b4 4d ago
A AtlantisPleb Author 4d ago

Done in 3b3a7b46fb. This one matters more than its number: coder-lite is the UI the owner will drive, so its rendering is the product.

crates/coder-lite/src/markdown/ carries StreamingMarkdownRenderer with checkpoint freezing, syntect highlighting, anstyle/supports-color adaptation, URL scanning with OSC 8, LaTeX→Unicode, tables, mermaid, MarkdownStyle, and CJK-aware word_wrap_lines_with_joiners(). Plus a wrap cache keyed by (width, generation) so frozen rows wrap once.

Identity held. markdown/theme.rs is ours, not ported: one amber on one background, elements told apart by weight/dim/italic/underline, syntect's colours flattened by amberize. A test asserts every painted cell on a page of mixed markdown is exactly #FFB000 on #080600. Spinner frames and the Entry/CoderUi frame are unchanged.

Mutation-verified rather than eyeballed. Disabling freezing → reparsed 4612595 bytes for a 23090-byte document. Breaking the wrap cache → word-wrapped 90599 lines to display 599. Batching instead of streaming → all three streaming tests fail.

Three bugs found by running it, two of them in the port itself. A hidden link_outer glued URLs to their labels (the forgehttps://…); a visible code_language swallowed the newline after a fence (rustfn main() {); and — caught only from a real frame dump — Entry::push_text appended a chunk to self.text before seeding the renderer from self.text, so the first chunk of every stream rendered twice. That last one is the mirror of the rule this repo keeps enforcing: there were tests for content dropped and none for content duplicated. Both now covered.

The rebase over four concurrent coder-lite commits kept all four, verified by a rebase_contract.rs suite; one had silently reverted the bullet spacing and was restored.

Judgement calls, flagged: edition 2024 for coder-lite only; new deps in the crate's own Cargo.toml rather than the workspace's; the fence language tag is hidden, matching grok-build's production theme, with the body never hidden and a test for unknown languages; ↳ markers are not painted because upstream has none and adding a visible marker would change how coder-lite looks.

642 tests pass, and cargo run -p coder-lite -- --dev builds and runs under a real pty.