Ingest
cms ingest <url> takes a site of under fifty pages and produces a site repo in this engine's format: markdown pages, a nav and site data file, imported media with sidecars, and a site.css that is as close to nothing but token overrides as the source will allow.
It ships as a skill, /ingest, because the work splits cleanly. The CLI does the mechanical stages, which are deterministic and re-runnable. The agent does the judgment stages, which is where a wrong guess is cheap to correct and expensive to automate.
Stages
1. Crawl
Sitemap first, then internal links, same origin only, honouring robots.txt, capped at --max (default 50). Pages render in a headless browser, so JavaScript-built markup and computed styles are both available.
Everything fetched lands in .cms/ingest/<host>/: raw HTML, response headers, computed style samples, and a manifest. That directory is the input to every later stage, so a re-run costs nothing and a mapping change can be re-applied without touching the source site again.
Provenance is recorded, and the run records who authorised it. Ingesting a site is copying its content, which is fine for a site you own or have been engaged to rebuild and not otherwise.
2. Separate chrome from content
The header, nav, and footer are found by diffing the DOM across pages: the subtrees that are identical or near-identical everywhere are chrome, and what remains is the page. This beats heuristics about <header> and <nav>, which plenty of sites do not use, and it degrades gracefully on a site with only two pages by falling back to landmark detection.
Chrome becomes data. The nav becomes nav.yml, the footer's link columns and legal links become site.yml, and neither one is ever markup again. On a site that duplicates its header across thirteen files, this stage is most of the value on its own.
3. Map content to blocks
Each page's remaining content is matched against the component catalogue in reverse:
| Source pattern | Block |
|---|---|
| `h1`, a lead paragraph, and one or two buttons at the top | `hero` |
| Repeated sibling structure of heading, text, and a link | `cards` |
| Numbered or visually sequential heading and paragraph pairs | `steps` |
| Alternating text and image rows | `features` |
| `figure` with `figcaption`, or an image with adjacent caption text | `shots` |
| Two balanced columns of heading, text, and a button | `split` |
| A list where every item carries a marker glyph | `ticks` |
| `details`/`summary` groups, or heading-and-answer pairs | `faq` |
| `table` with a header row | `table` |
| Anything else | `prose` |
The matcher is deliberately conservative. prose is always a correct answer, and a wrong component is worse than an unambitious one, so a match below the confidence threshold falls back to prose and gets listed in the report for the agent to look at.
Prose conversion is a whitelist: the elements the content model allows, and nothing else. Presentational spans, empty divs, and tracking attributes are dropped rather than carried over.
4. Media
Every referenced image is downloaded, deduplicated by content hash, and written into src/media/ with a sidecar. Existing alt text is carried over when it looks like alt text and flagged when it is empty, a filename, or a duplicate of the caption beside it. Intrinsic dimensions are recorded so the first build has what it needs.
The alt-text flags are a work list, which is the point: the gate will not let the site build until every one is resolved, so inherited alt-text debt gets paid at ingest rather than discovered in an audit.
5. Derive the theme
This is the stage the whole design turns on, because the constraint is that site.css stays close to empty and the token overrides do the work.
1. Sample. Render every crawled page and read computed styles for every
element, weighting each colour by the area it covers.
2. Cluster. Collapse near-duplicates in OKLCH.
3. Assign. Give each cluster a role by where it appears:
largest background area → --fp-bg
secondary section background → --fp-bg-alt
darkest large text → --fp-ink
body text → --fp-ink-soft
secondary and fine text → --fp-ink-mute
hairlines and card borders → --fp-rule
link text and small accents → --fp-accent candidate
button and chip fills → --fp-amber candidate
dark bands → the --fp-board family
4. Repair. Hold hue, hold chroma where it survives, and move lightness in
OKLCH until every pairing in the token graph clears 7:1.
5. Enforce. Apply the design system's role rules. A candidate accent that
cannot reach 7:1 as text without moving further than the
threshold becomes a fill instead, and the accent role takes a
repaired version of the same hue.
6. Emit. A :root block and a dark-scheme block. Nothing else.
Working in OKLCH is what makes the repair usable. Moving lightness while holding hue keeps a brand colour recognisable as itself, where a naive darken in sRGB shifts hue and comes back looking like a different colour. The report gives the lightness delta and the perceptual distance per token, so a person can see exactly how far each brand colour moved and decide whether to accept it, pick a different source colour, or take the change back to the client.
Typography is reported, then imported on request. The ingest names the faces the source site uses and how it serves them. Bringing one across is cms font add, which needs the font file and its licence, because the engine self-hosts every face and will not build against a licence it has no record of. A site that linked its face from a third-party host has to supply the file, and that conversation is better had at ingest than at launch.
6. Site CSS budget
site.css starts as the token block and nothing else. Anything the source site does that tokens cannot express falls into one of two piles.
A proposed component, if it is a real repeated pattern the catalogue lacks. The ingest writes a schema, a template, and a stylesheet fragment, and lists it for review. gtfs.media's departure board is exactly this shape.
A one-off, if it appears once. These are reported and not written. Almost all of them turn out to be either a component in disguise or something the site is better without.
.cms/budgets.json carries a line budget for site.css, and cms check reports the current count against it on every run. A number that has to be argued about is more durable than a rule everyone agrees with and nobody measures.
7. Report
The run ends with .cms/ingest/report.md, which is the agent's work list:
- Pages ingested, and any that were skipped and why.
- Blocks by confidence, with every low-confidence match named.
- Components proposed, with the markup that triggered each one.
- Tokens derived, with the contrast repair applied to each.
- Alt text missing or suspect.
- Accessibility failures inherited from the source, by criterion.
- Links that did not resolve, and redirects that need writing.
- Copy that reads against the writing guide.
Nothing in the report is fixed automatically. The agent works through it, the gate holds the line, and a person reviews the diff.
Two modes
Ingest defaults to keeping the copy. --vibe gives it permission not to.
| Default | --vibe |
|
|---|---|---|
| A component may claim a section it cannot hold in full | no | yes, down to half |
| Copy that does not fit a component | stays in prose | may be left behind |
| Text a match drops | cannot happen | recorded verbatim in the report |
| The agent afterwards | preserves wording, changes structure | may rewrite and reorganise |
Nothing is lost in either mode. In --vibe, every sentence a component left behind appears in the report under "Text a component left behind", and the crawl snapshot stays on disk regardless. The difference is what ends up on the page, not what survives.
--vibe still has a floor. Below half the section's text, a match is not a rendering of that section, it is a different section, and taking it would be a bug rather than a choice.
The mode is not a one-off flag. apply writes it into .cms/prompts/site.md, so a session opening the repo tomorrow knows whether it may rewrite the copy. Under --vibe that file says paraphrasing is expected; by default it says preserve the wording and change structure freely.
Measured on gtfs.media, the difference is smaller than it sounds: 24.6% prose in exact mode against 21.9% in vibe, with two sections matched loosely and two sentences set aside. Most fallbacks are not close calls about fidelity, they are components the catalogue does not have, and no amount of licence to paraphrase invents one.
What ingest does not do
It does not preserve a design. It reads a site's content and its brand colours and rebuilds both inside a design system that already meets AAA. A client who wants their existing layout reproduced pixel for pixel wants a different service, and saying so early is cheaper than discovering it at review.
It does not carry over behaviour. Scripts, embeds, forms, and third-party widgets are listed in the report and left out of the build. Each one is a decision with an accessibility cost attached, and the embed component exists so that decision gets made deliberately.
Acceptance
Two tests, in order.
gtfs.media, against ground truth. By the time ingest is built, gtfs.media has been migrated by hand. Ingesting the live site and comparing against that migration gives the matcher a real score: which blocks it got right, which it dropped to prose, and whether the derived tokens land near the FivePaths palette the site already uses. The hand migration is the answer key.
An arbitrary small site. A real site of twenty to fifty pages that nobody involved has seen before. It builds, it passes the gate, and its site.css is inside budget. The report is judged on whether working through it produces a site worth shipping, which is a human call and the only one that matters.