Skip to content

feat(llms-txt): write each page's Markdown and llms-full.txt - #1127

Open
ovflowd wants to merge 3 commits into
feat/graphviz-diagramsfrom
feat/llms-full
Open

ovflowd wants to merge 3 commits into
feat/graphviz-diagramsfrom
feat/llms-full

Conversation

@ovflowd

@ovflowd ovflowd commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Description

This PR adds two opt-in options to the llms-txt generator:

  • writeMarkdown: writes the Markdown of every page at {path}.md, which is where llms.txt links by default (pageURL: '{baseURL}{path}.md'), so those links resolve without a separate copy step.
  • writeFull: writes llms-full.txt with every page, each preceded by its URL. It's the same format VitePress' llms plugin produces.

Pages are rebuilt from the metadata entries' content with doc-kit's own remark stringify (the MDX one for MDX pages), skipping synthetic pages.

This is part of a proof of concept of migrating rolldown.rs from VitePress to doc-kit. rolldown.rs serves both today, and this keeps parity without a post-build script.

Validation

  • Tests for building the pages' Markdown (entries joined per page, in order; synthetic pages left out).
  • Rolldown's build produces llms-full.txt and a .md file per page.

Related Issues

Part of the Rolldown docs migration PoC: rolldown/rolldown#11072. It's the top of a stack: #1125 (TypeDoc plugin) → #1126 (Graphviz diagrams) → #1127 (llms-full), each targeting the one below. The changes don't depend on each other, the stack just keeps them reviewable one at a time while the Rolldown PoC builds from the top branch.

Check List

  • I have read the Contributing Guidelines and made commit messages that follow the guideline.
  • I have run node --run test and all tests passed.
  • I have check code formatting with node --run format:check & node --run lint.
  • I've covered new added functionality with unit tests if necessary.

@vercel

vercel Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
api-docs-tooling Ready Ready Preview Oct 3, 2026 2:24pm UTC

Request Review

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Failed ❌

View logs ↗
64217c9 2026-10-03T14:24:00.485Z View logs ↗
  • Build: Failed ❌

View logs ↗
762bf78 2026-10-03T14:14:18.955Z View logs ↗
  • Build: Failed ❌

View logs ↗
9566b83 2026-10-03T13:25:49.860Z View logs ↗
  • Build: Failed ❌

View logs ↗
bbfc21c 2026-10-03T13:14:34.515Z View logs ↗
  • Build: Failed ❌

View logs ↗
0f386b2 2026-10-01T14:25:32.775Z View logs ↗
  • Build: Failed ❌

View logs ↗
82490d7 2026-10-01T13:27:00.445Z View logs ↗

@codecov

codecov Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.13402% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.86%. Comparing base (253bd41) to head (64217c9).

Files with missing lines Patch % Lines
packages/react/src/llms-txt/generate.mjs 12.90% 27 Missing ⚠️
packages/react/src/llms-txt/utils/buildPages.mjs 96.66% 1 Missing ⚠️
Additional details and impacted files
@@                    Coverage Diff                     @@
##           feat/graphviz-diagrams    #1127      +/-   ##
==========================================================
- Coverage                   92.94%   92.86%   -0.08%     
==========================================================
  Files                         262      264       +2     
  Lines                       25341    25435      +94     
  Branches                     2589     2600      +11     
==========================================================
+ Hits                        23553    23621      +68     
- Misses                       1766     1792      +26     
  Partials                       22       22              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

api-links Generator

Performance estimate (single CI run)

  • Generation time: 4.2% faster (1.43 s → 1.37 s)
  • Peak memory: 17.1% lower (423.48 MB → 351.26 MB)

json Generator

Performance estimate (single CI run)

  • Generation time: 1.2% slower (10.17 s → 10.29 s)
  • Peak memory: 15.3% higher (1.45 GB → 1.67 GB)

legacy-html Generator

Performance estimate (single CI run)

  • Generation time: 53.0% slower (28.34 s → 43.37 s)
  • Peak memory: 0.0% lower (2.45 GB → 2.45 GB)

legacy-json Generator

Performance estimate (single CI run)

  • Generation time: 28.5% slower (6.43 s → 8.26 s)
  • Peak memory: 0.3% lower (1.58 GB → 1.58 GB)

llms-txt Generator

Performance estimate (single CI run)

  • Generation time: 14.3% slower (6.83 s → 7.81 s)
  • Peak memory: 9.0% lower (1.56 GB → 1.42 GB)

orama-db Generator

Performance estimate (single CI run)

  • Generation time: 27.8% slower (6.41 s → 8.19 s)
  • Peak memory: 4.2% higher (1.63 GB → 1.70 GB)

web Generator

Output size: 2 files changed · net +201.00 B

File size details
File Main PR Change
assets/style-Dpi-h_PZ.css — 137.38 KB +137.38 KB
assets/style-CgRaeUUe.css 137.19 KB — -137.19 KB (-100.0%)

Performance estimate (single CI run)

  • Generation time: 12.8% faster (51.81 s → 45.16 s)
  • Peak memory: 23.2% higher (3.25 GB → 4.00 GB)

@ovflowd
ovflowd marked this pull request as ready for review October 1, 2026 14:17
@ovflowd
ovflowd requested a review from a team as a code owner October 1, 2026 14:17
@ovflowd
ovflowd changed the base branch from main to feat/graphviz-diagrams October 1, 2026 14:24
@ovflowd
ovflowd added this pull request to stack #1143 October 1, 2026 14:28
defaultConfiguration: {
templatePath: join(import.meta.dirname, 'template.txt'),
pageURL: '{baseURL}{path}.md',
writeMarkdown: false,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be in the generator, or can it just be cp input output, this feels out of scope.

- `templatePath` {string} Path to the template file.
**Default:** `'template.txt'`.
- `pageURL` {string} URL template for documentation page links.
**Default:** `'{baseURL}/latest/api{path}.md'`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure to update the Node.js config since this default changed

Comment on lines +52 to +58
if (config.writeFull) {
const full = pages.map(
({ path, markdown }) =>
`---\nurl: ${populate(config.pageURL, { ...config, path })}\n---\n${markdown}`
);

await writeFile(join(config.output, 'llms-full.txt'), full.join('\n'));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

llms-full-txt as a separate generator might be smarter, since these don't actually share any data, wdyt?

Two options: `writeMarkdown` writes the Markdown of every page at the `.md` URLs llms.txt links by default, and `writeFull` writes llms-full.txt, holding every page preceded by its URL.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview – api-docs-tooling — 64217c99 Deployed Oct 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants