How to Convert a Web Page to Markdown for an LLM or RAG Pipeline

Turn web pages into clean Markdown without menus, cookie banners or footers, ready for a language model, a search index or a RAG pipeline.

Raw HTML is a poor input for a language model. A typical article page carries navigation, footers, cookie notices, scripts and tracking markup around a few hundred words of real text. That wastes tokens and adds noise to embeddings. Markdown keeps what matters (headings, lists, links, tables, code) in a compact, readable form, which is why it is the common format for RAG pipelines and LLM context.

Here is how to do the conversion for one page, for a handful, and for a long list.

For one page: reader view and copy

Open the page in a browser with a reader view (Firefox, Safari and Edge have one).
Switch to reader view. This keeps the article body and drops most of the page chrome.
Copy the text into a Markdown editor, or paste the HTML into any HTML to Markdown converter.

This works, but headings and links do not always survive the copy, and it does not scale.

For a few pages: a command line converter

Pandoc can fetch a URL and write Markdown:

pandoc -f html -t gfm https://example.com/article -o article.md

It converts the whole page, menus included. To keep only the main text, run the HTML through a main content extractor first. In Python, libraries such as readability-lxml or trafilatura find the article body, and markdownify or html2text turn the result into Markdown.

Clean up before you embed

Whatever tool you use, check these before the text goes into your index:

Main content only. Strip navigation, footers, sidebars, forms and cookie banners.
Absolute links. Relative links like /docs/intro are useless once the text leaves the site. Make them full URLs.
Images. Decide whether to keep image URLs, keep only alt text, or drop images. Alt text is often enough for a model.
Split by heading. Chunking at headings keeps each chunk on one topic, which usually retrieves better than fixed size chunks.
Keep metadata. Store the URL, title and language next to each chunk so you can cite the source.
Check for empty pages. Pages that build their content with JavaScript can come back nearly empty from any tool that reads the server HTML. A word count check catches them.

Respect the site

Check robots.txt before fetching, pause between requests to the same site, and only process pages you are allowed to use.

Doing it for a long list of URLs

URL to Markdown Converter, an Apify actor, does the steps above for a list of public pages. For each page it checks robots.txt, removes scripts and page chrome, keeps the main content with a reader mode style algorithm (falling back to the whole page when there is no clear article body, and saying so in the extraction field), and converts headings, lists, code blocks, quotes and tables to Markdown with absolute links.

Each result has title, description, language, word_count and markdown. Options you will use for RAG:

imageMode: keep images as Markdown, write [Image: alt text], or drop them.
includeSections: adds a sections array split at each heading, ready for chunking.
includeLinks: adds a list of every unique link in the content.

It costs $0.002 per converted page, about $2 for 1,000 pages. Failed and skipped pages are not charged and are listed with a reason. You can also add it to an AI agent as a tool through Apify's MCP server.

Its limits: it does not run JavaScript, log in, or crawl. It reads HTML and plain text only (PDFs are skipped), skips pages over 5 MB or slower than 30 seconds, and fetches about one page per second on a single site. Main content extraction is a heuristic and can keep a sidebar or drop a short intro on unusual layouts; turn it off to get the whole page with only the chrome removed.

FAQ

Why Markdown and not plain text?

Plain text loses structure. Markdown keeps headings, lists and tables, which helps both chunking and the model's reading of the page, at only a few extra characters.

How do I get every page of a site?

List the URLs first. Most sites publish a sitemap; see how to get every URL from a sitemap, then pass the list to the converter.

What about PDFs?

PDFs need a different tool. See how to extract text from PDF links in bulk.

Will it work on single page apps?

Usually not. Pages that render in the browser return little text to any tool that does not run JavaScript. Check word_count and use a headless browser for those pages.

This guide was written with AI (Claude) and checked against the product's documentation.

All guides All products