How to Extract Text and Metadata From PDF Links in Bulk

Get the text of every page and the title, author, dates and page count from a list of PDF URLs, with free command line tools or in one batch run.

You have a list of PDF links: reports, papers, manuals, forms, a public document archive. You want the text inside them for a search index or a language model, and the metadata (title, author, creation date, page count) in a table. Opening each file by hand is not an option past a dozen or so.

Here is the free way to do it, what goes wrong, and a faster option for long lists.

Step 1: Collect the links

Put one PDF URL per line in a text file, urls.txt. Make sure each link points at the PDF file itself, not at a web page that shows or links to it. A quick test: open the link and check that the browser shows the PDF viewer, or that the address ends in the file.

Step 2: Download the files

With wget or curl, politely, with a pause between requests:

wget --wait=1 --random-wait -i urls.txt -P pdfs/

Some sites ask for a longer delay in their robots.txt Crawl-delay line. Respect it.

Step 3: Extract text and metadata

Poppler's command line tools are free and fast. pdftotext writes the text and pdfinfo prints the metadata:

for f in pdfs/*.pdf; do
  pdftotext -layout "$f" "${f%.pdf}.txt"
  pdfinfo "$f" > "${f%.pdf}.info.txt"
done

pdfinfo shows fields such as Title, Author, Creator, Producer, CreationDate, ModDate, Pages and PDF version. In Python, pypdf gives you the same data per page if you want JSON instead of text files.

Step 4: Check what you got

Empty text. If a file returns no text, it is probably a scanned document: the pages are images. You need OCR for those, for example Tesseract or ocrmypdf, which adds a text layer you can then extract.
Missing metadata. Many PDFs leave title and author blank, especially ones made by scanners or LaTeX. That is normal.
Odd order. Text comes out in the order it is stored in the file. For single column documents that is reading order. Multi column layouts and tables can come out with columns mixed together.
Not a PDF. If the server returned an HTML login or error page, the tools will fail on it. Check the first bytes or the file type.
Your rights. The text includes whatever is printed in the document, names and contact details included. Make sure you may use it.

Where the manual way gets slow

It works, but with hundreds of links you also end up writing code to handle failures, skip oversized files, respect each site's robots.txt and crawl delay, and merge everything into one dataset.

PDF to Text and JSON, an Apify actor, does that in one run. You give it the list of PDF URLs. For each file it checks robots.txt (and waits the site's Crawl-delay), stops the download early if the first bytes are not a PDF, reads the text layer page by page with pdf.js (the PDF engine in Firefox), and reads the document info and XMP metadata.

Each result has title, author, subject, keywords, creator, producer, created_at, modified_at, pdf_version, page_count, file_size_bytes, word and character counts, has_text and pages_without_text (to flag scans), the full text, and a pages array with each page's text. You can read only the first N pages with maxPagesPerPdf and skip big files with maxFileSizeMb.

It costs $0.005 per PDF whatever the page count, so 100 PDFs is $0.50. Failed links are not charged and are listed with a reason, such as HTTP 404, Not a PDF, or PDF is password protected.

Its limits: there is no OCR, so scanned pages return empty text (they are still counted, because page count and metadata come back). It does not log in or open password protected files. It has no table detection, files over 100 MB are always skipped, and slow crawl delays make some sites slow (arXiv asks for 15 seconds between requests).

FAQ

Can I extract tables from PDFs this way?

Not as tables. Both pdftotext and the actor return text in stored order, so cells may land on separate lines. For real tables, use a table extraction tool such as Camelot or Tabula on the downloaded files.

How do I tell which PDFs are scanned?

With pdftotext, look for empty output files. With the actor, filter rows where has_text is false.

Does it work on PDFs behind a login?

No. Both methods above assume public links. Download private files yourself and process them locally.

What about web pages, not PDFs?

Use a web page converter instead; see how to convert a web page to Markdown.

This guide was written with AI (Claude) and checked against the product's documentation.

All guides All products