How to Convert an RSS or Atom Feed to JSON

Turn RSS 2.0, RSS 1.0, Atom or JSON Feed items into flat JSON rows with clean dates, with a few lines of Python or a scheduled run that returns only new items.

RSS and Atom feeds are still the simplest way to follow blogs, news sites, release notes and podcasts. They are XML, though, and every format names its fields differently: RSS has pubDate and guid, Atom has updated and id, and dates come in several styles. If you want feed items in a database, a spreadsheet, or a language model's context, the first job is to turn them into one consistent JSON shape.

Step 1: Find the feed URL

• Look for an RSS icon or a "feed" link on the site.
• Try common paths: /feed/, /feed.xml, /rss.xml, /atom.xml, /index.xml.
• View the page source and search for application/rss+xml or application/atom+xml. The <link rel="alternate"> tag that contains it holds the feed URL.

Step 2: Convert with a few lines of Python

The free feedparser library reads RSS, Atom and other variants into one Python structure. Install it with pip install feedparser, then:

import json, time, feedparser

feed = feedparser.parse("https://blog.rust-lang.org/feed.xml")
rows = []
for e in feed.entries:
    t = e.get("published_parsed") or e.get("updated_parsed")
    rows.append({
        "title": e.get("title"),
        "url": e.get("link"),
        "id": e.get("id"),
        "author": e.get("author"),
        "published_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", t) if t else None,
        "summary": e.get("summary"),
    })
print(json.dumps(rows, indent=2))

feedparser normalises dates to UTC in the *_parsed fields, which saves a lot of date handling.

Step 3: Clean up the fields

• Summaries are often HTML. Strip tags if you want plain text.
• Not every item has a date. Keep those items, but do not rely on sorting by date.
• Links can carry tracking parameters such as utm_source. Remove them before you compare links.
• Podcast audio is in the item's enclosure (e.enclosures in feedparser), with a URL, type and length.
• Feeds only hold recent items. Most keep somewhere between 10 and 50 entries. If you need history, poll on a schedule and store what you see.

Step 4: Keep only new items

To watch feeds over time, save a key for every item you have stored (the id, else the link, else the title and date) and skip items whose key you already have. Run the script on a schedule with cron or Task Scheduler.

When you have many feeds

The script above is fine for a few well formed feeds. With dozens of feeds you start handling wrong character encodings, broken dates, HTML entities inside XML, sites that give you a web page instead of a feed, robots.txt, and deduplication across feeds.

RSS and Atom to JSON, an Apify actor, does that work. It reads RSS 2.0, RSS 1.0, Atom and JSON Feed, and returns one flat row per item with the same fields whatever the format: title, url, author, published_at and updated_at in ISO 8601 UTC, a plain text summary, categories, image_url, enclosure fields, and a stable item_key for your own deduplication. If you give it a web page instead of a feed, it follows the first feed the page declares. It fixes common date problems and works out the text encoding, so accented letters come through.

Two options go further than the script:

• seenItemsStore: give it a name and run on a schedule, and each run returns only items earlier runs with that name have not returned.
• fetchFullText: opens each item's link and adds the page's main text as Markdown, for feeds that only carry a teaser.

It costs $0.0005 per item, plus $0.0015 per item when full text is fetched. 1,000 items is $0.50 without full text. Failed feeds and skipped items are free.

Its limits: public feeds only (no logins), only what the feed holds at the moment of the run (no archive fetching), feeds over 20 MB are skipped, and full text works on articles but returns little from pages built with JavaScript.

FAQ

What is the difference between RSS and Atom?

Both are XML feed formats. Atom is a later standard with stricter rules on dates and ids. For converting to JSON the difference is mostly field names, which a parser or the actor maps for you.

Can I get older posts than the feed shows?

Not from the feed. It only contains recent items. Start polling now and keep what you collect, or list older pages from the site's sitemap; see how to get every URL from a sitemap.

How do I get the full article text?

Fetch each item's link and extract the main content. The actor's fetchFullText does this, or see how to convert a web page to Markdown.

Does this work for podcast feeds?

Yes. Podcast feeds are RSS, and the audio file is the item's enclosure.

This guide was written with AI (Claude) and checked against the product's documentation.

All guides   All products