FEED CRAWLER
Part of LLMTEXT.com

RSS/Atom Feed Crawler

Discover and extract content from RSS/Atom feeds

How it works:

1. Discovers RSS/Atom feeds from the hostname

2. Extracts markdown content using Parallel AI

3. Only re-extracts updated posts on subsequent runs

Crawl Results


                

Index RSS feeds from the Kagi Smallweb list. This will enable periodic feed fetching for each domain.

What this does:

1. Fetches the smallweb.txt file from GitHub

2. Stores feed metadata for each domain

3. Enables hourly feed refresh for each domain

4. With API key: Automatically extracts new/updated posts each hour

5. Without API key: Only discovers items; manual extraction required

Index Results


                

Posts