Firecrawl Scraper

Firecrawl Scraper is a skill for pulling clean, structured content out of websites that fight back — JavaScript-rendered SPAs, sites behind bot challenges, and endpoints that throttle aggressively. It wraps the Firecrawl API and adds retry logic, structured-data extraction, and crawl pacing so you can feed reliable data into research notebooks, monitoring jobs, or ETL pipelines. Built for data analysts and backend engineers who need scraped pages as Markdown or typed JSON rather than raw HTML soup.

When to use it

• You need the rendered content of a JS-heavy page (React/Next/Vue) where a plain HTTP GET returns an empty shell.
• A target site returns 403s, CAPTCHAs, or rate-limit 429s and a naive scraper keeps stalling.
• You want a whole section of a site crawled (docs, a blog archive, a catalog) without writing your own frontier/dedupe logic.
• You need fields pulled into a schema — prices, article metadata, author, publish date — instead of parsing the DOM by hand.

What it does

The skill calls Firecrawl's core endpoints and shapes the results for you:

• Scrape (/v1/scrape) — fetches a single URL and returns Markdown, cleaned HTML, links, and metadata. Renders JS and waits for content before extracting.
• Crawl (/v1/crawl) — submits an async crawl job, polls the job id, and streams results as pages complete, honoring includePaths/excludePaths, limit, and maxDepth.
• Extract / JSON mode — pass a JSON Schema or a natural-language prompt and get typed objects back. Also surfaces JSON-LD, OpenGraph, and microdata it finds on the page.
• Map (/v1/map) — quickly enumerate a site's URLs before committing to a full crawl.

Retries use exponential backoff on 429/5xx and respect the Retry-After header; crawl orchestration throttles concurrency to stay under your plan's rate limit.

Usage

curl -fsSL --create-dirs -o ~/.claude/skills/firecrawl-scraper/SKILL.md https://claudeskil.com/skills/firecrawl-scraper/SKILL.md
export FIRECRAWL_API_KEY=fc-...

# single page as markdown
> scrape https://example.com/pricing and give me the plans table

# typed extraction with a schema
> crawl https://blog.acme.com (limit 50, only /posts/*) and extract
> { title, author, publishedAt, tags[] } from each post

Tips & gotchas

• Crawl is asynchronous — large jobs return a job id and complete over time; the skill polls for you, but set a sane limit to avoid burning credits.
• Each scraped page consumes credits; use map first to scope URLs before a broad crawl.
• For flaky SPAs, ask for a waitFor delay or an action sequence (click/scroll) so dynamic content loads before extraction.
• excludePaths is your friend for skipping /tag/, /login, and pagination loops that bloat results.

FAQ

Do I need a Firecrawl account? Yes. Set FIRECRAWL_API_KEY; the free tier works for testing, but crawls and JS rendering draw down credits faster than plain scrapes.

Markdown or raw HTML? Default is LLM-friendly Markdown plus metadata. Request html or rawHtml formats explicitly if you need the original markup.

Is it polite to target sites? Firecrawl handles rendering and proxies, but you remain responsible for robots.txt, terms of service, and reasonable rate limits.