---
name: firecrawl-scraper
description: "Production web scraping via Firecrawl. Handles JS sites, anti-bot, rate-limits. Wraps the Firecrawl API with smart retry, structured-data extraction (JSON-LD, OpenGraph, microdata), and rate-limit-aware crawl orchestration. Use for research, monitoring, or data pipelines."
license: MIT
---

# Firecrawl Scraper

Firecrawl Scraper is a skill for pulling clean, structured content out of websites that fight back — JavaScript-rendered SPAs, sites behind bot challenges, and endpoints that throttle aggressively. It wraps the Firecrawl API and adds retry logic, structured-data extraction, and crawl pacing so you can feed reliable data into research notebooks, monitoring jobs, or ETL pipelines. Built for data analysts and backend engineers who need scraped pages as Markdown or typed JSON rather than raw HTML soup.

## When to use it

- You need the rendered content of a JS-heavy page (React/Next/Vue) where a plain HTTP GET returns an empty shell.
- A target site returns 403s, CAPTCHAs, or rate-limit 429s and a naive scraper keeps stalling.
- You want a whole section of a site crawled (docs, a blog archive, a catalog) without writing your own frontier/dedupe logic.
- You need fields pulled into a schema — prices, article metadata, author, publish date — instead of parsing the DOM by hand.

## What it does

The skill calls Firecrawl's core endpoints and shapes the results for you:

- **Scrape** (`/v1/scrape`) — fetches a single URL and returns Markdown, cleaned HTML, links, and metadata. Renders JS and waits for content before extracting.
- **Crawl** (`/v1/crawl`) — submits an async crawl job, polls the job id, and streams results as pages complete, honoring `includePaths`/`excludePaths`, `limit`, and `maxDepth`.
- **Extract / JSON mode** — pass a JSON Schema or a natural-language prompt and get typed objects back. Also surfaces JSON-LD, OpenGraph, and microdata it finds on the page.
- **Map** (`/v1/map`) — quickly enumerate a site's URLs before committing to a full crawl.

Retries use exponential backoff on 429/5xx and respect the `Retry-After` header; crawl orchestration throttles concurrency to stay under your plan's rate limit.

## Usage

```bash
curl -fsSL --create-dirs -o ~/.claude/skills/firecrawl-scraper/SKILL.md https://claudeskil.com/skills/firecrawl-scraper/SKILL.md
export FIRECRAWL_API_KEY=fc-...

# single page as markdown
> scrape https://example.com/pricing and give me the plans table

# typed extraction with a schema
> crawl https://blog.acme.com (limit 50, only /posts/*) and extract
> { title, author, publishedAt, tags[] } from each post
```

## Tips & gotchas

- Crawl is asynchronous — large jobs return a job id and complete over time; the skill polls for you, but set a sane `limit` to avoid burning credits.
- Each scraped page consumes credits; use **map** first to scope URLs before a broad crawl.
- For flaky SPAs, ask for a `waitFor` delay or an action sequence (click/scroll) so dynamic content loads before extraction.
- `excludePaths` is your friend for skipping `/tag/`, `/login`, and pagination loops that bloat results.

## FAQ

**Do I need a Firecrawl account?** Yes. Set `FIRECRAWL_API_KEY`; the free tier works for testing, but crawls and JS rendering draw down credits faster than plain scrapes.

**Markdown or raw HTML?** Default is LLM-friendly Markdown plus metadata. Request `html` or `rawHtml` formats explicitly if you need the original markup.

**Is it polite to target sites?** Firecrawl handles rendering and proxies, but you remain responsible for robots.txt, terms of service, and reasonable rate limits.