RAG Pipeline Builder

A scaffolding skill for engineers building retrieval-augmented generation systems that have to survive contact with real traffic. Instead of a toy embed -> cosine -> stuff-into-prompt loop, it wires up the full retrieval stack — chunking, embeddings, a vector store, hybrid search, a reranker, a cache, and an offline eval harness — and lets you swap each component by config. It's aimed at people who already have a prototype that "kind of works" and now need recall numbers they can defend.

When to use it

• Your demo RAG returns plausible-but-wrong chunks and you can't tell whether the bug is chunking, embeddings, or ranking.
• You're choosing between Pinecone, pgvector, and Qdrant and want one interface so the decision isn't permanent.
• Pure dense retrieval misses exact-match queries (error codes, SKUs, function names) and you need BM25 in the mix.
• You want a regression gate in CI so a chunk-size tweak can't silently tank recall@5.

How it works

The skill lays out a modular pipeline with pluggable stages:

• Chunking — recursive (token-aware splits on separators), semantic (embedding-distance breakpoints), and parent-document (retrieve small, return the enclosing parent) strategies.
• Embeddings — adapters for OpenAI text-embedding-3-large, Voyage voyage-3, and Cohere embed-v3, with batching and dimension config.
• Vector store — a common upsert/query interface over Pinecone, Postgres pgvector (HNSW index), and Qdrant.
• Hybrid search — dense + BM25 (sparse) candidates fused with Reciprocal Rank Fusion.
• Reranking — Cohere rerank-3 or Voyage rerank-2 over the fused candidate set.
• Caching — embedding and query-result caches keyed by content hash.
• Eval — golden-set scoring (recall@k, MRR, nDCG, plus answer faithfulness) that fails the build on regression.

Usage

curl -fsSL --create-dirs -o ~/.claude/skills/rag-pipeline-builder/SKILL.md https://claudeskil.com/skills/rag-pipeline-builder/SKILL.md

# then, in your project
"Build a RAG pipeline: parent-document chunking, Voyage embeddings,
 pgvector, hybrid + Cohere rerank, with an eval set from ./qa.jsonl"

# rag.config.yaml
chunking: { strategy: parent-document, child_size: 256, parent_size: 1024 }
embeddings: { provider: voyage, model: voyage-3 }
vector_store: { backend: pgvector, index: hnsw, metric: cosine }
retrieval: { hybrid: true, fusion: rrf, top_k: 40 }
rerank: { provider: cohere, model: rerank-3, top_n: 8 }

Tips & gotchas

• Embedding dimensions are baked into the index — switching providers means a full reindex, not a config flip.
• Run BM25 and dense over the same chunk IDs; mismatched corpora make RRF fusion meaningless.
• Rerank a wide candidate set (top_k ~40) then narrow to top_n ~8; reranking only what dense already ranked adds little.
• Build the eval golden set before tuning, or you'll overfit chunk size to vibes.

FAQ

Do I need a reranker if I already use hybrid search? Usually yes — fusion improves candidate recall, reranking fixes ordering. The biggest faithfulness gains typically come from the rerank stage.

pgvector or a dedicated DB? Start with pgvector if your corpus is under a few million chunks and you already run Postgres. Move to Qdrant/Pinecone when filtered queries or scale make HNSW tuning painful.

Can I run eval in CI? Yes — the eval harness emits recall@k/MRR/nDCG and exits non-zero past your threshold, so it drops into a GitHub Action.