Iceberg Lakehouse Engineer

Operate a production Apache Iceberg lakehouse end to end: wire up REST catalogs, keep tables healthy with maintenance procedures, design hidden partitioning, and evolve schemas without rewriting data. Built for data engineers and platform teams running Iceberg on Spark, Trino, or Flink against object storage (S3, GCS, ADLS) who are tired of guessing which maintenance job to run and why query planning has gotten slow.

When to use it

• A streaming or CDC pipeline has produced thousands of tiny files and scans are crawling — you need rewrite_data_files with the right target file size and a sane rewrite strategy.
• Snapshot metadata and old data files are piling up storage cost, and you need a safe expire_snapshots + remove_orphan_files cadence.
• You're standing up Polaris or Lakekeeper as a REST catalog and need vended credentials, namespace layout, and the correct catalog properties.
• A column type or partition scheme needs to change and you want to do it in place via schema/partition evolution.

What it does

Configures Iceberg REST catalogs (type=rest, uri, warehouse, credential/OAuth2, header.X-Iceberg-Access-Delegation for vended creds) for Apache Polaris and Lakekeeper, including namespace creation and storage config. Drives the Spark stored procedures in the system namespace: rewrite_data_files (with sort/zorder strategies and target-file-size-bytes), expire_snapshots (older_than, retain_last), remove_orphan_files, and rewrite_manifests. Designs hidden partitioning with transforms — bucket(N, col), truncate(W, col), year/month/day/hour(ts) — set via ALTER TABLE ... ADD PARTITION FIELD. Handles schema evolution (add/drop/rename/widen columns) by field ID, so changes are metadata-only.

Usage example

curl -fsSL --create-dirs -o ~/.claude/skills/apache-iceberg-lakehouse/SKILL.md https://claudeskil.com/skills/apache-iceberg-lakehouse/SKILL.md

-- Compact small files into ~512MB targets, sorting by event_time
CALL catalog.system.rewrite_data_files(
  table => 'analytics.events',
  strategy => 'sort',
  sort_order => 'event_time ASC',
  options => map('target-file-size-bytes','536870912','min-input-files','5')
);

-- Then expire old snapshots and clean orphans
CALL catalog.system.expire_snapshots('analytics.events', TIMESTAMP '2026-06-23 00:00:00', 10);
CALL catalog.system.remove_orphan_files(table => 'analytics.events');

Tips & gotchas

• Always run rewrite_manifests after heavy partition evolution — stale manifests bloat planning even when data files are fine.
• remove_orphan_files defaults to a 3-day lookback to avoid deleting files from in-flight writers; never lower it below your longest-running job.
• Adding a partition field does not rewrite existing data; old files keep their old spec, so expect mixed-spec scans until compaction.
• Vended credentials expire — set the access-delegation header and refresh tokens rather than baking long-lived keys into catalog config.

FAQ

Does expiring snapshots break time travel? Yes, for expired snapshots. Tune retain_last and older_than to keep the history window your audits and rollbacks require.

Sort vs. z-order compaction? Use sort for single dominant filter columns; use zorder when queries filter on several columns with no clear leader.

Can I rename a partition column safely? Yes — Iceberg tracks fields by ID, so renames and type widening are metadata-only and don't rewrite data.