---
name: apache-iceberg-lakehouse
description: "Operate an Apache Iceberg lakehouse — REST catalogs, compaction, snapshot expiry, partitioning, and schema evolution. Production Iceberg lakehouse ops: wire REST catalogs (Polaris, Lakekeeper) with vended credentials and namespaces, run table maintenance via rewrite_data_files compaction, expire_snapshots, remove_orphan_files, and rewrite_manifests. Covers hidden partitioning with partition transforms (bucket, truncate, day), partition-spec and schema evolution, sort orders, V3 features (deletion vectors, row lineage), copy-on-write vs merge-on-read, and engine wiring across Spark, Trino, dbt, and Snowflake-managed Iceberg."
license: MIT
---

# Iceberg Lakehouse Engineer

Operate a production Apache Iceberg lakehouse end to end: wire up REST catalogs, keep tables healthy with maintenance procedures, design hidden partitioning, and evolve schemas without rewriting data. Built for data engineers and platform teams running Iceberg on Spark, Trino, or Flink against object storage (S3, GCS, ADLS) who are tired of guessing which maintenance job to run and why query planning has gotten slow.

## When to use it

- A streaming or CDC pipeline has produced thousands of tiny files and scans are crawling — you need `rewrite_data_files` with the right target file size and a sane rewrite strategy.
- Snapshot metadata and old data files are piling up storage cost, and you need a safe `expire_snapshots` + `remove_orphan_files` cadence.
- You're standing up Polaris or Lakekeeper as a REST catalog and need vended credentials, namespace layout, and the correct catalog properties.
- A column type or partition scheme needs to change and you want to do it in place via schema/partition evolution.

## What it does

Configures Iceberg REST catalogs (`type=rest`, `uri`, `warehouse`, `credential`/OAuth2, `header.X-Iceberg-Access-Delegation` for vended creds) for Apache Polaris and Lakekeeper, including namespace creation and storage config. Drives the Spark stored procedures in the `system` namespace: `rewrite_data_files` (with `sort`/`zorder` strategies and `target-file-size-bytes`), `expire_snapshots` (`older_than`, `retain_last`), `remove_orphan_files`, and `rewrite_manifests`. Designs hidden partitioning with transforms — `bucket(N, col)`, `truncate(W, col)`, `year/month/day/hour(ts)` — set via `ALTER TABLE ... ADD PARTITION FIELD`. Handles schema evolution (add/drop/rename/widen columns) by field ID, so changes are metadata-only.

## Usage example

```sql
-- Compact small files into ~512MB targets, sorting by event_time
CALL catalog.system.rewrite_data_files(
  table => 'analytics.events',
  strategy => 'sort',
  sort_order => 'event_time ASC',
  options => map('target-file-size-bytes','536870912','min-input-files','5')
);

-- Then expire old snapshots and clean orphans
CALL catalog.system.expire_snapshots('analytics.events', TIMESTAMP '2026-06-23 00:00:00', 10);
CALL catalog.system.remove_orphan_files(table => 'analytics.events');
```

## Tips & gotchas

- Always run `rewrite_manifests` after heavy partition evolution — stale manifests bloat planning even when data files are fine.
- `remove_orphan_files` defaults to a 3-day lookback to avoid deleting files from in-flight writers; never lower it below your longest-running job.
- Adding a partition field does not rewrite existing data; old files keep their old spec, so expect mixed-spec scans until compaction.
- Vended credentials expire — set the access-delegation header and refresh tokens rather than baking long-lived keys into catalog config.

## FAQ

**Does expiring snapshots break time travel?** Yes, for expired snapshots. Tune `retain_last` and `older_than` to keep the history window your audits and rollbacks require.

**Sort vs. z-order compaction?** Use `sort` for single dominant filter columns; use `zorder` when queries filter on several columns with no clear leader.

**Can I rename a partition column safely?** Yes — Iceberg tracks fields by ID, so renames and type widening are metadata-only and don't rewrite data.