# Parquet and Iceberg: An Overview

> A pile of containers is not a shipment. A shipment is containers plus a manifest.Apache Iceberg is the manifest. It is not a file format, it is a table format:…

## The Container and the Manifest

For most of my career, data moved between systems as a CSV file. Everyone reading this has been burned by one. A comma hiding in an address field, a date that parses three different ways, a file that is secretly Latin-1, a header row that is there on Tuesday and gone on Wednesday. The CSV is the handshake agreement of data formats. It works fine right up until something valuable is riding on it. There is a better container, and most of the industry has already standardized on it: Parquet.

 **What a Parquet file is**

Before the 1950s, cargo went onto ships as loose barrels and sacks, hand-carried by longshoremen, packed differently on every dock. Then the shipping container arrived: one standard box, and suddenly every crane, every ship, every train, and every truck on earth could handle your freight without asking what was inside.

Parquet is the shipping container of data.

It is an open, binary, column-oriented file format from the Apache project. Inside the file, values are stored column by column in row groups, each column encoded and compressed to a fraction of its raw size. The footer carries the schema and minimum and maximum statistics for every chunk, so an engine (such as [pgColumnar](<https://commandprompt.github.io/pgcolumnar/>)) reading the file knows the types without guessing and can skip whole sections that cannot match a query. A Parquet file cannot lie to you about the data it contains.

And because it is a standard container, every crane can lift it. Spark, DuckDB, pandas, Trino, Snowflake, BigQuery, Athena, Flink, and yes, Postgres through extensions, all read and write the same file. Write it once from one system and query it from six others, with no conversion and no argument about what a date looks like. One format, every platform.

 **What it is good for**

Column questions over large data. Sum this, count that, average the other, filtered by time. The columnar layout means an engine reads only the columns a query asks for, compression makes the files small and cheap to store, and the footer statistics let engines skip most of the file entirely. Ten times smaller than the same data as CSV is normal, and scans run faster on top of it. Park the files in object storage like S3 and you have an archive that costs pennies and answers questions. With Parquet the file is immutable. You write it once and you never update it.

 **The value of Apache Iceberg**

That immutability can be a problem. One Parquet file is a container. A data lake is thousands of them in a bucket. So which files make up the customers table right now? What happens when two jobs write at the same time? What happens when the schema adds a column?

A pile of containers is not a shipment. A shipment is containers plus a manifest.

Apache Iceberg is the manifest. It is not a file format, it is a table format: a metadata layer that records exactly which Parquet files make up a table at every point in time. That one idea buys you things we used to think required a warehouse. Transactions on object storage, so writers never corrupt readers. Schema evolution without rewriting the data. Time travel, so you can query the table as it stood yesterday. And because Iceberg is open and engine-neutral, the table stops belonging to any one vendor. Netflix built it for exactly that reason, and now nearly every engine and warehouse speaks it.

 **Using them together**

Parquet holds the data. Iceberg holds the truth about the data. Together they give you a warehouse’s behavior, transactions, history, and evolving schemas, on storage that costs pennies, readable by every tool you own and every tool you have not bought yet. Store the data once. Query it from anywhere.

To be clear, if your data fits comfortably in Postgres and one team queries it, you do not need any of this. Do not build a port for a rowboat. But the moment your data outgrows one system, or one engine, or one vendor’s pricing page, the container and the manifest are how you keep owning it.

The next time you are about to export a CSV, stop and write a Parquet file instead. And when that bucket of files starts acting like a table, give it a manifest.

---
[View this page online](https://www.commandprompt.com/blog/parquet-and-iceberg-an-overview/)