Choosing Between JSONL and Parquet

DuckDB

The shortest path: a single binary, no runtime to install. DuckDB documents read_ndjson as an alias for read_json with the format parameter set to newline-delimited.

COPY (SELECT * FROM read_ndjson('data.jsonl')) TO 'data.parquet' (FORMAT PARQUET);

Glob a whole partition, pick a codec, and pin the types instead of letting them be inferred:

-- every file in a partition, zstd-compressed output
COPY (SELECT * FROM read_ndjson('events/2026-07-*.jsonl.gz'))
  TO 'events-2026-07.parquet' (FORMAT PARQUET, COMPRESSION ZSTD);

-- declare the schema instead of trusting inference
COPY (
  SELECT * FROM read_ndjson('data.jsonl',
    columns = {id: 'BIGINT', user: 'VARCHAR', amount: 'DECIMAL(12,2)', ts: 'TIMESTAMP'})
) TO 'data.parquet' (FORMAT PARQUET);

Declaring columns is worth the extra lines on any recurring job: inference samples the file, so a rare string in an otherwise-numeric column can quietly change the output schema between runs.

Polars

Use the lazy API so the file is never fully materialized. Polars documents sink_parquet as evaluating the query in streaming mode, which "allows streaming results that are larger than RAM to be written to disk."

import polars as pl

# streaming: input larger than memory is fine
pl.scan_ndjson("data.jsonl").sink_parquet("data.parquet", compression="zstd")

# eager, for files that comfortably fit in RAM
pl.read_ndjson("data.jsonl").write_parquet("data.parquet")

Prefer scan_ndjson plus sink_parquet in any pipeline: the lazy version keeps peak memory bounded regardless of input size, which is usually why you had JSONL in the first place.

pandas and PyArrow

pandas reads line-delimited JSON with lines=True, which is also the only mode in which chunksize is accepted - the docs are explicit that it "can only be passed if lines=True".

import pandas as pd

df = pd.read_json("data.jsonl", lines=True)
df.to_parquet("data.parquet", engine="pyarrow", compression="zstd", index=False)

For a file too large to hold in memory, stream it through PyArrow's incremental writer. The first chunk fixes the schema, so an inferred type that shifts later will raise:

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

writer = None
for chunk in pd.read_json("data.jsonl", lines=True, chunksize=100_000):
    table = pa.Table.from_pandas(chunk, preserve_index=False)
    if writer is None:
        writer = pq.ParquetWriter("data.parquet", table.schema, compression="zstd")
    writer.write_table(table)
if writer:
    writer.close()

Pass an explicit pa.schema(...) to ParquetWriter when the data is at all irregular. Letting the first 100,000 rows decide the types of a ten-billion-row file is the kind of implicit contract that fails in production.