How to inspect a Parquet file: a practical guide
Parquet is the default storage format for data lakes on AWS: Athena, Glue, Redshift Spectrum and EMR all read and write it. Because it is a binary, column-based format, you can't open it in a text editor. This reader shows its structure and contents in a few seconds, which is usually all you need when a query returns the wrong results.
What you see when you open a file
- Schema: every column with its physical type (INT32, INT64, DOUBLE, BYTE_ARRAY…) and converted type (STRING, DATE, TIMESTAMP_MILLIS, DECIMAL…). This is the first thing to compare against your Glue table definition.
- Metadata: row count, number of row groups, the compression codec (Snappy, Gzip, Zstd) and the tool that wrote the file (
created_by). - Rows: the data itself, which you can filter and export to CSV or JSON.
Worked example: Athena returns NULLs for one column
A common case: a pipeline writes Parquet to S3 and Athena shows NULL for a column that clearly has data. Open one of the files here and compare the schema with the table:
- Column name: Athena matches Parquet columns by name by default. A file column called
customerIddoesn't map to a table columncustomer_id. - Type: the file stores
event_timeas INT64 milliseconds, but the table declarestimestamp. Fix the table type, or convert when writing. - Mixed files: open an older file from the same prefix too. If its schema differs, the writer changed, and the table must handle both versions.
Common Parquet problems
- Too many small files. Thousands of tiny files slow every query. Aim for roughly 128 MB–1 GB per file, and compact with Glue, Spark or Athena
CTAS. - Wrong row group size. Row groups that are too small reduce compression and skipping. Writers usually default to 128 MB.
- Timestamp precision. INT96 timestamps from older Spark versions and microsecond/nanosecond INT64 values are read differently by different engines. Check the column's type here.
- Decimals stored as doubles. Money values written as DOUBLE pick up rounding errors. Use DECIMAL(precision, scale).
Creating test files
Need a file to try the reader, or a fixture for a pipeline test? Download one of the ready-made datasets from the sample Parquet files page, or create your own with pandas:
import pandas as pd
df = pd.DataFrame({"id": [1, 2], "name": ["a", "b"]})
df.to_parquet("sample.parquet", engine="pyarrow", compression="snappy")
Converting JSON logs before loading them? The JSON to NDJSON converter prepares line-delimited JSON for Athena and Glue.
About file handling
Files up to 50 MB are supported. Your file is sent to our server only so it can be parsed, and it is deleted when the request ends; nothing is stored. For sensitive production data, use a sample or a masked extract.