Reading the archive

Every other page here is a client. So is a DuckDB shell, a notebook, and a BI tool pointed at the directory. None of them is privileged, because the contract is the directory, not a library — and this page is where you learn to read it.

Every result below is a real query, run by DuckDB in your browser against the same parquet the dashboard reads. The SQL is above each one, and you can change the last one.

What the solve job wrote

One archive per scenario. This is base, by shape rather than by file — every quantity is its own directory of per-period slices:

runs/base/
├── model.yaml                 3.0 kB   the spec, as solved
├── sources.parquet            1.4 kB   (run, source, digest)
├── sources/                 11 files   every input, as solved
└── answer/
    ├── objective.parquet      2.9 kB   one row per period: status, objective
    ├── metrics.parquet        3.5 kB   one row per period: size, seconds
    ├── primal/                     3   build, p, total — 4 slices each
    ├── dual/                       4   accumulate, balance, capacity, carbon — 4 slices each
    └── expression/                 3   capex, emissions, opex — 4 slices each

Three kinds of thing are in there. model.yaml and sources/ are what was solved — the spec and every input, so the run reproduces. answer/ is what came back: primal/ per variable, dual/ per constraint, expression/ per named quantity, one directory each. objective.parquet, metrics.parquet and sources.parquet are the record: one row per period saying how it terminated, what it cost to build and solve, and what each input's bytes digest to.

What a query gets back

A value frame carries the model's own dimensions and a value. Nothing else, and no index:

Those column names — year, generator — are the model's, not this repository's. They come from the spec that was solved, which is why a reader who has never seen the model can still group by generator, and why two quantities keyed the same way join without a mapping table.

Three rules, one query each

The record tables carry run on every row, so they concatenate across archives with a single glob and need no path parsing:

A value frame does not carry run, because it carries the model's columns only. Reading across archives, you derive it from the path — filename = true in DuckDB, one regexp_extract. The site's loader has already done that here, which is why run is a column above.

The catalogue is the tree. Which quantities exist and which dimensions key each is read off the directory names and the parquet schema. Nothing is declared twice:

Run one yourself

The tables above are registered; edit the query and it re-runs. objective, metrics, digests, total and emissions are in scope.

The same thing, in your own tools

Neither of these imports lpspec, and neither imports this repository's warehouse.py. Both answer the four numbers the pathway page leads with, and tests/test_clients.py holds them to each other on every archive in the directory — so a drift between them fails CI rather than reaching this page.

"""The four headline numbers off an archive, in polars, with no lpspec and no client library.

    uv run python clients/headline.py runs/base

The archive is a table per glob, so reading it needs nothing this repository
ships. `headline.sql` answers the same four questions in DuckDB, and
`tests/test_clients.py` holds the two to each other.
"""

import sys
from pathlib import Path

import polars as pl


def headline(run: Path) -> dict[str, float]:
    frames = lambda name: pl.read_parquet(run / 'answer' / name / '*.parquet')
    at = lambda frame, year: frame.filter(pl.col('year') == year)['value'].sum()

    objective = pl.read_parquet(run / 'answer/objective.parquet')
    first, last = objective['year'].min(), objective['year'].max()
    emissions, fleet = frames('expression/emissions'), at(frames('primal/total'), last)
    clean = pl.read_parquet(run / 'sources/rate.parquet').filter(pl.col('value') == 0)['generator'].to_list()

    return {
        'pathway_cost': objective['objective'].sum(),
        'emissions_cut': 1 - at(emissions, last) / at(emissions, first),
        'zero_carbon_share': frames('primal/total')
        .filter(pl.col('year') == last, pl.col('generator').is_in(clean))['value']
        .sum()
        / fleet,
        'carbon_price': -at(frames('dual/carbon'), last) + 0.0,
    }


if __name__ == '__main__':
    for name, value in headline(Path(sys.argv[1] if len(sys.argv) > 1 else 'runs/base')).items():
        print(f'{name:<18} {value:>15,.4f}')

uv run showcase-solve --runs runs     # once, if runs/ is not there yet
duckdb -c ".read clients/headline.sql"