Tiny model, full provenance — where Apache Sourcelume is, and where it is heading

By Achim Nierbeck and Jamie Goodyear Ask anyone who ships a language model a simple question: which data was this trained on, and what license comes with it?…

By Achim Nierbeck and Jamie Goodyear Ask anyone who ships a language model a simple question: which data was this trained on, and what license comes with it? The honest answer today is a shrug with a bibliography.

Training corpora are assembled from dozens or hundreds of sources, licensing information is missing, wrong, or unverifiable — a 2024 audit found license-omission rates above 70% across popular dataset-hosting sites — and what a producer does publish is usually a claim you have to take on faith: a paragraph in a paper, a model card, a PDF. Nothing you can check. What Sourcelume is That is the gap Apache Sourcelume is built for.

It is not a certification authority and it does not decide whether anyone’s licensing claims are true. It is plumbing: a versioned, JSON-LD metadata specification for dataset origin, custody chain, and licensing; a reference registry built on Apache Atlas that stores and serves those records through plain REST APIs; and tooling that turns “documenting your data” from a favor into a build step.

The point is narrower and more radical than a licensing database: when a claim is published as a record, in a shared shape, at a stable identifier, then others can finally check it — resolve it, compare it, count it — instead of trusting it. Datasets, and then models The dataset side of this is already real. The spec’s ProvenanceRecord has been exercised against a public example we could find: the DFM Mímir training corpus.

We verified dataset by dataset, straight from the report’s own tables. But records about datasets were only ever half of the question. On the model side, the situation is inverted: model producers know exactly what they trained on, but there has been no neutral place to say it. When a downstream consumer recently asked whether they could fetch structured provenance for a model — machine-readable, not prose — the piece that would answer them did not exist.