Nutrition labels for AI tools and datasets
A standard, glanceable label on every civic tool: what it does, what data it uses, who is accountable, where it falls short. A user choosing a tool and an agent invoking one make the decision on the same disclosed facts.
The impact of agents
A registry entry tells a user or their agent that a tool was approved, not what it is. Before relying on a tool, a user needs to see at a glance what it does, what data it uses, who is accountable, and where it is known to fall short. An agent needs the same facts in a form it can read.
What serves both is a standard, glanceable disclosure: one schema, rendered for the user's eye and the agent's parser alike.
What must be verified
A structured, glanceable label has to be accurate and kept current. This is the 'ingredient list' that lets a user or agent see what they are consuming, beyond the bare fact that it was approved. The tool's publisher or maintainer must complete that label and keep it updated.
Protecting access
A disclosure standard that demanded specialist effort would exclude the small builder whose tools most often go undocumented. Users inherit that gap: the tools they meet most often disclose the least. An accuracy claim expressed as a raw statistic misleads users who aren't specialists in reading it.
Keeping the path open
- Set a minimum label small enough to publish without specialist help.
- Make it machine-readable, so an agent consumes the same label a user reads.
- State the accuracy claim as what the tool can and cannot reliably do, rather than as a raw figure.
- Make the rendered label work with a screen reader, not only display visually, or the disclosure never reaches a user who can't see it.
Response surface
A fixed label states the same required fields for every tool, extended for the higher-risk domains.
A higher-risk decision-support tool, such as a debt calculator, shows the same five fields plus a sector extension: the tested error rate against reference cases, known failure modes, the population the test data covers, and the date of the last independent check. That is the standard’s documentation, not this low-risk label’s — so the label links out rather than reproducing it.
‘Accuracy claim’ and ‘last updated’ are the two fields most often left vague. The label makes both explicit and checkable.
Maturity
- Emerging Headline
For model cards and datasheets, common yet unevenly completed even on platforms built around them, with healthcare labels the most advanced sector application.
- Frontier
For a government-specific nutrition label for civic technology tools, which has yet to be built.
Precedents
Model Cards (Mitchell et al.). The template runs nine sections: model details, intended use, factors, metrics, evaluation data, training data, quantitative analyses, ethical considerations, and caveats. Adoption is concentrated rather than universal, with models carrying a card accounting for roughly 90 percent of Hugging Face download traffic while well under half of repositories have one. A systematic analysis of 32,111 cards found training details most consistently completed, and environmental impact, limitations, and evaluation lowest.
Datasheets for Datasets (Gebru et al.). By analogy to electronics datasheets, the template runs seven sections: motivation, composition, collection process, preprocessing and labeling, uses, distribution, and maintenance. A second version focuses on interoperability and reuse, and Europeana has adapted the format for cultural heritage datasets. The template moved from preprint to Communications of the ACM, which is when it became citable as the authoritative text.
Dataset Nutrition Labels. The free, public-facing, voluntarily disclosed standard is modeled on food labels and covers provenance, quality, and intended use. It is hosted at Consumer Reports as an innovation initiative, which places a consumer-advocacy organization behind the format.
CHAI health AI nutrition labels. The open-source applied model card for healthcare AI carries developer identity, intended uses, target patient populations, model type, data types, performance metrics, security and compliance accreditations, maintenance, known risks and out-of-scope uses, bias, ethical considerations, and third-party information such as relevant clinical studies. A domain has specified the field list its own regulators and buyers need.
What carries over to agent use
The nutrition label metaphor reads clearly to users and suits government digital services. The design challenge is making the label machine-readable for agents as well as human-readable. A government pattern library can specify a minimum label schema for any tool that processes public data or advises users. The CHAI model, sector-specific labels with mandatory fields tailored to the risk domain, is a workable template for government adaptation.
Where things go wrong
The failure mode is a tool whose accuracy claims and currency are left implicit. A nutrition label mandating an 'accuracy claim' and 'last updated' field forces an explicit, checkable statement of what the tool can and cannot reliably compute. A builder can game the label itself: overstate the accuracy claim, or leave 'last updated' stale once the tool has moved on. Because the claim names a specific, checkable capability rather than a vague assurance, a user or downstream agent can test it against the tool's actual behavior and catch the mismatch.
Sources
9 references
The instrument, the operating deployment, or the official record itself.
- Model Cards for Model Reporting (Mitchell et al., FAT* '19)
- Datasheets for Datasets (Gebru et al., CACM 64(12))
- Dataset Nutrition Labels (Data Nutrition Project)
- CHAI — assurance lab certification and nutrition label for health AI (field list)
- Datasheets for Digital Cultural Heritage Datasets — Journal of Open Humanities Data
- IEEE 7003-2024 — Algorithmic Bias Considerations
Writing about the subject rather than the framework itself, including vendor commentary.