When valid documentation meets the reality of enterprise deployment, the gap between a model’s ingredients and its actual safety is where real governance begins.

The documentation for enterprise AI is developing rapidly. The formats for describing what a model is made of—the AI bill of materials (AIBOM)—are already maturing. However, suppose you hold a complete, accurate, independently validated AIBOM for every model you run. Do you know whether any of them is fit to deploy, who is accountable when one is not, or who decided what “validated” means in the first place? Documentation is not assurance.

AIBOM progress

The security community has made significant progress in standardizing elements of an AIBOM. For example, CycloneDX is now an Ecma International standard (ECMA-424) whose second edition covers machine learning models as well as software, hardware, and services, and SPDX has shipped AI and Dataset profiles in version 3.0.1 of its specification. The major governance regimes are converging on a shared vocabulary of provenance and lifecycle fields, whether the regime is a binding law like the EU AI Act, a certifiable management standard like ISO/IEC 42001, or a voluntary framework like the NIST AI Risk Management Framework. Adoption is still building, as Marek Grác and Martin Ukrop’s article “Unpacking AI’s black box,” (RHRQ Winter 2025-26) described, but the direction is no longer in question. Within a few years, holding a bill of materials for a deployed model will be commonplace.

That is good news. Grác and Ukrop’s article went beyond the paperwork, setting out the bigger challenge of validating that a model matches what its AIBOM claims and describing the work that Red Hat and Brno University of Technology (BUT) are doing on model structure inspection and on detecting malicious fine-tuning. That research answers a question I am not going to try to answer here: is this model what it says it is?

What the AIBOM misses

A bill of materials is essentially an ingredient label. It may be accurate, useful, and necessary, but does it tell you whether the dish is safe to serve? The gap is not in the documentation. It’s between a high-level obligation—be transparent, manage risk, ensure human oversight—and an implemented, evidenced, auditable practice that somebody outside your organisation can check. That gap, the translation layer between an obligation and an implemented practice, is what the rest of this article discusses.

The current AIBOM formats show where the join should be. The SPDX AI Profile includes a field for the safety risk assessment of an AI package, taking values of high, medium, or low, which is a governance conclusion stored in a documentation artifact. The field holds the answer, but it doesn’t show the criteria used to reach it, the evidence behind it, the role that signed off on it, or what should happen when the model is quantized and the answer then needs updating. None of that is a criticism of SPDX: a schema is the right place for a conclusion and the wrong place for the process that produced it. But it does show that the documentation layer has run up against the layer above it and left something missing.

Open weight models are where this matters most.

Open weight models are where this matters most. Once weights are public, anyone with a GPU can fine-tune them into a variant the original release didn’t cover. The obligation to govern the model remains, but the methods the author would normally use to meet it have gone; it can no longer recall the weights, patch them, or restrict who runs them. What is left is a diffuse collection of deployers, each with obligations written as though one party still held control of the artifact. That makes open weights the hardest test case of a general problem, not a separate problem of their own.

To clarify this article’s scope: a malicious actor is a real problem, but that problem belongs to security and counter-proliferation. This article is about the good-faith deployer who wants to do the right thing and needs help working out what the right thing actually is.

What software supply chains already learned

Software supply chains addressed this issue a decade ago, and the bill of materials turned out to be just the bottom layer of a governance pyramid.

Three layers were added above the BOM (see Figure 1).

• Provenance and signing first, because an inventory without that history is just a list: build attestations and signatures made the claim checkable.
• Then lifecycle and vulnerability management: advisories, errata, patch cadence, end-of-life dates, the part that turns “we know what we have” into “we know what needs attention and when.”
• Finally, policy as code: rules written as executable automation that runs, rather than a document that asserts.

Red Hat Satellite is lifecycle governance in production. Content Views pin a known set of packages to a known set of hosts, errata management tells you which of those hosts is exposed to which advisory, and the result is “what is deployed where, and what does it need” becomes a customizable and automatable report. Ansible is policy as code in production. The difference between an asserted control and an implemented one is a playbook that runs on a schedule, logs what it changed, and can be handed to an auditor with no need for explanation. In both cases the value is not the inventory; it’s that the obligation has somewhere to live.

The infrastructure and processes, not just the file formats, are what AI governance still needs to build.

That process took years, and it wasn’t created by standards bodies. Instead, it was built by admins who had inventories but still couldn’t answer the questions the auditors were asking. The AIBOM formats are already moving in the same direction; ECMA-424 now has structures for asserting claims, attestations, and supporting evidence. That is a good step forward, but a container for an attestation isn’t the same thing as the process that produces one, the role that signs it, or the processes that keep it accurate.

Open source didn’t stop at documentation; it went on to create the infrastructure and processes that made obligations implementable. The infrastructure and processes, not just the file formats, are what AI governance still needs to build.

Five tests for any AI governance obligation

Outside those systems and processes, I believe any AI governance obligation has to pass five tests before it counts as an implementable practice.

• Specifiable. Can it be written as a defined control? “Ensure robustness” is a hope. “Reject any model whose provenance chain is unsigned” is a control, because it has a subject, a condition, and an action.
• Evidenceable. What proves it happened? A log, an attestation, or a pipeline record are all evidence. If compliance isn’t recorded then your auditor might well have questions.
• Accountable. Whose name is on it when it fails? Specifically, it must be a named role that signs off deployment, not “the organization.” Diffuse accountability is the failure mode that open weights make structural. The model’s author is known, but the party answerable for any given deployment is the deployer running it, not the lab that published it, and there are many such deployers.
• Verifiable. Can a third party check it without having to trust the vendor’s word, through reproducible checks, published criteria, or independent re-validation? In governance, you may reasonably rely on data a trusted partner attests to. With verification, you rely on nobody: you reproduce the check yourself. 
• Maintainable. Does it survive the model’s lifecycle, including fine-tunes, quantized variants, upstream releases, or hardware changes? A control that is only valid at download time means nothing, because not everything you run in production is exactly what was downloaded.

The tests judge the obligation, not the model. That distinction is what makes this framework complementary to verification research rather than competing with it. Model structure inspection and behavior testing of the kind Grác and Ukrop describe are ways of satisfying test four for a particular claim; they are inputs to the framework. The framework itself is agnostic about which technique it’s based on.

We can walk through these five tests with a real obligation. The EU AI Act requires providers of general-purpose AI models to draw up and keep up-to-date technical documentation and to make specified information available to downstream providers. Specifiable: yes, the annexes list the fields. Evidenceable: yes, the document is the artifact. Maintainable: partly, since “keep up to date” is a time-based obligation with no timing in it. Accountable: named at the provider, not at the deployer, and that is where much of the practical risk really is. Verifiable: this is the weak point, because a downstream deployer receiving documentation has no clear way of checking it. That is an indication pointing directly at where verification research is needed. Take a different regime, ISO/IEC 42001 or the NIST AI Risk Management Framework or one of the emerging US state statutes, and the pattern of strong and weak tests changes, but the tests do not.

Who decides what “validated” means?

When a vendor publishes a monthly collection of third-party open models declared “validated for enterprise use,” that word is making a huge claim, and most of it is not technical.

Red Hat’s own validated models collection makes a good object of study, partly because I work there, and partly because it is remarkably well documented, which is exactly what makes it testable. Models are published on Hugging Face, refreshed roughly monthly in line with upstream vLLM releases, and validated using open source tooling including GuideLLM, the Language Model Evaluation Harness, and vLLM itself, specifically so that customers can reproduce the results.

What do the tests say? Specifiable: strong, because validation means a defined set of evaluations against defined hardware and configurations. Evidenceable: strong, because the collection, the benchmarks and the accuracy evaluations are all public. Verifiable: better than most, since open source evaluation tooling means a customer can rerun the checks, although the vendor still sets the bar for what counts as a pass. Maintainable: reasonable, and admits its limitations, with forward compatibility offered for at least two subsequent vLLM minor versions and an explicit reservation of the right to stop validating a model at any time.

“Accountable” is where it gets interesting. The published terms state clearly that these third-party models are not supported, indemnified, certified, or guaranteed in any way, which is a perfectly reasonable position for any vendor to take on models it didn’t train. It also means the word “validated” here carries engineering assurance and no liability at all. Both of those things are true, and a deployer who reads “validated” as “somebody else is answerable for this” has misread it.

While I believe this is a great thing, I also think there’s a structural question worth asking: as trustworthy-use governance emerges vendor by vendor, each with its own bar, its own tooling, and its own disclaimer, does an ecosystem built on open weights end up with a set of competing private definitions of validated?

Where do we go from here?

The supply chain analogy has limits: you can’t read billions of weights the way you read source code, you can’t patch a model the way you patch a package, and a fine-tune is more of a fork than an update. Some of the workings transfer cleanly, signing and provenance most obviously. Some of it breaks, and mapping that boundary carefully is a research task rather than an engineering one.

What I am sure about is the nature of the problem. The documentation layer is being solved, and solved well, by standards bodies. The verification layer, which asks whether a model matches its claims, is being worked on by researchers, including the Red Hat-BUT team behind  “Unpacking AI’s black box.” The translation layer above it, which asks whether an obligation has become an implemented practice, has no obvious owner, and it is the layer where enterprises are actually stuck. Figure 2 marks the point where “validated” changes hands. Below that, “validated” is a technical property of the model: does it match its claims? Above it, “validated” is a judgement someone makes and answers for: who sets the bar, who signs?

That makes it a good place for practitioners, researchers, and standards people to meet. If you are working on any part of it, or you think one of the five tests is wrong, I’d love to hear from you at mott@redhat.com.