Picture a quality technician at a mid-sized pharmaceutical manufacturer. She has a thirty-page document about cleaning and line clearance between production batches, and her manager wants it turned into a course for the operators on the floor by Friday. She uploads it to the learning platform, describes the course she wants, and a few minutes later she has six modules, a quiz and a summary card.
Somewhere in that document, a hold time is wrong. Maybe it was copied from an older version. Maybe a unit was mistyped. She didn’t notice, because she wasn’t looking for it. The model didn’t notice either, or it did and quietly “fixed” it without telling anyone. The course is published. Two hundred operators complete it.
This is a scenario, not an incident report. But it is the scenario that has been sitting in the back of my head since we started putting generative AI into the hands of the people who build training inside our clients’ organisations. This week we shipped the thing we built to deal with it. This post is about why, how we made the main decisions, and what it still doesn’t solve.
We moved the work. We also moved the risk.
Over the last few years we made a deliberate choice at GFoundry: give clients AI tools to produce their own content in the backoffice, and make those tools fast enough that they actually use them. The Learn Designer, where you upload a document and get a course back, is the clearest example. Today, most of the content on the platform is produced by the clients themselves.
What we were slower to say out loud is that we had also moved the responsibility. The person writing a course is now often a team lead, an HR generalist, or a technician who knows the subject but has never had to think about what happens when a mistake in a learning module is followed to the letter. In most domains that mistake is embarrassing. In pharmaceutical manufacturing, food production, occupational safety or emergency response, it can hurt someone.
AI didn’t create this problem. People have always written wrong things in training material. What AI changed is the friction. Producing a course could take weeks, and somewhere in those weeks a second person usually read it. When it takes ten minutes, the second reader is often the first thing to go.
Two ways to be wrong
When we looked at how errors actually reach learners through an AI-generated course, we found two distinct failure modes, and they pull in opposite directions.
The first is the obvious one. The source document contains a wrong statement, and the model carries it into the course intact, now formatted nicely, split into modules and reinforced with a quiz question. A wrong instruction with a quiz attached is more persuasive than a wrong instruction in a PDF.
The second is less obvious, and in some ways worse. The model notices that something in the source looks wrong and silently corrects it. The course is now factually better than the document it came from, and nobody knows. The SOP still says one thing, the training says another, and the auditor who compares them finds a discrepancy no one can explain. There is a name for this in the research literature: a paper at EACL 2026 calls it harmful factuality hallucination, and reports that it gets worse, not better, as models get larger.
A tool that only catches the first failure makes the second one more likely. If you tell a model “make sure nothing in here is wrong”, it will happily start fixing things. So whatever we built had to do something slightly counterintuitive: find the problems, and refuse to fix them silently.
“Human in the loop” assumes there is a second human
The regulatory conversation is converging on a clear answer. In April 2026, the FDA issued what DLA Piper describes as the first warning letter citing a drug manufacturer for relying on AI in its cGMP obligations. The firm had used AI agents to draft specifications, procedures and master production records without adequate review by its quality unit. When FDA pointed out a missing process validation, the firm said it hadn’t known about the requirement because the AI agent hadn’t flagged it. The draft EU GMP Annex 22 goes further: generative AI and LLMs are out of scope for critical GMP applications, and in non-critical uses, qualified personnel stay responsible for the output.
As a piece in BioPharm International put it, none of this is a new rule. It’s the old rule, applied to a new kind of draft: someone qualified reviews it, someone accountable approves it.
I agree with all of it. My problem is with the assumption underneath. “Human in the loop” works when the human in the loop is not the same person who made the mistake. In a large pharma company with a dedicated quality unit, that’s a reasonable assumption. In a manufacturer of a few hundred people in Portugal, the person who writes the training, reviews the training and publishes the training is often one person, on a Thursday afternoon, with three other things to finish.
We can’t fix that organisational reality from a software platform. What we can do is make sure the loop has something in it besides the author.
What we built
It’s called Gi Guardian. It reads learning content and looks for statements that contradict established external consensus or regulation, in two places: in the Learn Designer, before a course is generated from a document, and when someone presses Publish on a piece of learning content.
The interesting part isn’t that it checks. It’s the set of decisions about what it’s allowed to do with what it finds. That’s where the real design work was.
It never checks your own procedures. Internal deadlines, roles, approval chains and processes belong to the organisation. We have no way of knowing whether your batch record review happens in 24 hours or 48, and a tool that pretends to know will be wrong often enough to be ignored. Gi Guardian only checks statements against external references: consensus and regulation that exist outside the client.
Severity follows risk, not topic. This was the hardest call. A checker that blocks too often gets switched off, and then it protects nobody. So a block requires three things at once: the content is in a critical area (health, medication safety, food safety, occupational safety, fire and emergency response, transport safety), the contradiction is confirmed, and following the statement could cause serious harm. Anything short of all three is a note. A wrong date in a history-of-the-company module is never a block, however confidently wrong it is.
It explains, it doesn’t rewrite. Every finding says where it is, what the content says, what the reference establishes, why it matters and how to correct it. The correction is a suggestion. The author makes the change in their own content, so the course and the source stay aligned, and the second failure mode above doesn’t get a way in.
Nothing unread is approved. If part of the content can’t be read yet, the content stays “In verification” instead of being published. We would rather be slow than pass something we didn’t look at.
It says what it didn’t check. Gi Guardian verifies text. It does not verify images. Slides that contain only images are listed explicitly in the review, so a clean result can never be read as “the diagram on slide 12 is fine.” We wrote recently about what happens when a system’s doubts don’t reach the person who decides. This is the same rule, applied before publishing.
It works in the content’s language. English, Portuguese, Spanish, French, Italian and German. Operational training is written in the language of the people doing the work, and a safety checker that only works in English would miss the content that matters most.
Every verification is recorded against the version it checked, with its history. When an auditor asks “what checks did this module go through before it was published?”, there is an answer.
The trade-offs we accepted
There are things in this design I’m not fully comfortable with, and I’d rather name them than pretend otherwise.
The “confirmed contradiction” threshold means some real errors will come through as notes rather than blocks. We chose that on purpose, because the alternative is a tool that blocks on suspicion, and suspicion from a language model is cheap. But it means the notes matter, and notes can be ignored.
Blocking at all is a form of paternalism. We are telling a client’s employee that they can’t publish their own content until they change it. We limited this to the narrow band where the consequence is physical harm, but the line between “serious” and “not serious” is a judgement, and it’s a judgement made by a model.
And the asymmetry in activation is deliberate but imperfect. The check before generation is on for every organisation that uses the Learn Designer. The check on publish is switched on per organisation, at the client’s request, because it changes their publishing workflow and some of them have their own review processes that we shouldn’t override. That means the protection depends partly on a client knowing to ask for it.
What’s still unresolved
The checker is also a language model. We designed Gi Guardian around the idea that a generative model shouldn’t be the last line of defence for content that can hurt someone, and then we built the second line out of the same material. It’s a different job, with different instructions, explicitly asked to find problems rather than produce content, and it explains every decision so a human can disagree with it. But it can be wrong, and I don’t think anyone building this kind of system should pretend otherwise.
Consensus moves. A dosing guideline that was correct when a course was published can be superseded a year later. Today, Gi Guardian checks at generation and at publish. It doesn’t go back and re-read everything already live when the world changes. That’s an obvious next problem, and not a small one.
It is not a validated GMP control, and we shouldn’t let anyone treat it as one. Annex 22 is clear that LLMs don’t belong in critical GMP applications, and the FDA’s position is that the quality unit approves, not the software. Gi Guardian is a net under the author, not a signature on the document. A regulated client still needs a qualified person to own what gets taught.
The question I’m left with is the one we started with, in a different form. We made it fast for anyone in an organisation to produce training. The more successful that is, the more content there is, written by more people, reviewed by fewer. A tool like this narrows the gap. I’m not sure any tool closes it. At some point the answer is an organisational one, about who is allowed to teach what, and I don’t think software gets to make that decision for anyone.
But it can make sure that when the hold time is wrong, someone finds out before the operators do.