Over the last four articles in this series, I’ve argued a single, slightly stubborn idea: in regulated work, you don’t make AI trustworthy by chasing a bigger, smarter model. You make it trustworthy by wrapping it in structure you can read, version, and sign off on - the Skill, the harness, the small specialized models, the maintenance discipline. The instruction layer, not the model, is where compliance lives.
That’s a comfortable thing to claim on a personal blog. It’s a different thing when a research team at Microsoft builds a system that does exactly that - and 11,000 people star it on GitHub in a matter of weeks.
The project is called SkillOpt. It was never written for pharma. Nobody involved was thinking about GxP, audit trails, or medical inquiry responses. And that’s precisely why it’s the strongest piece of outside evidence I could ask for. When people solving a completely different problem independently arrive at your architecture, the architecture is probably right.
Let me show you what they built, why it’s a direct proof of everything this series has argued, and - the part that matters most for you - why reading and improving these files does not require you to be a programmer or a machine-learning expert.
Why This Matters for You
Here’s the whole article in one paragraph, in case you’re reading this between two meetings:
Microsoft built a system that makes an AI agent measurably better at its job without touching the AI model at all. Instead, it improves the instruction file the agent follows - a plain-text document, close to an SOP - and it only keeps an improvement if that improvement passes a validation check. The model stays frozen. The thing that gets better is a document a human can open and read. That is the harness thesis, built by someone else, proven on public benchmarks. And the artifact at the center of it is text you could review in your browser right now.
If you take one thing from this: the improvable, auditable part of a compliant AI system is a document, not a black box. Microsoft just demonstrated it at scale.
What Microsoft Actually Built
Strip away the machine-learning vocabulary and SkillOpt does something almost mundane.
An AI agent is given a task and a skill file - written instructions telling it how to do that task. SkillOpt runs the agent, watches where it succeeds and where it fails, and then makes small, bounded edits to the instruction file: add a line here, delete a redundant one there, rewrite an ambiguous step. Crucially, it keeps an edit only if the edited instructions score better on a held-out set of test cases the agent hasn’t seen. If an edit doesn’t provably help, it’s thrown away.
Repeat that loop, and you converge on a compact, high-performing instruction file - the authors report it’s typically only 300 to 2,000 words long. The model underneath never changes. The reported results are substantial: on one frontier model, the optimized skill file lifted accuracy by more than 23 points in direct chat, with no added cost at run time.
Now, the one moment of machine-learning framing, because it’s genuinely useful and then I’ll drop it: the team borrowed the discipline of how you train a neural network - work in rounds, make small controlled changes, always check against data the system hasn’t seen, keep only what verifiably improves the result. They applied that rigor not to the model’s internal weights, but to a text document. They “train” the instructions.
That’s the clever part. Forget epochs and learning rates. The takeaway is one sentence: they optimize the thing you can read, and they leave the thing you can’t alone.
This Is the Harness Thesis, Independently Confirmed
Read that mechanism again and map it onto the four articles you’ve already read here. The overlap isn’t loose or metaphorical. It’s point for point.
-
The Skill is the trainable layer (Part 1). I argued that the reusable, plain-text instruction set - your SOP written for the AI - is the real unit of work. SkillOpt treats that exact file as the thing worth improving. Not the model. The document.
-
A validation gate decides what counts (Part 2). The core of the harness is that nothing advances unless it passes a software check. SkillOpt’s rule - keep an edit only if it strictly improves a held-out score - is that same validation gate, now pointed at the instruction file itself. An improvement isn’t accepted because it sounds good. It’s accepted because it’s proven.
-
Optimize the glass box, not the black box (Part 3). I made the case that a frozen, inspectable setup beats an opaque one you can’t explain to an auditor. SkillOpt freezes the model - the opaque part - and pours all the improvement into the transparent, readable part. Glass box by construction.
-
Improvement is a deletion problem too (Part 4). SkillOpt’s edits aren’t only additions. It deletes redundant instructions when deletion helps the score. The very shape of the maintenance argument - that a better system is often a smaller, cleaner one - is baked into how the tool works.
I did not know SkillOpt existed when I wrote those pieces. That’s the point. A general-purpose research tool, built for coding agents, converged on the same four principles this series derived from pharma’s quality culture. Two very different starting points, one architecture. That convergence is the evidence.
Let Me Correct My Own Framing
When I first looked at SkillOpt, the easy headline was “see - you don’t need to fine-tune the model.” I want to be honest and not sell you that, because most Medical Affairs teams aren’t fine-tuning models in the first place. You’re working with retrieval and prompting on top of a model someone else trained. “No fine-tuning” isn’t the revelation.
The real revelation is narrower and stronger: the thing that carries the improvement is a document you can read, diff, version, and approve - like a controlled SOP revision - instead of a weight update no human can inspect. Whether or not fine-tuning is on your table, that property is what makes an AI system reviewable in a regulated environment. SkillOpt makes the improvable surface a text file on purpose. That’s the part worth building on.
You Don’t Need to Be a Programmer to Read This File
This is the part I most want a Medical Affairs reader to hear, because it’s where a lot of people quietly opt out.
A skill file is not code. It has no programming logic, no syntax you need to compile, nothing to debug. It’s structured plain text - headings, numbered steps, quality criteria - and it reads like a procedure because that’s exactly what it is. If you can read an SOP, you can read a skill file. Here’s the medical inquiry response Skill from Part 1 of this series:
## Skill: Medical Inquiry Response
### Procedure
1. Identify the specific clinical question
2. Classify as on-label or off-label inquiry
3. Retrieve relevant SmPC sections (4.1, 4.2, 4.8, 5.1)
4. Search approved standard response library
5. If no standard response exists, summarize relevant clinical data
6. Format response using approved template
7. Flag for medical review if off-labelNow imagine an optimizer proposes one edit: it adds a new line to step 5.
5. If no standard response exists, summarize relevant clinical data,
include safety information, and add the adverse event reporting reminderYou do not need to understand a single thing about how the AI works to evaluate that change. You need to understand medical inquiry responses - and you already do. Should step 5 require the AE reporting reminder? Yes, obviously. Approve it. That’s the entire skill needed to govern this system: reading a proposed change to a procedure and judging whether it’s correct.
A medical reviewer reading that diff is doing precisely what they already do when an SOP is revised: read the redline, check it against requirements, sign or reject. The machine-learning happening underneath is irrelevant to that judgment. Understanding the principle - that the instructions are the controllable layer - matters far more than understanding the model. You bring the domain expertise. The file is just where it gets written down.
What SkillOpt Does Not Solve
I’d lose your trust if I stopped at the good news, so here are the honest limits.
The model is still a black box. Optimizing the instructions does not make the underlying AI transparent, and it does not stop it from hallucinating. A better skill file reduces failures. It does not eliminate them, and it does not substitute for the validation gates, the human-in-the-loop touchpoints, or the audit logs described in Part 2. SkillOpt improves the Skill. It is not the harness. You still need both.
A machine proposing an edit is not the same as a human approving it. This is the subtle one, and it cuts straight to compliance. If an optimizer autonomously rewrites your procedure, a fair auditor will ask: who authored this instruction, and why? “A tool found it improved a score” is not, by itself, an acceptable answer for a regulated workflow. The resolution is one you already own: proposal is not approval. Let the optimizer draft candidate improvements and let the validation gate filter out the ones that don’t work - but a qualified human still signs the change before it goes live. That’s not a limitation of the tool. That’s change control (GAMP 5, EU GMP Annex 11, 21 CFR Part 11), applied to your AI’s instructions exactly as it’s applied to every other controlled document you touch.
Used that way, SkillOpt isn’t autopilot. It’s a faster, evidence-driven way to draft better instructions - with the human review it needs kept firmly in place.
The Pharma Advantage, One More Time
I keep coming back to the same conclusion because the evidence keeps pointing at it. The discipline SkillOpt implements - propose a bounded change, test it against held-out cases, keep it only if it’s provably better, discard the rest - is not a novel AI invention. Pharma has a name for it. It’s change control with acceptance criteria. You’ve run that loop your entire career, on documents, methods, and processes.
Microsoft built it for coding agents and put it on GitHub. You’ve been living it for decades. The difference now is that the same loop runs on the instructions your AI follows - and, unlike a model’s weights, you can read every line of what changed.
The compliant, improvable core of an AI system was never going to be a black box you have to trust. It’s a document you can review. That was the thesis. Microsoft just handed us the proof.
This is Part 5 of a series on AI-driven SOPs in pharma. Part 1: SOPs for AI - How Pharma’s Most Underrated Skill Became the Key to Agentic Workflows · Part 2: The AI Harness - How to Make Agentic SOPs Audit-Proof in Pharma · Part 3: Nutcrackers, Not Sledgehammers: Why Small Models Belong in Your AI Harness · Part 4: Maintenance Is a Deletion Question
Source: Yang et al., “SkillOpt: Executive Strategy for Self-Evolving Agent Skills”, arXiv preprint 2605.23904, 2026. Project: github.com/microsoft/SkillOpt. Reported performance figures are the authors’ own, on public benchmarks.
This article was co-authored with Anthropic’s Claude Opus 4.8 model. The ideas, domain expertise, and editorial direction are mine - the AI helped structure, draft, and refine the text.
