You ask an LLM how to calculate something. It gives you an answer. You use it. It works.

Next month, same question, different answer. Also works. But differently. The new one passes surface checks but fails an edge case the first one handled.

Which one was right? You don’t know. You can’t compare them because you don’t know how either answer was produced. Different model version? Different parameters? Different batch size affecting floating-point reduction order?

Yes, that’s a real thing. Thinking Machines Lab demonstrated that identical prompts produce different outputs on the same model because of batch-sensitive kernels in the inference pipeline. Tiny floating-point differences cascade through layers and flip the answer. They built batch-invariant kernels that produce bitwise-identical output across 1,000 runs. The nondeterminism isn’t mystical — it’s engineering.

You got a result without a method. That’s not knowledge. That’s asking the oracle.

The Pre-Scientific Problem

This is older than computers.

For most of human history, knowledge came from authority. The priest said so. The oracle said so. The master said so. Then a shift began — slowly, unevenly, across different cultures and centuries. Show me your method. Write it down. Let me try it. If I get the same result, maybe we’re onto something.

That was the scientific method. Not a buzzword. A discipline. Observation, hypothesis, experiment, falsification, peer review. The whole point was reproducibility. If you can’t reproduce it, it’s not knowledge. It’s anecdote.

We’re back to oracles. We just call them models now.

What Skill Creation Actually Does

An AI agent skill is a markdown file. Steps, commands, pitfalls, verification. When the agent loads it, it follows the procedure instead of improvising.

Sounds simple. But look at what just happened: you took an undocumented interaction with an LLM and turned it into an inspectable procedure. You wrote down the method.

Now, a skill is still a prompt interpreted by an LLM. It doesn’t guarantee bitwise-identical output. But it narrows variance. A well-structured skill with explicit steps, expected outcomes, and verification checks gives the model less room to improvise. And if you pair it with a model that sticks to the prompt (lower temperature, instruction-tuned, less creative freedom), you reduce the nondeterminism further. You won’t get a chemistry lab’s reproducibility. But you get something you can re-run, compare, and diagnose. That’s partial reproducibility. Enough to work with.

That’s not a code trick. That’s the first step of the scientific method. You moved from “the model said so” to “here’s the procedure, here’s what it produces, here’s how to verify it.”

What Exists Today

Current frameworks already approach skill improvement in different ways.

Hermes Agent has a self-improvement loop: do a task, verify the result, save what worked as a skill, reuse it next time. Their /learn command watches what you do and distills it into a reusable procedure. Their self-evolution engine uses GEPA, a genetic algorithm that mutates skill variants and selects the best performers. GEPA’s selection pressure functions as a weak form of falsification: variants that fail get discarded. It optimizes for past performance — what worked, not why it worked.

OpenClaw has a skill workshop with a governance lifecycle: propose, review, apply, reject, or quarantine. That’s peer review — a form of experimentation at the process level.

Anthropic’s “Dreaming” extracts patterns from past sessions into playbooks. All feedback loops. All useful.

The LLM evaluation field already has methodologies: BLEU, ROUGE, human eval, LLM-as-judge, arena rankings. These test model behavior at scale.

But here’s the thing none of these do.

None of them treat a skill as a hypothesis.

Think about what a skill file actually claims. “This procedure will produce the correct result for this class of task.” That’s not a vague aspiration — it’s a falsifiable claim. You can test it. You can break it. You can record the conditions under which it holds and the conditions under which it fails. That’s the next step beyond feedback loops, and it’s a fundamentally different posture. A feedback loop says “keep what works.” A hypothesis says “here’s what I expect to happen, and here’s exactly what would prove me wrong.”

What would that look like? Imagine a skill file that starts with three lines before the procedure:

1
2
3

Hypothesis: Given a sales Excel file with consistent formatting and complete data, this skill produces a quarterly review presentation that correctly identifies period-over-period changes, flags anomalies against the prior quarter, and generates a summary with accurate variance figures.
Falsification: The source file has missing values, inconsistent category naming, or broken formulas that the skill does not flag before building the deck, OR a chart plots the wrong column, OR a category in the spreadsheet is missing from the presentation, OR a data point in the deck doesn't match the source file.
Expected variance: Slide layout, chart types, and wording may differ; what counts as an "anomaly" is a judgment call; numbers and names must match the source file.

Three lines. That’s the difference between a documented procedure and a testable claim. The hypothesis states what should happen — and includes its own preconditions. The falsification tells you exactly what failure looks like, in terms the agent can check itself: diff the deck against the source file, verify every category appears, compare chart data ranges to column headers. The expected variance acknowledges that nondeterminism exists and defines which differences matter and which don’t. Now when the model changes next month, you re-run the skill and check against the falsification metric. Pass or fail. Not vibes.

Why It Matters

Consider LLM nondeterminism. Thinking Machines Lab showed that identical prompts produce different outputs not because of some mystical AI uncertainty, but because of batch-sensitive kernels in the inference pipeline. Tiny floating-point differences cascade through layers and flip the answer. With batch-invariant kernels, 1,000 identical runs produce bitwise-identical output.

So the “the model is just probabilistic” excuse is partly engineering laziness. But even with deterministic inference, you still have model version drift. You still have context window changes. You still have the system prompt that grew by 200 tokens since last month.

The answer changes. Unless you’ve documented the method, you can’t tell whether the new answer is better or just different.

Skills give you a controlled re-run protocol. A skill says: these are the steps, these are the expected outcomes, this is how we verify. When the model changes, you re-run the skill. If it still produces the right result, the method holds. If it doesn’t, you’ve caught a regression. The value isn’t eliminating nondeterminism. It’s making it detectable and diagnosable. Not through vibes. Through a test.

That’s the scientific method. Not abstract philosophy. A practical response to the fact that you can’t trust answers you can’t reproduce.

The College Class I Didn’t Enjoy

I took a research methodology class in college. Hated it. Hypothesis formulation, experimental design, control groups, statistical significance. It felt like bureaucracy for people who couldn’t just do the work.

Turns out, the bureaucracy is the point. Without it, you’re just collecting answers and hoping they’re right. With it, you can trust your own results.

I’m not building a chemistry lab. I’m writing markdown files that tell an AI agent how to deploy a static site. But the discipline is the same. Write down what you did. Predict what should happen. Check if it did. If not, figure out why. Write it down again.

Same method. Different lab.

The scientific method isn’t one procedure — it’s a family of them. More on the version that fits AI agent work next time.