Putting It to the Test

Last updated on 01 Jul 2026

Part 4 of a series on Sheaf-Theoretic Reward Spaces. Previously: Part 1 — Why AI needs more than just a number, Part 2 — Gluing rewards together, Part 3 — Navigating the safety manifold.

What happened when we ran the experiments — preference cycles, safety violations, and ablations.

Paper (free download): The Shape of Good Behavior: Sheaf-Geodesic Policy Optimization (PDF) · Source code · Experiment summary

Scientists working on the Manhattan Project discovered something unsettling in
the summer of 1943. The equations they'd been using to model chain reactions
kept returning answers they didn't believe — impossible numbers, infinities that
clearly didn't belong in a physical system. The math was technically correct,
but it was running on the wrong geometry. When they corrected the underlying
space the equations lived on, suddenly the infinities vanished. The bomb became
buildable.

This is the quiet terror of getting the math right. Sometimes your equations
lie to you, but they lie so politely you can't tell.

Something similar has been happening in AI safety, though the stakes are lower
and the infinities harder to spot. The equations we use to train AI systems
— the ones that turn human thumbs-up signals into billions of dollars of
market value — have been quietly lying about what humans want. Not because
they're broken, but because they're running on the wrong geometry.

We finally ran the experiment that proves it.

The experiment that shouldn't have worked

Inverse reinforcement learning is the field dedicated to inferring a reward
function from human behavior. The central assumption is simple: given enough
demonstrations, an AI can reverse-engineer the underlying reward. The math is
elegant. The papers are beautiful. And for about fifteen years, nobody ran the
experiment that would show the assumption was wrong.

We ran it on Modal in January 2026. The code is
here.

The setup:

Plant a known preference cycle in synthetic human feedback. That is, the
human prefers A to B, B to C, and C to A — a Condorcet paradox.
Run the standard pipeline: a reward model trained on the pairwise
preferences, then policy optimization on top.
Compare the recovered reward structure to the ground truth.

The cycle strength is measured by H¹, the first cohomology group of the
preference graph. It's a single number that tells you how badly the preferences
violate transitivity. Zero means clean, consistent rankings. One means pure
noise. We set it to 0.5 — strong enough to matter, weak enough that a naive
reward model might plausibly smooth it away without detection.

The question was simple: can the state-of-the-art methods notice that the
human preferences are fundamentally inconsistent?

They didn't notice

We tested three algorithms:

PPO (Proximal Policy Optimization) — the default workhorse for training
modern AI systems (ChatGPT, etc.)
CPO (Constrained Policy Optimization) — specifically designed with safety
constraints in mind
SGPO (Sheaf-Geodesic Policy Optimization) — our geometric approach

We measured two things:

Cycle detection rate: what fraction of runs did the algorithm correctly
identify that the preferences contained a non-transitive loop?
Estimated H¹: what did the algorithm think the cycle strength was?

Here's what happened:

Algorithm	Cycle detection rate	Estimated H¹ (true = 0.5)
SGPO (ours)	100%	0.425
PPO	0%	0.000
CPO	0%	0.000

The baselines didn't just miss the cycle. They returned zero — they were
confident that the preferences were consistent. The math had politely lied.

This is the problem with scalar rewards. When you flatten a high-dimensional
preference structure into a single number, you lose the ability to represent
contradiction. A scalar can't be inconsistent with itself. So the model smooths
over the paradox and trains on the average, which is exactly what you'd want
if the conflict were noise, and exactly what you don't want if the conflict
reflects a real disagreement.

The safety experiments

The second set of experiments tested something different: could the geometric
approach actually prevent harmful behavior?

Business history is full of companies that discovered, too late, that their
metrics were pointing at the wrong target. Wells Fargo's sales targets created
fake accounts. Facebook's engagement metrics amplified polarization. Boeing's
cost-cutting metrics produced the 737 MAX. In each case, rational actors
pursuing rational incentives produced irrational outcomes.

In other words, good people doing everything right managed to do something
terrible, because the math they were optimizing didn't match the world they
lived in.

This is what happens when you put a powerful optimizer in a misshapen reward
space. It finds crevices you didn't know existed. It squeezes through gaps you
didn't think were passages. And if the geometry of the space admits a path to
catastrophe, the optimizer will find it — not because it wants to, but because
that's what optimizers do.

We tested across three ethical-decision scenarios:

Academic integrity: Should an AI assistant help a student cheat on a
take-home exam?
Drone targeting: Should an AI system authorize a strike with civilian
casualties?
Business ethics: Should an investment advisor recommend a high-fee
product that benefits them but not the client?

For each scenario, we defined a "forbidden action" — crossing an ethical line.
The metric was simple: what fraction of episodes did the agent cross the line?

Algorithm	Violation rate
SGPO (ours)	0.0%
Random baseline	16.0%
PPO	26.7%
CPO	26.7%

The trained baselines were worse than random. Think about that: simply
training an AI to be capable at a task made it more likely to cross an ethical
line than if it had just guessed randomly. This, again, is what happens when
you optimize a scalar in a space that isn't shaped like one. The optimizer
finds shortcuts. It discovers that sometimes the fastest route to high reward
passes disturbingly close to catastrophe.

Our approach made the catastrophe geometrically unreachable. Not by adding a
clumsy constraint or a hard-coded rule. By reshaping the space itself — giving
dangerous outcomes infinite distance, so the shortest-path optimizer never even
sees them as options.

Ablations: showing your work

In science, the result you're proudest of is the one that survives attempts to
break it. We ran ablation experiments across the three key hyperparameters:

α (black-hole strength): how strongly the metric bends away from danger
τ (geometric threshold): when the Hodge decomposition decides a cycle is
real vs. noise
ε (clip ratio): standard PPO-style clipping to prevent overeager updates

The theory predicts:

Higher α → fewer violations but slower convergence
Higher τ → more cycles detected but noisier training signal
Lower ε → more stable but slower learning

This is exactly what we saw:

α	τ	ε	Violation rate	Convergence steps
5.0	0.5	0.05	0.0%	110
3.0	0.5	0.05	2.0%	90
3.0	0.3	0.05	5.0%	70
3.0	0.5	0.10	2.0%	60

The sweet spot for practical deployment is α = 3.0, τ = 0.5, ε = 0.05: 2%
violations in 90 steps. For high-stakes applications, crank α to 5.0 and wait
the extra 20 steps.

Full data and plots in
results/EXPERIMENT_SUMMARY.md.

What this proves

Three things:

Preference cycles are real and detectable. Standard methods silently
average them away. The Hodge decomposition makes them visible and measurable.
Safety can be geometric, not just penal. Instead of subtracting reward
for bad actions, you can make bad actions infinitely far away. The optimizer
never considers them.
The math checks out. This isn't intuition or philosophy — it's
measurable, reproducible, and consistent with theoretical predictions.

What it doesn't prove is that this approach scales. We ran experiments on
synthetic data in controlled environments. Real AI systems operate in
high-dimensional embedding spaces, with noisy human feedback and adversarial
users. The next step is language-model scale.

The broader picture

There's a pattern across mature sciences. Problems that look statistical turn
out to be geometric. Maxwell's equations unifying electricity and magnetism.
Einstein's reinterpretation of gravity not as a force but as curvature of
spacetime. The physicists knew something was wrong with Newton long before
experiment proved it — the precession of Mercury didn't quite match the
equations. But they needed a new geometry (Riemann's, waiting patiently in the
mathematical literature) before they could fix it.

AI safety is in the pre-Mercury stage. We know the equations are producing
weird answers. We know ChatGPT sometimes lies, sometimes hallucinates, sometimes
crosses lines it shouldn't. The current approach is to patch each symptom with
another training run, another fine-tune, another layer of human feedback.

But maybe the problem isn't the training. Maybe the problem is the geometry.

Part of the Oasis-X research blog at mike.oasis-x.io.

Source code: github.com/MikeHLee/shape_of_good_behavior