Sign in Subscribe

Gluing Rewards Together: How Math Solves Paradoxes

Last updated on 

Part 2 of a series on Sheaf-Theoretic Reward Spaces. Previously: Part 1 — Why AI needs more than just a number. Next: Part 3 — Navigating the safety manifold.

How a branch of math built for stitching local data into a global picture turns out to be the right tool for "good behavior depends on the situation."

Paper (free download): The Shape of Good Behavior: Sheaf-Geodesic Policy Optimization (PDF) · Source code

In Part 1, we saw that human values are messy, context-dependent, and sometimes outright circular — Rock-Paper-Scissors preferences are real, and squashing them into a single score loses the structure that mattered.

That leaves us with an awkward question: how do you build an AI that understands context without inventing a new rule for every situation? The answer comes from a branch of mathematics with an unhelpful name and a very helpful idea — sheaf theory.

The Data is Local

Imagine you're trying to draw a map of the Earth. You can take a flat piece of paper and draw your neighborhood perfectly. You can draw your city. You can even draw your country.

But if you try to draw the entire Earth on a single flat sheet, you have to distort things. Greenland gets huge. Antarctica gets stretched. There is no single flat map that accurately represents the round Earth.

Sheaf theory accepts this limitation. It says: "Don't try to make one global flat map. Instead, keep a collection of local maps, and know how to glue them together."

In AI alignment, our "local maps" are specific contexts.

Consider the act of cutting a person's chest open with a saw.

  • Context A (Street Fight): This is attempted murder.
  • Context B (Heart Surgery): This is a life-saving procedure.

A simple scalar rule like minimize_harm(action) fails here. If the penalty for using a saw is -100, the surgeon is paralyzed. If the reward for saving a life is +1000, the street fighter might rationalize "preemptive surgery." This is what Amodei et al. refer to as the challenge of Safe Exploration and Avoiding Side Effects in Concrete Problems in AI Safety (2016)—safety isn't a static property of an action, but a dynamic property of the context.

Or take Michael Burry, profiled in Michael Lewis's The Big Short (2010). In the mid-2000s, he bet against the housing market. To his investors (and their risk models), this looked like insanity—burning money on premiums for an asset class that "never goes down." But Burry was operating in a different local section of the market's truth. Within his thesis, the "safe" bet (long housing) was actually the maximum risk.

In robotics, this subtlety is non-negotiable. A robot holding a steel beam needs to grip it with thousands of pounds of force. If it applies that same force to holding a baby, it's a tragedy. The action "apply force" isn't good or bad—its validity depends entirely on the sheaf of context it lives in.

An action like "Delete all files" might be valid in Context A (if the user asks to clean a directory) but catastrophic in Context B.

Restriction Maps: The Art of Zooming In

The key tool in sheaf theory is the Restriction Map. It’s a fancy name for "zooming in."

If I have a rule for the whole kitchen ("Keep it clean"), I can restrict that rule to the sink ("Don't leave dirty dishes").

  • Global Rule: "Be Helpful."
  • Restriction to Coding: "Write correct syntax."
  • Restriction to Chat: "Be polite."

A Sheaf is just a system that checks if these rules match up. If the global rule says "Be Helpful," but the specific coding rule says "Delete the user's hard drive," the sheaf says: "Error: These sections do not glue."

Cohomology: measuring the glitch

Here is where it gets useful. We can measure exactly how much the rules disagree.

Remember the "Escher staircase" from Part 1 — the infinite loop of preferences? In sheaf theory, that loop shows up as a literal hole in the logic: a place where local rules refuse to glue into a global one.

The tool for spotting and counting those holes is called cohomology — specifically a quantity written $H^1$. Skip the symbol if it helps; what it does is simple:

  • If $H^1 = 0$, the local rules all agree. A clean global reward function exists.
  • If $H^1 \neq 0$, there is a contradiction somewhere, and $H^1$ tells us how much.

Instead of blindly following the loop and getting confused, our system computes $H^1$. When it comes back non-zero, the system stops and reports: "There is a logical paradox in your training data. Humans say A > B > C > A. I cannot optimize this." That refusal is the feature.

Decomposing the flow

To separate the signal from the noise, we use a technique called the Hodge decomposition. The name is intimidating; the picture is not.

Imagine the flow of rewards as water moving across a landscape:

  1. Gradient flow — the water that flows steadily downhill. This is the consistent, rankable part of the reward signal. This is what we want the AI to follow.
  2. Curl — the water that spins in tight little vortices and never makes progress. These are the paradoxes. We want the AI to recognise them without trying to optimise around them.

The Hodge decomposition splits the reward signal into exactly those two pieces. The agent then pursues the gradient — the genuine direction of improvement — without getting trapped in the swirls that look like reward but lead nowhere.

That handles the contradictions. But it does not yet handle the cliffs — the actions an AI should never take, no matter how much short-term reward they promise. In Part 3 — Navigating the safety manifold, we turn those dangerous outcomes into geometric "black holes" the agent physically cannot route through.