THE CORRELATION BUDGET

Take the Alice–Bob–Carol example above. Each of the three statements is perfectly reasonable on its own. Together they cannot be true, and you can feel it without doing any arithmetic: if Alice tracks Bob and Bob tracks Carol, Alice cannot be unrelated to Carol. Relationships constrain each other, the way the three sides of a triangle do — you are free to choose two of them, and then the third is boxed in.

A correlation matrix is exactly that kind of table, written out for every pair at once. And the trap scales up badly. With three quantities you can spot the contradiction by eye. With fifty, a table can be full of individually sensible numbers and still describe a world that cannot exist, and nothing about looking at it will tell you. The technical name for a table that can exist is positive semi-definite. The plain version: no combination of the quantities is allowed to come out with a negative amount of variability. Negative variance is not a small error to be tolerated — it is nonsense, like a negative length.

This matters well outside mathematics. Portfolio risk models, weather and mining models, sensor networks, machine-learning kernels and Gaussian-process models are all built on such tables, and all of them break when somebody adjusts one entry by hand to reflect a judgement. Entire software libraries exist for nothing but taking a table that a human has quietly ruined and pushing it back to the nearest valid one.

So here is a natural question which, pleasingly, has an exact answer. Start with a table that is valid. Pick two of the quantities and start adding extra correlation between them — turning a knob up from zero, changing nothing else. At zero the table is fine by assumption. Turn the knob far enough and it is certainly nonsense. Where precisely is the cliff edge?

It is worth saying where the question came from, because the origin is unusual and the answer is not. It arose in a physics programme where the table is a correlation structure and distance is defined from it, so that adding correlation between two far-apart regions is a statement about geometry. None of that motivation is needed below. What follows is finite-dimensional linear algebra and applies to anyone's covariance matrix.

Fix a valid base structure K₀. Pick two localised profiles — think of them as two regions, two sensors, two clusters of assets — and add extra correlation strictly between them, in the smallest possible way: a symmetric rank-two term, scaled by the knob λ. At λ = 0 the structure is valid by assumption; at large λ it is not. The question is where the transition happens, and the answer should be exact rather than found by trial and error.

Two words carry weight in what follows. Positive semi-definite is the validity condition from section 01, written as a condition on eigenvalues: every one of them must be at least zero. And rank-two means the perturbation is the simplest object that can couple two profiles symmetrically and nothing else — the minimal edit, so that whatever we find is a property of the base structure rather than of an elaborate perturbation.

Write the perturbed structure as K(λ) = K₀ + λ(uAuBᵀ + uBuAᵀ) with unit-norm profiles and λ ≥ 0. Assume K₀ is symmetric and strictly positive definite. Define three numbers — the two self-terms and the cross-term, all measured in the metric induced by the inverse of the base:

That is the whole result. Below the boundary the perturbed structure is strictly positive definite; at the boundary it is semi-definite with exactly one new null direction; above it, indefinite — no longer a covariance structure. The endpoint is included, which matters, because pre-registered tests tend to be written at the endpoint.

The proof is short enough to give in full outline. Whiten: set x = K₀−1/2uA and y = K₀−1/2uB. Congruence by K₀−1/2 preserves inertia — it cannot change how many eigenvalues sit on each side of zero — and turns the problem into I + λ(xyᵀ + yxᵀ), which is the identity everywhere except on the two-dimensional span of x and y. On that span the matrix is 2×2, and its determinant factors:

For λ ≥ 0 the first factor is at least one. The second is non-negative exactly up to 1/(√(ab) − c). Cauchy–Schwarz guarantees |c| ≤ √(ab), so the denominator is non-negative and the bound is meaningful. At the boundary the null direction is √b·K₀⁻¹uA − √a·K₀⁻¹uB, written out explicitly because a test that wants to detect the boundary numerically needs to know what it is looking at.

The two excluded cases are the interesting ones, because they say what the budget is really measuring.

One structural remark worth extracting: spatial separation never enters the algebra. The bound is identical whether the two profiles sit at opposite ends of a domain or overlap completely. Distance only enters through the base matrix, via the three inner products. The budget is a statement about consistency of the correlation structure, not about geometry — and any claim that it constrains something geometric has to earn that separately.

The executed test used a smooth base on a periodic chain: correlation falling off as a Gaussian in the circular distance between sites, evaluated on grids of 256 and 512 points at several correlation lengths. This is about as benign as a covariance matrix gets. It is also, at the declared parameters, numerically catastrophic.

Because the base is circulant, its eigenvalues are available in closed form as a cosine sum, and they can be bounded against a theta series. The result at the worst declared cell: the smallest eigenvalue is positive, but positive at the scale of 10−76, while the largest is about 15. The condition number exceeds 7 × 10⁷⁶. Every one of those matrices is mathematically, provably strictly positive definite — and completely unresolved in ordinary double precision, where the smallest eigenvalues are indistinguishable from zero and forming the inverse would produce numerical noise dressed as an answer.

The bound is a theorem, so it does not need an experiment. What needed testing was everything around it: whether the boundary is where the code says it is, whether the perturbation does anything geometrically visible, and whether the effect is localised to the pair it was injected between. Those are empirical questions about an executable, and they were pre-registered — profiles, grids, tolerances, contrasts and decision rules all hashed before the scan existed.

The capsule completed twice with byte-identical output — a 2.4 MB result file with a recorded SHA-256, inside a repository manifest of 108 checksummed entries. All 24 cells completed; all 24 precision certifications passed; all 24 distance-validity and spectral-trace checks passed. Then the decision rules were applied exactly as written.

In the parent programme there is a proposition stating that the observable correlator is real-analytic regardless of how singular the underlying structure is — smoothing screens out sharp features. It is tempting to reach for it here and conclude that the whole question is closed by fiat.

It is not, and the note says so. The perturbation used here is built from analytic bump profiles, so the perturbation is itself jointly analytic — a finite sum of products of analytic functions. The smoothing proposition screens non-analytic sharpness; it has nothing to say about a smooth rank-two off-diagonal term. Analyticity neither excludes nor establishes the effect being tested. Recording that is a small thing, but it is exactly the kind of small thing that otherwise gets used later as a proof of something it never proved.

As a piece of applied linear algebra the bound is reusable well outside its origin: any setting where a covariance matrix is edited, stress-tested, or synthesised with an imposed dependence between two blocks has the same question and the same closed-form answer, including the pseudo-inverse variant for rank-deficient estimates. It is short-note grade mathematics, offered as such.

As part of a physics programme it is one executed capsule among several, and it is a good example of why that programme publishes its negative verdicts: the theorem is the cheap part.