Methodology & limits
How the score is built, what it cannot see, and the mistakes we made
getting here.
How a score is produced
Headlines are collected from public news indexes, deduplicated by headline similarity,
and scored against a finance lexicon. Each word carries an intensity weight. A
"surge" is not the same as a "rise". And a negator in the three preceding words flips
polarity. The score for a session is the mean across that session's clustered stories.
The score is deliberately not produced by a language model. A lexicon is auditable:
every number traces back to specific words in specific headlines, and it can be
recomputed identically years later. A model that cannot be audited cannot be trusted with
a public claim.
What a lexicon cannot see
Word-counting cannot resolve meaning that depends on what the word attaches to.
"Yields spike" is bearish for gold and bullish for the dollar, the same word, two
opposite readings, and a bag of words cannot tell them apart. We removed the worst
offender but the limitation is structural, and it caps how good any score of this kind
can be.
Defects we found in our own pipeline
These were found by an adversarial audit and by a power calculation, not by us
pre-emptively. They are listed because a publication that quietly fixes its own mistakes
is not trustworthy.
1. A look-ahead in the historical backfill. News was assigned to
sessions using whole UTC calendar days, while prices used each instrument's local session
date. For the Nikkei that meant 18 hours of the news window fell after the session
had closed, the window overlapped the very move it was being tested against. This
manufactured a predictive correlation of +0.068 that vanished (−0.016) once the windows
were derived from the price bars instead. A result computed on the old data was invalid;
it has been withdrawn and recomputed.
2. A lexicon with almost no resolution. The original formula
divided by the count of directional words, which yields only about five possible values.
On a live page that collapsed 59 independent stories onto three distinct positions, so
the indicator's disagreement channel was arithmetically incapable of showing
disagreement. Now intensity-weighted with a smoothed denominator: 1,469 distinct values
in the current series.
3. A null reported without its power. The per-instrument test
found nothing across 17 instruments, and we first published that as a finding. It was
not. With ~130 sessions per instrument the smallest detectable correlation at 80% power
is |r|≈0.25, while the published effects are r≈0.02, 0.08, giving 4, 14% power. The
test could not have found a real effect, so its null meant nothing. The properly powered
test pools every observation into one panel (N≈2,100, power 0.96), and that test
is the one whose null counts.
What survived all of this. The contemporaneous relationship is
strong and correctly measured, pooled r = +0.236, and on US indices up to +0.61. The
predictive relationship is not detectable at any horizon we tested, in any specification
we could construct from the literature. That is the finding, and it is now properly
powered.
Date discipline
Sessions are bounded by consecutive price-bar closes, not by calendar dates.
That invariant holds across timezones, exchange holidays and daylight-saving shifts,
because it is read off the price series rather than assumed. Items are placed by their
publication time, never fetch time, and anything undated, future-dated or outside
the stated window is rejected rather than guessed at.
Open an Axi Account →
Methodology published so the work can be checked. Not investment
advice, not a recommendation.
