Data breaks silently — the 4.5-month outage and the fake crash caught 90 minutes before publication (Part 4)

Part 4 of five. Stories from our incident log — because the real contest isn't building the system, it's noticing when it breaks.

Most writing about system-building is about the building. What months of operation taught me is different: the real game is whether you notice when things break. Here are three representative entries from our incident log.

Incident 1: price ingestion, silently dead for 4.5 months

One day the backtests started looking wrong. Digging in, I found something hard to believe: daily price ingestion had been effectively stopped for about four and a half months. Each day, a small fraction of stocks got ingested; the rest were silently skipped.

The frightening part: there were no errors. The job reported "success" every single day. It did a little bit of work, and called itself successful. The scariest failure mode in human organizations, too.

The lesson: detecting failures is not enough. You need to detect silent stops — "is the thing that always runs doing its usual volume of work?" Today a watchdog checks exactly that, and if prices go stale for a few days, a warning lands in my Slack the next morning.

Incident 2: an EPS that could not exist

Financial data had its accident too. A listed company's earnings-per-share turned into a number that was off by orders of magnitude. The cause: format quirks in disclosure documents dropped some fields, and raw pre-consolidation values got mixed in.

The policy born that day became the system's backbone: broken data must never reach paying readers — period.

We built a "corruption firewall": one place that flags suspicion — impossible EPS, operating profit larger than revenue, absurd ratios — and automatically excludes suspect stocks from all products and rankings until the cause is fixed. When unsure, exclude. A missed opportunity costs little; charging money for a broken number costs everything.

Incident 3: the fake -91% crash

And then the closest call of all.

The weekly report's data showed a major cable manufacturer down -91.8% in one week. Other blue chips down 70–90%. Had the market collapsed? No. A wave of stock splits had just taken effect across large caps that week.

In a split, one share becomes, say, ten, and the price divides by ten. Holders lose nothing. But if you naively compare prices across the split, you get "a 90% crash." The database held pre-split and post-split values side by side with a seam between them, and the system honestly computed the seam as a decline.

The arithmetic was right. The data was lying.

It was caught in the final pre-publication check — by the human common-sense brake ("companies that size don't do that"), which the machine didn't yet have. In the ~90 minutes that followed, we added: (1) logic to mechanically detect split-seam suspects, (2) a firewall excluding them from products, (3) three regression tests so the same accident can never pass silently again — then rebuilt the weekly with correct numbers and published. The next morning, "any split seams left?" became a permanent item in the daily self-inspection.

Defense in three layers

The design philosophy the incident log produced is simple:

• Detect — every morning at 9:45, a 25-point self-inspection reports anomalies to Slack
• Firewall — suspicious data is locked out of products first, before root-causing
• Cure — fix the cause, and leave behind regression tests that would catch the same accident (the pile is now 1,800+ tests)

The order matters. Cures take time; detection and firewalls can stand guard today. The realistic goal is not "never be wrong." It's "wrong numbers stop before they reach a reader."

Next, the finale: automating the publishing itself — and the honest numbers on what this all costs and what it earns.

Tsukiyo Research publishes a weekly systematic report on Japan's uncovered small caps — every issue opens with the scorecard. Subscribe here.

This series is a personal build-and-operate log. Nothing here is investment advice or a recommendation to buy or sell any security.