My Data Was Lying Before I Wrote a Line of Logic

For a long time I believed my problems lived in my code. When results came out wrong, I went hunting through my logic, my models, the clever parts I was proud of. The actual culprit was almost always sitting one layer below all of that, quiet and unexamined: the data itself. It looked clean. It was not. And everything I built on top of it faithfully inherited the lie.

This is the most boring kind of failure there is, which is exactly why it is so dangerous. Nobody wants to spend their time auditing rows of numbers. So nobody does, and the rot sits there at the bottom, holding up everything.

The myth of clean data

Raw market data looks reassuringly tidy. Neat rows, orderly timestamps, sensible-looking numbers marching along in sequence. That tidiness is a costume. Real-world data, gathered from real systems that occasionally break, is riddled with small corruptions that do not announce themselves: gaps where moments are simply missing, duplicate entries, timestamps that drift or misalign, values that freeze and repeat during an outage, occasional impossible numbers, whole stretches subtly shifted out of place.

None of this arrives with a warning label. It looks exactly like the good data, because it is sitting in the same neat rows. The format is honest even when the contents are not, and the format is all you see at a glance.

Garbage in, confident garbage out

The thing to internalize is that the system does not know the data is wrong. It cannot. It takes whatever you hand it and processes the corruption with precisely the same diligence it applies to the truth, and then it produces a clean, confident, professional-looking result built on top of rot.

That is the whole trap, right there. The output looks fine. Corruption does not propagate as obvious, screaming errors that point you to the problem. It propagates as plausible wrong answers — results that are entirely believable, that pass your gut check, that you happily build the next thing on top of. By the time the wrongness surfaces, if it ever does, it is buried under layers of work that all assumed the floor beneath them was solid.

The errors that hide inside plausibility

I learned, slowly, that the dangerous data errors are not the obvious ones. A price of zero, an entire missing day, a value that is clearly nonsense — those you catch, because they are absurd on their face. They trip the first sanity check.

The errors that actually hurt you are the plausible ones. A value that is wrong but still well within a believable range. A timestamp off by a single unit. A gap that something upstream silently filled in with a guess. These sail straight through every sanity check you have, because they look exactly like legitimate data — they are indistinguishable from the truth without specifically going to verify them against something else. The most expensive corruption is the kind that is polite enough to look correct.

I was debugging the wrong floor of the building

For an embarrassingly long time, my instinct when something came out strange was to go looking for a bug in my logic — in the interesting, sophisticated part of the system, the part I found worth thinking about. The bug was almost never there. It was underneath, in the data, in some assumption about the input that I had simply never bothered to verify.

I was, in effect, searching the top floors of a building for the source of a crack that was running through the foundation. I would inspect the elegant code over and over, find nothing wrong with it, and grow more and more baffled — because nothing was wrong with it. It was correctly processing bad inputs into bad outputs, exactly as designed. The logic was innocent. The ground it stood on was guilty.

Data quality is not a task you finish

One of the more humbling realizations was that you do not clean data once and then move on. That is a fantasy. New data keeps arriving, and it arrives with new corruptions. A source quietly changes its format. An upstream provider has an outage and then backfills the gap with something questionable. Something that was reliable for months breaks in a novel way on an ordinary Tuesday.

Data quality, I came to understand, is not a one-time cleanup you check off a list. It is a continuous discipline — an ongoing posture of suspicion toward every input, maintained forever, because the moment you stop watching is precisely the moment a new and creative corruption walks in unchallenged.

Validate ruthlessly, trust nothing

The practical fix was to treat every input as guilty until proven clean. Check for gaps. Check for duplicates. Check for impossible values, for timestamps that fail to move steadily forward, for suspicious stretches where a number stops changing in a way real data never would. And crucially, build these checks directly into the pipeline, so that corruption is caught loudly at the front door, the instant it arrives, rather than discovered months later as the unexplained weirdness in a result you can no longer trace.

It is tedious, defensive, deeply unglamorous work. It is also the work that decides whether everything downstream is built on rock or on sand. A loud failure at the door is a gift. A silent one that you inherit and build upon is a slow catastrophe.

The deeper lesson: the foundation sets the ceiling

The principle underneath all of this is simple and unforgiving. No amount of brilliance in the model on top can compensate for rot in the data underneath. The quality of everything you build is capped, absolutely, by the quality of what you feed it. You cannot reason your way out of bad inputs; you can only produce more confident, more elaborate wrongness on top of them.

So the dull, thankless labor of guaranteeing clean inputs is not beneath the real work, the way I used to treat it. It is the real work’s foundation, and the entire structure above is only ever as honest as that base. I used to want to skip past it to get to the interesting parts. Now I understand that the interesting parts are worthless if the boring part underneath them is lying.

— No signals, no returns, not investment advice.