How I Keep Critical Systems Running

Nobody notices infrastructure until it fails. Then it is the only thing anyone notices.

For nearly four years at AE Global Solutions, I worked on financial data infrastructure that ran 24/7 for clients who needed live market data every hour the markets moved. There was no quiet season and no acceptable excuse. The standard was simple: the data is there, and it is right.

Reliability Is the Product

Clients buying market data are not buying servers or pipelines. They are buying the confidence that the number on their screen is current and correct. Every architecture choice I make starts from that promise.

That meant maintaining 99% system uptime across round-the-clock data center operations, and treating every minute of downtime as a client’s decision made blind.

Validate Before It Ships

Bad data is worse than no data, because people act on it. So validation can’t be a final step. It has to live at every stage.

Every day, I processed and validated 200GB+ of financial data: record counts, schema conformance, value range checks, and reconciliation across sources. Anomalies triggered alerts before downstream consumers were affected. The result was 99% accuracy across all outputs, sustained while the volume kept growing.

Find the Real Cause, Not the Nearest One

On September 9, 1947, the team running Harvard’s Mark II found a moth wedged between the contacts of Relay #70 and taped it into the logbook with the note “First actual case of bug being found.”¹ Engineers had called faults “bugs” for decades by then. What makes the page famous is that someone went all the way down to the physical cause, and wrote it down.

That is the habit I bring. A failed job is a symptom. I trace failures down to the root, from the pipeline to the query to the operating system, using the same discipline as Toyota’s “five whys”: keep asking why until the answer is something you can actually fix.² Patch the symptom and the same problem comes back next week. Fix the cause and it doesn’t come back at all.

Make It Faster Without Making It Fragile

Speed that costs accuracy is not an improvement. At AE Global, I improved pipeline efficiency by 30% while holding accuracy at 99%, by profiling the slowest transformation steps, rewriting bottleneck queries, and adding parallel processing only where the data flow allowed it. Lower latency during peak market hours, with no new ways to fail.

The same instinct drove the custom ingestion pipeline I later architected at Root Labs: exponential backoff on rate limits, dead-letter queues for failed records, and alerts for anomalies above a set threshold. Failure is planned for, so it is never a surprise.

Build It to Run Without You

The cover photo shows flight director Gene Kranz at his console during a 1965 simulation for Gemini IV. Simulations like that were how Mission Control learned to solve problems calmly under pressure, and five years later Kranz’s White Team helped bring Apollo 13 home.³ The heroics were built on rehearsal, procedure, and documentation.

A system that only one person understands is a single point of failure with a pulse. So I build for reliability first: monitoring that pages the right person, documentation someone else can follow at 3 a.m., and clean handoffs that let the system keep running when I’m not in the room.

Watch: Capt. Grace Hopper's 1982 lecture on data and systems (NSA)

What You Get on Day One

• Someone who has kept 24/7 financial infrastructure at 99% uptime, and knows what that takes.
• Validation built into every stage, so bad data gets caught before it reaches a report or a decision.
• Root causes, not patches, from the pipeline down to the operating system.
• Systems documented and monitored well enough to run without me in the room.

Sources

1. “Log Book With Computer Bug”, Smithsonian National Museum of American History (catalog 1994.0191.01).
2. “Five whys”, Wikipedia.
3. “Gene Kranz”, Wikipedia.