EV Data Pipelines

The state of things

I joined a five-engineer team operating Shell’s EV and e-mobility data pipelines. The pipelines ran, but they failed often enough that a meaningful share of the team’s week was spent on recovery rather than on building.

Two things were true at once: the failures were repetitive, and a lot of ingestion was still being done by hand.

What I changed

Root-cause fixes over restarts. The standing practice for a failed run was to re-run it. I traced the recurring failures back to their actual causes — schema drift, unhandled null shapes, upstream timing assumptions — and fixed those instead.

Validation at the boundary. Bad data was being detected late, deep inside transforms, where the error message told you nothing. Moving validation to ingestion meant failures became legible: which source, which field, which expectation.

Together these cut the error rate by roughly 75% per pipeline.

Automated ingestion. The manual ingestion steps were mechanical, which meant they were automatable. Removing them cut manual effort by 50% and removed a class of human error along with it.

Why it mattered

None of this is glamorous work. But the team got its week back, and the data downstream became something people trusted by default rather than checked.