Migrating IoT Telemetry Ingestion to a New Schema with Zero Event Loss
Problem
A third-party device server that streams telemetry from the fleet was moving to a new cloud version. The change repointed writes from the existing location/alarm tables to new v2 tables. We ingest that data via CDC → Kafka, so the cutover risked dropped or duplicated pings across every device on the platform.
Constraints
- No lost events — telemetry is the source of truth for utilization and safety
- Live devices (~200+, each sending a ping per minute) can't be paused
- Pre-existing reliability bugs in the ingestor had to be fixed as part of the work
Architecture
I updated the ingestor to consume from the new location_v2 / alarm_v2 CDC streams and map the new shape onto the existing internal model, cutting over without gaps. Along the way I fixed a class of Kafka reliability bugs that predated the work: offsets that weren't being committed correctly, and failure handling that let a single bad message stall or replay the whole batch — replacing it with proper per-message handling and commit semantics.
Trade-offs
Why: Downstream consumers stayed unchanged, so the migration stayed contained to ingestion.
Cost: Careful mapping and validation to guarantee parity between old and new shapes.
Why: The cutover would have amplified these latent bugs; fixing them first made zero-loss achievable.
Cost: Larger scope than a straight repoint.
Outcome
- Cut over to the v2 telemetry schema with zero event loss
- Removed a class of Kafka reliability bugs (uncommitted offsets, poison-message stalls)
- More reliable ingestion underpinning utilization and safety metrics