Recovery as a principle, not a patch.

01 / TODAY

A different recipe at every station.

Gap handling is not new. Exchange protocols have solved it for decades: Nasdaq’s MoldUDP64, for example, numbers every packet so a feed handler can detect a gap and re-request the missing range. The trouble starts behind the feed handler, where each station brings its own answer.

• Transport. The feed handler re-requests what the network lost. Its guarantee ends where its output begins.
• Distribution. After a reconnect, many distribution layers send a fresh image of the current state. The ticks in between are not delivered again.
• Capture. A tick-capture process can replay its log to subscribers that restart. If the capture process itself is down, KX’s own documentation is frank: that data “typically will not be recoverable”.
• History. Holes in charts and tick history get filled by backfill scripts, re-import jobs, or someone checking the next morning.

Each recipe works on its own. Together they cost time, specialist knowledge, and attention that would be better spent on the product.

02 / OPENMDTA

One principle, from feed to app.

In OpenMDTA, a gap is not an error path. It is a normal part of the data, and every station handles it the same way.

• Live first. When a connection drops, the system records the gap and resumes live delivery immediately. Nobody waits for the past to be recovered.
• Recovery in the background. An independent recovery worker asks the source for exactly the missing range, wherever the vendor offers a replay. Recovered records are joined to the live data by their source positions.
• History follows by itself. Candle windows are sealed only after coverage is proven. When recovered data arrives, the affected candles are recalculated.
• Consumers resume, too. A client that loses its own connection resumes from its last position through the generated SDK, with gaps and replacements delivered explicitly.
• What cannot be recovered stays visible. If a vendor cannot replay a range, the gap remains in the data and is reported, instead of disappearing quietly.

03 / CONSEQUENCES

What the guarantee lets us leave out.

Once every station can rely on recovery, it no longer needs its own protection against loss. That removes a surprising amount of machinery.

Fast and complete, not one or the other.

Classic distribution systems relay prices immediately and accept that a crash loses data. Systems with a delivery guarantee wait for an acknowledgement, from replicas, a broker’s spool, or an object store, before a message moves on, and pay for it in latency. OpenMDTA relays live data immediately. Whatever had not been written when a process crashed simply becomes a gap, and is recovered like any other.

Less infrastructure.

Many designs put a broker cluster such as Kafka between feed and consumers so nothing is lost on the way, with its own replication, leader election, and acknowledgements. In OpenMDTA, an archive of immutable segments in object storage, published through a compare-and-swap manifest, is the log. Recovery is the delivery guarantee. Object-storage-backed streams are becoming popular for good reason; they usually wait for each write to land. On the live path, OpenMDTA does not have to.

No mandatory standby.

Neither the feed nor the derived data needs a second installation that can take over at any moment. The architecture allows several ingestion replicas, for example in different data centers, to write to the same archive; overlap is harmless and no leader is elected. A second replica makes gaps smaller. It is not a prerequisite for complete data.

04 / BOUNDARIES

Where the limits are.

• Recovery depends on the source. Where a vendor offers no replay, a gap can be made explicit, but not filled.
• Recovery is scoped to each source and its retention. There is no global exactly-once promise: SDK delivery after a reconnect is at least once, keyed by source message IDs so consumers can apply it idempotently.
• Inputs that cannot be fetched again, such as manual corrections, do wait for durable storage before they are acknowledged.

The details of delivery, retention, and gap reporting are part of each pilot’s API contract. The Technical Overview describes the components involved.

Running feeds that drop?

I am looking for pilot customers: fintechs with traction, several data feeds, and a new market on the roadmap.

Let’s talk ↗