openMDTA.

Under the hood / Technical Overview

The hard parts of market and reference data.
Handled.

Market and reference data come with hard backend work: identifiers that disagree, feeds that drop out and leave gaps, and access rights for every user. OpenMDTA takes care of it, built on more than twenty years of experience founding and running a fintech, so your team can focus on what makes your product unique.

  1. 01 / Data in Your reference and market feeds, through existing importers or your own mapping plugins.
  2. 02 / Process & remix Durable ingestion and recovery, identity resolution, and remixing into the Datasets your products need.
  3. 03 / Developer experience One Gateway and typed, schema-generated SDKs for records, live values, tick history, and calculated figures.
  4. 04 / Your application out Quote pages, watchlists, instrument search, screeners, and customer APIs. You add the users.

Search, live prices, tick history, key figures, and corporate actions come built in. Adding a provider means adding its parser and mapping; the rest of the platform is already in place.

01 / DATA IN

Your feeds. On a shared foundation.

You should choose feeds for your product—not to fit our connector list. We implement or adapt the parsers, protocol handling, and mappings for the providers you need, using the shared ingestion, recovery, storage, and delivery infrastructure already in place. Your pilot defines the exact interfaces and the integration work still required.

Choosing a feed doesn’t grant rights to its data. Provider access, use, and redistribution still depend on the applicable data agreements.

02 / THE DATA MODEL

A Dataset defines what you cover.

Which entities do you have information about, and what can your product do with it? A Dataset brings together a Catalog and, optionally, one canonical Stream for the entities it covers. The Catalog describes those entries; the Stream carries their time-ordered updates.

Each Dataset has its own coverage: a broad instrument universe, a venue’s listings, or a specialist set of structured-product terms.

Catalog — Model the things your product needs to understand.

A Catalog holds structured records: instruments, listings, issuers, financial observations, classifications, and relationships. Define typed fields, identifiers, and dimensions around your data. Combine provider information with your own fields while retaining provenance and access requirements.

  • Instruments and listings: identifiers, names, currencies, venues, and product terms.
  • Companies and issuers: legal entities, classifications, and issuer relationships.
  • Fundamentals and estimates: observations organized by subject, metric, period, and provider.
  • Analyst data: structured ratings, price targets, and recommendations.
  • Funds: holdings and constituent records linking funds to instruments.
  • Corporate actions: dated events, split ratios, statuses, and distributions.

A Catalog is not a fixed instrument schema or a static file. Its records change as information arrives; your schema defines what those records mean.

Stream — Model what happens over time.

A Stream carries a source-ordered sequence of typed updates: quotes, trades, or other changing values. Your schema defines the payload. OpenMDTA supplies live delivery, retained history, explicit gaps, and recovery around it.

Source positions make continuity explicit. Live processing continues alongside background recovery; recovered history does not overwrite newer live state. Anything still missing remains visible.

For example, a Catalog describes an instrument and its listings. A Stream carries quote and trade updates. Services expose the current values, individual historical ticks, and aggregated candles.

The distinction is how you model and consume information—not simply “slow data” versus “fast data.” A Catalog-only Dataset can provide specialist information without a live Stream.

03 / CONNECT THE IDENTIFIERS

Different identifiers. The same entity.

One provider uses an ISIN, another a WKN, a third its own identifier. Connect their information through shared identity resolution instead of rebuilding cross-provider mapping tables in every application.

Typed identifiers. Traceable connections.

Identifiers carry their type: ISIN(…), WKN(…), CUSIP(…), or a configured vendor-specific type—not an unexplained string. Source records assert which identifiers refer to the same entity. Identity resolution connects those claims within the appropriate entity type and selects a leading identifier using configured priorities.

Alternative identifiers and contributing source records remain available. Your product gets a common reference without discarding provider IDs or rewriting authoritative source data. An issuer’s LEI identifies a legal entity, not the instrument it issued.

Same instrument does not mean same listing.

An instrument can have several trading lines. Keep its identity separate from venue, currency, and other source-defined dimensions. Resolve the instrument, then filter the selected Catalog’s records for the trading context your product needs. A lookup can return several records; even ISIN, MIC, and currency are not universally sufficient to identify one listing.

When you need one exact source record, address it by {catalog, record_key}. Its key stays source-owned—not a universal instrument ID or a vendor-independent listing ID.

Changing claims, priorities, and exact references

Resolution uses source-supplied claims and configured policies, not name matching or an assumed global identifier table. Source updates and withdrawals recompute affected mappings; downstream remixers follow the published Identity Catalog. Conflicts and rebalances remain inspectable.

Actual identity resolver: SAP's German ISIN DE0007164600, WKN, and FIGI share-class identifier resolve to one entity from four source records.
Different identifiers. One resolved entity. SAP's German ISIN (DE0007164600), WKN, and FIGI share-class identifier, connected through four source records. Actual screenshot from our local evaluation deployment, cropped to the resolver's selected outcome. Click to view full size.

04 / REMIX YOUR UNIVERSE

Remix your data. Your rules.

Remixing turns your inputs into a maintained, product-specific Catalog—sometimes called a golden copy—of the entities you cover and the fields you choose from each source. You can build several, and one remixed Catalog can feed another. Market Streams remain separate from these structured records; derived values can combine selected Catalog and price inputs.

  1. Exchange data
  2. Regulatory data
  3. Your own data
Remixed Catalog Identities resolved, your rules applied
  • Your applications Through the Gateway and typed SDKs
  • Specialist Catalog Remixed again, e.g. structured products
One remixed Catalog serves your applications and can feed further specialist Catalogs.

Combine your sources into one view.

Choose which input Datasets determine membership in a broad master Dataset. Resolve their records to shared entities, combine fields with your own source priorities and rules, and enrich them from other inputs. Remixing materializes its Catalog and updates affected entities as inputs change, retaining provenance and access requirements.

Remix again for specialist products.

Use a broad master as an input to a more focused Catalog, such as one for structured products. Add specialist sources and fields where that product needs depth, while the general master stays lean. Each output has its own membership and model.

Discover depth without copying everything.

Publish a per-entity overview of membership in configured input Catalogs, including record keys and lifecycle state. Your general master can link to in-depth information in a specialist Dataset without copying all its fields.

For example, show a “Structured-product details” link in your instrument navigation header only when that Dataset covers the instrument and the user has access—not simply because the instrument has a structured-product category label.

05 / MAKE IT YOURS

Your fields. First-class everywhere.

Your product does not have to stop at our base data model. Add the information that makes it yours—from the source to your developers’ SDKs, without changing OpenMDTA’s core.

Configuration first. Code when you need it.

Most source mappings and remixing rules fit our DSL: YAML or JSON that the platform validates and runs, with no compiler involved. When you need full flexibility, write the same logic in Rust, compile it to WebAssembly, and deploy it as a plugin. Either way, your logic ships separately from OpenMDTA’s service images, which stay unchanged.

Add your own fields when remixing.

Define custom fields in your schema and populate them in your remixing rules or plugin. Combine source values, derive product-specific classifications, or add the business rules that make your master Dataset useful to your team.

Extend the importer. Keep the platform.

Reuse an existing importer and extend its mapping to capture more from the exchange feed. When the base model does not cover your needs, derive custom fields directly from the decoded feed messages, in the DSL or in code. Build and deploy your mapping plugin independently—without recompiling the importer or OpenMDTA’s service binaries.

First-class fields. All the way to your SDK.

Custom fields travel through the same schema-driven infrastructure as standard fields. The Gateway exposes them through your configured data model, and your auto-generated SDKs include their types and accessors automatically. No separate API or hand-written client wrapper for every field you add.

Lists and index families, ready for selectors.

Select instruments by list instead of maintaining member tables. Index families such as GER40, US500, TECH100, and EU50 are Catalog lists with metadata and identifier-only members. Use the base lists, publish your own, or remix them. A venue selection resolves each member to matching records, subject to coverage and access rights.

const selected = selector.list("GER40").venue("XETR");
const quotes = connection.select(selected).latestStream();

Composition is fixed for a request. A new request or reconnect reads it again. Weights, index calculation, and index values are separate concerns.

Evolve your schema. Keep your clients working.

Every schema change is checked at build time and classified as compatible or breaking.

  • Automatic when possible. Optional fields can be added under the same name. Retired optional fields keep their wire slots, so historical values stay readable and generated SDKs keep their deprecated accessors.
  • Explicit when breaking. Removals and changes of type, unit, or meaning are rejected under the existing name. Introduce a new field version with a typed migration in the schema or in Rust; lossy conversions require consumer opt-in.
  • Clients read the version they know. Gateway reads and the generated SDKs request a supported version and receive converted records.

06 / THE SERVICES

One Gateway. Purpose-built services.

Your team builds against one interface.

The Gateway brings the services together behind one customer-facing interface. It authenticates requests, applies access rules, and routes them to the right service. Your developers use a consistent, schema-generated SDK for structured records, current values, tick history, and calculated figures—not a separate integration for every service.

READER / CATALOG ACCESS

Find and retrieve your structured data.

Turn a Catalog into a usable application interface. Retrieve selected fields, look up records by keys or identifying dimensions, and expose configured search and discovery capabilities.

  • Record and field retrieval; identity-aware lookup through the platform.
  • Autocomplete, text search, facets, and numeric filters.
  • Configured search projections and access rules.

Build an instrument finder, an issuer profile, or a searchable fundamentals dataset.

LAST / LATEST VALUES

The current picture, continuously updated.

Read the latest available values, then follow updates through a subscription. Shared upstream processing serves multiple consumers without duplicating the entire ingestion path.

  • Current snapshots followed by typed updates—not just the last traded price.
  • Applicable real-time, delayed, and end-of-day delivery policies.
  • Continuity and gap information alongside the data.

Build a watchlist, a quote page, or a customer-facing live API.

TIMESERIES / HISTORY & AGGREGATION

Tick history and OHLC. Built to scale.

Query individual historical observations and open/high/low/close candles on the same platform. Partitioned storage, compressed pages, and indexed range reads keep queries focused on the data they need.

  • Tick-level history, OHLC aggregation, historical queries, and live-following interfaces.
  • Explicit coverage and gaps; supported intervals follow the configured model.
  • Separate query and materialization workers: add serving capacity without repeating historical processing.

Build charts, historical analysis, or a vendor’s historical-data API.

KEYFIGURES / CALCULATIONS & SCREENING

Turn data into useful calculations.

Combine Catalog information with selected price inputs. Define calculation logic and expose results for individual instruments or searchable result sets.

  • Configurable calculations, input selection, and derived numeric values.
  • Filtering, sorting, and screening with price-freshness, availability, and access policies.
  • Midpoint and spread are implemented examples; additional financial formulas and their inputs are scoped separately.

Build a product screener or a customer-specific calculated dataset.

OBSERVATION / WATCHES & TRIGGERS

Turn changing data into useful notifications.

Build persistent watches around the conditions your users care about.

  • Durable task state, trigger transitions, and protection against rapidly oscillating conditions.
  • Task state and pending notifications committed together; delivery retries use a durable outbox.
  • Historical recovery produces separate summaries, not misleading live alerts, and does not overwrite the current live state.

Your application supplies instrument resolution, typed rules, and notification delivery. The reusable core is implemented; a ready-made Gateway observation API or email/push integration is not implied.

Build price alerts, threshold watches, or customer-specific notifications that bring users back at the right moment.

Delivery semantics and integration boundaries

Catalog import cursors advance after writes complete; replayed writes are idempotent. Immutable payloads precede manifest publication. These are scoped semantics, not a blanket exactly-once promise. Ordering is source-local; SDK reconnect delivery is at least once. Clients handle gaps and replacement values according to the selected API contract.

Catalog search exposes a configured projection to holders of its search grant; per-record detail permissions do not filter that projection. Review its fields before enabling it. Keyfigures search checks the caller’s permitted inputs before returning results. Search freshness and ranking follow the selected service’s contract; there is no universal guarantee of exact global live ranking.

07 / BUILD

Typed data. Familiar code.

OpenMDTA generates type-safe SDKs in multiple languages from your schema, so your fields reach your developers’ editors with their types. Prefer plain HTTP? Every Gateway also publishes an OpenAPI document for the same data, with a built-in explorer to try it.

This example from our generated Globex SDK streams MSFT and AAPL quotes with the default blocks. Each typed message keeps the selector that produced it, and prices stay exact Decimal values.

import { connect, selector, type ConnectOptions } from "@openmdta/globex";

export async function readQuoteSample(options: ConnectOptions) {
  const connection = await connect(options);
  try {
    const requested = [selector.us("MSFT"), selector.us("AAPL")] as const;
    // No options: the stream carries every block the Dataset publishes.
    const quotes = connection.select(requested).latestStream();
    let received = 0;
    for await (const msg of quotes) {
      const quote = msg.fields.bidAsk;
      console.log(
        msg.selector.expression,
        quote?.bid?.price.toString(),
        quote?.ask?.price.toString(),
      );
      if (++received === 10) break; // Leaving the loop cancels the stream.
    }
  } finally {
    await connection.close();
  }
}

The whole market, not just one quote.

Beyond single-instrument snapshots, a stream feed delivers the market data of every instrument in a Dataset: in source order, with gaps made explicit, in real-time, delayed, or end-of-day quality.

All your reference data, in one feed.

A Catalog feed delivers every reference-data record with the fields you choose and marks progress with cursors. Use it to fill your own database, cache, or search index.

08 / ENGINEERING TARGET

Two million instruments.
One fast CPU core.

Our goal: handle a feed of 2 million instruments at 200,000 updates per second, including ingestion, latest-state maintenance, and time-series processing, on one fast CPU core.

Only the separate S3 storage server’s resources are excluded. OpenMDTA’s encoding, compression, and client-side storage work remain in scope.

09 / OPERATE

Durable storage. Your operations.

STORAGE / DURABLE STATE

Durable storage. Replaceable services.

OpenMDTA’s durable state lives in S3-compatible storage, not in an irreplaceable service instance. Restart or replace services, reconnect them to the same storage, and resume from persisted state. Local caches can be rebuilt; there is no OpenMDTA-specific backup-and-restore procedure for each service.

Reliable storage is the foundation of recovery. It must provide the required consistency and conditional-write semantics. Availability, retention, and data protection remain part of your storage design. Keep configuration and credentials independently available.

OPERATIONS / CHOOSE YOUR CONTROL LAYER

Your data plane. Optional managed control plane.

The data plane provides the services, schemas, connectors, SDK tooling, and a static local configuration. Run those binaries with the deployment system your team already operates. Our commercial control plane adds the Kubernetes operator, declarative topology reconciliation, coordinated rollouts, and managed operations.

Prefer bare metal? Run and manage the service binaries directly. Kubernetes is an operating choice, not a requirement of the data platform.

Our goal: independence you can demonstrate.

Give an external engineering team the data-plane release and its documentation. Can they install it, run the static local example, ingest a feed, generate an SDK, reconnect services to existing storage, upgrade, and operate it without the commercial Kubernetes control plane? That is our acceptance test for independent self-hosting.

10 / MORE

More stuff

Different users. Different data access.

Your server creates and signs user-specific access tokens, limiting each user to their licensed subset of data packages and delivery quality. Signing keys stay on your server; clients receive only short-lived, signed access tokens. The Gateway and services enforce the applicable access rules.

For example, your real-time NASDAQ Level 2 license does not mean every user should get that access. Grant premium exchanges or specialist Datasets only to entitled users, while others keep access to your standard offering.

Your application maps subscription plans to grants and handles billing. OpenMDTA handles data access within the rights you hold.

Corporate actions. Consolidated and applied.

Consolidate dated corporate actions and distributions across sources with deterministic source priorities and retained provenance. Request raw values or opt into split adjustment through the Gateway for supported snapshot, live, and time-series fields. Confirmed split ratios adjust prices and quantities at delivery; stored source data stays unchanged.

Raw is the default. Distributions are modeled and consolidated, but are not yet applied as a total-return adjustment. Feed coverage and the financial rules required by your product remain part of the integration scope.