Kdb Tick Data Storage

Kdb+Tick is a standardised architecture by kx for storing, analysising and retrieving tick data. The product provides a framework for subscribing to data, distributing the data to your C++/Java processes and storing it efficiently long term.

Contents

  1. Market Data System Overview
  2. Event Sequence - Typcial sequence of events when accepting data, storing it and processing at end of day.
  3. Data Feed Handlers - Reuters, Bloomberg, Tibco.
  4. Tickerplant

Market Data System Overview

Architecture for storing market tick data in kdb+tick

A kdb+tick system consists of the following components

  • Feedhandler - Subscribes to exchange feeds and sends that data to the kdb+ tickerplant.
  • Tickerplant - Accepts feed data, logs it to file and pushes out the relevant data to subscribers.
  • RDB - An in-memory Realtime DataBase that subscribes to all symbols on the tickerplant and stores them in memory.
  • HDB - An on-disk Historical DataBase that stores tick data for previous days.

Data Feed Handlers

A feedhandler subscribes to data from an exchange, processes, cleans, normalizes the data and sends it to a kdb tickerplant. Examples of kdb java feedhandlers and C feedhandlers are available.

Event Sequence

  1. Tickerplant tick.q starts
    1. Load .q file that contains our table schemas.
    2. Load u.q that contains our .u functions: .u.init,.u.del,.u.sel,.u.pub,.u.add,.u.sub,.u.end
    3. Define our tickerplant specific functions for:
      1. .u.endofday - Handles end of day saving data down and starting new log.
      2. override .z.ts - To publish updates to subscribers in a timed loop
      3. .u.upd - Accepting data updates from feedhandlers, adding time column where needed and buffer for later publishing.
    4. tick[]
      1. Create empty subscriber list.
      2. Check all tables have time/sym columns, apply g# attribute.
      3. Open Log file etc.
    5. Waits for .upd updates to be received, buffer them, publishes to subscribers on a timed loop.
  2. RDB r.q starts
    1. Define Functions:
      1. .u.end - Save RDB contents to HDB directory at end of day.
      2. .u.rep - Replay log file, goto HDB directory.
      3. .u.upd - Accept updates by inserting into in memory table.
    2. Sit appending updates to in-memory tables until another command received.
  3. HDB starts
    1. Load directory as on-disk database. Answer queries as and when received.

Tickerplant

Important Variables

  • .u.t - tables that exist
  • .u.l/L - log file handle and directory
  • .u.i - log file message count
  • .u.w - contains a dictionary from tablenames->(handle;subscribed syms). e.g. Shows one subscribe to the quote table for the syms IBM,AA on socket handle 396.

.u.upd

.u.upd[tableName; tableData] accepts two arguments, for inserting data to a named table. This function will normally be called from a feedhandler. It takes the tableData, adds a time column if one is present, inserts it into the in-memory table, appends to the log file and finally increases the log file counter.

.u.sub

.u.sub accepts two arguments, the first argument is the tables that you want to subscribe to, the second is a list of syms that you are interested in. Once this function has been called, the remote process will receive any updates on them tables for the selected lists of syms. An empty symbol ` for any argument acts as a wildcard and returns data for all tables r syms.

End of Day

At end of day (when the date changes) a special sequence of events occurs, in order to save the data from the RDB to the HDB etc.

  1. Tickerplant - sends all subscribers a `.u.end message.
  2. RDB - saves all in-memory tables to disk using .Q.hdpf
  3. reload the HDB.

See Also