TimeStored Blog

jkdb – kdb Java – Running q language code within the java runtime

While doing the project euler programming challenges  it annoyed me how verbose the java answers would have to be compared to kdb. Then I got to wondering if I could create functions like til,mod,where,asc etc. in java and use them to create really short answers. Once I had the basic functions working, I wondered if I could get a working q) prompt…

I ended up with a prompt that let me run the following valid q code within the java runtime:
where[or[ =[0; mod[til[1000];3]]; =[0; mod[til[1000];5]] ]] 

Download the jkdb functional kdb code

The Code

How it Works

My Jkdb class has a number of statically defined functions that accept arrays of ints/doubles and return arrays/atoms of int/double. An example is asc:
/** perform ascending sort **/ public static double[] asc(double[] x) { Arrays.sort(x); return x; }; public static int[] asc(int[] x) { Arrays.sort(x); return x;}; public static double asc(double x) { return x; } public static double asc(int x) { return x; }
In the background each line you type has the square brackets and semi-colons converted to () and comma which are now valid java function calls. The amended string gets added to a .java file, compiled, ran and the output shown each time you press enter. Since I statically imported my class that has the til(x),max(x) etc functions, the code is valid java and works. This is why we can only use functional[x;y] form calls and not the infix notation x func y as I didn’t want to write a parser.

Interesting Points that would occur in Kdb

• All of the functions usually accept int,double, int[],double[] and boolean[] as their arguments and the various combinations. It is annoying having to copy paste these various combinations, I assume in kdb they use C macros to generate the functions, we could do something similar by writing java code to write the java functions.
• The functions I wrote often modify the array passed in, this gives us insight to why in the kdb C api you have to reference count properly, when the reference is 0 the structure is probably reused.

What use is this?

Almost entirely useless 🙂 I use the java class to occasionally spit out some sample data for tests I write. It was however insightful to get an idea of how kdb works under the covers. If you found this interesting you might also like kona an open source implementation of kdb.

I’d be interested in hearing:

• Have you tried kona? Did it work well?
• Have you found a functional library that gives you similar expressiveness to kdb for java?
• Tried Scala? Clojure?

The Jkdb provided functions are:

kdb+ 3.1 new STAC Benchmarks

http://www.kx.com/press-releases-130610.php

Palo Alto (10 Jun 2013) – Kx Systems, the leader in high-performance database and time-series analysis, has announced today the release of kdb+ v3.1 as well as its latest benchmark results for the STAC (Securities Technology Analysis Center) summit in Chicago. The results of the STAC M3™ vendor-independent market data benchmark suite demonstrate very significant performance improvements in v3.1, with results up to 8x faster than any previous STAC M3 benchmarks.

STAC ReportThe tests highlighted Kx’s unparalleled performance in such real-world calculations as generating the complete NBBO (National Best Bid and Offer) for one day, on thousands of symbols. Compared to previous best results, v3.1 runs over 4x faster, taking about half a minute to read, calculate, and write the entire table. In other benchmarks, calculations such as VWAPs (Volume Weighted Average Prices) on tick data were more than 8x faster….

It’s great to see kx continue to make speed improvements.

q kdb math library 0.4 released

Andrey Zholos announced a new version of q Math Library:

qml 0.4 is out:
http://althenia.net/qml

Now compatible with kdb+ 3.0 and whatever implementation of LAPACK your
distro provides.

Thanks to everyone who prompted me to continue working on this, and
especially to Jim Schmitz for his invaluable ideas, Kim Kuen Tang
for providing an interim version, and Kx for making a binary-
incompatible change to the interface

If there’s a permissively-licensed numerical library you’d like to see
included in the next version, let Andrey know. Or just submit a patch.

q for mortals – Version 3

The excellent Q For Mortals: A Tutorial In Q Programming by Jeffry Borror will soon be updated to version 3.
Jeff mentioned it at the recent NY user meeting.

You can read q for mortals online for free at: http://code.kx.com/wiki/JB:QforMortals2/contents
version 2 – added a new chapter on kdb database disk storage.

He said to get in touch if there’s anything you feel you would really like added.

UPDATE December 2015

q for mortals 3 is now out.

Q for Mortals Version 3 is a thorough presentation of the q programming language and an introduction to the kdb+ database. It is a complete rewrite of the original Q for Mortals that is current with q3.3. The presentation is derived from classes taught by the author at international financial institutions over the last decade. It is a series of tutorials based on q snippets intended to be entered interactively into the q console by the reader. The text takes its subject seriously but not itself. Technical explanations are augmented by mathematical observations, references to general programming concepts and other programming languages, and bad jokes. Coding style recommendations and advice to avoid gotchas appear liberally throughout. Examples are as simple as they can be but no simpler.

• Chapter 1, Q Shock and Awe, provides a piquant panorama of the power of q and its dazzling zen-like nature.
• Chapter 2 describes the base data types of q.
• Chapter 3 discusses lists, the fundamental data structure of q
• Chapter 4 presents the basic operators.
• Chapter 5 introduces dictionaries, which associate keys and values.
• Chapter 6 presents an in-depth description of functions and q’s constructs for functional programming.
• Chapter 7 demonstrates transforming data from one type to another.
• Chapter 8 introduces tables and keyed tables, the fundamental data structures for kdb.
• Chapter 9 describes q-sql and all the methods to manipulate tables.
• Chapter 10 presents ways to control execution of q programs.
• Chapter 11 covers file and interprocess communication I/O
• Chapter 12 describes workspace organization and management.
• Chapter 13 discusses system commands and command line parameters.
• Chapter 14 serves as an introduction to the kdb+ database. M

Available at all good bookstores.
http://www.amazon.com/For-Mortals-Version-Introduction-Programming/dp/0692573674
qTips is also good: http://www.amazon.co.uk/Tips-Fast-Scalable-Maintainable-Kdb/dp/9881389909

want to learn kdb? here’s some new tutorials

Added four new tutorials for those starting to learn kdb:

• kdb database – why a column-oriented in-memory database is best for timeseries analysis
• q Programming Style Best Practices – coding standards for effective maintainability
• Kdb database errors and limitations – Kdb has its limits.
• History of Kdb – What is kdb? k4? q kdb+ kdb+tick kdb+taq?

qStudio for Kdb 1.26 Released

q Code File Browser and Adding Multiple Kdb Servers

Added IDE Features:

• Add File Tree that allows browsing directory and providing autocomplete
• qDoc supports custom user tags (Thanks Aaron)
• Allow adding/exporting whole lists of servers at once (much quicker)
• Installers are now signed.
• Ctrl-D “goto definition” of function to open that file/position
• (PRO) Unit Testing and function profiling partially integrated.

Download qStudio 1.26 for Kdb Database

qStudio Kdb IDE 1.25 Released

Added qStudio Features:

• Faster chart drawing (~1.6x faster)
• Added No Redraw chart option for those who want extra speed
• Numerous bugfixes to charts that froze
• Allow setting code editor font size
• Fix display of boolean/byte lists

kdb+ New York User Meeting – March 11th 2013

Speakers Included:

Dennis Shasha – Fun with Timeseries

Interesting fun talk on finding the highest correlated streams among thousands of streams extremely efficiently by comparing streams against random generated data rather than exhaustively against each other. Followed up by pattern detection over different time windows efficiently which I think is this paper: .

Aaron Davies – (dis-)functional select and de-queueing bugs

Slides available here.
To those that have ever been forced to write functional selects, the shortcut notation allows a much more readable form.

Joe Landman – Performance vs. IO & Memory etc.

A topic I find extremely interesting, unfortunately the material was probably too densely packed and I couldn’t follow along well. Hopefully the slides will be released so I can get the details at my own pace. Joe has a great blog http://scalability.org full of high performance articles.

Simon Garland – Kdb 3.0 3.1f

Multithreaded slaves as standard – start kdb with an overridden -s and you can run separate kdb processes on different parts that will each perform part of the query. Standard slaves with par.txt were proving less worthwhile especially for SSD’s. Curious to see how this will compare to mserve.
-23! – optimise reading entire tables by mapping them in all at once. Really I want this to happen automatically (query optimizer please) but I guess we have to be happy with the faster speed.

discussion panel: “Components, Frameworks and Nifty Internal Tools – The Good, Bad and Ugly?”

UK Kdb Developers have longer attention spans

On TimeStored we monitor visitor statistics, so assuming that most visitors are going to work in kdb this gives us insight into where kdb consultants/developers are located, what OS/browser they use and how long they spent on the site. For example:

kdb locations

If you work in kdb your probably underneath a dot on this map.

Kdb Developers by Country

Kdb Developers by City

We can see if you work in kdb your probably located in either the UK or US, mostly London or New York. It’s interesting to see Belfast and Newry show up but easily explained as First Derivatives HQ is in Newry and both Citi and First Derivatives have offices in Belfast with kdb staff.

Now that we have seen where kdb developers are located let’s consider what OS/browser they use:

Kdb Developer Attention Span

kdb developer web browsers

kdb user operating systems

UK users spent 1 minute longer on the site than their US counterparts 🙂 Though this could partially be explained by the site being more responsive in the UK as the hosting was based there.

Surprisingly (to me) chrome was the number one browser and someone was using freebsd!