qStudio 1.40 is now available to download.
The latest changes include:
An intellij keyword file is now available to provide syntax highlighting of kdb code in intellij:
q Code Intellij Highlighting
To install it copy this xml file to this directory:
C:\Users\USERNAME\.IdeaIC14\config\filetypes
Where USERNAME is obviously your username. Then restart intellij and open a .q file.
We’ve updated our notepad++ qlang.xml to provide code folding and highlighting of the .Q/.z namespaces.
We’ve now posted all source code from this website on our github kdb page.
Additionally we are open sourcing qunit, our kdb testing framework.
We look forward to receiving pull requests to fix our (hopefully few) bugs.
Since our last qStudio kdb+ IDE announcment we have added a lot of new features:
Bulk importing kdb server lists
There’s a lot of new features to allow supporting a huge number of servers efficiently:
Based on user requests we have released a number of new features with qStudio 1.36:
Download the latest ->qStudio<- now.
Which can be set under settings->preferences
To expand a panel into a new window click the “pop-out” icon.
This will bring up the result in a new window:
Newry-based financial software firm First Derivatives has acquired a majority shareholding in big data analytics company Kx Systems for £36m (€44m).
KX has historically had a hard time penetrating markets outside finance, FD have a good sales team and previously acquired a marketing company in Philadelphia, hopefully this is the chance for kdb+ to go mainstream.
However it’s a worry that FD (First Derivatives) may increasingly “encourage” purchasing of the delta platform bundle rather than stand-alone kdb+. With the smaller margins outside of finance, will FD take a risk and open up the database. (Would FD have been a supporter of the 32-bit version becoming free for commercial use?) There’s a large number of individuals in off-shore locations that want to learn kdb+, FD could be incentivized to discourage that as it would hurt their consulting business.
It’s also interesting to consider companies that have already invested in KX technology and whether they will continue to do so
What do you think? Will this lead to wider adoption? a growing platform?
kdb+ London Contract – £800 p/day
kdb+ belfast Citigroup Contract £400 p/day
Java Poland £150 p/day
sqlDashboards are included as a bundle with qStudio, part of that package is a command line utility called sqlChart that allows generating customized sql charts from the command line.
Checkout the video to see how you can create a chart based on data from a kdb+ database in 2 minutes:
The sqlChart page has all the documentation you need, Download the qstudio.zip to try it now.
I always like to investigate new technology and this week I found a nice automatic technique for improved cache use that I had previously seen some people manually write.
Consider a database query with three steps (three SQL SELECTs), some databases may pass results of each step to temporary tables in main memory. When the first step is finished, these intermediate results are passed back into CPU cache to be transformed by the second step, then back into a new temporary table in main memory, and so on.
To eliminate this back-and-forth, vector-based statistical functions can be pipelined, with the output of one function becoming input for the next, whose output feeds a third function, etc. Intermediate results stay in the pipeline inside CPU cache, with only the full result being materialized at the end.
This technology is part of ExtremeDB, they have a video that explains it well:
This is what the actual code would look like to calculate the 5-day and 21-day moving averages for a stock and detect the points where the faster moving average (5-day) crosses over or under the slower moving average (21-day):
This approach eliminates the need to create, populate and query temporary tables outside CPU cache in main memory. One “tile” of input data is processed by each invocation of ‘mco_seq_window_agg_avg_double()’, each time producing one tile of output that is passed to ‘mco_seq_sub_double()’ which, in turn, produces one tile of output that is passed as input to mco_seq_cross_double(), which produces one tile of output that is passed to mco_seq_map_double(). When the last function, mco_seq_map_double() has exhausted its input, the whole process repeats from the top to produce new tiles for additional processing.
A very cool idea!
And yes, ExtremeDB are the same guys that posted the top Stac M3 benchmark for a while (in 2012/13 I think).
Kdb does not have built-in functions for bitwise and,or,xor operations, we are going to create a C DLL extension that provides bitwise operators.
To download the source and definitions for and/or/xor bitwise operations click here
If your new to extending Kdb+ see this tutorial on writing a DLL for Kdb+
I also thought this would be a good opportunity to look at using q-code to generate C code, to generate Kdb Dll’s. When converting an existing library to work with Kdb or writing similar functions for different types, there’s a lot of repetitive coding. Either you can use macros or you can use one language to generate another, to save a lot of typing. Let’s look at bitwise operations as an example of what I mean:
If I wanted to write a bitwise and function band that takes two lists and performs a corresponding operation, we could write the C functions like so:
Notice the similarity between the bandJJ and bandII functions. Plus if we want to handle the case where the second argument is an atom we need an entire other set of C functions similar to bandJ to handle that. This is soon beginning to spiral beyond an easy copy paste job. Instead I used the following 20 lines of q code to generate the 130 lines of C code:
This generates my C code, a .def file exporting the functions that I want to provide and creates a .q file that loads those functions into kdb. Which means any time I add a function or want to support a new type, everything is done for me. let’s look at our functions in action:
Yes the q code is messy (it could be improved) but the concept of dynamically generating your imports can be a real time saver.
This code is not intended for reliability or performance, use at your own risk! If however bitwise operations are a topic that interest you I recommend this article on: Writing a fast vectorised OR function for Kdb.