Archive for the 'kdb+' Category

qStudio 1.41 Release with Custom Security

qStudio 1.41 is now available to download.

It adds the ability to use custom Security Authentications and custom JDBC drivers.
By automatically loading .jar plugins from libs folder.

After a few users reported issues around “watched expressions” we are removing the ctrl+w shortcut as it was often getting used by mistake. The last change was some internal work to improved startup/shutdown logging for debugging purposes..

Standard SQL sucks and this is Why

dbfire

Recently there was a post on SQL tips by the JOOQ guys. I love their work but I think standard SQL is not the solution to many of these problems. What we need is something new or in this case old, that is built for such queries. What do I mean, well let’s look through their examples reimplemented in qsql and I’ll show you how much shorter and simpler this could be.

Everything is a table

In kdb we take this a step further and make tables standard variables, no special notation/treatment, it’s a variable like any other variable in your programming language. Instead of messing about with value()() we define a concise notation to define our variables like so:

Data Generation with Recursive SQL

This is the example syntax they have used to define two tables and then join them:

What to hell! If I want variables, let’s have proper variables NOT “Common Table Expressions”.

I created two tables a and b then I joined them sideways. See how simple that was.

Running Total Calculations

Oh dear SQL how badly you have chosen your examples. Running calculations are to APL/qSQL as singing is to Tom Jones, we do it everyday all day and we like it. In fact the example doesn’t even give the full code. See this SO post for how these things get implemented. e.g. Standard SQL Running Sum

qSQL table Definition and Running Sum:

Finding the Length of a Series

This is their code:

This is KDB:

In 1974 Ken Iverson gave a talk on APL. He described how he reduced it down to a core set of operations that everything could be made from. Using these simple building blocks you could make some really cool things. It’s sad to think we may not have came that far.

qSQL/kdb is a database based on the concept of ordered lists, carrying over many ideas from APL that make array operations shorter and simpler. If you like what you see we provide tutorials on kdb to get started, this intro is a good place to get started.

We also have free online kdb training for students.

kdb standard SQL support s)

First, in case you haven’t heard about it kdb has a standard SQL mode, you can send queries prefixed with s) and they will be interpreted as standard SQL like so:

Notice how the standard “and” syntax worked when I used s) but without it, q’s right to left evaluation causes problems. It’s about now that a lot of people get very excited, they think great I can skip learning that q-sql and use my standard SQL. Sometimes the look of joy on their faces transforms to frustration once they start using it. So let’s look at what works:

Operation Works?
Standard SQL Inserts Yes
ORDER BY half works
COUNT Yes
DELETE Yes
UPDATE Yes
String matching slightly works
NOT NO
IN Yes
GROUP BY Yes
LIMIT / TOP NO
Date Times NO

Standard SQL Inserts work

ORDER BY half works

“ORDER BY” will sort the columns in ascending order, attempting to use DESC has no effect.

COUNT works

DELETE works

UPDATE works

String matching slightly works

NOT fails

Modifying our String query slightly by adding NOT throws an error. My guess is that the interpreter has got confused.

IN works

GROUP BY works

LIMIT / TOP does not work

Date Times Don’t Work Right

Overall standard SQL support in kdb has got much better. However I would still recommend only using the s) syntax for plugging into an existing jdbc/odbc visualization tool and getting some immediate simple results. For any form of complex queries on strings, joins etc. support is either not there or the result may not be what you expect.

qStudio kdb IDE 1.40 Released

qStudio 1.40 is now available to download.

The latest changes include:

  • No need to save changes before shutdown, unsaved changes stored till reopened.
  • Add sqlchart to system path.
  • Fix display of tables with underscore in the name.
  • Database documenter/report enhancements
  • Improved code printing
  • FileTreePanel much more efficient at displaying large number of files.

KX closes down commercial 32 bit kdb, open alternatives?

Previously on our blog we had a lively debate about a possibly Open Sourced kdb+ , unfortunately kx now seems to be moving the opposite direction. In a recent announcement they are now restricting “32-bit kdb+ for non-commercial use only”. The timing is particularly unfortunate as:

Alternative (far less enterprise proven) solutions are available:

  • MAN AHL have released Arctic an open source Market Data platform based on python and MongoDB
  • Kerf Database – A DB aimed at the same market as kdb has now partnered with Briarcliff-Hall and is making greater sales inroads

This renewed interest in kdb alternatives hasn’t so far delivered a kdb+ killer but I fear in time it will.

kdb code highlighting in intellij

An intellij keyword file is now available to provide syntax highlighting of kdb code in intellij:

q Code Intellij Highlighting

q Code Intellij Highlighting

To install it copy this xml file to this directory:
C:\Users\USERNAME\.IdeaIC14\config\filetypes
Where USERNAME is obviously your username. Then restart intellij and open a .q file.

We’ve updated our notepad++ qlang.xml to provide code folding and highlighting of the .Q/.z namespaces.

kdb qunit testing now open source on github

We’ve now posted all source code from this website on our github kdb page.

Additionally we are open sourcing qunit, our kdb testing framework.
We look forward to receiving pull requests to fix our (hopefully few) bugs.

qStudio adds Nested Server Folder Support

Since our last qStudio kdb+ IDE announcment we have added a lot of new features:

Bulk importing kdb server lists

Bulk importing kdb server lists

There’s a lot of new features to allow supporting a huge number of servers efficiently:

  • Support importing HUGE number of servers:
    • 5000+ server connections are now supported
    • To prevent massive memory use, the object tree for a server is no longer refreshed at startup only on connection.
    • Allow specifying default username/password once for all servers
    • Allow nested connection folders
    • Add critical color option – servers with prod in name get highlighted in red
  • Sort File Tree Alphabetically
  • Numerous bugfixes including:
    • Fix critical Mac bug that prevented launching in some instances
    • Fix query cancelling

Open Sourced kdb+

In a world overran with open source big data solutions is kdb+ going to be left behind? I hope not…

Every few weeks someone comes to me with a big data problem often with a half-done mongoDB/NoSQL solution that they “want a little help with”. Usually after looking at it I think to myself

“I could solve this in 20 minutes with 5 lines of q code”

But how do I tell someone they should use a system which may end up costing them £1,000s per core in licensing costs. Yes it’s great that there’s a free 32-bit trial version but the risk that you may end up needing the 64-bit is too great a risk.

kdb+ vs mongoDB database popularity

Given the ever-increasing number of NoSQL solutions and in particular the rising popularity of Hadoop, R, python and MongoDB it’s not hard to see that open-source is taking over the world. Right now kdb+ still has the edge, that it’s faster, sleeker, sexier..but I don’t think that will save it in the long run. The power of open-source is that it let’s everyone contribute, witness the 100’s of libraries available for R, the 1000’s of javascript frameworks. The truly sad thing is that it’s not always the best underlying technology that wins. A 1000 amateurs creating a vibrant ecosystem of plug-ins, add-ons, tutorials… can beat other technologies through sheer force of numbers.

  • APL was a great language yet it remains relegated to history while PHP flourishes.
  • PostgreSQL was technically superior to MySQL yet MySQL is deployed everywhere

I believe kdb+ is the best solution to a large number of “big data” problems (small data to kdb+), When you stop and think, time-series data is everywhere, open sourcing kdb+ would open up entirely new sectors to kdb+ and I hope it’s a step kx take before it’s too late.

What do you think? Leave your comments below.

qStudio kdb+ GUI adds Dark Theme and Chinese Language

Based on user requests we have released a number of new features with qStudio 1.36:

Download the latest ->qStudio<- now.

Dark Code Editor Themes

qstudio-kdb-dark-theme-gui

Which can be set under settings->preferences

qstudio-settings-preferences

Open Results and Charts in New Window

To expand a panel into a new window click the “pop-out” icon.
pop-out

This will bring up the result in a new window:

mutliple-chart-windows

UTF-8 Chinese Language Support

qstudio-utf8