Open Sourced kdb+
In a world overran with open source big data solutions is kdb+ going to be left behind? I hope not…
Every few weeks someone comes to me with a big data problem often with a half-done mongoDB/NoSQL solution that they “want a little help with”. Usually after looking at it I think to myself
“I could solve this in 20 minutes with 5 lines of q code”
But how do I tell someone they should use a system which may end up costing them £1,000s per core in licensing costs. Yes it’s great that there’s a free 32-bit trial version but the risk that you may end up needing the 64-bit is too great a risk.
Given the ever-increasing number of NoSQL solutions and in particular the rising popularity of Hadoop, R, python and MongoDB it’s not hard to see that open-source is taking over the world. Right now kdb+ still has the edge, that it’s faster, sleeker, sexier..but I don’t think that will save it in the long run. The power of open-source is that it let’s everyone contribute, witness the 100’s of libraries available for R, the 1000’s of javascript frameworks. The truly sad thing is that it’s not always the best underlying technology that wins. A 1000 amateurs creating a vibrant ecosystem of plug-ins, add-ons, tutorials… can beat other technologies through sheer force of numbers.
- APL was a great language yet it remains relegated to history while PHP flourishes.
- PostgreSQL was technically superior to MySQL yet MySQL is deployed everywhere
I believe kdb+ is the best solution to a large number of “big data” problems (small data to kdb+), When you stop and think, time-series data is everywhere, open sourcing kdb+ would open up entirely new sectors to kdb+ and I hope it’s a step kx take before it’s too late.
What do you think? Leave your comments below.
I like that kdb is niche.
I worry that with a flood of developers the premium wage that kdb commands would fall.
I don’t particularly care for kdb+ at this point in time (very green in respect to the technology), but k and inherently q (and hence kdb+) are fascinating beasts that deserve far more attention.
I hope that kparc move to a more open model (at least for the language), this will hopefully promote more attention the community, apl’s and cull the attitudes of those in comfortable and perhaps unchallenged positions in the hierarchy of the banking industry.
I can’t help but feel that technologies such as datomic are doing a great job of drawing away from kdb+ because of their very vocal core team and community not to mention the ever charismatic Rich Hickey. (although this might be from a personal perspective).
I’ve never seen a job posting for a junior or intermediate q/kdb developer. I usually associate such things with an incestual marketplace (or too many self certified ‘seniors’ in a very small industry).
I hope that this changes, and I feel that it slowly is. I’m not sure that open-sourcing kdb will do it. But perhaps k will. (breadcrumbing people to kdb). The community seems to really be open and helpful, it really just comes down to education and awareness.
Promoting to the developers space won’t yield a fiscally rewarding outcome, and will probably dilute the current wages of kdb developers, but these are costs that must be weighed; and I’m sure Arthur and the team at Kx/kparc have been doing this for some time.
Either way I still think kdb is far underused in the broader community (outside finance/insurance) and one of the key ways to rectify this is currently being done (free introductory courses).
Offering kdb+ under an open source licence is an interesting move from kx. Think this will increase the use of kdb+ in other sectors than the finance industry. I’m looking forward to the new ideas this will bring into the ecosystem.
I’m totally agree with Ryan. Kdb+’s potential is much bigger than its current niche in financial sector. I don’t think that Kx spends a lot for development, not for the product which is the single binary with a size of modest JavaScript framework, so I don’t expect that community efforts would offload Kx’s development team a lot, but community can create forks with such abilities which Kx never dreamt about. Ryan is right about BigData, kdb+ could do a lot of work and save vast amounts of money being used there, but current licensing scheme makes massive deployments (onto hundreds of cloud instances) impossible.
It seems to me, that even being locked in financial sector kdb+-related consultancy can generate even more money than selling licenses and official support, otherwise First Derivatives wouldn’t be able to buy the major stake in Kx. So, why to not grow up out of current niche?
I think open sourcing kdb+ would be a good things. I see the following two new areas that would benefit from it:
FPGA-accelerated computing
————————–
Intel will soon release a E5 Xeon with integrated FPGA. See:
http://www.extremetech.com/extreme/184828-intel-unveils-new-xeon-chip-with-integrated-fpga-touts-20x-performance-boost
This Xeon can enable a 20x performance boost for kdb+ applications.
Non-x86 based hardware
———————-
An open sourced kdb+ would allow to compile kdb+ for example
for MIPS64 processors. The Octeon III processors are MIPS64
processors that are used in high end ethernet hardware (10 GBit
routers for example). This would be a new market that
can be accessed.
It’s a great idea, one that Kx should seriously consider. It’s also an opportunity to go mainstream and let others experience what KDB+ is capable of and leverage that to grow its business. This is what is driving all the growth in today’s market – open source with premium services.
I don’t know, but I find it pretty comical that a piece of shit like MongoDB is valued at $1.6B in its latest round of funding, while First Derivatives buys 46% of KX for £26M. Can’t help but think that Arthur and KX have cost themselves big-time with their small minded, short-term thinking.
I kind of agree, too. The specific mindset of “I am the niche” of the current kx always seems a bit outdated to me. I always believe that opening sourcing a piece of fantastic code only makes it better going forward. Even with the risk of branching, etc., it’s still worth the efforts.
I think there is absolutely no risk fo Kx Systems to publish the source – nobody will be able to read and will need expensive support anyway. It would boost sales for sure for additional illusion what one can read code if something goes wrong.
I don’t hing closed source solutions will survive 15 years and corporate policies will have statement “only software with source code available allowed”.
[…] on our blog we had a lively debate about a possibly Open Sourced kdb+ , unfortunately kx now seems to be moving the opposite direction. In a recent announcement they are […]