kdd 2019 Roadmap – Cloud and Community
This article was drafted in 2019, given it’s 2021 it made an interesting look back and sadly still a look forward….
There have been two big changes in the software world:
- The Cloud
- Community collaboration
Winners and Losers
From these shifts, there have been winners and losers
- Community helped Wikipedia build the best encyclopaedia
relegating Britannica and Encarta to history. - Community helped linux become the dominant operating system
Solaris/OS2 systems are now, only used in legacy niches - Community developed python is replacing matlab
- Cloud has seen atlassian/github/amazon/salesforce etc . win by offering SAAS solutions
to replace what would previously have been locally installed software (SAP/perforce) - Cloud hosted Gmail/Hotmail has replaced companies running their own mail servers
If kdb doesn’t change it will become a legacy platform with developers maintaining legacy systems that over time will be replaced with modern cloud alternatives.
Therefore we are starting two initiatives:
Cloud native KDB
- A fully-managed time-series database hosted on google cloud
- Able to be signed up for and used within 10 minutes
- Clear predictable pricing based on storage and query usage
- Hiding all the complexity of kdb (no par.txt/segments/sym file manipulation)
- While providing access to the speed and expressiveness of the language
- Taking advantage of modern load balancing (kubernetes) And cheap storage (S3)
We have a skunkworks team based in their own office, tasked with making a kdb database cloud solution so reliable and feature rich your kdb expert can now stop working to keep the database running and instead focus on business problems.
Community Driven q
We want kdb to run everywhere, for the barriers to adoption to drop and for the language to expand what it can do. A new kdb user will be able to run kdb on their machine through their standard package manager and to access a whole library of utilities to help them with whatever task they are trying to achieve.
KDB everywhere
To do this, we’ve formed a committee including representatives from finance/education and the wider community to:
- Open source the q language
- Development possibilities will be opened up to the wider community as anyone can submit ideas or even PRs for experimental functionality
- Being open source allows kdb to be bundled with linux and we see this as allowing wider use of q scripts
- Create a hosted packaging system that allows reusing code easily similar to NPM/maven
- Providing a wider library of community maintained packages that are easily reusable
- Work with aquaq to migrate parts of their torq framework to provide a kdb standard library
- Work with the community to onboard some of their code as packages
e.g. TimeStored is donating qunit
- Provide a recommended SDLC for kdb. Over the years we’ve developed processes for end to end development of q code at scale and we will be providing that same tooling to everyone.
By both open sourcing the language and allowing easier development of shareable packages we accelerate the pace at which kdb can help all developers solve problems and share solutions. Making the kdb platform stronger for everyone.
The Future of kdb is with you
It’s an exciting time and the demand for storing and analysing large time-series is growing. We believe by becoming cloud first and community driven we can continue to provide solutions for many years to come.