kdb Python API - qPy PyQ

Article updated 2023-05-15. You now have two main options:

• qPython3 - Is an API for querying or uploading data to/from kdb and python.
  This is an open source project now under FINOS.
• PyKX - Is a closed source library allowing you to run python and kdb in the same memory space.
  Previously kx purchased the open source PyQ but now only continues PyKX development commercially

Python Libraries Available

Library	License	Last Updated	Website	Description	Python Versions Supported
qPython3	Open - Apache2	2023	site	qPython is a Python library providing support for interprocess communication between Python and kdb+ processes Synchronous and asynchronous queries, kdb+ v4.0	2.7 3.6 3.9 3.10* and numpy 1.8+
Kola	Closed	2023	Site	Python Polars Interface to kdb+/q written in rust language, is released to Linux/macOS/Windows. Authored by Jo Shinonome. More info..	3.8 to 3.12
PyKX	Closed / Commercial	2023	site	Brings Python and Q interpreters in the same process allows code written in either of the languages to operate on the same data. Seems to be kx's successor to PyQ, however where pyq was apache license, PyKX is commerical.	2.7 3.9 3.10*
PyQ	Open - Apache2	2020	site	Brings Python and Q interpreters in the same process allows code written in either of the languages to operate on the same data.	2.7 3.9 3.10*
Dan Nugents Python Library	Open	2013	site	Python interface for querying kdb servers, successor to qpy. Not updated in 10 years.	2.4 2.7 3.3
qpy	Open	2010	site	First Python interface for querying kdb servers, modeled after the Java interface.	2.7 3.3

Dan Nugents Python Library

Dans Python Library can be downloaded here.
Successor to qpy, this library is based heavily on it. It is quicker than qpy however you are unable to easily display tables. Daniel is no longer actively maintaining it but does accept bug fixes, there are no plans to support v3.x of Python.

2017 Update from Dan

For anyone else who needs a Python library, I am highly recommending the exxeleron qpython library (though it does require numpy, which requires 2.6 As a minimum, I believe, which can be a limitation)

1. Create a folder in your Python working directory so that you can separate modules from other files. You can find out Python's working directory by: #start Python on your OS >>> import sys,os >>> os.getcwd() #returns your python working directory, for example #/Users/carlosbutler/ >>> sys.path.append("python") #my own python folder inside the working directory
2. In another terminal window, download the files to your working directory, and unzip. Then run kdb. $ cd /Users/carlosbutler/python #changing to a folder in my working directory $ wget --no-check-certificate https://github.com/DannoHung/q/archive/master.zip $ unzip master.zip $ rm master.zip #removes downloaded file $ mv q-master/ q/ #rename the unzipped folder to make it easier to work with $ q -p 5000
3. In Python, import the module, connect and send query: >>> import q >>> #The __init__.py is a Python standard that Python will search for when you import a folder. >>> #This file in-turn imports other things. >>> #Create the connection and save the handle to a variable >>> c=q.conn(host="127.0.0.1",port=5000) >>> #Try the connection >>> c('1+1') 2
4. Example of sending / receiving data and tables. >>> #Sending and receiving data >>> c('t:([]a:til 10;b:10?100.0)') #send data and create a table >>> aVar=c('max t.a') #returns a straight number >>> -3+aVar*5 42 >>> >>> #Working with tables >>> aVar=c('select from t') >>> print aVar <q.collections.table object at 0x10ce1f090> >>> #the above happens as the API has no adequate table printing function, you are better off getting individual rows as they will display as dictionaries. Either by: >>> aVar[0] {'a': 0, 'b': 39.27523819729686} #or by doing >>> c('t[0]') {'a': 0, 'b': 39.27523819729686} >>> aVar[0]['b']#you can then use nested indexing to access table columns 39.27523819729686

qpy

qpy can be downloaded here.
Originally developed back in 2009 qpy is easy to setup and contained within one Python module. You can display tables in the Python command line with ease however it is no longer under development and can be slow if displaying a lot of data. qpy's successor is Daniel Nugent's Q.

Setup

1. Create a folder in your Python working directory so that you can separate modules from other files. You can find out Python's working directory by: #start Python on your OS import os os.getcwd()
2. You can then create a folder if you wish (you may want to do this to separate Python modules from other files and folders): os.makedirs("pythonFolderName") Remember that in Python you must append any folders you wish to access to Python's list of paths: import sys sys.path.append("pythonFolderName") sys.path #just to show you why you have done the previous command
3. Download kdb.py from https://bitbucket.org/halotis/qpy/src/, qpy's SVN, and move it to your relevant Python folder
4. Start your Q instance and open it up to an appropriate port number, such as 5001
5. Back in the Python instance: import kdb #in Python, when importing something you do not use the extension .py conn=kdb.q('localhost',5001,'') #three inputs are serverAddress(string),portNumber(int),userName(string) #you have just created a connection to the Q server, you can send commands by doing conn.k('1+1') conn.k('\l aFile.q') #etc...
6. Receiving and Sending Data To a Q Instance You will need to use the print command some of the time:
   print conn.k('select from table') #for example when receiving tables
7. You can send data by running a query: conn.k('insert [`tableName;(123;123.321)]') [12] #returns the row number just inserted

Recommendations

When accessing a couple of thousands of rows it can be quite slow, if that is a common occurrence you may wish to use another Python interface. However, in general it is best to avoid using Python and write your functions in Q, only later transferring the specific data to Python.

PyQ

Loads python as a dynamic library inside of kdb, allows entering code in either language using shared memory and data.

As recommended on Kx's Wiki page. If you are using a non Linux platform, it requires more effort to get up and running as you will need to compile it for a specific platform. PyQ can pre-compile Q queries and reuse them to speed up processing. But it is still recommended to minimise the computing done in Python, regardless of library being used, as Python is consierably slower than Q.

Install instructions are available here.