Kdb IPC Inter-Process Communication

Communication amongst kdb processes is simple, efficient and supports both synchronous and asynchronous communication.

The Basics

To start q on port 5000 we could use the command q -p 5000, or we can use \p 5000 from within the q console.

From one q instance hopen can be used to open a handle to another q process. Straight from the code kx website the formats supported are:

q)h:hopen `:10.43.23.198:5010                    / using ipaddress
q)h:hopen `:mydb.us.com:5010                     / using hostname
q)h:hopen `::5010                                / localhost
q)h:hopen (`:mydb.us.com:5010:elmo:sesame;10000) / full argument list, 10 second timeout
        

Message Formats

Once we have a connection there are two message formats:

handle "message"	Send a string, the entire message is evaluated on the server
handle (function; arg1; arg2..)	Send a list where the first item is either a locally defined function symbol/string for a remotely defined function The remaining items in the list are arguments for the function.

The string format is useful when entering commands manually at the console, however since it doesn't allow passing generated values easily, the list format is more useful.

(A)Synchronous messaging

Once we have a handle we can either send a message:

Synchonously	handle "message"	Send message, wait on and return the result.
Asynchonously	neg[handle] "message"	Send message, return to processing next statement immediately do not wait on or return a result.

Messages that require a response like function calls or select statements will normally use the synchronous form while message that need not return like inserting updates to a table will be performed asynchronously. Inside kdb there is a messaging thread which handles the actual sending of data. All messaging is actually TCP, the messaging thread takes care of message queueing/buffering. The processing loop of kdb itself is single threaded.

Example client/server communication

Client q)h:hopen `::5000 q)h "show `helloWorld" q)h "2+2" 4 // retrieve a remote value q)trade 'trade q)h "trade" sym time price size ------------------------------ kh 00:12:08.999 8.388858 12 hj 00:54:08.192 19.59907 10 lh 01:53:10.321 37.5638 1 mg 02:00:42.990 61.37452 90 mi 02:27:13.385 52.94808 73 // set a remote value q)h "a:11" q)neg[h] "a:12"; / asynchronous // alternative format is (functionName; arg1; arg2; ...) q)h (`f;100) 200000 // notice not a symbol, as function local q)f:{x+23} q)h (f; 100) // list format call 123

Server - q -p 5000 `helloWorld q)trade:([] sym:5?`2; time:5?.z.t; price:5?100.0; size:5?100) q)a 'a q)a 11 q)a 12 f:{x*2000}

Important Variables - .z.W .z.w .z.a .z.u

.z.W	Dictionary of open handles, to a list of message sizes. e.g. 3 -> 100 200 means handle 3 has two messages queued, size 100 then 200 in bytes
.z.w	When called inside a synchronous call, gives the return handle back to the caller.
.z.a	Returns the current IP address, if called inside a remote call this will be the remote IP address.
.z.u	Return the current users name, if called inside a remote call this is the name supplied when the connection was created.

Message Handlers - .z.pg .z.ps

When a message arrives into kdb it is processed by message handlers, these have a default setting but can be custom set by the user. For example we could override the the handler for synchronous message handling by defining .z.pg ourselves. By default .z.pg returns the value of the query. Handlers include:

.z.pg	Synchronous or Get call handler, x argument is the incoming message. Default handling is {value x} ie parse and run the query, returning result.
.z.ps	Asynchronous or Set call handler, x argument is the incoming message. Default handling is {value x;} ie parse and run the query, returning no result.
.z.po	Port Open, called when a new connection is made.
.z.pc	Port Close, called when an existing connection is closed.
.z.ph	HTTP GET called for web requests on port 80.
.z.pp	HTTP POST
.z.ws	Web Sockets (new in kdb 3.0)