Shakti - replacement for kdb+ ? And MQTT

The main (very large) brain behind kdb+, Arthur Whitney, has been working on a new project since January 2019. The firm is called Shakti which is:

"Shakti is designed to manage streaming, in-memory and historical data in any format. Structured data, such as relational or time-series data can co-exist alongside, semi-structured and unstructured data creating a single platform for storage and analytics. Our technology takes advantage of the latest in hardware advancements from multi-core architectures, CPU cache, RAM, SSD and Intel® Optane™ technology all the way to cloud storage.

Parallelism lies at the core of Shakti. With the deceleration of Moore’s Law coupled with slowly increasing CPU clock speeds, distributed architectures have evolved to fill the gap. Our platform is parallel down to the lowest level. The distribution model extends out to multiple jobs and machines, enabling fast deployments for on-premises, containerized and/or cloud-based applications."

Source: here

Keen readers will be aware of this blogs long term interest in fast cars and fast data...

Hence a covid-19 lockdown project to create an internet-of-things platform using Shakti as a back-end.

At a very high level the implementation will be:

Automotive CANBUS data 
+
Precise GPS data 
==>MQTT client 
==>TCP/IP 
==>MQTT Server 
==> Shakti

From Shakti there will be streaming analytics and visualisations.

Why use MQTT?

If you take the time to compare the MQTT specifications to the FIX Trading Community SBE specifications you can see that there is a degree of commonality in implementation.  Having recently completed a fully-featured SBE-FIXP-SOFH application stack for a client the greenfield implementation of an MQTT stack with the lessons learnt clearly in mind was an easy choice.

The more generic nature of MQTT as an internet-of-things protocol means that there are some interesting twists, such as the variable byte integer. The smaller number of message types supported and the smaller number of datatypes supported also mean that the protocol is easier to implement on a feature complete basis.  The caveat here is that the client-side may well be a very resource constrained device, so as much work as possible must be offloaded to the server-side.  

Anyway, this is very much a work-in-progress, so I will try and update when I have anything interesting to write...

This would be even better with some IOT goodness