Programming/Kdb

Time series databases are increasingly being recognized as a powerful way to manage big data. They can be used to instrument, learn, and automate applications and systems, enable real-time and predictive analytics across manufacturing processes, and provide operational intelligence to make improved and faster decisions based on what is occurring now, what has occurred in the past, and what is predicted to take place in the future.

Kdb+ is a time series database optimized for big data analytics. The columnar design of kdb+ means it offers greater speed and efficiency than typical relational databases and its native support for time series operations vastly improves both the speed and performance of queries, aggregation, and analysis of structured data.

Kdb+ is also different from other popular databases because it has built-in proprietary languages, k and q, allowing it to operate directly on the data in the database, removing the need to ship data to other applications for analysis. Kdb+ and q make full use of the intrinsic power of modern multi-core hardware architectures. Kdb+ supports a number of different interfaces with a very simple API, for easy connectivity to external graphical reporting, and legacy systems.

In the world of high-frequency trading, the kdb+ database and its underlying programming languages, k and (especially) q, have risen to the top of the ranks as tools for implementing quantitative analyses of all types.

According to eFinancialCareers, k and q, the languages behind kdb+, "will get you the best jobs in banking" and are "the hottest coding language[s] in finance".

"A lot of kdb+ jobs are project-based, and very well paid," says Olly Thompson, an electronic trading systems recruiter at GQR Global Markets. "The reason for that is that there are simply not a lot of kdb+ developers out there."

A senior quant developer at U.S. bank, who works in kdb+, confirms that he is one of a select tribe. "Skilled kdb+ engineers are hard to come by–and skilled kdb+ quants are even more rare."

Although kdb+/q has a steep learning curve, once mastered, it becomes an exceedingly fast, convenient, and indispensable tool.

As the volume of data and speed at which it arrives continue to grow, traditional relational database management systems are facing ever more challenging workloads that they were never designed to support. For completely independent and audited performance benchmarks, the Security Technology Analysis Center (STAC) Benchmark Council has a number of tests comparing low-latency, high-volume technologies, and kdb+ features well in STAC results.

Przemek Tomczak explains:

"Kdb+ is ideally suited for these demands because of its unique combination of a higher-performance in-memory, columnar and relational database with an integrated vector-oriented programming system. Our customers are using kdb+ to get significant improvements to the performance and scalability of their applications in the face of these data volumes, particularly for supervisory control and data acquisition, data historians, fault detection and prediction, advanced data warehouses, and capital markets trading and surveillance systems."