Programming/Kdb/Labs/Feedhandler and kdb tick environment

Cryptocurrency

A cryptocurrency, crypto-currency, or crypto is a digital asset designed to work as a medium of exchange wherein individual coin ownership records are stored in a ledger existing in a form of a computerized database using strong cryptography to secure transaction records, to control the creation of additional coins, and to verify the transfer of coin ownership. It typically does not exist in physical form (like paper money) and is typically not issued by a central authority. Cryptocurrencies typically use decentralized control as opposed to centralized digital currency and central banking systems. When a cryptocurrency is minted or created prior to issuance or issued by a single issuer, it is generally considered centralized. When implemented with decentralized control, each cryptocurrency works through distributed ledger technology, typically a blockchain, that serves as a public financial transaction database.

Bitcoin, first released as open-source software in 2009, is the first decentralized cryptocurrency. Since the release of bitcoin, other cryptocurrencies have been created.

Cryptocurrency exchanges

A cryptocurrency exchange, or a digital currency exchange (DCE), is a business that allows customers to trade cryptocurrencies or digital currencies for other assets, such as conventional fiat money or other digital currencies. Exchanges may accept credit card payements, wire transfers or other forms of payment in exchange for digital currencies or cryptocurrencies. A cryptocurrency exchange can be a market maker that typically takes the bid-ask spreads as a transaction commission for its service or, as a matching platform, simply charging fees.

Here is a list of some major cryptocurrency exchanges:

• Binance
• Binance Delivery
• Binance Futures
• Binance US
• Bitcoin.com
• Bitfinex
• bitFlyer
• Bithumb
• BitMax
• BitMEX
• Bitstamp
• Bittrex
• Blockchain.com
• Bybit
• Coinbase
• Deribit
• EXX
• FTX
• FTX US
• Gate.io
• Gemini
• HitBTC
• Huobi
• Huobi DM
• Huobi Swap
• Kraken
• Kraken Futures
• KuCoin
• OKCoin
• OKEx
• Poloniex
• ProBit
• Upbit

APIs

API is an acronym for Application Programming Interface. In general, an API is an interface that allows two unrelated systems to interact with each other. An API specification describes the way that each of the two systems must interact with the API itself (such as the language that must be used, the syntax of messages that are passed back and forth, how frequently the messages can be sent, etc.).

Specifically for cryptocurrency trading, an API enables the client to interact with the exchange programmatically (via software instead of a human interface), allowing the client to obtain real-time market data, make trades, and manage the user's account.

A few of the tasks that can be accomplished via an API:

• download historical market data;
• stream real-time market data;
• place and cancel trading orders;
• view account balances;
• download trading history;
• make funding transactions...

Cryptocurrency exchange APIs are typically implemented using REST or the WebSocket protocol.

REST

Representational state transfer (REST) is a software architectural style that was created to guide the design and development of the architecture for the World Wide Web. REST defines a set of constraints for how the architecture of an Internet-scale distributed hypermedia system, such as the Web, should behave. The REST architectural style emphasizes the scalability of interactions between components, uniform interfaces, independent deployment of components, and the creation of a layered architecture to facilitate caching components to reduce user-perceived latency, enforce security, and encapsulate legacy systems.

A web service that obeys the REST constraints is informally described as RESTful. Such a web service must provide its web resources in a textual representation and allow them to be read and modified with a stateless protocol and a predetermined set of operations. This approach allows the greatest interoperability between clients and servers in a long-lived Internet-scale environment which crosses organisational boundaries.

In a RESTful web service, requests made to a resource's URI elicit a response with a payload formatted in HTML, XML, JSON, or some other format. For example, the response can confirm that the resource state has been changed. The response can include hypertext links to related resources. The most common protocol for these requests and responses is HTTP. It provides operations (HTTP methods) such as GET, POST, PUT, and DELETE. By using a stateless protocol and standard operations, RESTful systems aim for fast performance, reliability, and the ability to grow by reusing components that can be managed and updated without affecting the system as a while, even while it is running.

WebSocket

WebSocket is a computer communications protocol, providing full-duplex communication channels over a single TCP connection. The WebSocket protocol was standardized by the IETF as RFC 6455 in 2011, and the WebSocket API in Web IDL is being standardized by the W3C.

WebSocket is distinct from HTTP. Both protocols are located at layer 7 in the OSI model and depend on TCP at layer 4. Although they are different, RFC 6455 states that WebSocket

is designed to work over HTTP ports 443 and 80 as well as to support HTTP proxies and intermediaries,

thus making it compatible with HTTP. To achieve compatibility, the WebSocket handshake uses the HTTP Upgrade header to change from the HTTP protocol to the WebSocket protocol.

Schematically the WebSocket connection can be represented as follows:

REST versus WebSocket

Priya Pademkar lists some differences between the REST and WebSocket APIs:

• WebSocket uses HTTP only to establish the initial connection, whereas HTTP is the common protocol in RESTful web services.
• WebSocket communication is based on the socket concept, whereas REST is based on the resources concept rather than commands.
• WebSocket relies on an IP address and port number, whereas REST is based on the HTTP protocol and uses HTTP methods to relay data.
• The cost of communication is lower for WebSocket than for REST.
• WebSocket is better with high loads, whereas REST is more commonly employed for occasional communication.
• WebSocket is a stateful protocol, whereas REST is based on HTTP, which is a stateless protocol.

The following diagram by Emmanuel Picard emphasizes the differences between REST and event-driven APIs (such as those based on WebSocket):

FIX and related protocols

Few of the cryptocurrency exchanges use FIX and other specialized financial protocols. For completeness we mention them here.

The Financial Information eXchange (FIX) protocol is an electronic communications protocol initiated in 1992 for international real-time exchange of information related to securities transactions and markets. With trillions of dollars traded annually on the NASDAQ alone, financial service entities are employing direct market access (DMA) to increase their speed to financial markets. Managing the delivery of trading applications and keeping latency low increasingly requires an understanding of the FIX protocol.

The following diagram by Gareth Randall shows how FIX messaging looks between Buyside/Customer and Sellside/Supplier:

Other finance-specific protocols include FIX Adapted for STreaming (FAST), ITCH, and OUCH. See this page for more details.

Kdb+tick

Kdb+tick is an architecture which allows the capture, processing, and querying of real-time and historical data. It can be used to capture, store, and analyze massive volumes of data in real-time.

It is shipped in the form of a q library and consists of three q files:

• tick.q — the tickerplant process script;
• tick/r.q — the vanilla real-time subscriber process, known as the real-time database (rdb);
• tick/u.q — loaded by tick.q, contains the definitions of the .u functions: .u.init, .u.del, .u.sel, .u.pub, .u.add, .u.sub, and .u.end.

Kdb+tick can be downloaded from https://github.com/KxSystems/kdb-tick.

Installation

To install kdb+tick, place these files under the QHOME directory (e.g. C:/q), so you end up with:

• .../q
  • tick
    • r.q
    • u.q
  • w64 (or w32, or m32, or ..., depending on your OS and whether you are using 32-bit or 64-bit kdb+/q)
    • ...
  • q.k
  • q.q
  • README.txt
  • s.k
  • sp.q
  • tick.q
  • trade.q

Architecture

The architecture of a kdb+/q system is as follows. (The diagram is reproduced from Building real-time tick subscribers by Nathan Perrem, Kx:

The components are as follows:

• The data feed — a stream of data updates from a data source, e.g. order book updates and trades from the BitMEX exchange.
• The feedhandler (fh) — a process that captures the data from the data feed and feeds it into the tickerplant.
• The tickerplant (tp) — a kdb+/q instance that publishes data records to the real-time database and other real-time subscribers. Constitutes the first element of the kdb+tick triplet.
• The real-time database (rdb) — a kdb+/q instance that subscribes to data records on the tickerplant and stores them in an in-memory table. Today's data can be queried on the rdb. The rdb constitutes the second element of the kdb+tick triplet.
• The historical database (hdb) — a kdb+/q instance that stores the data up to and including yesterday in a splayed and partitioned (by date) table on disc. Can be queried. Constitutes the third element of the kdb+tick triplet.
• The real-time subscriber (rts) — a kdb+/q instance or another process that subscribes to data records on the tickerplant. This process may compute certain metrics, such as the volume weighted average price (VWAP), visualize the data, etc. Optional.
• The binary log file contains the data for the current date written by the tp. Used to restore the rdb should the rdb process crash or require restarting for whatever reason.

Feedhandler

Many cryprocurrency exchanges support both the WebSocket and REST APIs.