Difference between revisions of "Programming/Kdb/Installing kdb"

From Thalesians Wiki
 
(9 intermediate revisions by the same user not shown)
Line 3: Line 3:
==Downloading==
==Downloading==


'''Kdb+/q''' can be downloaded from <span class="plainlinks">https://kx.com/developers/download-licenses/</span>.
'''Kdb+/q''' can be downloaded from https://kx.com/developers/download-licenses/.


In particular, there are two free versions of kdb+/q: the 32 Bit Personal Edition and the 64 Bit Personal Edition (On-Demand).
In particular, there are two free versions of kdb+/q: the 32 Bit Personal Edition and the 64 Bit Personal Edition (On-Demand).
Line 27: Line 27:
* Linux/Mac/Windows versions are available.
* Linux/Mac/Windows versions are available.


The 32 Bit Personal Edition can be downloaded immediately, whereas for the 64 Bit Personal Edition (On-Demand) you would need to request a license. Both editions are available via <span class="plainlinks">https://kx.com/developers/download-licenses/</span>.
The 32 Bit Personal Edition can be downloaded immediately, whereas for the 64 Bit Personal Edition (On-Demand) you would need to request a license. Both editions are available via https://kx.com/developers/download-licenses/.


==Installing==
==Installing kdb+/q==


On Windows, it is customary to install kdb+/q under <tt>C:\</tt>, so you end up with
Kdb+/q can be installed on Linux, macOS, or Windows.
 
There are three installation scenarios:
* You are a commercial user, with a <tt>k4.lic</tt> file.
* You have the 64 Bit Personal Edition On Demand for non-commercial use. This is enabled by a <tt>kc.lic</tt> license-key file and requires an always-on internet connection. (This version of kdb+ is not licensed for use on cloud servers.)
* You have the 32 Bit Personal Edition for non-commercial use. It does not require a license-key file.
 
The first step is to download kdb+/q from https://kx.com/developers/download-licenses/.
 
Once you have downloaded the archive, unzip it to:
* <tt>~/q</tt> on Linux;
* <tt>~/q</tt> on macOS;
* <tt>C:\q</tt> on Windows.
Here <tt>~</tt> represents your <tt>HOME</tt> directory.
 
You can use the command line to unzip the archive. On Linux:
<pre>
unzip l64.zip -d $HOME/q
</pre>
On macOS:
<pre>
unzip m64.zip -d $HOME/q
</pre>
On Windows (PowerShell):
<pre>
Expand-Archive w64.zip -DestinationPath C:\q
</pre>
 
It is possible to install kdb+/q to another directory, however, if you install it to a different directory from the ones listed above, you need to set the <tt>QHOME</tt> environment variable to point to that directory. You can install kdb+ anywhere as long as you set the path in <tt>QHOME</tt>.
 
Note that 32-bit applications will not run in macOS 10.15+ (Catalina and later).
 
Thus you end up with the following directory structure:
* <tt>C:\</tt> (for example)
** <tt>q</tt>
*** <tt>w32</tt> (or <tt>w64</tt>, or <tt>l32</tt>, etc.)
**** <tt>q.exe</tt>
*** <tt>q.k</tt>
*** <tt>q.q</tt>
*** <tt>README.txt</tt>
*** <tt>s.k</tt>
*** <tt>sp.q</tt>
*** <tt>trade.q</tt>
 
==Installing the license file==
 
If you have a license file, <tt>k4.lic</tt> or <tt>kc.lic</tt>, put it in the <tt>QHOME</tt> directory.
 
Then you will end up with the following directory structure:
* <tt>C:\</tt> (for example)
** <tt>q</tt>
*** <tt>w32</tt> (or <tt>w64</tt>, or <tt>l32</tt>, etc.)
**** <tt>q.exe</tt>
*** <tt>kc.lic</tt>
*** <tt>q.k</tt>
*** <tt>q.q</tt>
*** <tt>README.txt</tt>
*** <tt>s.k</tt>
*** <tt>sp.q</tt>
*** <tt>trade.q</tt>
 
By default, kdb+/q looks for a license file in <tt>QHOME</tt>. To keep your license file elsewhere, set its path in the environment variable <tt>QLIC</tt>.
 
==Tips and tricks==
 
===Running 32-bit kdb+ on 64-bit Linux===
 
Use the
<pre>
uname -m
</pre>
command to determine whether your machine is using the 32-bit or 64-bit Linux distribution.
 
If the result is
* <tt>i686</tt> or <tt>i386</tt> or similar, you are running a 32-bit Linux distribution;
* <tt>x86_64</tt>, you are running a 64-bit Linux distribution.
 
To install 32-bit kdb+ on a 64-bit Linux distribution, you need a 32-bit library. Use your usual package manager to install <tt>i686</tt> or <tt>i386</tt>, for example:
<pre>
sudo apt-get install libc6-i386
</pre>
 
For more details on how to run 32-bit programs on Ubuntu 64-bit, see https://askubuntu.com/questions/454253/how-to-run-32-bit-app-in-ubuntu-64-bit.
 
===<tt>rlwrap</tt>===
 
On Linux and macOS the <tt>rlwrap</tt> command allows the Up arrow to retrieve earlier expressions in the q session. It is very useful and we recommend installing it.
 
Ask for its version number:
<pre>
rlwrap -v
</pre>
If you see one, <tt>rlwrap</tt> is already installed. Otherwise, you will see
<pre>
rlwrap: command not found.
</pre>
 
Install <tt>rlwrap</tt> using your package manager. (Common package managers: <tt>apt</tt>, <tt>dnf</tt>, and <tt>yum</tt> for Linux; Homebrew and MacPorts for macOS.)
 
It is straightforward to "wrap" the <tt>q</tt> process into <tt>rlwrap</tt>:
<pre>
rlwrap ~/q/l64/q
</pre>
 
===<tt>taskset</tt>===
 
If your machine has many cores, you may encounter the following error:
<pre>
'licence error: cores
</pre>
 
This means you are trying to run kdb+/q on a machine with two many cores. The number of licensed cores appears in parentheses in the message that appears when you launch kdb+/q. For example:
<pre>
KDB+ 4.0 2020.05.04 Copyright (C) 1993-2020 Kx Systems
l64/ 72(16)core 1029691MB paul ...
</pre>
 
This means that kdb+/q is running on a machine with 72 cores, but only 16 of them are covered by the license.
 
In this situation you have to restrict the cores visible to the kdb+/q process using <tt>taskset</tt>. According to the documentation,
<blockquote>
<tt>taskset</tt> is used to set or retrieve CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs [or CPU cores] on the system. The Linux scheduler will honour the given CPU affinity and the process will not run on any other CPUs.
</blockquote>
 
For example, we could run the <tt>q</tt> process as follows:
<pre>
rlwrap taskset -c 0,1,2,3,4,5,6,7,8,9,10,11 ~/q/l64/q
</pre>
 
In our example, this will force kdb+/q to use only the cores from 0 to 11 (inclusive) and will resolve the "cores" error.
 
==Confirming success==
 
Confirm that kdb+/q is working: launch your first q session. On Windows:
<pre>
cd C:\q\w32
q.exe
</pre>
On Linux:
<pre>
cd ~/q
./q
</pre>
 
The q session opens with a banner like this:
<pre>
KDB+ 3.6 2019.04.02 Copyright (C) 1993-2019 Kx Systems
w32/ 8()core 4095MB paul desktop-efq7tt8 192.168.1.105 NONEXPIRE
</pre>
 
Try your first expression:
<pre>
q)3+5
8
</pre>
 
End the q session and return to the command shell.


=Integrated development environments (IDEs)=
=Integrated development environments (IDEs)=
Line 39: Line 195:
==Q Insight Pad (Windows-only)==
==Q Insight Pad (Windows-only)==


<span class="plainlinks">[http://www.qinsightpad.com/ Q Insight Pad]</span> (sometimes referred to simply as qPad) by <span class="plainlinks">[http://olegzakharov1974.narod.ru/ Oleg G. Zakharov]</span> is a kdb+ client, written in C++ to run under Windows; as a result, it is fast, memory-efficient and easy to use. The author makes a special emphasis on providing a Windows-style GUI that is visual, simple and intuitive.
[http://www.qinsightpad.com/ Q Insight Pad] (sometimes referred to simply as qPad) by [http://olegzakharov1974.narod.ru/ Oleg G. Zakharov] is a kdb+ client, written in C++ to run under Windows; as a result, it is fast, memory-efficient and easy to use. The author makes a special emphasis on providing a Windows-style GUI that is visual, simple and intuitive.


Q Insight Pad provides simple but useful visualization capabilities.
Q Insight Pad provides simple but useful visualization capabilities.


An unregistered but fully functional version (working with free versions of q) can be downloaded for free from <span class="plainlinks">http://www.qinsightpad.com/</span>.
An unregistered but fully functional version (working with free versions of q) can be downloaded for free from http://www.qinsightpad.com/.
 
This version expires every 3 months. If you want to license Q Insight Pad for use in your organization, contact the author: http://www.qinsightpad.com/contact.html.
 
===Connecting Q Insight Pad to kdb+/q===


This version expires every 3 months. If you want to license Q Insight Pad for use in your organization, contact the author: <span class="plainlinks">http://www.qinsightpad.com/contact.html</span>.
Start a kdb+/q session. Open a port, for example, 41822:
<pre>
q)\p 41822
</pre>
 
You can now connect to this port from Q Insight Pad.
 
Right-click on "KDB+ Servers". Click on "Add...". Give the server a descriptive name, e.g. "Local". Set the Server to <tt>localhost</tt> and the port to 41822.
 
Click on "Test". You should see "Successfully connected to <tt>`:localhost:41822::</tt>.
 
Click "OK".
 
Double-click on the new server, which should now appear under "KDB+ Servers".
 
In the text editor, enter an expression that you would like to evaluate, e.g.
<pre>
3+5
</pre>
Select that expression and press Ctrl + E. The result
<pre>
8j
</pre>
should appear under "Output".


==qStudio==
==qStudio==


<span class="plainlinks">[http://www.timestored.com/qstudio/ qStudio]</span> by <span class="plainlinks">[http://www.timestored.com/ TimeStored]</span> is an intelligent code editor for kdb+/q that provides:
[http://www.timestored.com/qstudio/ qStudio] by [http://www.timestored.com/ TimeStored] is an intelligent code editor for kdb+/q that provides:
* q syntax highlighting;
* q syntax highlighting;
* q code completion;
* q code completion;
Line 61: Line 244:
** qUnit &mdash; a unit testing framework for kdb.
** qUnit &mdash; a unit testing framework for kdb.


qStudio can be downloaded from <span class="plainlinks">http://www.timestored.com/qstudio/</span>.
qStudio can be downloaded from http://www.timestored.com/qstudio/.


qStudio Pro can be purchased from <span class="plainlinks">http://www.timestored.com/qstudio/buy</span>.
qStudio Pro can be purchased from http://www.timestored.com/qstudio/buy.


This application requires a Java Runtime Environment 1.8.0_66 or above.
This application requires a Java Runtime Environment 1.8.0_66 or above.
Line 69: Line 252:
==Studio for kdb+==
==Studio for kdb+==


<span class="plainlinks">[https://github.com/CharlesSkelton/studio Studio for kdb+]</span> is a rapid development environment for kdb+. In the style of commonly used SQL clients, it allows you to
[https://github.com/CharlesSkelton/studio Studio for kdb+] is a rapid development environment for kdb+. In the style of commonly used SQL clients, it allows you to
* connect to kdb+ processes;
* connect to kdb+ processes;
* execute selected text from the editor window;
* execute selected text from the editor window;
Line 86: Line 269:
* immediate charting of grid data.
* immediate charting of grid data.


Studio for kdb+ is free, provided under the Apache 2 license, and can be downloaded from <span class="plainlinks">https://github.com/CharlesSkelton/studio</span>.
Studio for kdb+ is free, provided under the Apache 2 license, and can be downloaded from https://github.com/CharlesSkelton/studio.


The program is portable and comes in the form of a JAR file. You need the Java runtime environment (JRE) to run it:
The program is portable and comes in the form of a JAR file. You need the Java runtime environment (JRE) to run it:
Line 92: Line 275:
java -jar studio.jar
java -jar studio.jar
</pre>
</pre>
==KX Developer==
[https://code.kx.com/developer/ KX Developer] is a visual environment used to manage, manipulate, and explore massive datasets in real-time by exploiting kdb+'s server-based analytics technology.
The Developer is comprised of the following components:
* Workspace &mdash; local and shared repository for managing entities;
* Importer &mdash; import datasets into the workspace;
* Exporter &mdash; export datasets into other systems;
* Visual Inspector &mdash; visualize massive datasets in real-time or near-real-time;
* Interactive editor &mdash; write applications using the q programming language;
* Debugger &mdash; debugger dramatically increases productivity;
* Code Differ &mdash; compare multiple versions of the functions or modules.
Contained within the KX Developer package is a set of highly useful q developer libraries and scripts for use outside of the KX Developer UI and for automated build systems. The libraries contained provide support for:
* Code coverage;
* Data visualization;
* Documentation generation from source annotations;
* Property-based testing;
* Q static linting and syntax checker;
* Regular expressions;
* Run time and memory profiling;
* Unit testing;
* KX Developer module to q script conversion.
KX Developer can be downloaded for free from https://code.kx.com/developer/.
'''Note that KX Developer requires a 64-bit version of kdb+/q.'''

Latest revision as of 22:11, 21 December 2021

Kdb+/q

Downloading

Kdb+/q can be downloaded from https://kx.com/developers/download-licenses/.

In particular, there are two free versions of kdb+/q: the 32 Bit Personal Edition and the 64 Bit Personal Edition (On-Demand).

Both editions are licensed for personal use.

The 32 Bit Personal Edition:

  • is permitted to run on-premises;
  • is permitted to run on cloud;
  • provides access to KX Fusion interfaces and integrations;
  • provides access to community support;
  • Linux/Mac/Windows/Raspberry pi versions are available.

The main limitation of the 32 Bit Personal Edition and, more generally, of any 32-bit program, is that it can only ever use up to 4 GB of memory. That's due to the 32-bit address space, since 2^32 = 4,294,967,296 bytes. For some use cases this may be a serious limitation.

The 64 Bit Personal Edition (On-Demand):

  • requires "always on" internet connection;
  • is permitted to run on-premises;
  • is compatible with KX Developer and KX Dashboards Direct;
  • provides access to KX Fusion interfaces and integrations;
  • provides access to community support;
  • gives regular access to upgrades;
  • Linux/Mac/Windows versions are available.

The 32 Bit Personal Edition can be downloaded immediately, whereas for the 64 Bit Personal Edition (On-Demand) you would need to request a license. Both editions are available via https://kx.com/developers/download-licenses/.

Installing kdb+/q

Kdb+/q can be installed on Linux, macOS, or Windows.

There are three installation scenarios:

  • You are a commercial user, with a k4.lic file.
  • You have the 64 Bit Personal Edition On Demand for non-commercial use. This is enabled by a kc.lic license-key file and requires an always-on internet connection. (This version of kdb+ is not licensed for use on cloud servers.)
  • You have the 32 Bit Personal Edition for non-commercial use. It does not require a license-key file.

The first step is to download kdb+/q from https://kx.com/developers/download-licenses/.

Once you have downloaded the archive, unzip it to:

  • ~/q on Linux;
  • ~/q on macOS;
  • C:\q on Windows.

Here ~ represents your HOME directory.

You can use the command line to unzip the archive. On Linux:

unzip l64.zip -d $HOME/q

On macOS:

unzip m64.zip -d $HOME/q

On Windows (PowerShell):

Expand-Archive w64.zip -DestinationPath C:\q

It is possible to install kdb+/q to another directory, however, if you install it to a different directory from the ones listed above, you need to set the QHOME environment variable to point to that directory. You can install kdb+ anywhere as long as you set the path in QHOME.

Note that 32-bit applications will not run in macOS 10.15+ (Catalina and later).

Thus you end up with the following directory structure:

  • C:\ (for example)
    • q
      • w32 (or w64, or l32, etc.)
        • q.exe
      • q.k
      • q.q
      • README.txt
      • s.k
      • sp.q
      • trade.q

Installing the license file

If you have a license file, k4.lic or kc.lic, put it in the QHOME directory.

Then you will end up with the following directory structure:

  • C:\ (for example)
    • q
      • w32 (or w64, or l32, etc.)
        • q.exe
      • kc.lic
      • q.k
      • q.q
      • README.txt
      • s.k
      • sp.q
      • trade.q

By default, kdb+/q looks for a license file in QHOME. To keep your license file elsewhere, set its path in the environment variable QLIC.

Tips and tricks

Running 32-bit kdb+ on 64-bit Linux

Use the

uname -m

command to determine whether your machine is using the 32-bit or 64-bit Linux distribution.

If the result is

  • i686 or i386 or similar, you are running a 32-bit Linux distribution;
  • x86_64, you are running a 64-bit Linux distribution.

To install 32-bit kdb+ on a 64-bit Linux distribution, you need a 32-bit library. Use your usual package manager to install i686 or i386, for example:

sudo apt-get install libc6-i386

For more details on how to run 32-bit programs on Ubuntu 64-bit, see https://askubuntu.com/questions/454253/how-to-run-32-bit-app-in-ubuntu-64-bit.

rlwrap

On Linux and macOS the rlwrap command allows the Up arrow to retrieve earlier expressions in the q session. It is very useful and we recommend installing it.

Ask for its version number:

rlwrap -v

If you see one, rlwrap is already installed. Otherwise, you will see

rlwrap: command not found.

Install rlwrap using your package manager. (Common package managers: apt, dnf, and yum for Linux; Homebrew and MacPorts for macOS.)

It is straightforward to "wrap" the q process into rlwrap:

rlwrap ~/q/l64/q

taskset

If your machine has many cores, you may encounter the following error:

'licence error: cores

This means you are trying to run kdb+/q on a machine with two many cores. The number of licensed cores appears in parentheses in the message that appears when you launch kdb+/q. For example:

KDB+ 4.0 2020.05.04 Copyright (C) 1993-2020 Kx Systems
l64/ 72(16)core 1029691MB paul ...

This means that kdb+/q is running on a machine with 72 cores, but only 16 of them are covered by the license.

In this situation you have to restrict the cores visible to the kdb+/q process using taskset. According to the documentation,

taskset is used to set or retrieve CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs [or CPU cores] on the system. The Linux scheduler will honour the given CPU affinity and the process will not run on any other CPUs.

For example, we could run the q process as follows:

rlwrap taskset -c 0,1,2,3,4,5,6,7,8,9,10,11 ~/q/l64/q

In our example, this will force kdb+/q to use only the cores from 0 to 11 (inclusive) and will resolve the "cores" error.

Confirming success

Confirm that kdb+/q is working: launch your first q session. On Windows:

cd C:\q\w32
q.exe

On Linux:

cd ~/q
./q

The q session opens with a banner like this:

KDB+ 3.6 2019.04.02 Copyright (C) 1993-2019 Kx Systems
w32/ 8()core 4095MB paul desktop-efq7tt8 192.168.1.105 NONEXPIRE

Try your first expression:

q)3+5
8

End the q session and return to the command shell.

Integrated development environments (IDEs)

There is no one-size-fits-all IDE for kdb+/q. Therefore we recommend that the reader download some/all of the tools listed above and give them a try.

Q Insight Pad (Windows-only)

Q Insight Pad (sometimes referred to simply as qPad) by Oleg G. Zakharov is a kdb+ client, written in C++ to run under Windows; as a result, it is fast, memory-efficient and easy to use. The author makes a special emphasis on providing a Windows-style GUI that is visual, simple and intuitive.

Q Insight Pad provides simple but useful visualization capabilities.

An unregistered but fully functional version (working with free versions of q) can be downloaded for free from http://www.qinsightpad.com/.

This version expires every 3 months. If you want to license Q Insight Pad for use in your organization, contact the author: http://www.qinsightpad.com/contact.html.

Connecting Q Insight Pad to kdb+/q

Start a kdb+/q session. Open a port, for example, 41822:

q)\p 41822

You can now connect to this port from Q Insight Pad.

Right-click on "KDB+ Servers". Click on "Add...". Give the server a descriptive name, e.g. "Local". Set the Server to localhost and the port to 41822.

Click on "Test". You should see "Successfully connected to `:localhost:41822::.

Click "OK".

Double-click on the new server, which should now appear under "KDB+ Servers".

In the text editor, enter an expression that you would like to evaluate, e.g.

3+5

Select that expression and press Ctrl + E. The result

8j

should appear under "Output".

qStudio

qStudio by TimeStored is an intelligent code editor for kdb+/q that provides:

  • q syntax highlighting;
  • q code completion;
  • ability to query servers direct from the editor;
  • server browser for viewing server objects;
  • charts for displaying your data;
  • database management tools to modify your database structure;
  • a data loader for importing large data files easily;
  • language programming tools to increase programmer productivity:
    • qDoc for generating HTML documentation;
    • qUnit — a unit testing framework for kdb.

qStudio can be downloaded from http://www.timestored.com/qstudio/.

qStudio Pro can be purchased from http://www.timestored.com/qstudio/buy.

This application requires a Java Runtime Environment 1.8.0_66 or above.

Studio for kdb+

Studio for kdb+ is a rapid development environment for kdb+. In the style of commonly used SQL clients, it allows you to

  • connect to kdb+ processes;
  • execute selected text from the editor window;
  • view results as tables, charts, or classic console style.

The editor component is based on the NetBeans editor component, and includes the following features:

  • find/search/replace;
  • cut/copy/paste;
  • undo/redo;
  • syntax highlighting for the q language;
  • most recent files menu.

Additionally, the application features:

  • export to Excel;
  • drag and drop;
  • immediate charting of grid data.

Studio for kdb+ is free, provided under the Apache 2 license, and can be downloaded from https://github.com/CharlesSkelton/studio.

The program is portable and comes in the form of a JAR file. You need the Java runtime environment (JRE) to run it:

java -jar studio.jar

KX Developer

KX Developer is a visual environment used to manage, manipulate, and explore massive datasets in real-time by exploiting kdb+'s server-based analytics technology.

The Developer is comprised of the following components:

  • Workspace — local and shared repository for managing entities;
  • Importer — import datasets into the workspace;
  • Exporter — export datasets into other systems;
  • Visual Inspector — visualize massive datasets in real-time or near-real-time;
  • Interactive editor — write applications using the q programming language;
  • Debugger — debugger dramatically increases productivity;
  • Code Differ — compare multiple versions of the functions or modules.

Contained within the KX Developer package is a set of highly useful q developer libraries and scripts for use outside of the KX Developer UI and for automated build systems. The libraries contained provide support for:

  • Code coverage;
  • Data visualization;
  • Documentation generation from source annotations;
  • Property-based testing;
  • Q static linting and syntax checker;
  • Regular expressions;
  • Run time and memory profiling;
  • Unit testing;
  • KX Developer module to q script conversion.

KX Developer can be downloaded for free from https://code.kx.com/developer/.

Note that KX Developer requires a 64-bit version of kdb+/q.