Skip to content

Commit bf5a32f

Browse files
dwalshqdwalshq
committed
1.1.0 release
1 parent ae78e06 commit bf5a32f

7 files changed

Lines changed: 346 additions & 82 deletions

File tree

‎docs/install.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,11 @@
55
* Logging: The module generates status and error logs. By default, it utilizes the KX Logging framework. If a custom logger is not provided via the configuration parameters, ensure the [KX logging](https://code.kx.com/kdb-x/modules/logging/overview.html) is installed and available in your [QPATH](https://code.kx.com/kdb-x/modules/module-framework/quickstart.html#search-path). You can install `logging` and `printf` by
66

77
```bash
8-
LATEST=`curl -s https://api.github.com/repos/KxSystems/logging/releases/latest | grep 'tag_name' | cut -d '"' -f 4` \
8+
LATEST=$(curl -s https://api.github.com/repos/KxSystems/logging/releases/latest | grep 'tag_name' | cut -d '"' -f 4) \
99
curl -L https://github.com/KxSystems/logging/archive/refs/tags/$LATEST.zip -o logging.zip && \
1010
unzip -j logging.zip "logging-$LATEST/log.q" -d $HOME/.kx/mod/kx/ && \
1111
rm logging.zip
12-
LATEST=`curl -s https://api.github.com/repos/KxSystems/printf/releases/latest | grep 'tag_name' | cut -d '"' -f 4` \
12+
LATEST=$(curl -s https://api.github.com/repos/KxSystems/printf/releases/latest | grep 'tag_name' | cut -d '"' -f 4) \
1313
curl -L https://github.com/KxSystems/printf/archive/refs/tags/$LATEST.zip -o printf.zip && \
1414
unzip -j printf.zip "printf-$LATEST/printf.q" -d $HOME/.kx/mod/kx/ && \
1515
rm printf.zip

‎docs/reference.md‎

Lines changed: 68 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# NYSE TAQ Data Loader module
22

3-
This module provides high-performance utilities for parsing [NYSE TAQ (Trade and Quote) PSV files](https://ftp.nyse.com/Historical%20Data%20Samples/DAILY%20TAQ/), performing data transformations, and persisting the results into a date-partitioned kdb+ database (aka. HDB).
3+
This module provides high-performance utilities for parsing [NYSE TAQ (Trade and Quote) PSV files](https://ftp.nyse.com/Historical%20Data%20Samples/DAILY%20TAQ/), performing data transformations, and loading the results either into a date-partitioned kdb+ database (HDB) or directly into in-memory tables (RDB).
44

55
## Prerequisites: Input PSV Files
66

@@ -30,7 +30,7 @@ SIZE=small ./scripts/getCSVs.sh /tmp/nysetaqpsv "$DATES"
3030

3131
### Dataset Statistics (Reference: 2025.07.01)
3232

33-
The following table provides an estimate of the data footprint based on the `SIZE` parameter:
33+
The following table estimates the data footprint by `SIZE` parameter:
3434

3535
| `SIZE` | Symbol Range (First Letter) | Uncompressed PSVs Size | Uncompressed HDB Size | Symbol Nr | Quote Nr |
3636
| --- | ---: | ---: | ---: | ---: | ---: |
@@ -41,12 +41,19 @@ The following table provides an estimate of the data footprint based on the `SIZ
4141

4242
## Quickstart
4343

44-
The primary interface for this module is the `parseToDisk` function, which requires at least three parameters.
44+
This module exposes two primary functions:
45+
46+
- **`parseToDisk`** — parses TAQ data and persists it into a date-partitioned HDB on disk.
47+
- **`parseToMemory`** — parses TAQ data and loads it directly into in-memory tables, suitable for RDB-style workflows.
4548

4649
```q
47-
([parseToDisk]): use `kx.taq
50+
([parseToMemory; parseToDisk]): use `kx.taq
4851
```
4952

53+
### parseToDisk
54+
55+
`parseToDisk` requires at least three parameters.
56+
5057
To create the `trade`, `quote`, `master` tables and `exnames` dictionary for October 2, 2025, and save them to `/tmp/kdbdb`:
5158

5259
```q
@@ -82,19 +89,68 @@ AMZN 0D04:00:00.010640417 219.98 100 219.41 219.98
8289
..
8390
```
8491

92+
### parseToMemory
93+
94+
`parseToMemory` requires at least two parameters — no destination path is needed as data is loaded directly into memory.
95+
96+
To load the `trade`, `quote`, `master` tables and `exnames` dictionary for October 2, 2025, into memory:
97+
98+
```q
99+
(trade; quote; master; exnames): parseToMemory["/tmp/nysetaqpsv"; 2025.10.02]
100+
```
101+
102+
The resulting `trade` and `quote` tables are sorted by `time` and carry a grouped attribute on `sym`, matching the layout of a typical RDB. Unlike the HDB tables produced by `parseToDisk`, in-memory tables do not include a `date` column.
103+
104+
```q
105+
/ Perform an as-of join (aj) between the in-memory trades and quotes
106+
q)aj[`sym`time; select sym, time, price, size from trade where sym in `MSFT`GOOG`AMZN; select sym, time, bid, ask from quote]
107+
sym time price size bid ask
108+
---------------------------------------------------
109+
AMZN 0D04:00:00.009709706 219.5 3 219.41 219.98
110+
AMZN 0D04:00:00.010213563 219.7 3 219.41 219.98
111+
AMZN 0D04:00:00.010379075 219.7 2 219.41 219.98
112+
AMZN 0D04:00:00.010640417 219.98 100 219.41 219.98
113+
...
114+
```
115+
116+
Storing a full day of NYSE TAQ data in memory is RAM-intensive. The table below shows approximate memory requirements by `SIZE` parameter, measured against data from 2025.10.02.
117+
118+
| `SIZE` | Symbol Range (First Letter) | Memory need |
119+
| --- | ---: | ---: |
120+
| `small` | Z | ~2 GB |
121+
| `medium` | I | ~14 GB |
122+
| `large` | A-H | ~79 GB |
123+
| `full` | A-Z | ~170 GB |
124+
85125
## Configuration Options
86126

87-
You can pass a dictionary as the fourth argument to `parseToDisk` to customize the ingestion process.
127+
Both `parseToDisk` and `parseToMemory` accept an optional dictionary as their last argument to customize the ingestion process.
128+
129+
### Common parameters
130+
131+
| Key | Default | Description |
132+
| --- | ---: | --- |
133+
| `letters` | `"A-Z"` | Restricts ingestion to symbols whose first letter falls within the specified range (e.g., `"K-N"`). |
134+
| `includetestsymbols` | `0b` | If `1b`, includes instruments flagged as test symbols in the `master` PSV. |
135+
| `batchsize` | `10 000 000` | Number of rows processed per chunk. Set to `0` to read the entire file in one pass for faster throughput if RAM permits. |
136+
| `logger` | logger created by `.logger.createLog[]` of the [KX log module](https://code.kx.com/kdb-x/modules/logging/overview.html) | Logger used for status updates during the ingestion process. |
137+
138+
### parseToDisk extra parameters
139+
140+
| Key | Default | Description |
141+
| --- | ---: | --- |
142+
| `compparam` | `([master: 0 0 0; trade: 0 0 0; quote: 0 0 0])`, i.e. no compression | Table-specific compression settings for [.z.zd](https://code.kx.com/q/ref/dotz/#zzd-compressionencryption-defaults). Example: `([master: 0 0 0; trade: 17 2 6; quote: 17 2 6])`. Pass a dictionary of dictionaries to specify column-level compression. |
143+
| `linked` | `0b` | Set `1b` to add a linked column `master` to the `trade` and `quote` tables, linking via `sym` to the `master` table. |
144+
145+
### parseToMemory extra parameters
88146

89147
| Key | Default | Description |
90148
| --- | ---: | --- |
91-
| `letters` | "A-Z" | Restricts ingestion to symbols starting with specific letters (e.g., "K-N") |
92-
| `includetestsymbols` | 0b | If `1b`, includes instruments flagged as test symbols in the `master` PSV. |
93-
| `batchsize`| 10 000 000 | Number of rows processed in memory per chunk. Set to `0` to read the entire file at once for faster processing if RAM permits. |
94-
| `compparam` | `3#0`, i.e. no compression | Compression settings for [.z.zd](https://code.kx.com/q/ref/dotz/#zzd-compressionencryption-defaults). Example: `(17;2;6)` for logical block size, algorithm, and level. You can also pass a dictionary if you would like to specify column-level compression |
95-
| `logger` | logger created by `.logger.createLog[]` of the [KX log module](https://code.kx.com/kdb-x/modules/logging/overview.html) | A logger used for status updates during the long running process |
149+
| `grouped` | `1b` | If `1b`, applies the grouped attribute to the `sym` column in the `trade` and `quote` tables. |
150+
| `sortbytime` | `1b` | If `1b`, sorts `trade` and `quote` by `time` and applies the sorted attribute. |
151+
96152

97153
## Performance Notes
98154

99-
* **Multithreading**: The PSV parsing engine leverages multithreading. Ensure you start your ingestion process with the `-s` flag (e.g., `q -s 8`) to utilize available CPU cores.
100-
* **Memory Management**: If you encounter memory pressure, reduce the `batchSize` in the options dictionary. Conversely, increasing it (or setting it to `0`) will speed up the process.
155+
* **Multithreading**: The PSV parsing engine is multithreaded. Start your ingestion process with the `-s` flag (e.g., `q -s 8`) to make use of available CPU cores.
156+
* **Memory Management**: If you encounter memory pressure, reduce `batchsize` in the options dictionary. Conversely, increasing it (or setting it to `0`) will speed up the process.

‎docs/release-notes.md‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,24 @@
22

33
_This document provides the version history of the KDB-X Taq Module, detailing released versions, fixes, and improvements._
44

5+
## 1.1.0
6+
7+
**Release Date**: 2026-03-19
8+
9+
### Fixes and Improvements
10+
11+
- New function (`parseToMemory`) to parse data into memory.
12+
- New `parseToDisk` option `linked` to create linked columns to master.
13+
- **NUC**: Values of `exnames` are now strings instead of symbols.
14+
- **NUC**: `compparam` requires table-specific compression parameters.
15+
516
## 1.0.1
617

718
**Release Date**: 2026-03-12
819

920
### Fixes and Improvements
1021

11-
- Quote column name fix: Bsize-> bsize
22+
- Quote column name fix: `Bsize` -> `bsize`.
1223
- Improved input parameter validation.
1324
- Documentation fixes.
1425

@@ -24,7 +35,7 @@ The Taq module is designed for benchmarking and is based on the [KX Taq scripts]
2435
- Introduced a new parameter to filter by the first letter of the Symbol.
2536
- Enhanced error handling.
2637
- Improved code quality.
27-
- Added an option to drop test Symbols.
38+
- Added an option to drop test symbols.
2839
- Columns are written in parallel for better performance.
2940
- Batch processing support for a smaller memory footprint.
3041
- Function documentation now uses qdoc syntax.

‎taq/converters.q‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,4 +19,15 @@ masterTestSymbolFilter: ?[;enlist (not; `testFlag);0b;()]
1919
// @fileoverview conversion function to filter out based on the first letter of the sym column
2020
// @param l string in format like "LM" representing the range of the first letter of the sym column values to be included
2121
// @param t table with a sym column used for the filtering
22-
letterFilter: {[l;t] select from t where sym[;0] within l}
22+
letterFilter: {[l;t] select from t where sym[;0] within l}
23+
24+
// @kind function
25+
// @fileoverview adds linked column master to sym column of the master table
26+
// @param dst destination directory
27+
// @param date partition
28+
// @param t table that needs to be extended with a linked master column
29+
//
30+
// @return the updated table
31+
addLinkedColToMaster: {[dst:`s; date:`d; t]
32+
update master:`master!get[.Q.dd[.Q.par[dst;date;`master];`sym]]?sym from t
33+
}

0 commit comments

Comments
 (0)