Jd is Jsoftware's high-performance columnar RDBMS, written in C with a deep J integration. Non-commercial use is free; a non-commercial key is auto-installed on first run.
This entry uses Jd's native reads query language (SQL-ish but with J
syntax) rather than translating to ANSI SQL — Jd accepts SQL keywords
in a different order (reads <select> from <table> where <where> order by <order>) and uses by inside reads for GROUP BY. The
queries.sql file holds J expressions that wrap jd 'reads …' calls
plus J operators for things Jd's query layer doesn't ship (LIMIT,
DISTINCT).
./install:
- Clones
jsoftware/jsource@build96and usesjlibrary/as the J installation root, then overlays the platform-specific binary (jconsole,libj.so,libtsdll.so,libgmp.so) from the same tag's release zip (l64.zipon x86_64,rpi64.zipon aarch64). The release zip ships binaries only and won't run withoutjlibrary/'s standard library. - Installs a small
/usr/local/bin/ijconsolewrapper that re-execs the realjconsoleunderfaketime '2026-05-10 00:00:00'. Why: Jd's bundledjdkey.txtis an evaluation key Jsoftware refreshes periodically, and the copy injsoftware/data_jdexpired 2026-05-16. Until upstream pushes a new key (tracked in the data_jd repo asjdkey.txt), everyjconsoleinvocation needs to see a date before the expiry orjdlicensereturns_2("eval key") andjd.ijs:147asserts out. Backdating with faketime is the cheapest workaround that keeps the rest of Jd intact. - Uses J's package manager (
pacman/jpkg) to install thedata/jdaddon and its J-side dependencies (api/curl,ide/jhs,arc/lz4,general/misc,data/jfiles,data/jmf,net/jcs,net/socket,web/gethttp,convert/json,convert/pjson).
./load ingests hits.csv via Jd's CSV loader with an explicit
column schema instead of csvload_jd_'s auto-inference. The
default flow types every string column by sampling the first 5000
rows and then runs csvscan to widen any column it inferred as
byte to the full-file max width. ClickBench has very sparse text
columns (e.g. OpenstatServiceName, SocialNetwork) that look
empty in the 5000-row sample → typed as byte, then later widened
to hundreds of chars × 100 M rows. With ~30 such columns the
splayed table grew past 500 GB during the load and segfaulted.
Declaring text columns as varbyte (variable-length, per-row
offset + concatenated data) keeps storage proportional to actual
string content. The script writes a hand-rolled hits.cdefs file
into the csvload jdcsv folder, then calls csvrd directly,
skipping csvcdefs (auto-type) and csvscan (byte-width
widening).
Schema choices:
int(8-byte signed) for every numeric column. Jd'sint1/int2/int4leave per-row data asn,xchar matrices, and the<>predicate then sees a shape-2 column vs a shape-0 scalar, so we use the flat 8-byte JINT form everywhere.varbytefor TEXT / VARCHAR / CHAR.edateforEventDate,edatetimefor the three TIMESTAMP columns. Both are 8-byte epoch-nanos and Jd's csv loader parses iso8601 fromiso8601-charmode (CSV format isYYYY-MM-DD/YYYY-MM-DD HH:MM:SS).
The loader writes per-column files to a dedicated database under
~/j9.6-user/temp/jd/csvload/; that's the database ./query
opens.
./query reads a J expression from stdin and evaluates it via
ijconsole query.ijs. The query.ijs script opens the csvload
database, times the eval, prints the result to stdout, and emits
the runtime in fractional seconds to file id 5 (stderr).
The translations stay close to the SQL semantics but diverge in a few places:
LIMIT nisn't areadskeyword — we use J'sn {.after the query (e.g.10 {. jd '...').LIMIT n OFFSET musesn {. m }. jd '...'.COUNT(DISTINCT col)uses J's# ~.(count of unique items) after pulling the column withjd 'reads col from t'.min/avgonvarbyte: Jd's aggregators are numeric-only, so Q23'sMIN(URL)/MIN(Title)becomefirst URL/first Title(any value from each group, semanticallyANY_VALUE).- Q28 (
AVG(LENGTH(URL))), Q29 (REGEXP_REPLACE), and Q43 (DATE_TRUNC('minute', ...)) use facilities not in Jd'sreadslanguage; they currently return the literal'null'and the benchmark driver records them as missing. They could be expressed with a J-side computed column — contributions welcome. order byrequires the column inselect: Jd's parser rejectsreads SearchPhrase from hits order by EventTimebecause the order key isn't projected. Q25 / Q27 are rewritten to projectEventTime,SearchPhrase(timing unaffected; only the printed output has one extra column).COUNT(DISTINCT col): outsidereads, J's# ~. ; }. jd '…'(count of unique, after dropping the header row). The}.drops the header box so the unique scan only sees the data values.
J / Jd is single-threaded by default. Jd's columnar layout makes
single-column scans fast; cross-column where-then-aggregate paths
are also vectorised in the C core. There is no daemon — each query
call cold-starts ijconsole, loads the database (mostly memory-mapped
columns), and runs.