sqlite-zstd is an extension to SQLite that provides transparent dictionary-based row-level compression for SQLite. This allows you to compress entries in a SQLite database as if you were compressing the entire database file, but at the same time preserving random access.

Depending on the data, this can reduce the size of the database by up to 80% while keeping most of the performance the same (and even improve performance because the data read from disk is smaller).

The project can be built in two modes:

• as a Rust library
• as a pure SQLite extension

The SQLite extension binaries are available from the GitHub release, or the extension can be built manually:

cargo build --release --features build_extension
# should give you target/release/libsqlite_zstd.so

This library can be loaded as an SQLite extension or as a Rust library. Note that the sqlite extension is not persistent, so it needs to be loaded every time you connect to the database.

Sqlite CLI

Load it in the REPL:

$ sqlite3 file.db
SQLite version 3.34.0 2020-12-01 16:14:00
sqlite> .load .../libsqlite_zstd.so
[2020-12-23T21:30:02Z INFO  sqlite_zstd::create_extension] [sqlite-zstd] initialized
sqlite>

or:

sqlite3 -cmd '.load libsqlite_zstd.so' 'select * from foo'

C API

int success = sqlite3_load_extension(db, "libsqlite_zstd.so", NULL, NULL);

Rust

The recommended way is to add sqlite_zstdas a dependency of your project, then use

let conn: rusqlite::Connection;
sqlite_zstd::load(&conn)?;

Alternatively, the extension can also be loaded like any other extension:

let conn: rusqlite::Connection;
conn.load_extension("libsqlite_zstd.so", None)?;