All projects

Project case study

LSMSharp

Inside an embedded storage engine

A C# key-value storage engine built around a write-ahead log, sorted memory buffers, and immutable disk tables.

  • C#
  • .NET 8
  • LSM-tree
  • Storage
GitHub Related writing
EvidenceFunctional checks cover updates, deletes, binary values, and concurrent access.

The problem

A storage engine has to accept new writes while keeping old data readable and recoverable. LSMSharp makes those competing jobs explicit: collect writes in memory, preserve a recovery log, then reorganize sorted files in the background.

How it fits together

  1. 01

    Write-ahead log

    Queue an append-only record for recovery.

  2. 02

    Memtable

    Keep recent keys sorted in a concurrent skip list.

  3. 03

    SSTables

    Flush an immutable table into level zero.

  4. 04

    Compaction

    Merge tables and reconcile versions across levels.

Reads check the active and flushing memtables first, then search disk tables. Bloom filters reject definite misses before a block is read and decompressed.

Decisions & tradeoffs

Batch persistence work

The WAL groups writes around a 100-entry threshold and a 100 ms timer. This reduces per-write I/O, but a completed SetAsync call can precede persistence. Flush coordinates WAL synchronization and table creation; batching is a throughput–durability tradeoff.

Separate ingestion from reorganization

A memtable becomes read-only during a flush while a new one accepts writes. Readers can still inspect the flushing table. Background compaction keeps the disk layout manageable, at the cost of additional reads and writes.

Spend memory to avoid disk work

Bloom filters, block caching, and cached SSTable handles reduce unnecessary I/O. A Bloom-filter match still needs a real lookup: the filter can return false positives, so it is an optimization rather than the source of truth.

Evidence & validation

The repository includes functional checks for CRUD operations, repeated updates, deletes, binary values, and concurrent access, alongside performance and reopen workloads. A checked-in test run records a post-compaction integrity failure, so fresh correctness checks should precede performance comparisons.

The published sequential-write example reports 5,000 operations in 207 ms, or about 24,155 operations per second. It is a repository-reported sample: the timer covers write submission and excludes the final flush. Hardware details were not recorded, so this is not a durable-commit benchmark or a cross-machine comparison.

For a meaningful rerun, record the runtime, hardware, value sizes, cache state, and flush policy; verify retrieved values as well as throughput. Multi-operation transactions and snapshot isolation remain separate design work.

Reproduce the workload

From a checkout of the repository, run the functional suite before comparing performance. The read/write samples use 5,000 operations and fixed random-key seeds; the sequential-write sample uses 256-byte values.

cd Tests
dotnet run -c Release -- functional
dotnet run -c Release -- performance

Related writing