go-bitcask

Bitcask (Go Implementation)

A clean, from-scratch implementation of the Bitcask key–value storage engine in Go, inspired by the original Basho (Riak) design.

What is Bitcask?

Bitcask is a log-structured, append-only key–value storage engine optimized for:

• Fast writes — sequential, append-only I/O
• Fast reads — O(1) lookups via an in-memory index
• Crash recovery — deterministic rebuild from disk
• Operational simplicity — minimal on-disk structures

Core Design Principles

• Append-only datafiles (no in-place updates)
• In-memory KeyDir mapping keys → file offsets
• Immutable datafiles once rotated
• Background merge & compaction
• Atomic file replacement

What This Project Contains

Storage Engine

• Append-only Bitcask datafiles
• In-memory KeyDir
• File rotation and disk sync
• Background merge & compaction
• Hint files for fast startup

Client Library

A Go client for programmatic access:

• Simple API: GET, SET, DELETE, EXISTS, COUNT, LIST
• Binary protocol
• Designed for correctness and testability

CLI Tool

• Interactive REPL
• Full command coverage
• Useful for debugging and exploration

Quick Start

Prerequisites

• Go ≥ 1.23
• Make (optional)

Build

git clone https://github.com/0xRadioAc7iv/go-bitcask.git
cd go-bitcask
make build

This produces:

• bin/bitcask — server
• bin/bk-cli — CLI client

Running the Server

./bin/bitcask

Custom Configuration

./bin/bitcask \
  --dir ./data \
  --df-size 128 \
  --port 9999 \
  --sync 15 \
  --size-check 5 \
  --dead-ratio 0.4

Configuration Options

Using the CLI

./bin/bk-cli --host 127.0.0.1 --port 9999

Example session:

> SET foo bar
ok

> GET foo
bar

> COUNT
1

> DELETE foo
ok

Using the Client Library

go get github.com/0xRadioAc7iv/go-bitcask/bitcask

client, _ := bitcask.Connect()
defer client.Close()

client.SET("user:1", "Alice")
v, _ := client.GET("user:1")
fmt.Println(v)

How It Works (High-Level)

Write Path

1. Append record to active datafile
2. Update KeyDir in memory
3. fsync periodically

Read Path

1. Lookup key in KeyDir
2. Read record at offset
3. Validate CRC
4. Return value

Merge & Compaction

• Runs periodically
• Selects files with high garbage ratio
• Rewrites only live records
• Atomically replaces old files

Crash Recovery

• Rebuilds KeyDir from hint files (or datafiles)
• Truncates corrupted tails
• Resumes safely

Development

make build
make test
make fmt

A data generation script is provided for stress-testing merge and compaction:

make datagen

References

• Bitcask: A Log-Structured Hash Table for Fast Key/Value Data - Original PDF
• Bitcask - A Log-Structured fast KV store - Arpit Bhayani's Blog

NOTE: This is a learning-focused implementation, not production software.