ThebeDB — System Documentation

🗄️ What is ThebeDB?

ThebeDB is a purpose-built database system for blockchain applications, developed by JupiterMetaLabs. It solves a problem that most blockchain systems face: you need two completely different databases at the same time — one to store data permanently and tamper-proof (like a ledger), and another to make that data easy to search and analyse (like a regular database). Most systems bolt these two things together awkwardly. ThebeDB handles both in a single, unified package.

Think of it like a bank's bookkeeping system: transactions are written into a permanent, append-only ledger that can never be altered, but a separate set of indexed records lets accountants run reports, look up accounts, and answer questions quickly. ThebeDB provides exactly this — writes go to an immutable log, reads come from a smart SQL mirror that keeps itself up to date automatically.

✨ What It Does

Immutable Ledger

Every piece of data written to ThebeDB is stored permanently in an append-only log. Nothing can be edited or deleted — exactly what blockchain integrity requires.

SQL Query Layer

A background process automatically mirrors the ledger into a structured SQL database (SQLite or PostgreSQL), so you can run complex queries, joins, and analytics without touching the immutable store.

Self-Healing Projections

If the SQL mirror is ever deleted or corrupted, ThebeDB can rebuild the entire queryable state from scratch by replaying the immutable log. The source of truth is always the ledger.

Blockchain Explorer API

A built-in REST API lets any browser or application browse blocks, look up transactions, and inspect account activity — all in real time.

Backpressure Control

If the system falls behind processing new data, ThebeDB automatically slows down incoming writes rather than crashing — keeping the system stable under load.

Pluggable Profiles

The system supports swappable "profiles" — modules that define how raw blockchain data is interpreted and stored in SQL. This makes it adaptable to different chain formats and business rules.

⚙️ How It Works (Plain English)

Here is the lifecycle of data through ThebeDB, step by step:

Data Arrives

A blockchain node sends a new block (or transaction) to ThebeDB. This could be via a direct Go function call or through the system's API.

Backpressure Check

Before accepting the write, ThebeDB checks whether the query mirror is keeping up. If it's too far behind, the write is rejected with a "system overload" signal so the caller knows to slow down and retry.

Written to the Immutable Log

If all is well, the data is serialized and appended to the KV ledger (BadgerDB) with a unique sequence number. This is extremely fast — BadgerDB is optimized for high-throughput writes. The data is now permanently recorded.

The Projector Wakes Up

A background worker (the Projector) continuously polls the ledger for new entries. Every 100 ms it checks whether new records have arrived since it last ran.

Data is Transformed into SQL

The Projector reads the new record, passes it through the active "profile" (the jmdt_explorer profile for JupiterMetaDN), which parses the raw bytes into structured blocks, transactions, and participants, then inserts them into SQL tables inside a single database transaction.

Available for Queries

Within milliseconds, the new block is queryable via the Explorer REST API. A web dashboard (React) and any external client can now look it up by block number, hash, transaction hash, or account address.

🔗 What It Connects To

💎
BadgerDB Embedded key-value store that serves as the immutable ledger. Runs inside the same process — no separate server needed.
🗃️
SQLite (default) or PostgreSQL The queryable SQL mirror. SQLite runs in the same process; PostgreSQL is available for production deployments needing multi-client access.
🌐
Explorer Web UI A React-based frontend (built with Vite) served directly by ThebeDB at port 8080. Lets users browse blocks, transactions, and accounts in a browser.
📊
Prometheus (optional) Docker Compose includes a Prometheus instance for collecting metrics and monitoring system health in production.
📡
OpenTelemetry Distributed tracing and logging infrastructure wired in via the ion logging library and OTLP exporters, for observability in complex deployments.

👥 Who Should Care About What

Role	What matters to you
Product / Business	ThebeDB provides a tamper-proof audit trail for all blockchain activity, plus a live explorer interface for ops teams and end users. Data is never lost — even if the query database is wiped, everything can be recovered from the ledger.
Blockchain Engineers	You interact with the `db.Append(record)` API to submit blocks and transactions. Watch for `ErrSystemOverload` and implement exponential backoff. The CQRS separation means writes never block on query performance.
Backend / Platform Engineers	The Go library exposes clean interfaces for the KV store, SQL engine, and profile system. Profiles are the main extension point — you can add new ones to support different chain namespaces without touching core code.
DevOps / Infrastructure	A `docker-compose.yml` provides PostgreSQL and Prometheus. The binary is configured via a simple YAML file. For production, switch the SQL driver from SQLite to PostgreSQL for concurrent read access.
Frontend / UI	Five REST endpoints under `/api/` expose blocks, transactions, and accounts as JSON. All responses are CORS-enabled. The React web UI in `/web` is a working reference implementation you can extend.

🛠️ Technology Stack

Language

Go 1.25

Standard library + modules

KV Store

BadgerDB v4

github.com/dgraph-io/badger/v4

SQL (default)

SQLite (WAL mode)

github.com/mattn/go-sqlite3

SQL (production)

PostgreSQL 16

github.com/lib/pq

Serialization

Protocol Buffers

google.golang.org/protobuf

Logging

Zap + ion

go.uber.org/zap · JupiterMetaLabs/ion

Observability

OpenTelemetry

OTLP traces + logs

Cache

Ristretto v2

github.com/dgraph-io/ristretto/v2

Web Frontend

React + Vite

JSX components, served from /web/dist

Config

YAML

gopkg.in/yaml.v3

Monitoring

Prometheus

Docker Compose sidecar

Pattern

CQRS

Command/Query Responsibility Segregation

🏗️ Architecture Overview

ThebeDB implements CQRS in a single process. The write path (Ingestor → KV Log) is completely decoupled from the read path (Projector → SQL). All five major subsystems start together under the unified thebedb binary.

System Architecture

graph LR Client["🔗 Blockchain Client\n(Go SDK / External)"] WebUI["🌐 Browser\n(React UI)"] API["🔌 Explorer REST API\n:8080/api/*"] BP["⚖️ Backpressure\nMonitor"] KV["📦 BadgerDB\nImmutable KV Log"] Proj["⚙️ Projector\n(Background Worker)"] PReg["🧩 Profile Registry\n(jmdt_explorer)"] SQL["🗃️ SQL Engine\nSQLite / PostgreSQL"] Client -- "db.Append(record)" --> BP BP -- "Check lag OK" --> KV BP -. "ErrSystemOverload\n(lag too high)" .-> Client KV -- "Iterate(seq)" --> Proj Proj -- "dispatch(record)" --> PReg PReg -- "Apply → SQL Tx" --> SQL API -- "GET /api/blocks" --> SQL WebUI -- "HTTP JSON" --> API style KV fill:#fef3c7,stroke:#f59e0b style SQL fill:#dcfce7,stroke:#16a34a style BP fill:#ffe4e6,stroke:#e11d48 style Proj fill:#e0e7ff,stroke:#6366f1

Key insight: The SQL database is a derived view — it can always be rebuilt from the KV log. The KV store is the single source of truth and is never modified after a write. Profiles are the only user-configurable processing logic.

🔄 Data Flow Diagrams

Write Path — Appending a Block

sequenceDiagram participant C as Blockchain Client participant BP as BackpressureMonitor participant KV as BadgerDB (KV Log) participant P as Projector (goroutine) participant Prof as jmdt Profile participant SQL as SQL Engine C->>BP: Check() — is lag within limits? alt lag > HardLimit (5000) BP-->>C: ErrSystemOverload else lag OK BP-->>C: nil (proceed) C->>KV: Append(CanonicalRecord{namespace:"jmdt", type:"block", value:blockJSON}) KV-->>C: seq (monotonic uint64) Note over KV: Stored as __sys:log:<seq> loop Every 100ms polling P->>KV: Iterate(since=lastSeq) KV-->>P: records[] P->>BP: UpdateLogHead(seq) P->>Prof: Apply(ctx, record, sqlTx) Prof->>SQL: INSERT jmdt_blocks, jmdt_transactions, jmdt_participants P->>SQL: SetProjectionOffset(name, seq) SQL-->>P: commit OK P->>BP: UpdateProjectionOffset("jmdt_explorer", seq) end end

Read Path — Explorer API Query

sequenceDiagram participant B as Browser participant A as API Handler participant S as SQL Engine participant DB as SQLite / PostgreSQL B->>A: GET /api/blocks A->>S: GetBlocks(ctx, limit=50) S->>DB: SELECT * FROM jmdt_blocks ORDER BY height DESC LIMIT 50 DB-->>S: rows[] S-->>A: []map[string]interface{} A-->>B: 200 OK JSON array B->>A: GET /api/block/42 A->>S: GetBlockByNumber(ctx, 42) S->>DB: SELECT ... FROM jmdt_blocks WHERE height = 42 DB-->>S: row S-->>A: map[string]interface{} A-->>B: 200 OK JSON object

📁 Directory Structure

ThebeDB/ ├── cmd/ │ ├── thebedb/ │ │ └── main.go # Unified entry point: loads config, starts DB + HTTP server │ └── benchmark/ │ └── main.go # Standalone performance benchmark tool ├── internal/ │ ├── config/ │ │ └── config.go # YAML config loading + validation + defaults │ ├── core/ │ │ ├── types.go # Block, BlockHeader, Transaction domain types │ │ └── proto/ │ │ ├── canonical.proto # Protobuf schema for the immutable log record │ │ └── canonical.pb.go # Generated Go code │ ├── ingest/ │ │ └── backpressure.go # BackpressureMonitor: tracks log vs projection lag │ ├── profiles/ │ │ └── jmdt/ │ │ ├── profile.go # jmdt_explorer Profile: Apply() logic, SQL migrations │ │ └── queries.go # Address-level transaction cursor queries │ └── projector/ │ └── runner.go # Projection loop: polls KV log, dispatches to profiles ├── pkg/ │ ├── api/ │ │ └── handlers.go # REST handlers: status, blocks, block detail, tx, account │ ├── builder/ │ │ └── builder.go # Fluent builder for constructing CanonicalRecords │ ├── cache/ │ │ └── cache.go # Ristretto-based cache wrapper │ ├── kv/ │ │ ├── store.go # Store interface: Append, PutWorm, PutDerived, Get, Iterate │ │ ├── badger_store.go # BadgerDB implementation of Store │ │ └── schema.go # KV logical table schema helpers (CreateKVTable, KVKey) │ ├── profile/ │ │ ├── profile.go # Profile interface + Registry (thread-safe) │ │ └── template.go # Profile template helpers │ └── sql/ │ ├── store.go # SQL Store interface │ └── postgres.go # SQLEngine: InitSchema, migrations, offset tracking, queries ├── web/ │ └── src/ │ ├── App.jsx # Root React component │ └── components/ │ ├── Dashboard.jsx # Block list + stats view │ ├── BlockDetail.jsx # Block + transaction detail view │ └── Navbar.jsx # Navigation bar ├── thebedb.go # Public API: New(), Open(), Start(), Close(), Append() ├── config.yml # Active runtime config ├── config.example.yml # Annotated example config ├── docker-compose.yml # PostgreSQL + Prometheus sidecar services └── Makefile # build, run, test, clean targets

🧩 Key Modules

File	Role	Key Exports
thebedb.go	Public library API — the main entry point for embedding ThebeDB in Go projects	`New()`, `Open()`, `Start()`, `Append()`, `Close()`
pkg/kv/store.go	Defines the `Store` interface for the immutable KV log. All backends implement this.	`Store` interface, `NewStore()`, `ErrKeyNotFound`
pkg/kv/badger_store.go	BadgerDB implementation. Manages the `__sys:log:<seq>` namespace and hash chain.	`badgerStore` (private), implements `Store`
internal/projector/runner.go	Background worker loop. Polls KV log, dispatches to profiles, tracks projection offsets, handles quarantine.	`Runner`, `Start()`, `RunOnce()`, `Stop()`
pkg/profile/profile.go	Defines the `Profile` interface — the extension point for domain-specific projection logic.	`Profile` interface, `Registry`, `Register()`
internal/profiles/jmdt/profile.go	Concrete JupiterMetaDN profile. Parses blocks/txs, validates hash chain, writes to SQL tables.	`Profile`, `Apply()`, `GetMigration()`, `GetAddressTransactions()`
internal/ingest/backpressure.go	Monitors lag between the KV log head and the furthest-behind projection. Enforces soft/hard limits.	`BackpressureMonitor`, `Check()`, `ErrSystemOverload`
pkg/sql/postgres.go	SQLEngine: manages schema init, migrations, projection offsets, and all query methods.	`SQLEngine`, `GetBlocks()`, `GetProjectionOffset()`, `BeginTx()`
pkg/api/handlers.go	HTTP handlers for the Explorer REST API. All read from SQL; writes never go through the API.	`HandleBlocks()`, `HandleBlockDetail()`, `HandleTx()`, `HandleAccount()`, `CorsMiddleware()`

📐 Data Models

CanonicalRecord — the atomic unit of the immutable log (Protobuf):

// Every write becomes a CanonicalRecord in the KV log message CanonicalRecord { uint64 seq = 1; // Monotonic sequence number (assigned by Core) string namespace = 2; // e.g., "jmdt" string type = 3; // e.g., "block", "tx" bytes key = 4; // User-defined ID (e.g. "T:<hash>") — optional bytes value = 5; // Opaque payload (JSON-encoded Block or Tx) uint64 timestamp = 6; // Server ingest time (Unix nanoseconds) }

Go Domain Types (internal/core/types.go):

          type Block {
          Header BlockHeader
          Transactions []Transaction
          }

          type BlockHeader {
          Height uint64
          Hash []byte
          ParentHash []byte
          Timestamp int64
          }

          type Transaction {
          Hash []byte
          Nonce uint64
          Sender string
          To []byte
          Value []byte
          Data []byte
          }
        

SQL Tables (created by the jmdt_explorer profile migration):

Table	Primary Key	Purpose
jmdt_blocks	`hash BYTEA`	One row per block. Stores height, timestamp, and parent_hash.
jmdt_transactions	`hash BYTEA`	One row per transaction. Links to block_height + tx_index, stores sender and raw data.
jmdt_participants	`(tx_hash, address, role)`	Address-centric index. One row per address per transaction (as "sender" or "receiver"). Enables fast address lookup.
__sys_projections	`profile_name`	System table tracking the last-processed sequence number per profile. Enables resumable projection.
__sys_quarantine	`log_seq`	Stores records that failed to project after 3 retries, with failure reason. Prevents a single bad record from blocking the whole pipeline.

⚙️ Configuration & Environment

Configuration is loaded from a YAML file (default: config.yml). Pass a custom path with --config.

# config.yml
          kv:
          backend: badger # Only "badger" is
            currently supported
          path: ./data/badger # Directory for KV
            files

          sql:
          driver: sqlite # "sqlite" (dev) or
            "postgres" (prod)
          dsn: ./data/sqlite/default.sqlite

          # PostgreSQL example:
          # sql:
          # driver: postgres
          # dsn: postgres://thebe:thebe@localhost:5432/thebedb?sslmode=disable

          log:
          # ion logging config (level, format, output)
        

CLI Flags (passed to the binary):

Flag	Default	Description
--config	config.yml	Path to the YAML configuration file.
--port	8080	Port for the Explorer HTTP server.

Backpressure Thresholds (hardcoded in thebedb.go, configurable by modifying BackpressureConfig):

Parameter	Default	Description
SoftLimit	1000	Log warning when projection lag exceeds this number of records.
HardLimit	5000	Reject writes with `ErrSystemOverload` when lag exceeds this.
CriticalProjection	jmdt_explorer	The profile name whose offset is used for lag calculation.

🔌 REST API Endpoints

Method	Path	Description	Response
GET	/api/status	System health and last synced sequence number	`{"status": "ok", "last_seq": 123}`
GET	/api/blocks	Latest 50 blocks (ordered by height DESC)	JSON array of block objects
GET	/api/block/:id	Single block by height (integer) or hash (hex string)	JSON block object or 404
GET	/api/tx/:hash	Transaction receipt by hash	JSON receipt object or 404
GET	/api/account/:addr	Account information by address	JSON account object or 404

All endpoints are read-only. Writes happen exclusively through the Go db.Append() API. CORS is fully open (Access-Control-Allow-Origin: *). The static React UI is served from / with SPA fallback routing.

🔑 KV Key Namespace Schema

Key Pattern	Type	Description
__sys:log:<seq>	SYSTEM	The canonical append-only log. Each entry is a protobuf-encoded `CanonicalRecord`. Seq is zero-padded for lexicographic ordering.
__sys:hash:<seq>	SYSTEM	Hash chain entries for cryptographic integrity verification.
jmdt:chain_head	DERIVED	Latest block state: 8-byte height + 32-byte hash. Used by the jmdt profile for fast parent-hash validation without SQL queries.
tbl:<name>:__schema__	DERIVED	Logical KV table schema descriptor (JSON). Written once by `CreateKVTable()`.

The __sys:* namespace is reserved and protected — writes to it through PutWorm or PutDerived are rejected. Only Append() can write to __sys:log:*.

🚀 How to Run

Prerequisites: Go 1.25+, Make, CGO enabled (for SQLite)

# 1. Clone and enter the repo
          git clone https://github.com/JupiterMetaLabs/ThebeDB
          cd ThebeDB

          # 2. Copy the example config (SQLite for local dev)
          cp config.example.yml config.yml

          # 3. Build the binary
          make build # produces bin/thebedb

          # 4a. Run directly (SQLite — no dependencies)
          make run # runs with ./config.yml on :8080

          # 4b. Run with PostgreSQL + Prometheus (production-like)
          docker-compose up -d # starts postgres:5432, prometheus:9090
          # Edit config.yml to use postgres driver, then:
          ./bin/thebedb --config config.yml --port 8080

          # 5. Run tests
          make test
        

Embedding as a Go library:

// go get github.com/JupiterMetaLabs/ThebeDB

          db, err := thebedb.New(thebedb.Config{
          KVPath: "./data/badger",
          SQLPath: "./data/sqlite/default.sqlite",
          }, logger)

          ctx := context.Background()
          db.Start(ctx) // runs projector in background
          defer db.Close()

          seq, err := db.Append(&corepb.CanonicalRecord{
          Namespace: "jmdt",
          Type: "block",
          Value: blockJSON,
          })
        

⚠️ Error Handling & Resilience

Scenario	Behaviour
Projection lag > HardLimit	`db.Append()` returns `ErrSystemOverload`. Caller must implement backoff and retry.
Profile Apply() fails (up to 3×)	Projector retries with exponential backoff. On 3rd failure, the record is written to `__sys_quarantine` and skipped. Projection continues.
SQL database deleted/corrupted	Delete the SQL file, restart the process. The Projector will rebuild the entire SQL state from the KV log, starting from sequence 0.
Process crash mid-projection	On restart, the Projector reads the last committed `__sys_projections` offset and resumes from there. No data is lost or double-processed.
Hash chain violation	The jmdt profile validates `parent_hash` on every block using the `jmdt:chain_head` KV key. A mismatch returns an error, causing the record to fail and be quarantined.