Coredump – Coredump

Building a programming language using SQLite's VM - Pt 1

Mon, 20 Oct 2025 00:00:00 +0000

SQLite is cool.

It’s the most deployed database in the world, it’s tiny, expressive and its architecture is quite different from traditional relational databases. SQLite uses a Virtual Machine (VM) called Virtual DataBase Engine (VDBE) to interpret SQL, rather than Volcano-style or vectorization. Do you know which other project uses VMs to interpret code? Lua, Java, Erlang and ~~god forbid~~ JavaScript.

But what if we used SQLite’s VM to build a programming language? That’s what this series is about.

Introduction

So, what’s a VM?

It’s basically an emulation of how computers work, you can have emulators for real world computer systems with full OS and driver support (like QEMU), simple CPU emulators (like this 8080 emulator) or even ✨abstract machines✨ that only exist on paper but can, through emulation, run real code.

For instance, the instructions ran by Lua are defined here, this is the instruction to add two numbers:

// It adds the values in registers B and C and save it on register A
OP_ADD,/* A B C R[A] := R[B] + R[C] */

SQLite also defines its own instruction format, with simple instructions like Add and fancier ones like CreateBtree.

Choosing which instructions your Instruction Set should have is a non-trivial problem, it’s a balance between performance and simplicity specially considering other architecture decisions like: Stack vs Register based, dispatch strategy, whether or not to do JIT, etc.

VMs are powerful, they allow you to encode full Turing complete programs with a relatively simple structure, you just need a Program Counter (PC), an Instruction Set Architecture (ISA) and a form of storing your data/opcodes (e.g pilling them onto a stack or using virtual registers), and… you’re done.

We’ll start by building a tiny VM in Rust to keep things less abstract. Once we’ve got that foundation we’ll look at how SQLite’s VM works. But if you truly want to understand this I couldn’t recommend a better book than Crafting Interpreters – the second part :).

The DVM (Dumb Virtual Machine)

“What I cannot create I cannot understand” – Richard Feynman

We’re going to do a simple VM that just executes mathematical expressions. We can define our dumb VM and instruction format like this:

struct DVM {
 pc: usize,
 registers: [u32; 16],
 program: Vec<Insn>,
}

enum Insn {
 Add { lhs: u32, rhs: u32, dest: u32 },
 Sub { lhs: u32, rhs: u32, dest: u32 },
 Mul { lhs: u32, rhs: u32, dest: u32 },
 Div { lhs: u32, rhs: u32, dest: u32 },
 LoadImm { dest: u32, value: u32 },
 Return,
}

Running a program is dead simple, just execute the instruction in the position indicated by pc’s value and increment pc:

pub fn run(&mut self) {
 while self.pc < self.program.len() {
 match &self.program[self.pc] {
 Insn::Add { lhs, rhs, dest } => {
 let result =
 self.registers[*lhs as usize].wrapping_add(self.registers[*rhs as usize]);
 self.registers[*dest as usize] = result;
 self.pc += 1;
 }
 Insn::Sub { lhs, rhs, dest } => {
 let result =
 self.registers[*lhs as usize].wrapping_sub(self.registers[*rhs as usize]);
 self.registers[*dest as usize] = result;
 self.pc += 1;
 }
 Insn::Mul { lhs, rhs, dest } => {
 let result =
 self.registers[*lhs as usize].wrapping_mul(self.registers[*rhs as usize]);
 self.registers[*dest as usize] = result;
 self.pc += 1;
 }
 Insn::Div { lhs, rhs, dest } => {
 let result = self.registers[*lhs as usize] / self.registers[*rhs as usize];
 self.registers[*dest as usize] = result;
 self.pc += 1;
 }
 Insn::LoadImm { dest, value } => {
 self.registers[*dest as usize] = *value;
 self.pc += 1;
 }
 Insn::Return => break,
 }
 }
 }

Yep, the heart of our VM is just a match inside a for loop, no magic at all. The task of our parser is to transform the language defined by our syntax in a sequence of instructions. A simple architecture for this is:

Code
↓
Parser → AST
↓
Code generator
↓
VM bytecode
↓
Virtual Machine executes instructions

For instance, 2 + 2 results in the program:

[
 LoadImm { dest: 1, value: 2 },
 LoadImm { dest: 2, value: 2 },
 Add { lhs: 1, rhs: 2, dest: 3 },
 Return
]

> 2 + 2
4

You can check out the full implementation here

Real Stuff

Now that you just did VMs 101, let’s take a deeper look into how SQLite’s works. Each SQL statement is transformed in a sequence of instructions, for example:

sqlite> explain select 1 + 1;
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 4 0 0 Start at 4
1 Add 2 2 1 0 r[1]=r[2]+r[2]
2 ResultRow 1 1 0 0 output=r[1]
3 Halt 0 0 0 0
4 Integer 1 2 0 0 r[2]=1
5 Goto 0 1 0 0

It works a bit different than our VM but the concept is the same, let’s break this step by step (pun intended):

Every program start at 0 with the Init instruction, the value in the register p2 indicates where should we go next, in this case to instruction 4;
Here we define a integer constant, 1, in the register 2 r[2]. Note the code generator is smart enough to know that our lhs and rhs are the same number, so it only allocates a single constant.
Goto jumps to the instruction with the index defined in p2.
Add takes the value of p1 and p2 and stores the result in p3 (quite similar to what we did).
Roghtly ResultRow takes the registers p1 through p1+p2-1 and builds a single row of results.
Halt is equivalent to our Return, it ends the computation.

Now look what’s the program to create a new table:

sqlite> EXPLAIN CREATE TABLE t(x INTEGER);
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 28 0 0 Start at 28
1 ReadCookie 0 3 2 0
2 If 3 5 0 0
3 SetCookie 0 2 4 0
4 SetCookie 0 5 1 0
5 CreateBtree 0 2 1 0 r[2]=root iDb=0 flags=1
6 OpenWrite 0 1 0 5 0 root=1 iDb=0
7 NewRowid 0 1 0 0 r[1]=rowid
8 Blob 6 3 0 0 r[3]= (len=6)
9 Insert 0 3 1 8 intkey=r[1] data=r[3]
10 Close 0 0 0 0
11 Close 0 0 0 0
12 Null 0 4 5 0 r[4..5]=NULL
13 Noop 2 0 4 0
14 OpenWrite 1 1 0 5 0 root=1 iDb=0; sqlite_master
15 SeekRowid 1 17 1 0 intkey=r[1]
16 Rowid 1 5 0 0 r[5]= rowid of 1
17 IsNull 5 25 0 0 if r[5]==NULL goto 25
18 String8 0 6 0 table 0 r[6]='table'
19 String8 0 7 0 t 0 r[7]='t'
20 String8 0 8 0 t 0 r[8]='t'
21 Copy 2 9 0 0 r[9]=r[2]
22 String8 0 10 0 CREATE TABLE t(x INTEGER) 0 r[10]='CREATE TABLE t(x INTEGER)'
23 MakeRecord 6 5 4 BBBDB 0 r[4]=mkrec(r[6..10])
24 Insert 1 4 5 0 intkey=r[5] data=r[4]
25 SetCookie 0 1 1 0
26 ParseSchema 0 0 0 tbl_name='t' AND type!='trigger' 0
27 Halt 0 0 0 0
28 Transaction 0 1 0 0 1 usesStmtJournal=1
29 Goto 0 1 0 0

I won’t go through every instruction but I think you got the idea.

Its implementation lies at vdbe.c, the equivalent of our run function is sqlite3VdbeExec, which is basically a “massive switch statement”. The main interface of SQLite is sqlite3_step, every time we call it it will perform a step in the computation inside the VM, until it reaches an error or ends the result. Cool right?

Why?

Now that we’re SQLite experts we can start hacking on it. But as a great philosopher said once:

Why?

As far as I can tell nobody did this before, so it’s cool. Besides that, SQLite provides nice primitives and ACID semantics that no other VM do. For instance, SQL has transactions, where roughly a set of statements either all succesfully happens and none happens, we could use function coloring to create ✨transactional functions✨, so if something goes wrong we can rollback our execution context to the point before doing the operation. Also, we could steal some ideas from Lua and expose tables as a primitive data type.

I’m not a programming language expert (in fact this is my first PL), so I’m not concerned with full correctness or even if this is a good idea in the first place, so let’s get some fun :)

Carrot, our language

What I want here is to detach the VM of SQL and be able to write programs like:

fn fibo(x) {
 if x == 1 || x == 0 {
 return 1;
 }
 return fibo(x - 1) + fibo(x - 2);
}

/// Transactional functions(!). More on that later :)
txn fn pay() {
 let balance = user.balance();
 if balance < 0 {
 abort;
 }
 /// ...
}

print("Hello world");

Excited? Me too. Stay tuned to the second part where we’ll start to do it!

First Freelance

Wed, 05 Jun 2024 23:05:53 -0300

This’s my first blog post, so if you’re reading this, thank you! The idea is to write regularly, documenting a bit about my experience as a Computer Science student getting into the job market.

Let’s get straight to the point: how my first freelance went and what lessons I learned.

Freelance Definition

“Freelance is all, closed-scope, paid work where there’s no employment relationship between the payer and the one getting the job done.” – Me

Some Context

I started learning web dev nearly a year ago, following the usual path: HTML, CSS, jumping to vanilla JS, then to React (which I later regretted — learning about ‘addEventListener’ and other stuff first would’ve been helpful), Nodejs, SQL, hot to develop basic REST APIs, a little bit of docker and most recently Java.

Each project helped me to improve at some aspect: speed (in terms of delivery, not raw performance), maintainability, learning completely new stacks. I learned things on-demand and continue to live my life. In some of these projects I met a pretty nice guy who guided me to other roads of best development practices, the ~~infamous~~ clean code, and refactoring.

This particularly project was a v2 of another project that this guy worked on. As the v2 still in a development phase, the v1 has been used and it about it that we gonna talk about.

Problems

This project is a contest study platform, developed in Vue.js, PHP/Laravel and MySql (stacks that I never worked on). And it was having two major issues:

A field that should highlight at a specific date was not working;
Some options that should appear after a specific date was not showing up;

Simple don’t you think? That’s was the two bugs that I was paid to fix.

Solutions

Unfortunately I can’t show the source code as I like but basically the first problem as just a miss object constructor related with moment.js, the second was a trickier one, some obscure SQL query had a wrong conditional.

But how I came to these solutions? What I did wrong? I most important, what I’ve learned?

Learnings

1. Focus on the problem

With no doubts my biggest mistake was trying to understand all the system at once, instead of focusing exclusively on the problems.

Working with other people code it’s a funny thing, and surely the most common scenario, you have to understand the how’s and why’s, explore components, modules and classes in such way to create a mental model of the system the most close as possible of the previous developer. Doing this it’s not easy and can take days or even months, but I wont have too much time. When I realized that I was trying to understand the code of each page instead of focusing on the buggy page I pivoted.

With specific tasks comes specific approaches. In reality is much simpler than it seems, there is no need to refactor all the code, just need to fix this tiny problem. What bring us to the next learning.

2. Create and test hypothesis

A practical lesson. The scientific method[1] applies exceptionally well in debug sessions. For instance, a system as some unexpected behavior, after digging into the logs, requests, and any trail of the bug, you:

Create a hypothesis about the reason of the current behavior;
Create a test scenario that makes your hypothesis fail;
Repeat;

I particularly have a great time doing this kind of things, I feel like a crime detective in the crime scene, or more precisely a mechanic fixing a car already “fixed” by another mechanic. Create hypothesis, test, repeat, and be curious. At least you going to learn something new, and don’t forget:

“When you have eliminated the impossible whatever remains, however improbable, must be the Truth” – Mr. Spock

3. Don’t be afraid of new stacks

As you could see about my background, I’ve never has worked with any of the project techs, but besides these techs:

The HTTP protocol is the same;
The front-end -> back-end -> database model is the same;
Laravel’s structure with controllers, services and models is not too different of other web frameworks.

Mainstream techs are more similar than we want to admit. So, if you learned the problem that a tool proposes to solve, you can understand variations of such tools, and the different was that these variations some the same problem.

4. Take risks!

Last but not least, take risks! I’m saying to bet on shitcoins? No! Or lend money to your relative open a snake oil store? ~~Maybe~~ NO! I’m saying that putting yourself in situations that you have nothing to lose, but a lot to gain is always good[2], even if you think you’re not 100% ready. Each opportunity that takes you off your comfort zone is a opportunity to improvement.

Notes

A powerful concept, even routine in programming, is the Ockahm’s razor, that says: “[…] The simplest explanation is usually the best one”. So, before checking for race conditions or deadlocks, certify about that semicolon on line 67 :)
For those who read Taleb’s books, that’s the bread and butter of antifragility, I recommend any of Nassim’s book.