Using AI self-improvement loops to make raggio Vector Database even faster

The scorecard at the end of my raggio post kept bugging me. raggio won on memory, recall and vector tail latency, but Weaviate took most of the rest, and not by a little: 129 queries per second against 1,050, and a median hybrid search in 104 ms against 36. So that night, a little after midnight and before I’d even hit publish, I opened Claude Code in the raggio repo and typed this:

Analyze the codebase, gather info and reply only no code: does it make sense to port this app to C ? How much effort would that be? Which are the benefits we should expect?

Everyone who writes Python has had this thought at least once: it’s slow, it’s Python, let’s rewrite it in something with a real compiler. Four minutes later the answer came back, and it was no.

This is part 1 of a seven-part series about what happened next: five days in which Claude Code, my DGX Spark (the NVIDIA GB10 box from the first benchmark) and I took raggio apart and put it back together faster. Ingest is 2.3x faster, a true cold start went from 155 seconds to 4.6 (the 30 seconds I published weren’t a true cold start, part 2 has that confession), and hybrid search under load has a p99 7.5x lower. This part is about the loop that did it, what I did in it, and what the agents got wrong. Each of the next six takes one bottleneck and explains how it got fixed.

The answer was no

The answer started with a map of where raggio’s work actually runs, and for “a Python app” it’s not where you’d think:

Almost all the heavy lifting already runs in native code. The vector gap has nothing to do with the language: raggio scans every vector, while Weaviate walks an HNSW graph and visits only a small part of them. In the answer’s own words, “Mostly no. The scan is already Rust with SIMD. This is an algorithm gap.” A flat scan written in C is still a flat scan. The one real piece of work left in Python was the BM25 rescorer, and that’s why it tops the list below.

Rewriting it in C made no sense, also because, as the answer pointed out, turbovec has no C API at all, only a Rust crate and a Python package, so I’d have had to write that binding too. So maybe a rewrite in Rust? Nah. Surely we’d gain something, but we aren’t hitting a language barrier, we’re hitting an algorithm one, and there’s still plenty of room to play there. Moreover, I prefer using a language that I know (Python) rather than an unknown one where I can give recommendations on the architecture, still.

Instead, the answer made five recommendations:

1. Move only the BM25 rescorer and the tokenizer to native code, “a few hundred lines”
2. Fix the Python inefficiencies: numpy set operations in place of Python sets holding 2.5 million ids, batched upserts, binary job payloads, batched index syncs
3. Look at approximate search inside turbovec, upstream
4. Evaluate free-threaded Python
5. If a full native rewrite ever happens, use Rust, not C

I replied “Plan for items 1,2,3,4”, switched Claude Code to ultracode, the effort setting that orchestrates workflows of parallel agents, and told it the DGX Spark was free to use. Item 5 went in a drawer.

Where it ended

Here’s the finish line, so you know where the story is going. One A/B on the DGX Spark: the image from before the program against the final one, each ingesting the 2.55 million arXiv vectors into a fresh volume, both in a 4 GiB container, interleaved on one boot.

The gray rows matter as much as the red ones. Flat vector search, recall, memory under load and disk didn’t move, and nothing was meant to move them: the flat scan is turbovec’s job, and it was already good at it. Five days, 112 commits, and a test suite that went from 46 tests to 502.

The loop

What made this work was a loop:

1. Research. Seven agents in parallel on my Windows desktop, one question each, from the native BM25 scorer to free-threading. Then a critic agent read all seven and listed what they got wrong or left out. One hour and 42 minutes.
2. Probe. Four probes on the DGX Spark, one at a time, each written to settle a question the briefs had left open. Several of the desktop’s conclusions died here, more on that below.
3. Plan. A design spec with 16 default decisions, which held unless I overrode them (I didn’t override any), and seven plans with 73 tasks between them: A the benchmark and the packaging, B concurrency races, C the cold start, D ingest, E the native BM25 scorer, F IVF fan-out, G free-threaded Python. Each plan ships on its own branch (A and D took two), keeps the test suite green, and doesn’t depend on a later plan.
4. Review. Every edit to a plan went to an editor agent, then to a separate verifier whose only job was to break it. Nearly every verifier found a real defect in an edit that looked finished. An example: a memory gate that allows the new code’s peak memory to be at most 50 MB above the old code’s. The editor wrote pf - pb <= 50. The peaks carry one decimal, and in floating point 2050.3 − 2000.3 is 50.00000000000023, so a peak exactly 50.0 MB higher would have printed FAIL. The verifier added a rounding.
5. Build. A pre-flight first: agents reading the plan against the current code, checking that every piece of code the plan quotes still exists and that no two tasks fight over the same file. Then, one task at a time, a Sonnet implementer (failing test, fix, full suite, commit) and a Sonnet reviewer, and at the end an Opus review of the whole branch.
6. A/B. Every performance claim needed one, on the DGX, against the main from just before the plan, measured again rather than remembered. The gates were written down before the run. A gate that fails stays FAIL. If I disagree with it, I override it in writing, and the override goes under the FAIL line, not in place of it.
7. Record. The results, failures included, go into the decision records: addenda to ADR 0001 and three new ones, 0004, 0005 and 0006. The plan merges, and the new main becomes the baseline the next plan gets measured against. Then around again.

The loop also improved itself along the way, because every incident left a rule behind, in the spec, in Claude Code’s memory, or in the next plan’s pre-flight. “Pin the threads before you read a microbenchmark” came from a mess with big and little cores (part 6). “A reviewer’s suggested fix to a test is a claim like any other” came from a fix that would have hung forever (part 4). By the last lap, the loop had far fewer ways to fool itself than at the first one.

Local tests lied at first

The research briefs measured things on my desktop. Then the probes measured the same things on the DGX, the kind of Arm box raggio is built for, and several conclusions flipped:

Not all of that was the machine: some briefs measured a 100k-row copy instead of the full 2.55 million, and the cold-start estimate started from my own warm 30 seconds. But the rule stuck, in the critic’s words: any Windows timing “cannot predict GB10 NEON behavior”. The desktop is x86 with AVX2 for its vector math; the GB10 is Arm, with NEON and two kinds of cores. Every performance decision got an A/B on the target, and my desktop’s stopwatch went in the drawer with item 5.

The DGX wasn’t a quiet box either, even with nothing else running on it. The page cache, the CPU scheduler and the random sample behind every IVF build each moved the numbers more than some of the fixes did, so the measuring rules grew one incident at a time:

• A fresh container for every arm, never a reused one
• At least three runs per arm after a discarded warm-up, the median as the result, and max minus min as the noise band. A gain only counts when it’s wider than the band
• Memory claims only with swap off, because on that host --memory 4g quietly grants another 4 GiB of swap
• The SQLite version on every row, because it turned out to be a performance variable on its own (part 5)
• Tune on seed 7, publish on seed 42, so the defaults aren’t fitted to the queries they’re scored on

The first lap proved the point. Plan A’s first branch only touched the benchmark client, and still failed its own equivalence gate, with 8 rows “better” than the baseline. The baseline had run two hours after a reboot with 6.1 GB of page cache; a day later the host had 40 GB cached, and the IVF index build alone went from 213 seconds to 50 with nothing else changed. The host moved the numbers more than the code did.

Me vs the agents

The agents did the typing: the research, the spec, the plans, the code, the tests, the reviews, the DGX runs over ssh, and the journal this series is built on.

I made the big decisions. The program’s decision register has 89 entries: about 40 are mine, 18 are defaults I left alone, and the agents took the rest on their own. For most of mine the recommended option was the one I would have picked regardless, but not for everything. Three times a gate failed and I overrode it.

On day three I noticed Claude was still asking questions I had already answered the same way ten times, and decided to be a little more hands off:

just assume that the reply is the same and ask only really blocking questions. If something needs to be double checked, just do it, assume I’m not at the desk and you have time to do stuff

From then on routine decisions, like merging, pushing main, starting the next plan or taking the recommended option, went ahead without me. That cut both ways: a fourth failed gate was overridden by the agent itself, under that standing instruction, with my earlier override cited as precedent (part 6). It’s in the record with its reasons, and I’m fine with it, but “don’t ask me” also means “don’t ask me about this one”.

What the agents got wrong

Less than an hour into the first night, the DGX needed updates, so I asked: “The DGX needs a lot of updates, can I restart or there’s something running?” Claude didn’t answer. It killed the running baseline by PID, stopped the container, and then replied “You can restart the DGX now”. It apologized when I pointed that out, and saved a rule to its memory, “Answer, don’t act”: a question gets an answer and an offer, never an action nobody asked for.

The second one is worse. The first baseline script cleaned up after itself with podman rm -f bench-tv bench-wv, and bench-wv was my Weaviate container from the first benchmark. Its volume survived, the container didn’t. The script got fixed, and the session’s own summaries said I had been told. I hadn’t. When the context fills up, Claude Code replaces the older conversation with a summary; one summary said “told”, and nothing checked it against the transcript. An agent fact-checking the journal found the gap 21 hours later, by searching every message for the disclosure and finding none. That’s a rule now too: a claim of having told the user gets checked against the transcript, never against a summary.

Then the plumbing, because the plumbing of agents is still plumbing. Plan writers kept getting killed by a 180-second stall watchdog that can’t tell a long write from a hang, until writes were capped at about 8 KB. A safety classifier blocked one agent because a shared instruction asked it to “reason in writing”. And for 11 hours every shell command returned nothing at all, mine included, because a third-party plugin had bloated the session’s environment file until the generated shell script got truncated.

What I didn’t expect is that the agents’ best work would be catching each other’s mistakes. Every part of this series has at least one example.

At what cost?

All of it ran on my Claude subscription, so there was no per-token bill, and tokens are the only meter I have. The research and the planning, before a single line of raggio changed, used about 15 million tokens in subagents, on top of the main session. Sonnet 5.5 came out about a day in, and from then on the workflow agents ran on it, while the main session and the final reviews stayed on Opus 5.5. Execution was cheaper per step, 31 to 43 thousand tokens per task for a Sonnet implementer in plan B, but there were a lot of steps, and I didn’t keep a running total. The DGX did the rest: about 17 hours of A/B runs (a day, counting baselines and probes), a good part of it while I was asleep.

The series

One bottleneck per part, every Monday:

• Part 2, October 19: the cold start. 155 seconds to say hello, and a database reading the same 3.57 GB table three times.
• Part 3, October 26: memory. The guard that said yes before the kernel said no, and the malloc setting that saved the index build and slowed search by a third.
• Part 4, November 2: ingest, 2.3x faster without giving up durability, and the gate that failed while every run won.
• Part 5, November 9: hybrid search. One underscore behind the p99 tail, and a BM25 scorer in Rust that did nothing until SQLite changed.
• Part 6, November 16: the IVF fan-out. Sixteen shards, one thread, and a box with two kinds of cores.
• Part 7, November 23: free-threaded Python. A win I couldn’t ship, and the bugs it shook loose.

A C rewrite would have cost me days or weeks to make the flat scan exactly as fast as it already was. The loop took five days, and most of what it fixed I didn’t know was broken.