KeppyLab
public workbench
Small strange machines for understanding software — and for knowing when software is lying to you.
KeppyLab is the research bench of github.com/keppy: evaluation harnesses, RL training loops, knowledge graphs for rare disease, COBOL archaeology. Everything here is built in the open, scored against real cases, and released on GitHub.
latest from the bench · 40:33
Fine-tuning an encoder and getting a go/no-go verdict
watch it, and read the writeup
The Work
gonogo
The eval harness from real deployments, open sourced. Scores an agent on your actual cases and returns a deployment decision — including "not enough evidence yet." Numbers instead of vibes.
thomas
thomas.train() — a training harness. Case → reward → train: take a Case set and a score function, get a baseline card, run the loop, compare before and after. Pretrain on Modal GPUs or post-train with LoRA RL.
titans-mini
A streaming engine over a swappable test-time memory core: MLP-weights memory vs. generated-weights vector memory, behind one narrow interface. The engine owns the loop, the core owns storage — 21 tests pin the autograd claims.
cotfaith
Chain-of-thought (un)faithfulness, study one: hint-following and confession rates on Qwen3-1.7B. Pre-registered decision log, blind-labeled judge validation, byte-exact run artifacts.
recurse
A numbered visual series that remembers itself: plan → draw → render → reflect → remember. Each entry inherits a JSON motif memory from the one before it, and continuity is over images, not intentions.
Older Haunts
describe / MCPM
Speak systems into existence. An MCP capability manager: discover servers, write client config, read the capability map back as resources and prompts.
cobol-reporter
RAG and report generation for understanding COBOL systems — sixty years of gravity, interrogated in plain English.
disease-lab
Knowledge-graph AI for rare disease literature and discovery workflows, built during a rare-disease hackathon.
WorldEnder.ai
RAG-powered text adventures with coherent long-horizon world state. Predict the end of the world.
From The Lab Notebook
Recent writings on evals, fine-tuning, and shipping AI before it's ready:
- banking77 canary: 87.2% pass, two dead runs, one false alarm
- Your agent eval is lying to you
- Fine-tuning Pythia
- The rare disease hackathon
- I demoed too early
@yok0zuna | GitHub | Hugging Face | Contact