Use the left and right arrow keys to move through this carousel, or the previous and next buttons below it. You can also drag it sideways.
A computer science undergraduate at BITS Pilani working on systems that have to be right rather than merely finished. Order books, kernels, CRDTs, inference engines. In every case the interesting half of the work is the machinery built to prove the first half wrong.
Live work, not
shipped work. Open
in front of me now.
tip!Click on the image to explore
DivergenceIn progressReasoning-model behaviour · ICLR 2027An in-progress study of how reasoning models diverge from themselves: one prompt, one set of weights, sampled traces that reach incompatible conclusions. It follows a published result of mine on self-consistency failure modes and moves from measuring disagreement to characterising which internal states predict it. Targeting ICLR 2027. The harness and the dataset land before the paper does.
I build the system first, then I build the thing whose only job is to prove the system wrong. The harness is not scaffolding around the deliverable, it is the deliverable, because an engine nobody has tried to falsify is not an engine anybody should trust. What ships is the pair, and the claim is worth exactly what the adversary failed to break.
Published work, and what is currently open in front of me.
The standard way to make a language model more reliable is to sample several reasoning traces and take the most common answer. On hard science questions, that makes it worse more often than it helps. Across 198 GPQA Diamond problems, majority voting reduced accuracy on 56.6% of problems for one model and 65.7% for another. The obvious response is to detect the bad cases and route around them, so I tested three cheap confidence signals that could do that. All three fail, each for a different reason, and one of those failures I still cannot explain. Every hypothesis was registered and git-tagged before the confirmatory data was touched. Version two revised the mechanism claim in my own accepted paper rather than defending it.
In progress
The unexplained failure above is what I am working on now. The question is whether agreement between samples measures evidence at all, or just measures how strongly the model already believed the answer before it reasoned. To test it I needed reasoning-model data that the published study could not afford to buy, so I generated it myself: 12,672 samples across 198 problems, 82.7 million tokens, twenty-three hours on a consumer laptop GPU. The hypotheses and kill conditions are written down and tagged before anything is run, so the result is whatever it is. No conclusion yet, and possibly no paper. That is the arrangement.
A bug that only appears sometimes has not been found, it has been noticed. Every system here is built so that a failure carries a seed, and the seed replays the failure on any machine, in any order, as many times as you want.
That is the whole discipline: make the machine deterministic enough that its bugs can be handed to somebody else. contribution index.
tip!Hover, focus or swipe to read
A sync loop used its timer handle as its stopped flag. A stop left the handle holding a fired timer’s id, so the resume check never fired and any backgrounded tab died permanently. The local test passed because it never backgrounded the tab.
Config validation ran before base64 decoding. Go’s decoder skips newlines, so mandatory fields set to a newline passed validation and decoded to empty. The container was created successfully, then every subsequent operation rejected it as invalid.
A span link was encoded as a causal reference, then read back as a parent. A root linking to its own descendant became a two-span cycle. The UI rendered a fully chromed page with no spans, no error, no spinner, and a silent console.
let x = ::; segfaulted in type checking. The cause was in the parser: it returned an error sentinel with no diagnostic, and the sentinel was a zero-segment path indistinguishable from a real one, so it flowed silently through name resolution.
Merged!
13 merged · 8 repositories · 8 organisations
Ordered by the name
you would recognise,
not by the date.
tip!Expand a row for the detail
Implements the CodeAttack technique from Ren et al. for Microsoft’s AI red-teaming framework, across Python stack, list and string variants plus C++ and Go. Two and a half months of review across 43 review comments, 19 commits and 19 files. Review caught real bugs: json.dumps emitting lone surrogate escapes that Go and C++ reject, synchronous YAML loading inside an async method, machine-dependent absolute-path identifiers, and custom templates locked to one encoding. A language-aware escaper replaced json.dumps, template loading moved into the constructor, and the tests grew from 31 to 88. The architecture changed mid-review: I argued it needed its own attack class mirroring FlipAttack, and the senior maintainer decided otherwise, so it landed as a converter.
Open PR #1960 on Microsoft PyRITThe reported bug was that BedrockGenerator sent both temperature and topP, which Claude 4.x on Bedrock rejects, breaking scans for everyone on those models. The maintainer had marked it a duplicate but invited a generic mechanism mirroring the OpenAICompatible pattern, so I built that rather than the narrow fix: a parameter map translating garak’s attribute names to Bedrock’s camelCase fields, the call path refactored to a loop with suppression checks, and validation moved into the constructor. On review I was asked to flip the key convention to garak’s names, did, and a competing narrower pull request was closed in favour of this one.
Open PR #1842 on NVIDIA garakThe original check decided whether a target was local by matching substrings in the URL, which is not a decision string matching can make correctly. It now parses the URL properly and uses Python’s ipaddress module for loopback handling, with 15 unit tests covering the cases the old check got wrong.
Open PR #742 on Hugging Face OpenEnvWorth telling honestly: this started as a roughly thousand-line GenAI Observability Mode submitted for CNCF LFX. The maintainer rejected it as too large to review and it stalled. I rewrote it as a three-file, plus-76 minus-4 change, and that is what merged. Twenty-three comments across the review.
Open PR #4053 on CNCF jaeger-uiA bug in my own merged code, found after the fact. The trace page scanned raw attributes itself rather than calling the canonical detector, so every improvement to that detector was quietly skipped on the auto-activation path. Four-line fix plus a regression test for each direction of divergence. I filed it as an issue and the fix auto-closed it; merged about two and a half hours after I reported the finding.
Open PR #4271 on CNCF jaeger-uiThe dead-code pass never marked a type live when it appeared only as a generic argument. Three existing tests were asserting that wrong behaviour, so I corrected the tests rather than coding around them. This is the one that reached upstream GCC: it was sent to gcc-patches as part of a 77-patch gccrs sync, committed to GCC master with authorship preserved.
Open PR #4731 on GCC gccrsThe most review process of any of these. My first design stored extern-crate status on the resolver rib; the reviewer rejected it and asked for a mapping keyed on node id instead. Rebuilding it around the existing module-id precedent surfaced a critical ordering difference: the precedent records after its insert, but this has to record before, because the duplicate check runs inside the insert and needs the new node’s status. Then three history rewrites on request, rebase, partial squash, full squash, with the suite verified bit-identical at 11,362 pass across all three.
Open PR #4728 on GCC gccrsMy first fix for a crash I found and filed myself: `let x = ::;` segfaulted. The crash surfaced in type checking, but it came from three layers down in the parser, which returned an error sentinel with no diagnostic. Because the sentinel is built as a zero-segment path and the error check is literally "has no segments", it was indistinguishable from a legitimate empty path the whole way down. Another contributor suggested fixing it in the pattern parser; gdb showed the reported case routes elsewhere, so that would have left the crash live. My first attempt then regressed three tests by also firing on a bare metavariable, and narrowing the guard fixed it. A probe confirmed the narrow guard is complete rather than partial.
Open PR #4799 on GCC gccrsPart of the tracking issue migrating legacy typing constructs to built-in generics across the model-compression toolkit. A small diff, and a net-negative line count, which is the shape maintainers like.
Open PR #2777 on vLLM llm-compressorThe companion to the utils change, under the same tracking issue. Also net-negative.
Open PR #2779 on vLLM llm-compressorPurely additive: a regex type that actually conforms to RFC 4291, validated before submission against a 22-case matrix and through the library’s own regex machinery. outlines is downloaded around 1.9 million times a month, which makes this the widest-distribution change here.
Open PR #1867 on dottxt-ai outlinesContent and structure work on the AI Village site, the DEF CON AI security community. Web content rather than tooling, and worth listing as such.
Open PR #53 on AI Village aiv_websiteCompanion to the community page change, on the same site.
Open PR #54 on AI Village aiv_website
