I build assays with the schedule, coding route and reliability threshold frozen before any data exists — a 64-block precommitted schedule executed as 20,096 controlled model jobs.
AI Evaluation
I design evaluations that can return an answer you did not want.
I design controlled experiments for AI systems: pre-registered behavioural assays, reliability testing between independent raters, statistical validation implemented from first principles, and failure analysis when a measurement does not hold up. The same governed evaluation appears on the main portfolio; here it is ordered around the evaluation design first.
Evaluation designControlled LLM experimentationReliability testingStatistical validation
What I'd bring
Evaluation work an employer can check.
Each of these is a concrete practice, not a slogan, and each one shows up in a shipped harness, a deposited paper, or both.
I measure whether independent coders actually agree, using Wilson lower bounds and Cohen's κ rather than an impression of agreement, against a 0.90 threshold fixed before collection.
Wilson intervals, Hedges' g, two-sided sign-flip permutation tests and Cohen's κ implemented directly on NumPy, with seeds derived via HMAC-SHA256 so a resampling run reproduces exactly.
When a reliability check failed, I stratified by adjudication route rather than reporting one number: coder-agreed items reproduced at 92.4%, adjudicated items at 68.5%.
When the pre-committed bar is not cleared, the headline result is ASSAY_INDETERMINATE — not a weaker version of the finding I was hoping for.
Evidence
The governed behavioural evaluation.
Executed
Pre-registered
Indeterminate
- Problem
- Can a striking output-level pattern be measured reliably enough to support any claim about what it means?
- Built
- A pre-registered behavioural assay: frozen 64-block schedule, fixed coding and adjudication route, an independent audit, and a reliability gate committed before collection.
- Scale
- 20,096 controlled model jobs, zero failures; 4,736 coded instances from 2,400 distinct prompts; 98.4% reached a final label.
- Methods
- Wilson lower bounds on two independent agreement checks (audit vs final, coder vs coder), Cohen's κ, and a conjunctive 0.90 threshold fixed before collection.
- Result
- Both checks fell short of the pre-registered bar — 0.823 and 0.728 against a required 0.90 — so the assay returned ASSAY_INDETERMINATE. What was established: procedural closure (98.4%) and measurement reliability are different properties. What was not established: any substantive claim about the underlying behavioural pattern, in either direction.
Evidence
The reliability failure analysis, and the reanalyses.
Executed
Deduplicated
Terminal preserved
- Problem
- Once the reliability check failed, was that a one-off artefact, or something structural that could be localised and checked?
- Built
- A dependence-aware reanalysis pipeline: a duplicate-structure census, exact-input deduplication, and a stratified split by adjudication route, run against the same coded dataset.
- Scale
- 4,726 coder-vs-coder and 470 audit-vs-final comparisons re-examined; 829 duplicated coder-input groups checked for heterogeneity.
- Methods
- Duplicate census on exact response text; Wilson intervals recomputed on deduplicated data; stratification into coder-agreed versus adjudicator-resolved items.
- Result
- Deduplicated reanalysis: 349/411, 84.9%, lower bound 0.811 — not a duplicate-data artefact. Stratified: coder-agreed items reproduced at 92.4% (314/340), adjudicator-resolved items at 68.5% (89/130). What was established: where the measurement broke. What was not established: whether the underlying behavioural claim is true or false — the terminal stayed ASSAY_INDETERMINATE.
Capability stack
What the evaluation work is built from.
Grouped by what each tool was used for, not as a logo wall — every entry appears in a shipped harness or a deposited artifact.
Evaluation design
- Pre-registered assays
- Frozen content schedules
- Fixed coding & adjudication routes
- Reliability thresholds fixed pre-collection
Controlled LLM experimentation
- Python 3.11
- Pluggable model adapters
- Blinding & hashing
- 20,096 controlled jobs, zero failures
Statistical validation
- Wilson intervals
- Cohen's κ
- Sign-flip permutation tests
- HMAC-SHA256 derived seeding
- NumPy, no scipy/statsmodels/sklearn
Contact
Open to AI evaluation work.
If the work involves finding out whether a measurement can be trusted before it gets used to decide something, I would like to hear about it.
Email
nived.rajendran.ai@gmail.com
The fastest route. I read everything.
ORCID
0009-0002-2951-0162
The authoritative record of the deposited work.
GitHub
nived-rajendran
Public repositories: evaluation tooling and the reproducibility package.
Portfolio
The full site
Every project, not just the evaluation-shaped slice of it.