Research Engineer, Evaluation (Web Data AI Infrastructure)

A fast-growing Series B company that turns web pages into clean data for AI systems wants an engineer to build the evaluation layer that shows whether its output is actually correct. Judging quality across enormous numbers of sites, layouts and odd edge cases is a hard measurement problem, and the person in this seat will define how it gets solved.

What you’ll own

Define the metrics that capture what a good result looks like across a huge variety of websites and formats

Build the test harnesses and scoring flows that measure quality with rigor and at volume

Produce and maintain data collections that make those measurements trustworthy

Close the loop between measured output quality and decisions about models and product

Present results in a form the entire team can act upon

About you

Four or more years in ML, research engineering or data-heavy backend work, with real evaluation experience

Has built evals for judging LLM-based extraction, such as checking whether the right fields were pulled from messy pages

Designed the metrics and the evaluation system yourself rather than running someone else’s benchmark

Built ground-truth datasets with gold labels or reference extractions

Statistical rigor: significance, estimator bias and variance, sampling error, and agreement between automated judges and humans

Strong Python, and comfort with problems where you must invent the yardstick

A bias for shipping, measuring and iterating

Eval experience limited to agents, recommendation ranking, search, safety work or public model leaderboards is not what we need

Hybrid in the San Francisco Bay Area, with three or more days in the office. Visa transfers and certain work authorizations supported; new sponsorship is not available.