Machine Learning Engineer, Evals
Nous Research
Remote · Americas (US time zones) · Posted August 13, 2026
Just laid off? Start here: a plan for the first weeks
See every open role at Nous Research
More in All Machine Learning Engineer jobs (262) · Technology & Software · Data & Analytics jobs
More roles at Nous Research
Similar jobs
- Principal Platform Engineer · CentralReach
- Senior Machine Learning Engineer · Strava
- Lead Systems Engineer - 90416789 · National Railroad Passenger
- Associate Reliability Engineer (Remote) · Kohl's
- IT Infrastructure Systems Engineer · Onto Innovation
- Machine Learning Engineer · Interwell Health
Data & Analytics jobs in the rankings
Rankings about data & analytics jobs
- Best cities for data and analytics jobs #1 New York (1,883 openings)
- Best cities for data and analytics pay after cost of living #1 San Mateo ($233k)
- Most requested AI skills in data and analytics jobs #1 Machine learning engineering (17.1%)
- AI tools named most often in data and analytics jobs #1 Databricks (7.6%)
- Employers with the most data and analytics jobs #1 Amazon (606 roles)
Data & Analytics jobs · Top Data & Analytics employers · All rankings
Pay compared with the market
This listing does not state pay. Across 47 open machine learning engineer listings that state pay, the middle half runs $163,500 to $243,500 and the median is $200,000.
Listed pay is what employers advertise, not what people are paid.
Machine Learning Engineer salaries by state · Check your own pay
How AI is used in this kind of work
Among computer occupations, all other, AI is already used for a large share of the tasks. Measured exposure for computer occupations, all other: 31% (high).
Share of this job’s tasks that AI is already used for at work, from real usage. It is not a forecast of job loss, and a low number means no use seen yet. Source: Anthropic Economic Index.
Skills this job asks for
Say which you have and which you want to build. We use it to match you to jobs like this one.
About the job
The Role
You'll work across the lab on agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together. This is a high-growth, high-ownership role on a small team, and you'll ship evaluation infrastructure that researchers depend on from day one.
Responsibilities
- Run the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on your first task
- Build a judge calibration protocol: sample human-labeled decisions, measure agreement (κ, per-class P/R), identify drift zones, and document it so anyone can re-run it
- Extend an existing benchmark (GAIA, τ-Bench, SWE-bench slice, etc.) with new tasks targeting known capability gaps, including the prompt, environment, rubric, automated grader, and QA
- Run failure analysis on model outputs: categorize failure modes, quantify prevalence, and write up findings with recommendations for training data, judge prompts, or benchmark changes
- Own a recurring eval workflow (weekly regression suite, judge drift dashboard, red-team evaluation for a new capability) and ship tooling researchers actually use
Qualifications
- 3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job
- Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short
- Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents
- Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it
- Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job)
- Understanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen's κ measures, how to think about confidence intervals on a metric
- At least 3 of the following: you can explain why LLM-as-judge needs calibration; you've done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues; you know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each; you've designed or extended an eval dataset with happy paths, edge cases, and adversarial examples; you've thought about non-determinism in eval, how you sample, how many runs, how you report variance
- You communicate clearly to both researchers and engineers, in the right language for each
- You're comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help
Preferred
- RLVR / RLHF pipeline experience
- Training data curation experience
- Distributed eval orchestration experience
- Benchmark design from scratch
- Red teaming and adversarial eval experience
- Familiarity with psychometrics or measurement theory