Machine Learning Engineer, Evaluation (AI Memory Startup)

A research-driven startup building a memory and personalization layer for AI agents is hiring an engineer to take full ownership of how its system is measured. The product is an intricate system of cooperating agents with partly opaque behavior, and the central puzzle is deciding what improvement even means for representations of people and entities that evolve over time and have no ground truth.

Focus areas

Design the evaluations: define scores, work out what improvement actually looks like for an evolving representation, and revisit that definition as the product evolves

Construct the data pipelines and test harnesses behind them, covering data intake, labeling, versioning, reruns and automated judges, so new questions get answers within days

Run the evaluations, read traces and results, and identify what is really broken rather than merely easy to measure, then propose fixes

Create large-scale agent simulations, because the pace of iteration depends on the number of entities that can be modeled simultaneously

Own the loop from question to pipeline to rerun to writeup, with no handoffs

Credentials

Three or more years designing and building LLM evaluations, including complex evals, simulators or environments for RL post-training

Evidence of strong research: first-author or co-author work at a top ML venue, or open-source work of comparable quality

Production-quality Python, with tests and sensible release habits

Familiarity with PyTorch, Hugging Face and experiment tracking; data pipeline tools such as Kafka or SQL are a plus

Experience with synthetic users, memory or personalization for LLMs is a bonus

On-site in New York, NY. No visa sponsorship; TN visas can be accommodated, H-1B cannot.