Senior Applied Scientist - AI Platform (GenSim)

About the role

Datadog's AI Platform builds the foundations of its AI efforts, split into three pillars: training and serving, agents, and evaluation and experimentation. This role sits in the evaluation and experimentation pillar, which owns the shared annotation and evaluation infrastructure, including the evaluation scenario store and telemetry archival systems. Specifically, you will be the first applied scientist on GenSim, the team that builds the environments Datadog's agents learn in: real, fully instrumented applications driven by representative traffic, with controlled failures injected so the ground truth is known.

Key responsibilities

• Own the applied science direction for GenSim, setting the methodology and technical direction for building simulated environments and post-training data on a team with no prior precedent.
• Define, measure and raise the quality of post-training data (correctness, representativeness against real customer systems and telemetry, and difficulty), making these measures actionable release by release.
• Close the realism gap between simulated and real production systems, driving the research and engineering behind it.
• Build scalable, production-grade systems of synthetic environments that are reliable and invokable inside a training loop.
• Determine how the data is best applied in LLM post-training and evaluation, and own the agent and LLM application evaluation approaches.
• Work cross-functionally with engineers and applied scientists on adjacent teams such as Bits AI SRE, model training and the wider evaluation and experimentation pillar.

Required profile

• PhD, MS or equivalent research experience in a scientific field, with strong applied mathematics grounding.
• 6+ years of applied science or ML engineering experience, including setting technical direction for others.
• Hands-on experience with LLM and agent post-training data, creation, management and quality control.
• Domain expertise in LLMs and agentic applications, beyond classical ML fine-tuning.
• Experience evaluating agents or LLM applications and defining what "good" means before measuring it.
• Strong programmer and production software engineer, able to ship scalable systems and work with distributed systems.
• Comfortable collaborating across engineering and science teams and thriving in ambiguity.

Required skills

• Python (minimum)
• Production software engineering and distributed systems
• Bonus: LLM fine-tuning, post-training or model training experience
• Bonus: statistics, experiment design and data analysis
• Bonus: production ML infrastructure, observability or monitoring background, architecture-level understanding of LLMs

What we offer

• New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
• Continuous professional development, product training and career pathing
• Intra-departmental mentor and buddy program, and access to Inclusion Talks
• Spring Health benefits for employees and dependents age 6+, plus competitive global benefits