Member of Technical Staff - Evals

About the Role

You will play a key role in ensuring the quality and reliability of our AI-powered features. You'll design and maintain evaluation frameworks that measure accuracy, reliability, and regression behavior. You'll build automated test harnesses that operate on real accounting data, catching potential failures before they impact customers. You'll define metrics and benchmarks that give the team clear, quantitative insight into model and system performance. You'll create easy-to-use tooling that lets engineers write, execute, and interpret evaluation tasks as a seamless part of their workflow. You'll collaborate closely with the Applied AI and Agent Engineering teams to make sure every shipped capability has a well-defined standard of "good enough" and a robust way to verify it.

Requirements

• Experience writing and reviewing evaluation tasks used by AI researchers to assess and improve model performance
• Background in quality assurance across software, client deliverables, or financial analysis
• Exceptional written communication skills, with the ability to clearly articulate steps to achieve desired outcomes
• Familiarity with accounting, financial systems, tax preparation, or payment technologies preferred

Responsibilities

• Design and maintain evaluation frameworks to measure accuracy, reliability, and regression behavior of AI capabilities
• Build automated test harnesses that operate on real accounting data to identify potential failures before they impact customers
• Define metrics and benchmarks to provide quantitative insights into model and system performance
• Create tooling that enables engineers to write, execute, and interpret evaluation tasks within their development workflow
• Collaborate with Applied AI and Agent Engineering teams to define and verify standards for shipped capabilities