Product Quality Analyst

Step into a pivotal role where your strategic vision will directly shape the future of personalized digital experiences for millions of global users. Partnering with Aquent, you will join an industry-leading digital streaming and technology pioneer to bridge the gap between cutting-edge machine learning and human nuance, transforming how next-generation AI systems understand culture, taste, and human behavior.

About the Opportunity

Aquent has partnered with a global leader in digital personalization and streaming technology known for delivering world-class, intuitive user experiences. This pioneering organization relies on sophisticated machine learning and artificial intelligence to connect people with the content they love. By joining this forward-thinking team, you will play a vital role in grounding advanced AI in real-world habits and cultural context, helping to drive the evolution of their core technology platform.

The Role and Your Impact

In this dynamic and highly collaborative position, you will serve as the strategic and operational backbone for human-in-the-loop machine learning lifecycle development. Rather than performing direct labeling, you will act as a program-level owner, designing evaluation rubrics, establishing scalable workflows, and aligning large language model (LLM) evaluators. Your work will directly free up engineering and data science bandwidth, professionalizing how human intelligence grounds next-generation AI models.

What You'll Do

• Lead the end-to-end execution of human annotation projects, managing everything from initial scoping and intake to final delivery.
• Serve as the primary liaison between data science, engineering, and external partners to map annotation requirements to the most efficient pipelines.
• Maintain and prioritize a comprehensive roadmap of annotation needs, coordinating internal and external resources effectively.
• Manage vendor relationships, including contracts, onboarding, and coordination with legal, security, privacy, finance, and accounting teams.
• Develop, maintain, and consult on reusable annotation frameworks, task designs, and evaluation rubrics.
• Facilitate feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps.
• Systematically evaluate annotation quality, resolve disagreements, and continuously improve guidelines and outputs.
• Establish operational documentation and best practices for a newly formalized function, partnering closely with dedicated data scientists.

What You'll Bring

• Proven background in data operations (Data Ops), annotation operations, model evaluation, product quality, or technical program management.
• Hands-on experience managing, coordinating, or designing data annotation workflows at an operational or program level.
• Demonstrated experience working within a scaled technology company or directly collaborating with established data annotation vendors.
• Familiarity with machine learning-driven products and the human evaluation of AI/LLM-powered features.
• Strong ability to translate complex product and modeling needs into clear, scalable annotation guidelines and evaluation rubrics.
• Experience navigating vendor onboarding, procurement coordination, and cross-functional project tracking.
• Exceptional cross-functional communication skills, with a proven ability to align technical teams and navigate ambiguity.
• A self-starter mindset, comfortable stepping into a 0-to-1 environment to build, formalize, and streamline processes.

Bonus Points

• Experience interacting with AI collaboration tools (e.g., Claude, Codex).
• Basic coding abilities.
• Experience scaling annotation pipelines from early-stage setups to operational maturity.