Staff AI Product Analyst
This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Staff AI Product Analyst in the United States.
This role sits at the intersection of AI product innovation, advanced analytics, and data-driven decision-making, shaping how next-generation agentic systems are measured, trusted, and improved. You will define the analytical backbone of AI-native products, ensuring that emerging capabilities are grounded in clear metrics, robust evaluation frameworks, and reliable data systems. Acting as a senior individual contributor, you will influence product, engineering, and data teams by translating complex AI behaviors into measurable business outcomes. The role requires a systems-thinking mindset, combining deep analytics expertise with a strong understanding of LLM-based and agentic workflows. You will help establish how success is defined, tracked, and improved across AI-driven experiences. This is a highly cross-functional position where your work will directly shape product direction, model trust, and organizational confidence in AI systems. The environment is fast-moving, experimental, and focused on turning cutting-edge AI into dependable, scalable product value.
Accountabilities:
- Define AI-native product analytics strategies, including north-star and guardrail metrics for key AI-powered user journeys, focusing on adoption, quality, and business impact.
- Design and maintain evaluation frameworks for AI systems, including golden datasets, scoring rubrics, regression tests, and structured model/agent assessments.
- Partner with data science and product teams to support experimentation, causal inference, and interpretation of AI product performance.
- Build systems-level understanding of how agents interact with data sources, APIs, tools, and metrics to ensure analytics requirements are clear and actionable.
- Develop taxonomy frameworks to classify AI failure modes such as hallucinations, tool misuse, retrieval gaps, and edge-case behaviors.
- Monitor live AI product usage to track success rates, abandonment patterns, fallback behaviors, and human override signals.
- Own analytics enablement through playbooks, documentation, and frameworks that define how AI systems should be evaluated and interpreted.
- Collaborate with engineering and data teams to ensure reliable data pipelines, consistent definitions, and trustworthy metric systems.
- Produce recurring insights, dashboards, and executive readouts that guide roadmap prioritization and reduce high-severity AI system issues.
- Support cross-functional alignment between technical AI capabilities and business outcomes through clear narrative and data storytelling.
- 10+ years of experience in product analytics, data science, applied analytics, or technical product roles in high-scale environments.
- 1+ year of hands-on experience working with AI/LLM systems in applied production or evaluation contexts.
- Strong technical proficiency in Python and experience with data analysis, ML concepts, and analytics tooling.
- Proven experience designing AI evaluation systems, including structured evals, benchmarking datasets, or production monitoring frameworks.
- Strong understanding of data governance, experimentation design, and causal analysis methodologies.
- Ability to translate complex AI and data insights into clear, actionable recommendations for non-technical stakeholders.
- Experience working in cross-functional environments involving product, engineering, and data science teams.
- Strong systems-thinking mindset with the ability to reason about end-to-end data and AI workflows.
- Comfortable operating in ambiguous, fast-evolving AI product environments with iterative development cycles.
- Bonus: experience with agentic systems, multi-step AI workflows, or LLM orchestration frameworks.
- Competitive salary range with performance-based bonus and equity opportunities.
- Comprehensive medical, dental, and vision insurance coverage.
- Flexible vacation policy supporting work-life balance.
- 401(k) retirement plan with company matching.
- Pet discount programs and additional lifestyle benefits.
- Remote-friendly work environment with flexible working arrangements.
- Opportunity to work on cutting-edge AI-native product systems at scale.
- High-impact individual contributor role with strong visibility across product and leadership teams.
- Collaborative, data-driven culture focused on innovation and experimentation.
- Inclusive workplace valuing diverse perspectives and experiences.