QA Engineer for Generative AI

Title: QA Engineer for Generative AI

Reports To: QA manager, who reports to Parker Wightman, Senior Director of Engineering

Location: Draper, UT / Remote (USA)

About The Role

Jump is looking for a US–based QA engineer to help coordinate and run data labeling/annotation campaigns used to improve our AI/ML systems and evaluate/review production system outputs, such as meeting notes, recap emails, and tasks; answers in our Ask Anything feature; our pre-meeting prep product; and our AI agents.

This role blends process design and hands‑on testing. You’ll use AI evaluation rubrics prepared by our product managers or data team to improve our products so our customers get accurate transcripts, summaries, and action items every time they interact with Jump. You’ll go deep into AI best practices and limitations.

You’ll partner closely with Engineering, Product, and Customer teams to ship quickly and confidently. Familiarity with Jump (as a user, beta tester, or close to advisor workflows) is a big plus. If you already have AI evaluation experience, that’s great—but it’s not required. We’ll teach you our approach; candidates with AI evaluation experience will be compensated accordingly.

What You’ll Do

• Serve as the embedded QA engineer on two pods (Jump’s cross-functional teams), collaborating with product managers to evaluate AI outputs, run exploratory and regression testing, and unblock engineers and PMs.

• Learn and track AI/ML quality signals, including golden datasets, prompt/regression suites, and metrics such as WER, diarization accuracy, action-item precision/recall, summary faithfulness, hallucination rate, and PII handling.

• Build dashboards for quality KPIs (defect escape rate, flake rate, regression coverage, MTTD/MTTR, AI eval scores) and drive continuous improvement.

• Partner with Product and Engineering to ensure requirements are testable, edge cases are captured, and AI evaluation rubrics are clear and repeatable.

• Foster a no-drama, direct-and-kind culture that moves with high-quality velocity.

About You

• 3+ years in QA or Quality Engineering for SaaS products

• Strong exploratory testing skills and clear, concise written communication for reproducing issues

• Curiosity and aptitude to learn ML/AI evaluation (prompt testing, golden sets, offline evals, safety/guardrails)

• Familiarity with AI prompts, LLMs, and the Jump product (as a user or employee)

• You don’t need a traditional STEM background to excel here. You’ll thrive if you

  • Get excited about spotting patterns

• Have a strong grasp of human language and thought processes

• You might have a background in

  • Editing

  • Technical writing

Nice-to-haves:

• Comfortable reading software system logs and finding patterns in messy data

• Familiarity with fintech or other regulated environments

• Experience with BigQuery or other data warehouses

• Experience with web API testing

• Basic familiarity with query languages, relational databases, and other data storage systems

What You’ve Done

• Built or scaled a QA function (process, tooling, reporting), or partnered with product managers and engineers to identify and resolve AI-related bugs

• Written great documentation, bug reports, or other clear technical writing

• Interacted meaningfully with LLMs and AI outputs

• Nice-to-haves:

  • Designed and executed AI evaluation workflows (golden datasets, human-in-the-loop scoring, clear rubrics)—a plus but not required; candidates with this experience may be considered for higher compensation

  • Created risk-based test plans and lightweight automation that caught regressions early

Jump is empowering financial advisors and their clients to thrive in the age of AI. We're growing incredibly quickly with a team that comes from Google, JP Morgan, BILL, Snowflake, Fidelity, Bain Capital, Harvard, Stanford and other top companies and schools. We can't wait to hear from you!

Jump’s culture

• High Velocity

• World Class

• Direct + Kind + No Drama

We believe in building tight teams of extraordinarily capable people. Come join us to transform the advisor and enterprise experience with state-of-the-art technology.

Compensation

• Salary: $75k to 90k (DOE)

• Equity

• Health benefits