Build and maintain monitoring, testing, and observability systems for AI agents; design test coverage frameworks; partner with science and product to develop LLM evaluation and debugging tools; support production via on-call rotation; collaborate with customers and engineers to drive scalable, production-ready agent quality.
At Qualtrics, we create software the world’s best brands use to deliver exceptional frontline experiences, build high-performing teams, and design products people love. But we are more than a platform—we are the creators and stewards of the Experience Management category serving over 18K clients globally. Building a category takes grit, determination, and a disdain for convention—but most of all it requires close-knit, high-functioning teams with an unwavering dedication to serving our customers.
When you join one of our teams, you’ll be part of a nimble group that’s empowered to set aggressive goals and move fast to achieve them. Strategic risks are encouraged and complex problems are solved together, by passing the mic and iterating until the best solution comes to light. You won’t have to look to find growth opportunities—ready or not, they’ll find you. From retail to government to healthcare, we’re on a mission to bring humanity, connection, and empathy back to business. Join over 5,000 people across the globe who think that’s work worth doing.
When you join one of our teams, you’ll be part of a nimble group that’s empowered to set aggressive goals and move fast to achieve them. Strategic risks are encouraged and complex problems are solved together, by passing the mic and iterating until the best solution comes to light. You won’t have to look to find growth opportunities—ready or not, they’ll find you. From retail to government to healthcare, we’re on a mission to bring humanity, connection, and empathy back to business. Join over 5,000 people across the globe who think that’s work worth doing.
Senior Software Engineer - Experience Agents
Why We Have This Role
Agent Studio is transforming how enterprises solve customer problems in real time. Your work on the testing and monitoring team directly enables customers to confidently deploy AI agents that reduce support escalations.We need a senior engineer who can build the observability and confidence systems that let our customers measure success and iterate rapidly. You'll work closely with product, science, and customer teams to ensure every agent deployed meets quality standards and delivers measurable business impact.
How You’ll Find Success
- Takes ownership of features from day one, learning new domains quickly and driving them to completion without needing task-by-task direction
- Writes effective prompts and debugs LLM graphs using tools like LangSmith, bridging traditional engineering with AI/ML work
- Collaborates across teams (admin, runtime, product, science) to understand system behavior and customer needs
- Uses AI tools like Claude and Cursor fluently to accelerate development and problem-solving
- Communicates clearly with both technical and non-technical stakeholders, translating monitoring insights into actionable customer guidance
- Demonstrates customer obsession by prioritizing metrics and insights that help customers succeed
- Balances full-stack capability, comfortable moving between backend systems and UI work as priorities shift
How You’ll Grow
- Deepen expertise in AI/ML observability and monitoring systems as the product scales
- Lead increasingly complex initiatives as the team grows and takes on new responsibilities
- Develop product thinking by working directly with customers and understanding how monitoring drives business outcomes
- Expand full-stack skills across both backend infrastructure and customer-facing interfaces
Things You’ll Do
- Build and evolve monitoring systems that give customers real-time visibility into agent performance, enabling them to measure success and identify improvement areas
- Design and implement test coverage frameworks that give customers confidence their agents are production-ready before deployment
- Partner with the science team to develop LLM evaluation metrics and debugging tools that help customers optimize agent behavior
- Collaborate with forward deployment engineers and customers to identify monitoring gaps and translate feedback into product improvements
- Contribute to the platform's transition from incubation to general availability by ensuring monitoring and testing scale reliably across growing customer base
- Participate in on-call rotation and support customers in production, using real-world issues to drive product improvements
What We’re Looking For On Your Resume
- 5+ years of software development experience with demonstrated ownership of features or systems from conception through production
- Experience building or working with monitoring, observability, or testing systems at scale
- Proven ability to work across full-stack domains (backend and frontend) in a fast-moving environment
- Track record of learning new technical domains quickly and shipping quality work
- Experience working with AI/ML tools or systems, or demonstrated ability to pick up new technical areas rapidly
What You Should Know About This Team
- You'll work on a small, high-impact team with significant scope for ownership and growth
- The team is deeply embedded with customers and product, giving you direct visibility into how your work solves real problems
- You'll collaborate closely with the science team on LLM prompts, graphs, and debugging, blending traditional engineering with AI innovation
- The product is at an inflection point (moving from 0-to-1 to 1-to-N), meaning your contributions directly shape how the platform scales
Our Team’s Favorite Perks and Benefits
- Opportunity to work on cutting-edge AI agent technology with measurable customer impact
- Collaborative environment with direct access to product, science, and customer teams
- Flexible work arrangement with Seattle office location
- Professional development support including access to AI tools and learning resources
- Competitive salary, performance bonuses, generous 401(k) Match
- Wellness Reimbursement for $300 per quarter for wellness activities including gym memberships, spa massages, workout equipment, meditation apps, and much more.
- $1800 Experience bonus to be used for an “Experience” of your choosing
The Qualtrics Hybrid Work Model: Our hybrid work model is elegantly simple: we all gather in the office three days a week; Mondays and Thursdays, plus one day selected by your organizational leader. These purposeful in-person days in thoughtfully designed offices help us do our best work and harness the power of collaboration and innovation. For the rest of the week, work where you want, owning the integration of work and life. #hybrid
Qualtrics is an equal opportunity employer meaning that all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other protected characteristic.
Applicants in the United States of America have rights under Federal Employment Laws:Family & Medical Leave Act, Equal Opportunity Employment, Employee Polygraph Protection Act
Qualtrics is committed to the inclusion of all qualified individuals. As part of this commitment, Qualtrics will ensure that persons with disabilities are provided with reasonable accommodations. If reasonable accommodation is needed to participate in the job application or interview process, to perform essential job functions, and/or to receive other benefits and privileges of employment, please let your Qualtrics contact/recruiter know.
Not finding a role that’s the right fit for now? Qualtrics Insiders is the one-stop shop for all things Qualtrics Life. Sign up for exclusive access to content created with you in mind and get the scoop on what we have going on at Qualtrics - upcoming events, behind the scenes stories from the team, interview tips, hot jobs, and more. No spam - we promise! You'll hear from us two times a month max with fresh, totally tailored info - so be sure to stay connected as you explore your best role and company fit.
For full-time positions, this pay range is for base per year; however, base pay offered within this range may vary depending on location, job-related knowledge, education, skills, and experience. A sign-on bonus and restricted stock units may be included in an employment offer. Full-time employees are eligible for medical, dental, vision, life and disability, 401(k) with match, paid time off, a wellness reimbursement, mental health benefits, and an experience bonus. For a detailed look at our benefits, visit Qualtrics US Benefits.
Washington State Base Annual Pay Transparency Range
$168,000—$221,000 USD
Qualtrics San Francisco, California, USA Office
San Francisco, CA, United States
Similar Jobs at Qualtrics
Artificial Intelligence • HR Tech • Information Technology • Software • Business Intelligence
Lead development of a Semantic Brain: map raw event logs to industry ontologies, build concept/identity-anchored graphs, design reward extraction and off-policy evaluation, deploy agentic LLM orchestration and scalable data pipelines to enable autonomous enterprise decisioning.
Top Skills:
SparkAws NeptuneCdkCloudFormationEc2LambdaLangchainLlamaindexPysparkPythonS3ScalaSparqlTerraform
Artificial Intelligence • HR Tech • Information Technology • Software • Business Intelligence
Design, implement, and productize scalable ML models and services. Build and maintain large-scale data pipelines and microservices. Collaborate with product and engineering teams, prototype solutions, lead design reviews, mentor junior engineers, and stay current with ML research to improve customer-facing features and platform capabilities.
Top Skills:
AIC#Data PipelinesJavaMachine LearningMicroservicesPythonPyTorchTensorFlow
Artificial Intelligence • HR Tech • Information Technology • Software • Business Intelligence
The Principal Program Manager will lead cross-functional programs across Product, Experience Design, and Engineering, ensuring alignment and operational excellence. Responsibilities include defining strategy, managing dependencies, and driving program success through influence and accountability. The role focuses on process improvement and stakeholder communication to enhance product development and delivery.
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
