Top Staff Data Engineer Jobs in San Francisco, CA

Own Eve’s data platform end to end, including ingestion, orchestration, incremental dbt models, Snowflake administration, Terraform infrastructure, observability, access controls, and pipeline reliability. Build schema-change detection, freshness SLAs, CI environments, and incident processes while managing performance and compute costs. Partner with analytics engineers and stakeholders to maintain trusted data for reporting and AI systems, using AI-assisted development and documenting platform standards.

Staff Data Engineer responsible for setting technical direction, architecting scalable data platforms, leading cross-functional initiatives, improving reliability and performance, and enabling AI-assisted analytics and development. The role includes hands-on development with Python, SQL, cloud data warehouses, orchestration, infrastructure as code, Kubernetes, Kafka, and streaming pipelines, along with mentoring, technical advocacy, documentation, code reviews, and on-call troubleshooting.

Lead the architecture and standards for Adyen’s data foundations. Design shared data models, semantic layers, data contracts, lineage, and data products; establish metric definitions and quality standards across domains. Build prototypes, solve complex data-stack challenges, influence teams through technical leadership, partner with cross-functional stakeholders, and mentor engineers while remaining hands-on.

Design and build petabyte-scale data pipelines for genomic and imaging data, transforming public and proprietary datasets into AI-ready training data. Develop agent-based curation and quality-control systems, data cataloging tools, and reliable observable infrastructure. Collaborate with AI researchers and scientists to define data requirements and integrate diverse biological datasets while addressing performance, bandwidth, scalability, and scientific correctness.

Lead the architecture and roadmap for Slack’s metrics platform, including ingestion, canonical datasets, semantic layers, APIs, and metric serving. Design scalable pipelines and data models processing billions of records with strong quality, reliability, freshness, and cost controls. Govern metric certification, lineage, and ownership while enabling people and AI agents to discover and query trusted metrics. Lead cross-functional technical initiatives, influence organization-wide standards, and mentor senior engineers.

Own end-to-end client data migrations: reverse-engineer legacy ETL, design and build Airflow DAGs and Spark jobs, validate and reconcile data at scale, document architecture and decisions, and coordinate cross-functional teams through go-live and hypercare.

Lead Cohere’s Data Foundation pillar, managing a team responsible for the full data lifecycle. Own the roadmap, BigQuery warehouse, Airflow and Cloud Run orchestration, dbt modeling, Looker and API serving, Terraform infrastructure, governance, access controls, reliability, incident response, and data freshness SLAs. Lead development of an agentic analytics platform and evaluation framework, establish company-wide analytics standards, and partner with product, engineering, security, legal, and business teams.

Lead financial data engineering initiatives involving commissions, incentive programs, and backend financial systems. Build scalable, accurate, and reliable data pipelines, platform tooling, partner integrations, analytics infrastructure, and data science infrastructure. Shape technical strategy and roadmaps, collaborate with Finance, Product, Data Science, and Engineering, mentor engineers, contribute code, lead incident response, and establish engineering standards.

Own the architecture and operation of 9amHealth’s data platform, including Python and SQL ETL/ELT pipelines, AWS infrastructure, Redshift data models, semantic layers, governance, quality, monitoring, and reliability. Establish technical standards, optimize performance and costs, support AI/ML initiatives, build integrations, and mentor data team members. Partner with business and clinical stakeholders to deliver reliable data products and translate complex questions into actionable analytics.

Own the technical design and operation of Palmetto’s governed data platform. Build trusted data products, semantic models, ingestion and transformation pipelines, secure access interfaces, and agent-ready data services. Establish governance, masking, authorization, observability, CI/CD, and audit readiness. Partner with finance, operations, product, and engineering teams while mentoring engineers and setting cross-team technical direction.

Lead the architecture, scalability, reliability, performance, and cost efficiency of Cisco’s data platform. Define multi-quarter technical direction for storage, streaming, compute, observability, and performance tooling. Establish standards for schemas, migrations, SLOs, tenancy, retention, and capacity planning; lead major distributed-systems initiatives and production investigations; make technology and cost decisions; and mentor engineers across teams.

Lead design and build of scalable batch and streaming data pipelines for multimodal and spatial AI. Enable dataset curation, versioning, lineage, and data quality for research-to-production ML workflows. Partner with researchers and architects, run proofs‑of‑concept, optimize cost and performance, and mentor engineers to maintain observable, reproducible AI data systems.

Own Moxie’s end-to-end data infrastructure, including pipelines, Snowflake warehousing, dbt and Omni modeling, orchestration, observability, and documentation. Partner with Engineering and Finance on schema changes, accounting logic, metrics, and business-critical reporting. Deliver financial or historical EMR data models, establish SOPs and reliability KPIs, optimize performance and cost, and help advance Practice Intelligence toward predictive and prescriptive products with appropriate AI safeguards.

Leads the architecture and evolution of scalable data platforms, pipelines, and data products across multiple teams. Designs Snowflake and cloud-based solutions for batch, streaming, CDC, ingestion, transformation, orchestration, and data delivery. Establishes engineering standards, governance, observability, testing, CI/CD, and reliability practices; mentors engineers; guides architecture and roadmap decisions; resolves complex incidents; and evaluates AI, automation, and MCP-enabled data capabilities.

Design, build, and scale reliable data infrastructure, including event ingestion, stream processing, data warehouses, ETL pipelines, analytics tools, and machine learning systems. Optimize performance, reliability, security, and data quality across AWS regions and data centers. Collaborate with analytics and data science teams, troubleshoot complex systems, influence architecture, conduct code reviews, and mentor data engineers while helping establish engineering best practices.

Lead operational reliability and platform enablement for Databricks: build monitoring, CI/CD, deployment standards, compute and job policies, observability, runbooks, and governance to support secure, cost-aware, production data workloads across regulated environments. Mentor engineers and align platform with cloud/infrastructure and compliance requirements.

Lead architecture, development, reliability, and improvement of foundational data platforms covering storage, streaming, caching, indexing, and warehousing. Collaborate with cross-functional stakeholders, deliver platform features, improve engineering practices and tooling, present technical guidance, and mentor engineers. The role requires extensive distributed systems, cloud, database, containerization, deployment, and large-scale data platform experience, with emphasis on technical leadership and organizational collaboration.

Own the architecture, roadmap, and production operation of GridCARE’s AWS-based data platform. Design ingestion, data models, storage, compute, APIs, distributed processing, governance, observability, and reliable serving layers for energy data. Build and operate AI coding agent teams, establish platform standards, mentor engineers, and partner with product and domain experts. The role requires hands-on backend, distributed systems, data infrastructure, ETL/ELT, AWS, and technical leadership experience.

Own the technical direction and delivery of WEX’s data acquisition platform. Build reusable, high-performance batch, CDC, and streaming ingestion services with schema validation, reconciliation, recovery, lineage, security, observability, and auditability. Lead architecture and code reviews, establish testing and CI/CD practices, mentor engineers, resolve cross-team dependencies, and support production escalations. The role requires hands-on development of scalable distributed systems and configuration-driven connectors.

Lead the architecture and delivery of GitLab’s data products, including near-real-time data ingestion, knowledge graph integrations, data marketplaces, observability, and data-sharing APIs. Translate ambiguous 0-to-1 initiatives into roadmaps, resolve cross-team dependencies, ensure operational readiness across deployment surfaces, and mentor engineers. The role requires deep full-stack expertise, technical leadership, and experience with AI-powered orchestration and multi-tenant SaaS systems.

Provides hands-on technical leadership for foundational data engineering and analytics engineering. Owns multi-year data architecture, production pipelines, dimensional models, and trusted metrics spanning cloud costs, traffic, infrastructure, and AI usage. Designs new datasets integrating cost, utilization, performance, and reliability signals; establishes standards, coaches engineers, aligns stakeholders, and drives adoption across Infrastructure, Finance, Data Science, and Product Engineering.

Lead the technical direction and hands-on development of scalable data pipelines, data-serving platforms, and AI infrastructure. Design systems for financial data ingestion, transformation, and consumption; improve performance, reliability, security, observability, and cost efficiency. Lead cross-team initiatives, establish engineering standards, resolve complex production issues, mentor engineers, and guide the adoption of AI-assisted development and governed AI applications.

Designs and maintains scalable ETL pipelines, data models, integrations, cloud databases, and healthcare data infrastructure. Ensures security, HIPAA/HITRUST compliance, reliability, monitoring, and documentation while collaborating with product, engineering, and clinical teams. Requires strong SQL and Python expertise, modern data warehousing and orchestration experience, cloud platform knowledge, and familiarity with healthcare interoperability and claims data.

Build and operate a scalable data platform supporting pipelines, reporting, and analytics across business teams. Responsibilities include defining canonical data models, implementing workflow orchestration with tools such as Airflow, improving platform infrastructure, troubleshooting complex data and system issues, enforcing data security controls, and mentoring junior engineers. The role requires strong software engineering, cloud, infrastructure, communication, and data pipeline expertise.

Lead the design, development, deployment, monitoring, and improvement of interconnected customer lifetime value models covering conversion, retention, premiums, and claim losses. Serve as a hands-on technical leader, guide modeling strategy, evaluate experiments and business impact, partner with engineering and business teams, support production ML systems, establish technical standards, and coach other data scientists.