Top Staff Data Engineer Jobs in San Francisco, CA

Build and modernize Carrum Health’s data platform by developing Python and SQL pipelines, AWS services, Terraform infrastructure, dbt transformations, observability, testing, and CI/CD. Automate carrier integrations, reduce operational toil, improve reliability, and support Databricks adoption. Extend AI-assisted ingestion and classification using Bedrock and collaborate across engineering, analytics, data science, and client-facing teams. The role includes production ownership, incident response, technical documentation, and healthcare data platform development.

Leads the design, development, deployment, and operation of scalable machine learning solutions and high-performance microservices. Builds real-time inference systems using Triton and TensorRT, guides distributed cloud architectures, participates hands-on in coding and reviews, and mentors engineers. Drives technical strategy, operational excellence, system resilience, and business-wide innovation across generative AI, graph machine learning, and big data initiatives.

Lead design and operation of a cloud-native data platform: set technical direction, build scalable ingestion/processing/warehousing pipelines (batch & streaming), support ML data needs, mentor engineers, drive projects end-to-end, run planning/standups/on-call, and eventually lead a data engineering team.

Own production-grade analytics pipelines from design through maintenance, build dbt-based analytical models using Kimball dimensional modeling, and deliver reliable data features. Partner with Product, Engineering, and Data Science teams to translate retail signals into business insights. Establish standards for monitoring, documentation, reproducibility, and code quality. Use AI tools to accelerate development and prototype conversational data experiences for natural-language retail data exploration.

Own and scale the company’s analytics infrastructure, including BigQuery data warehouses, dbt data marts, external data exchange, DataOps, data quality, governance, security, and cloud cost optimization. Partner with application database teams, analytics engineers, and data scientists to enable reliable metrics and scalable client data delivery.

Leads the design and delivery of scalable data pipelines, dbt models, orchestration workflows, and analytical data models. Establishes data architecture, engineering standards, quality frameworks, governance, observability, and self-service analytics practices. Partners with stakeholders, leads technical reviews, mentors engineers, supports hiring, oversees analyses, and participates in incident response and continuous improvement.

Staff Data Engineer responsible for defining data architecture strategy, building fault-tolerant batch and real-time streaming infrastructure, and managing scalable lakehouse and database systems. The role processes telemetry from millions of connected devices, drives performance tuning and data quality, supports on-call operations, evaluates modern technologies, and mentors engineers. It requires strong expertise in Python, SQL, cloud data platforms, streaming systems, infrastructure as code, and enterprise-scale architecture.

Conduct frontier AI research and build research-grade datasets, prototypes, tooling, reinforcement learning environments, benchmarks, and evaluation frameworks. Design rigorous experiments, train and evaluate models, investigate synthetic and agentic data generation, and translate findings into scalable products. Collaborate with research, engineering, product, and operations teams, communicate technical insights, contribute to publications and open-source work, and mentor peers. This is a fully onsite role based in San Francisco, Palo Alto, or Seattle.

Own the architecture and technical direction of Northbeam’s data ingestion and integration platform. Design scalable API-based ETL, event-driven and batch processing systems; establish standards for observability, freshness, and failure handling; and guide build-versus-buy decisions. Collaborate across engineering, infrastructure, product, and business teams while mentoring engineers, leading design reviews, and driving complex platform initiatives to completion.

Lead the technical strategy for Coinbase's Data Platform, architecting data systems for AI/ML workloads and ensuring data reliability, performance, and cross-functional alignment.

Design, build, and own business-critical data infrastructure, including AWS Glue and Spark pipelines, cloud warehouse or lakehouse systems, orchestration, and Tableau BI platforms. Define event-driven contracts, data models, schemas, observability, and batch or streaming architectures across multi-tenant, multi-region environments. Collaborate across producer and consumer teams while owning production data infrastructure and outcomes end to end.

Design, build, optimize, and scale Ray Data’s distributed data processing infrastructure for large-scale AI training and inference. Responsibilities include distributed execution, scheduling, resource management, data partitioning, fault tolerance, performance optimization, and system architecture. The role also involves collaborating with customers and AI-native companies to solve complex workload-scaling challenges.

Lead design and implementation of Jane's customer data infrastructure: own event collection, pipelines, CDP tooling, schema governance, identity resolution, and attribution. Work cross-functionally with marketing, BI, product, and engineering to set standards, evaluate tooling (e.g., RudderStack), ensure privacy-compliant instrumentation, and build reliable, consistent data delivery to the warehouse.

As a Senior Staff Data Engineer, you'll set the technical vision for data architecture, lead initiatives across teams, mentor engineers, and ensure platform reliability, while integrating data systems with ML pipelines effectively.

Build and operate full-stack interfaces, backend services, APIs, data pipelines, dashboards, and monitoring for human feedback and RL data collection. Partner with RL researchers to translate data needs into reliable collection campaigns and training-ready datasets. Own projects end to end, improve system latency and usability, and identify bottlenecks between data collection and model training.

Lead architecture, systems, and roadmap for Coinbase's Compliance Reporting Data Platform. Design and build production-grade data models, pipelines, quality frameworks, validations, and certification processes. Establish data contracts, monitoring, reconciliation, and audit-ready processes. Partner with engineering and compliance stakeholders, respond to regulatory exams, and mentor engineers while shaping standards and documentation.

Build and operate reusable cloud data-platform infrastructure across engineering teams. Design, secure, monitor, optimize, and maintain relational, NoSQL, columnar, streaming, and analytical data systems. Establish database, schema, access, backup, recovery, privacy, and observability standards; improve infrastructure-as-code and operational tooling; and guide datastore selection and architecture. Lead cross-team initiatives, reduce technical debt, mentor senior engineers, document best practices, and influence technical strategy without direct management authority.

As a Staff Software Engineer, you'll lead the architecture and implementation of core systems and data platforms, ensuring adaptability and planning technical strategies.

Leads the architecture, development, operation, and evolution of PlayStation’s real-time analytics platform. Builds scalable distributed data systems using stream processing and high-performance analytics technologies, integrates batch and streaming workloads, applies AI/ML to improve platform operations, and drives reliability and performance. The role owns critical platform capabilities, establishes technical standards, mentors engineers, influences architecture across teams, supports production incidents, and shapes long-term data platform strategy.

Technical leader for Data Engineering responsible for designing, operating, and optimizing Snowflake-based data platforms, self-hosted orchestration (Prefect), dbt practices, AWS infrastructure, and data pipelines. Drive architecture, mentor engineers, partner cross-functionally, ensure observability and production-grade practices, and enable model-ready datasets and AI/LLM workflows.

The Staff Engineer will lead technical direction on data infrastructure, focusing on ingestion frameworks, storage architecture, and orchestration to support scientific workflows. Responsibilities include mentoring, ensuring reliability, and collaborating with cross-functional teams to establish coding standards and data integrity.

Lead design and optimization of data-intensive distributed systems and query infrastructure. Improve query plans, indexing, caching, and schema/access patterns to reduce P95/P99 latency, ensure scalability, and support AI-powered data products. Mentor through design reviews and operate backend systems in the JVM ecosystem.

Design, implement, and maintain scalable distributed systems and storage for petabyte-scale 3D geospatial data. Build high-performance APIs, data ingestion and processing pipelines, monitoring/operational tooling, and collaborate with cross-functional teams to define requirements and drive the backend roadmap.

Lead architecture and technical direction for data integrations moving clinical and financial data at scale. Design orchestration, pipelines, data stores, observability, security/compliance, and mentor engineers while owning end-to-end cross-system initiatives and reliability.

Lead architecture and implementation of agentic AI systems powering Zuma's platform. Rebuild onboarding and integration frameworks, implement analytics and observability for LLM performance, translate product needs into technical solutions, mentor engineers, establish engineering standards, and work directly with customers to enable scalable, production-grade AI experiences.