Top Staff Data Engineer Jobs in San Francisco, CA

Build and maintain ELT pipelines, data models, semantic layers, and personalization signals for client retention and marketing tests. Partner with machine learning, product, marketing, design, and architecture teams to prepare predictive data, connect signals to CRM channels, support real-time event processing, and document a scalable data foundation. Use AI coding assistants daily to accelerate development, testing, and reviews in a fast-moving discovery environment.

Designs and operates batch, streaming, and real-time data platforms and pipelines using Spark, Kafka, Iceberg, Airflow, and cloud infrastructure. Owns data lake evolution, data quality, observability, reliability, governance, event instrumentation, schema management, and privacy controls for PII, retention, deletion, and residency. Leads complex initiatives end-to-end, troubleshoots production issues, mentors engineers, and develops AI agent harnesses that enforce data engineering standards and quality gates.

Lead design, build, and deploy of large-scale data platforms for LLMs, RAG, and agentic AI systems. Hands-on coding, architecting fault-tolerant pipelines, establishing MLOps/DataOps best practices, mentoring engineers, and operationalizing research into production across Exabyte-scale distributed systems.

Own and evolve Afresh’s analytics platform, including data architecture, dimensional models, semantic-layer metrics, pipelines, data quality, governance, and AI-assisted analytics tooling. Collaborate with data, product, engineering, and operations teams to support internal and customer-facing data products. Lead high-impact projects, optimize billion-row datasets, establish reliable analytics practices, and mentor other engineers.

Lead design and scaling of the data platform and analytics pipelines. Build and maintain ETL/ELT workflows, integrations, serverless data services, and reliable data models. Mentor teammates, improve automation, observability, and data governance to support analytics and operational insights.

Architect and operate BILL’s enterprise data platform across ingestion, lake storage, batch and streaming processing, query, feature store, knowledge graph, and search layers. Drive platform-wide technical decisions, migrations, standards, reliability, cost efficiency, and self-service capabilities. Partner with ML, Risk, Payments, and Analytics teams to deliver scalable solutions, lead complex initiatives from inception through production, and mentor senior and staff engineers.

Build and operate production machine learning systems for ranking, retrieval, recommendations, search, propensity, churn, LTV, and next-best-action decisioning. Design reliable signal contracts covering freshness, provenance, confidence, eligibility, and calibration. Lead feature pipelines, model serving, experimentation, monitoring, and feedback loops while evaluating fairness, risk, compliance, trust, and long-term customer impact. Collaborate across product, growth, data, platform, modeling, risk, and compliance teams.

Build and operate production machine learning systems for ranking, retrieval, recommendations, search, propensity, churn, lifecycle intelligence, and next-best-action decisioning. Design reliable signal contracts with freshness, provenance, confidence, and calibration guarantees. Lead experimentation, monitoring, feedback loops, and impact evaluation focused on fairness, trust, risk, compliance, and long-term engagement. Partner across product, growth, data, platform, modeling, risk, and compliance teams.

Architect, build, and operate low-latency market data systems, including feed handlers, normalization, distribution, and exchange connectivity. Design high-throughput infrastructure for real-time trading data, improve reliability and performance, participate in observability, on-call, and incident response, and deliver well-tested features end to end. Partner with engineering, product, and cross-functional stakeholders while leading a lean, high-impact team.

Own and architect large-scale data pipelines, orchestration, and integrations. Drive IaC strategy (Terraform), third-party API and AI/LLM tool integrations, build export pipelines and internal tooling, mentor engineers, and align cross-team technical strategy for reliable, scalable data infrastructure.

Lead the architecture and scaling of Overstory’s data platform for AI-powered vegetation analysis. Design and operate large-scale geospatial and temporal data pipelines, orchestration systems, event-driven workflows, service-oriented architecture, data quality frameworks, and observability practices. Drive architectural standards, system design, scalability, reliability, and traceability while mentoring engineers and collaborating with data, ML, and product teams.

Design and implement scalable data integrations and transformation layers for cybersecurity products. Build data contracts, dimensional/incremental models, observability, and lineage tooling. Collaborate with product engineering to integrate SIEMs (Sentinel, Splunk) and deploy cloud/on-prem solutions on Kubernetes/Docker, processing medium-to-large datasets and stream/message systems.

Build and ship production AI systems across retrieval, knowledge graphs, inference, agentic workflows, and data platforms. Develop customer-context retrieval, scored signals, evaluation and observability capabilities, and workflows that turn insights into action. Work hands-on with large and small models, durable orchestration, cloud deployment, inference-cost optimization, and high-volume observability data. Establish reusable engineering patterns, collaborate with product and platform teams, and mentor through strong technical contributions.

Design and maintain data pipelines, data contracts, and transformation layers for analytics. Model complex sources, enable observability and lineage, collaborate with product teams, and optimize data structures for scalable cybersecurity and industrial datasets.

Lead the architecture and technical direction of a cloud data platform spanning infrastructure, Databricks, CDC streaming, pipelines, data products, developer experience, and governance. Resolve cross-team technical challenges, establish platform standards, influence roadmaps, drive adoption, mentor senior engineers, and lead complex multi-squad initiatives with minimal direction.

Lead architecture and hands-on engineering to design, scale, and govern a cloud-native lakehouse using Databricks and Microsoft Fabric. Build batch/streaming pipelines, ensure HIPAA-compliant PHI handling, produce analytics-ready datasets, mentor engineers, and drive platform strategy, governance, and reliability.

Designs and leads scalable data platforms, event-driven services, orchestration pipelines, and DataOps practices using Python, Airflow, Snowflake, Kafka, and related technologies. The role establishes engineering standards, governance, security, observability, cost efficiency, and AI-assisted development practices. It also mentors engineers, drives cross-functional initiatives, modernizes data ingestion, and ensures reliable, compliant, and well-governed systems.

Lead the data engineering strategy and architecture for distributed pharmaceutical data systems. Design and scale pipelines, APIs, and data platforms while ensuring reliability, governance, quality, and PII/PHI protection. Drive technical decisions, improve developer efficiency, deliver business impact, influence cross-functional roadmaps, mentor engineers, communicate with executives, and promote organizational adoption of new technologies and processes.

Leads the design and development of Mozilla’s data platform, processing tens of terabytes daily. Builds GCP workflows using Dataflow, BigQuery, Pub/Sub, GKE, and Airflow; evaluates technologies; establishes engineering standards; partners with Data Science, Product, Legal, Policy, and Engineering; develops data abstractions; mentors engineers; and supports evolving data products.

Design and improve large-scale data processing and platform systems handling tens of terabytes daily. Build GCP workflows using Dataflow, BigQuery, Pub/Sub, GKE, and Airflow; evaluate technologies; establish engineering standards; lead cross-functional projects; mentor engineers; collaborate with Data Science; and develop data abstractions and active data products.

Build and operate AI-first full-stack applications, internal tools, service connectors, automations, and data products supporting business functions. Translate operational and financial reconciliation needs into scalable software solutions, integrate business systems, improve engineering processes, and own products across the full lifecycle. The role requires production engineering experience, data warehouse and SQL expertise, and familiarity with frontier AI tooling, with opportunities to evaluate LLM systems and work with platforms such as Snowflake, dbt, and enterprise business applications.

Build and operate streaming and batch data pipelines for energy-storage telemetry. Design reliable, query-ready time-series and event datasets using open table formats, orchestration frameworks, AWS, and infrastructure-as-code. Lead technical designs, partner with field operations and engineering teams, support on-call operations, and mentor engineers while improving data reliability and engineering practices.

Lead architecture and delivery of scalable data infrastructure: design and optimize ETL/ELT pipelines, enterprise data models, and data quality frameworks. Define platform standards, mentor engineers, drive cross-functional initiatives, and partner with analytics, data science, and engineering stakeholders to ensure reliable, production-grade data assets.

Sets technical direction and architectural coherence across Netflix’s Ads Data Engineering organization. The role guides cross-domain data system decisions, partners with engineering and business leaders, mentors data engineers, shapes platform-wide initiatives, and advances AI-native data engineering. Responsibilities span modern data foundations, ad-tech data flows, privacy and governance, platform migrations, scalable data products, and collaboration with Ads Platform Engineering, Data Science, and Product teams.

Lead design and operation of a cloud-native data platform: set technical direction, build scalable ingestion/processing/warehousing pipelines (batch & streaming), support ML data needs, mentor engineers, drive projects end-to-end, run planning/standups/on-call, and eventually lead a data engineering team.