Top Senior Data Engineer Jobs in San Francisco Bay Area, CA

Designs and operates an AWS-native lakehouse supporting analytics, reporting, visualization, and AI/ML. Builds batch and streaming ingestion pipelines using Python and PySpark, Apache Iceberg data structures, governance and metadata services, quality controls, security, observability, and CI/CD automation. Optimizes performance, scalability, reliability, and cost while collaborating with cross-functional technical teams and documenting architecture, operations, and secure configurations.

Design and maintain data architecture and pipelines to support compliance and risk teams. Build and optimize data models, standardize metrics, and create data dictionaries. Implement data quality, lineage monitoring, AI-driven agents for false-positive reduction and automation, and participate in on-call rotations to ensure SLAs are met.

Build, deploy, and operate production-grade data pipelines and data products for healthcare audiences. Design transformations, data models, and governed views using Python, SQL, Airflow, S3, Snowflake, and EMR. Implement data-quality, monitoring, and privacy-by-design controls for PHI/PII. Partner with product, analytics, and platform teams to onboard sources, support audience discovery, segmentation, activation, measurement, and troubleshoot production issues.

Design, build, and deliver scalable data pipelines and products end-to-end. Partner with analytics, product, and engineering to ingest diverse data sources, implement dbt transformations, ensure data quality, CI/CD, observability, monitoring, and cost/performance optimization. Leverage AI/automation for self-service tooling and intelligent pipelines while driving stakeholder alignment and reliable production operations.

Own and evolve Afresh’s analytics platform, including data architecture, dimensional models, semantic-layer metrics, pipelines, data quality, governance, and AI-assisted analytics tooling. Collaborate with data, product, engineering, and operations teams to support internal and customer-facing data products. Lead high-impact projects, optimize billion-row datasets, establish reliable analytics practices, and mentor other engineers.

As the founding engineer of the Data Intelligence team, you will design semantic layers, develop AI tooling, and automate data processes to drive business insights.

Lead design and scaling of the data platform and analytics pipelines. Build and maintain ETL/ELT workflows, integrations, serverless data services, and reliable data models. Mentor teammates, improve automation, observability, and data governance to support analytics and operational insights.

Design and implement scalable data pipelines and a SOC2-compliant data warehouse. Support analytics and ML teams, enable real-time and batch ETL, mentor data engineers, and collaborate cross-functionally to drive data-driven decisions.

The Senior Lead Data Engineer designs and develops technical solutions, leads a team of developers, and utilizes big data and cloud technologies to deliver robust solutions for Capital One.

Own and evolve the data infrastructure powering identity and fraud products. Build and operate scalable ETL/ELT pipelines, ensure data quality and observability, optimize storage and access, support product squads, respond to production incidents, mentor engineers, and drive data platform innovation.

Lead the Data Platform Engineering team owning ETL/reverse-ETL pipelines, replication, and production datastores. Define architecture, roadmap, standards (IaC, observability, schema contracts), mentor engineers, drive incident response and durable fixes, and partner on cross-team technical decisions and operational reliability for data correctness, completeness, and timeliness.

Design, build, and operate large-scale, fault-tolerant real-time and batch data pipelines powering an ad-tech DSP. Collaborate with engineers and data scientists to define data models and SLAs, optimize performance and capacity, enforce data quality and monitoring, mentor engineers, and evaluate new data technologies to scale analytics and ML workloads.

Designs and maintains healthcare data pipelines, ETL workflows, databases, data models, and analytical frameworks. Analyzes claims and payer data to identify trends and insights, while ensuring data quality and integrity. Creates reports, dashboards, and visualizations for nontechnical stakeholders and collaborates with healthcare policy experts, analysts, and other stakeholders to translate requirements into data solutions. The role also improves data infrastructure and analytics capabilities.

Build and own the unified data layer: ingestion pipelines into ClickHouse, data modeling and entity resolution, canonical metric definitions, quality checks, and internal data tools. Support ad-hoc analyses and work closely with engineering to ensure stable, trustworthy data powering customer dashboards and product BI.

Design and operate a scalable data platform for high-volume robotics telemetry and control-plane data. Responsibilities include data capture, storage tiers, ingestion, transformation engines, durable processing, schema and ID-key management, retention and egress policies, replication, CI/CD, monitoring, and alerting. The role supports cloud, on-premises, edge, and outage-resilient deployments while enabling downstream analytics and machine learning.

Lead design and operation of large-scale data pipelines and data APIs. Build Spark/PySpark workflows on Databricks, optimize job performance, manage data quality and observability, and develop MCP servers and AI-agent integrations. Mentor engineers, define standards, support production incidents and on-call rotations, and collaborate with stakeholders to deliver scalable data platform products.

Build and maintain IICS ETL/ELT pipelines, develop Amazon Redshift data models, and use SQL and Python for data transformation and automation. Create Power BI reporting models, implement data quality checks, document pipelines, and collaborate with client stakeholders and engineering teams. The role requires autonomous work, strong communication, and the ability to translate technical solutions into business context, with bonus value for Veeva or life sciences experience.

Build and maintain modern data-centric applications, pipelines, transformations, enrichment processes, provisioning layers, and user interfaces supporting clinical and operational healthcare processes. Analyze clinical data and report trends using cloud, big data, data science, and modern software development practices. Collaborate with Product, Platform, and Architecture teams, mentor Data Engineers, document solutions, manage source control, and research emerging technologies.

Design and implement scalable data models and ETL pipelines (batch and streaming), ensure data quality/SSOT, collaborate with product and platform teams to resolve issues, contribute to data governance, and explore practical integrations between data engineering and AI.

Designs and scales cloud-native data platforms and ingestion pipelines using AWS, Databricks, and Spark. Builds APIs, file-upload mechanisms, ETL/ELT workflows, and Lakehouse architectures; supports SQL Server data loading and legacy C# .NET processes. Partners with analytics, software engineering, architecture, and product teams to ensure data quality and availability for reporting and machine learning. Troubleshoots pipelines and applies governance, security, compliance, documentation, and reliability best practices.

Architects and operates petabyte-scale telemetry ingestion, processing, and storage systems. Builds resilient distributed systems and data pipelines using streaming or batch frameworks in Kubernetes. Establishes observability with ELK and Grafana, collaborates across engineering teams, and applies AI-assisted tools to improve development, security, validation, and operational reliability.

Architect, build, and operate low-latency market data systems, including feed handlers, normalization, distribution, and exchange connectivity. Design high-throughput infrastructure for real-time trading data, improve reliability and performance, participate in observability, on-call, and incident response, and deliver well-tested features end to end. Partner with engineering, product, and cross-functional stakeholders while leading a lean, high-impact team.

Design and deliver modern data platforms, scalable and secure ingestion and transformation pipelines, durable data models, streaming systems, and reporting layers. Partner directly with clients and business stakeholders to translate requirements into architecture, explain tradeoffs, and own delivery outcomes. The role includes production implementation across cloud data platforms, orchestration, governance, real-time technologies, and analytics or AI data consumption.

Build and maintain governed Silver/Gold data models and semantic assets on Databricks. Translate stakeholder KPIs into testable, reusable transformations, apply enterprise modeling standards, enable domain self-service, ensure data sensitivity and governance, and review domain-contributed models for production readiness.

Build Sailor Health’s foundational data platform from scratch. Responsibilities include developing Snowflake and dbt pipelines integrating EHR, CRM, and communications data; creating data-powered web applications; automating clinical and billing workflows with Python; analyzing therapy transcripts and patient data using AI and machine learning; tracking clinical outcomes; and designing scalable data architectures in collaboration with clinicians, engineers, founders, and other teams.