Top Senior Data Engineer Jobs in San Francisco Bay Area, CA

Build and maintain production data pipelines that transform robot logs and telemetry into reliable, queryable datasets. Design Spark-based ETL workflows, orchestrate processing with Airflow or similar tools, manage gold-layer datasets, and implement data quality, schema evolution, monitoring, and metadata practices. Collaborate with robotics, hardware, reliability, analytics, and ML teams on data modeling, fleet insights, debugging, and root-cause analysis. The role may also contribute to Python, Java, or C++ libraries supporting data capture and consumption.

Designs and leads production-grade batch, incremental, streaming, and real-time data pipelines across Microsoft Fabric and Azure. Builds Delta Lake medallion architectures, dimensional and Data Vault models, CDC and incremental-loading solutions, CI/CD automation, governance, security, observability, and data-quality controls. Supports migrations to Fabric, prepares governed data for AI and RAG workloads, and leads client-facing engineering workstreams, estimates, risk management, mentoring, and operational handover.

Own PushPress’s data platform, including ingestion, orchestration, cloud warehouse performance, governance, privacy, data quality, monitoring, and AI-ready data foundations. Build reliable pipelines integrating product, payment, and third-party data; establish data contracts, access controls, auditability, and semantic definitions; support real-time analytics and trustworthy reporting. Partner with Engineering, Data, and AI teams while maintaining infrastructure, incident response, documentation, and operational runbooks.

Design, build, operate, and optimize large-scale data pipelines and foundational data platforms for healthcare datasets. The role focuses on Python, SQL, Spark, Airflow, AWS, storage optimization, schema evolution, data quality, observability, troubleshooting, CI/CD, and production support. The engineer will develop trusted data products, improve platform reliability and efficiency, contribute to architecture and technical direction, collaborate cross-functionally, and mentor teammates while using AI-assisted engineering tools.

Build and own production data infrastructure for a healthcare AI platform. Responsibilities include REST API ingestion, Azure Event Hubs streaming into Databricks, Delta Lake medallion pipelines, entity resolution, semantic metrics, data-quality instrumentation, reconciliation, schema-drift handling, and troubleshooting source-to-warehouse discrepancies. The role requires strong Python and SQL skills, independent pipeline ownership, and experience with Databricks, Spark, PySpark, streaming, and healthcare-related data is a plus.

Designs and maintains data pipelines, data lakes, ETL processes, data integration workflows, and enterprise data platforms. Transforms and optimizes data using Databricks, Python, and SQL. Develops dashboards, reports, scorecards, and KPI solutions with Amazon QuickSight and Tableau. Works with PostgreSQL, Oracle, AWS RDS, and Starburst while ensuring data quality, performance, security, governance, and effective data modeling.

Build and scale production data systems spanning laboratory, manufacturing, battery, and business data. Responsibilities include developing ELT pipelines, architecting secure AWS data movement, modeling warehouse data, optimizing SQL, managing infrastructure as code, implementing monitoring and data quality controls, and partnering with scientists and engineers. The role requires strong Python, SQL, cloud, software engineering, and end-to-end ownership skills.

Lead Cohere’s Data Foundation pillar, managing a team responsible for the full data lifecycle. Own the roadmap, BigQuery warehouse, Airflow and Cloud Run orchestration, dbt modeling, Looker and API serving, Terraform infrastructure, governance, access controls, reliability, incident response, and data freshness SLAs. Lead development of an agentic analytics platform and evaluation framework, establish company-wide analytics standards, and partner with product, engineering, security, legal, and business teams.

Build and operate a Kubernetes-based data platform spanning lakehouse storage, ingestion, transformation, entity resolution, lineage, and data quality. Develop batch and streaming pipelines across heterogeneous and disconnected environments, create full-stack tools for subject matter experts, and expose documented APIs for applications and AI agents. Collaborate with product, AI/ML, infrastructure, mission partners, and field teams while mentoring engineers and supporting secure, air-gapped deployments.

Design and operate scalable data pipelines, transformation workflows, layered data models, and production data infrastructure supporting analytics, experimentation, and AI/ML. Implement monitoring, data quality, observability, and incident response processes; modernize legacy systems; collaborate with product, analytics, and engineering teams; and mentor engineers through reviews and architectural discussions.

Build and guide data platform capabilities supporting marketing, finance, analytics, and data science. Develop data collection, integration, scoring, experimentation, analytics, and infrastructure solutions; contribute code, lead cross-functional technical initiatives, improve system resiliency through incident response, and establish engineering standards. Partner with business and technical stakeholders to define strategy and long-term roadmaps.

Lead financial data engineering initiatives involving commissions, incentive programs, and backend financial systems. Build scalable, accurate, and reliable data pipelines, platform tooling, partner integrations, analytics infrastructure, and data science infrastructure. Shape technical strategy and roadmaps, collaborate with Finance, Product, Data Science, and Engineering, mentor engineers, contribute code, lead incident response, and establish engineering standards.

Own and evolve UJET’s data platform, including MySQL CDC replication to BigQuery, dbt transformations, Terraform infrastructure, pipelines, data quality, observability, and Looker models. Build reliable, analytics-ready datasets and trusted metrics for internal and customer-facing reporting. Partner with Analytics, Finance, Product, and Engineering to deliver scalable data solutions, improve platform reliability, optimize performance and costs, and establish development standards and guardrails.

Designs, develops, and maintains scalable cloud-based data pipelines and enterprise data solutions. The role covers data ingestion, transformation, orchestration, integration, data lakes, cloud warehouses, batch and streaming processes, performance optimization, governance, and lineage. The Senior Data Engineer collaborates with analysts, data scientists, engineers, stakeholders, contractors, and vendors while mentoring junior staff. Core tools include Databricks, dbt, Azure Data Factory, SQL, Python, and Azure.

Own and evolve WorkOS’s internal data platform, including ingestion, orchestration, Snowflake, dbt models, data governance, reverse ETL, semantic views, and AI-enabled data workflows. Ensure pipeline reliability, freshness, security, and scalability while partnering with Product, Finance, RevOps, GTM, Engineering, and Security. Build monitoring, runbooks, CI/CD gates, access controls, masking policies, and infrastructure automation across AWS and Kubernetes.

Build and scale secure data lakehouse and warehouse infrastructure supporting analytics, machine learning, clinical AI performance monitoring, and self-service insights. Develop scalable data pipelines, models, orchestration, quality controls, and LLM-enabled analytics tools using heterogeneous clinical and EHR data. Collaborate with engineering, data science, clinical, product, and client success teams to establish trustworthy metrics and technical solutions.

Design, build, and operate production data pipelines using Python, Spark SQL, and Airflow. Canonicalize healthcare and customer data into trusted internal models, optimize distributed processing, enforce schemas, and implement quality checks and observability. Support customer onboarding, streaming pipelines, troubleshooting, on-call operations, and platform best practices while collaborating with product, analytics, engineering, data science, and customer teams.

Own the analytics data layer by translating stakeholder needs into certified dimensional models, governed metrics, semantic layers, dashboards, and reports. Build with dbt, SQL, Databricks, Power BI, and Airflow while ensuring data quality, observability, reconciliation, documentation, and dependable orchestration. Partner closely with finance, operations, and client-facing teams, automate recurring workflows, and use AI-assisted engineering practices in a fully remote environment.

Own the technical design and operation of Palmetto’s governed data platform. Build trusted data products, semantic models, ingestion and transformation pipelines, secure access interfaces, and agent-ready data services. Establish governance, masking, authorization, observability, CI/CD, and audit readiness. Partner with finance, operations, product, and engineering teams while mentoring engineers and setting cross-team technical direction.

Own and scale Collective’s data platform by building reliable batch and event-driven pipelines into BigQuery, modeling data with dbt, and enforcing quality, observability, security, and cost standards. Partner with engineers, analysts, and business stakeholders to deliver trustworthy datasets, metrics, reporting, and AI-enabled warehouse access. The role includes production ownership, incident response, governance, infrastructure practices, and support for analytics and LLM-based tools.

Designs, builds, and maintains Databricks data pipelines and analytics solutions using SQL and Python. Collaborates with development and IT teams, applies DevOps and CI/CD practices, supports Power BI dashboards, and contributes to data modeling, management, and governance. The role requires extensive Databricks and cloud-platform experience, strong communication skills, fluent English, and at least five years of relevant experience.

Build scalable data ingestion, transformation, and loading infrastructure using Spark, SQL, AWS, and Databricks. Develop entity resolution systems, CI/CD pipelines, anomaly detection, and high-quality data products. Design scalable processing and storage patterns, orchestration workflows, and data warehousing solutions while collaborating remotely and documenting technical designs.

Designs, develops, maintains, and optimizes complex data systems, SQL databases, ETL processes, data warehouses, reports, and .NET applications supporting pharmacy benefit management. Responsibilities include database performance tuning, troubleshooting pharmacy claims processes, developing scalable solutions, documenting technical designs, participating in Agile ceremonies, collaborating with stakeholders, mentoring teammates, and ensuring healthcare data compliance.

Build and operate reliable data systems and products supporting clinical operations, drug evaluation, business development, analytics, machine learning, and AI agents. Responsibilities include designing pipelines, canonical data models, data contracts, warehouse transformations, quality and observability practices, governance for regulated data, and incident response. The role partners closely with Product Engineering and Data Science, uses AI-assisted development, contributes to architecture, and mentors other engineers.

Design, build, deploy, and maintain AWS-based data and AI/ML infrastructure for enterprise agentic AI platforms. Develop data lake pipelines, observability systems, agent-enablement tooling, and production MLOps/LLMOps capabilities. Own features end-to-end, contribute to architecture and design decisions, optimize reliability, scalability, security, and cost, and collaborate with AWS Professional Services, client teams, and junior engineers.