Top Senior Data Engineer Jobs in San Francisco Bay Area, CA

Lead design and implementation of enterprise data architecture and cloud-based data solutions. Manage large data engineering projects, develop data models and pipelines, ensure governance and security compliance, advise clients strategically, and build high-performing inclusive teams.

Build and operate reliable data systems and products supporting clinical operations, drug evaluation, business development, analytics, machine learning, and AI agents. Responsibilities include designing pipelines, canonical data models, data contracts, warehouse transformations, quality and observability practices, governance for regulated data, and incident response. The role partners closely with Product Engineering and Data Science, uses AI-assisted development, contributes to architecture, and mentors other engineers.

Build and scale batch and streaming data pipelines using Databricks, Spark, Kafka, AWS, Python, SQL, dbt, and Fivetran. Develop ETL/ELT and reverse ETL workflows, curated datasets, dashboards, data quality controls, and data activation integrations with Salesforce, HubSpot, and Braze. Partner with Finance, Marketing, Product, and Engineering on reporting, forecasting, segmentation, and AI/ML applications, including RAG, embeddings, and agent-based systems.

Build and maintain data pipelines to ingest, normalize, and deliver messy healthcare data (FHIR, HL7, CCDAs, PDFs) into structured models for AI and clinical use. Implement ETL, real-time webhooks, document extraction, PII sanitization, storage on AWS, and monitor data quality while ensuring compliance with healthcare interoperability standards.

Design, develop, deploy, and operate scalable ELT pipelines and data architectures for developer experience products. Integrate and validate diverse data sources, improve pipeline and query performance, establish data governance and quality practices, and build reliable data sources for proactive recommendations. Collaborate with data scientists, analysts, engineers, product managers, and customers while mentoring teams and promoting scalable data engineering standards.

Build and operate scalable data platforms and pipelines supporting AI-driven drug discovery. Ingest, transform, validate, version, and provide traceable chemical, biological, laboratory, and experimental datasets for machine learning workflows. Partner with scientists, engineers, laboratories, and instrument teams to optimize data delivery, ensure accuracy and reproducibility, integrate event-driven systems, and support production database migrations and platform operations.

Design and build maintainable data pipelines, ETL processes, and data architecture standards. Collaborate with product managers, analysts, and machine learning engineers to translate requirements into automated, scalable solutions. Apply software engineering best practices, asynchronous systems, APIs, streaming, testing, monitoring, and Python and SQL expertise. Help prioritize data initiatives based on business impact and communicate complex technical concepts to technical and non-technical stakeholders.

Build and maintain recruiting analytics data infrastructure using BigQuery, dbt, Fivetran, SQL, and Python. Design dimensional models, ETL/ELT pipelines, semantic layers, governance controls, data security, monitoring, and standardized recruiting metrics. Integrate Workday, Greenhouse, and internal systems while supporting batch and real-time analytics. Partner with data scientists, engineers, recruiters, and business stakeholders to deliver trusted data products and AI-enabled workforce insights.

Lead the architecture and development of scalable AWS-based data ingestion, transformation, and orchestration pipelines. Build reliable data infrastructure using Python, SQL, Airflow, Lambda, ECS, SQS, and Terraform. Establish data modeling, quality, lineage, monitoring, testing, and observability practices while partnering with analysts, scientists, and backend engineers. Mentor senior engineers, guide technical decisions, and provide hands-on leadership for complex data platform initiatives.

Build and maintain Databricks-based data platforms, including ingestion, transformation, storage, governance, data modeling, and serving pipelines. Establish medallion architecture standards, canonical data models, quality controls, lineage, schema evolution, and reliable batch or incremental processing. Improve pipeline observability, scalability, idempotency, and recoverability while moving curated data to systems such as ClickHouse. Collaborate across application and analytics teams to create durable, governed production datasets.

Architects, deploys, and operates large-scale Apache Kafka and Confluent streaming platforms. Responsibilities include cluster sizing, configuration, security, automation, observability, HA/DR, troubleshooting, and developer enablement. Partners with application and cross-functional teams to deliver reliable event-driven solutions, translates ambiguous requirements into production-ready designs, participates in code and design reviews, and mentors junior engineers.

Develop and optimize enterprise Oracle SQL and PL/SQL solutions, including data models, database designs, APIs, ETL processes, batch jobs, and performance tuning. Collaborate with cross-functional teams to translate requirements into scalable implementations, support testing and CI/CD, troubleshoot production issues, and document solutions. Provide technical leadership, mentor engineers, participate in architecture and code reviews, and drive engineering best practices across data modernization initiatives.

Designs, builds, and supports scalable batch and streaming data pipelines, integration solutions, data models, APIs, and enterprise data platforms. The role focuses on cloud data architecture, data quality, governance, platform performance, and graph capabilities using Neo4j. It partners with stakeholders, analytics, and data science teams to support operational insights, AI, machine learning, fraud detection, and connected-data use cases.

Design, build, and operate scalable ETL/ELT pipelines and backend data services using Go. Integrate relational, NoSQL, blockchain, API, web, and social data sources; develop NLP pipelines for text extraction and classification; and support production APIs, dashboards, alerts, and intelligence systems. Collaborate with engineering, data science, and investigations teams on architecture, data quality, lineage, validation, monitoring, and client-facing products.

Leads data engineering and data science teams, setting strategy and delivering scalable pipelines, data architecture, models, analytics infrastructure, and data governance. Partners with product, analytics, and business stakeholders to prioritize initiatives, assess upstream changes, maintain data quality, and support Snowflake-based warehousing. This hands-on leadership role includes mentoring staff, contributing to technical codebases, and advancing data science and engineering solutions.

Build and govern Anthropic’s quote-to-cash data foundation. Design canonical models from Salesforce, CPQ, and billing data; establish data integrity standards and SLAs; collaborate with systems engineers on schema changes and ingestion; and create data products, dashboards, and self-service analytics tools for GTM and Finance teams. The role also owns data architecture, translates stakeholder needs into technical requirements, and influences roadmaps.

Design, build, and maintain scalable batch and streaming data pipelines using Python, SQL, Kafka, and GCP. Develop REST and GraphQL data services, CI/CD automation, observability, and data-quality frameworks. Integrate ML and NLP models into production workflows, including continuous retraining and real-time inference. Support structured and unstructured healthcare datasets, distributed systems, microservices, and reliable data infrastructure while collaborating with data science and MLOps teams.

Build and configure a government Databricks workspace, develop ingestion and integration pipelines from enterprise and legacy systems, and prepare data for reporting. Implement Unity Catalog governance, metadata management, auditing, and lineage; configure MLflow workflows; assess security controls for sensitive data; and create architecture documentation, data dictionaries, and SOPs to support long-term sustainment.

Lead the design and optimization of healthcare data pipelines integrating EHR, payer, HIE, and vendor data. Build AWS-based ETL/ELT infrastructure, reusable integration tooling, data quality monitoring, and ML enablement foundations. Ensure HIPAA-compliant handling of clinical data, including security controls, audit logging, encryption, and de-identification. Provide technical leadership, mentor engineers, influence platform architecture, and lead complex partner integrations involving FHIR, HL7v2, CCDA, and healthcare terminology standards.

Design, build, and maintain scalable Spark-based ETL pipelines and computed tables in a central data lake. Integrate structured and unstructured IoT, sensor, and external data for analytics, model training, and dashboards. Collaborate with Data Science, Analytics, and ML teams to ensure reliable, high-quality customer-facing datasets.

Designs, builds, tests, and documents scalable data solutions and pipelines for CMS healthcare fraud, waste, and abuse initiatives. Responsibilities include data warehouse development, CI/CD deployment, orchestration, data quality, governance, AWS-based automation, testing, architecture support, and technical leadership. The role partners with analysts and business stakeholders, leads project initiatives, improves workflow automation, and ensures requirements and milestones are met.

Build and support scalable data platforms and pipelines, leading migrations such as Snowflake to BigQuery and transitioning reporting to Looker. Responsibilities include data architecture, ETL/ELT, API and marketing integrations, data quality, production troubleshooting, warehousing, performance optimization, and platform modernization. The role partners with analytics and business teams, owns projects through production, documents solutions, and provides technical guidance.

Design and lead scalable, secure, highly available database infrastructure for multi-terabyte SQL Server environments. Set technical direction for data architecture, performance, real-time pipelines, security, and growth. Troubleshoot complex production issues, optimize queries and indexes, guide cross-functional projects, mentor engineers, review changes, reduce technical debt, and evaluate emerging technologies. Monitor database health using Splunk, Datadog, and Grafana while influencing engineering and Site Reliability strategy.

Build scalable, secure data pipelines, models, platforms, and governance systems that enable AI applications to access trusted enterprise data. Partner with analytics, marketing, and product teams to translate requirements into technical solutions, independently deliver projects through production, and support sensitive-data security and access controls. Preferred experience includes LLM architectures, cloud data platforms, infrastructure as code, regulated environments, and technical leadership.

Design and scale lakehouse architecture, streaming and batch pipelines, and reliable data platforms using Kafka, Spark, Airflow, Iceberg, Databricks, and related technologies. Build Medallion-layer data systems, manage open table formats, optimize distributed queries, monitor platform reliability, resolve data quality issues, and collaborate with analysts, data scientists, and product teams.