Top Senior Data Engineer Jobs in San Francisco Bay Area, CA

Build and maintain data pipelines to ingest, normalize, and deliver messy healthcare data (FHIR, HL7, CCDAs, PDFs) into structured models for AI and clinical use. Implement ETL, real-time webhooks, document extraction, PII sanitization, storage on AWS, and monitor data quality while ensuring compliance with healthcare interoperability standards.

Staff Data Engineer responsible for setting technical direction, architecting scalable data platforms, leading cross-functional initiatives, improving reliability and performance, and enabling AI-assisted analytics and development. The role includes hands-on development with Python, SQL, cloud data warehouses, orchestration, infrastructure as code, Kubernetes, Kafka, and streaming pipelines, along with mentoring, technical advocacy, documentation, code reviews, and on-call troubleshooting.

Data Engineer II will modernize BlueLabs’ data platform by leading Kimball dimensional modeling and legacy warehouse migrations, building reliable production data pipelines, and enforcing data quality standards. The role requires advanced SQL, Python, and dbt expertise, hands-on Snowflake and Redshift experience, and ownership of testing, documentation, observability, and scalable architecture. The engineer will collaborate cross-functionally, guide Data Engineer I team members, contribute to architectural decisions, and translate business logic into maintainable, cloud-agnostic data layers.

The AVP, Data Engineer will design and build data ingestion, transformation, and replication pipelines using SQL, Python, dbt, and Dagster. Responsibilities include translating requirements into technical designs, developing data products, debugging production issues, supporting testing, and improving data engineering practices. The role also involves data modeling, mentoring junior engineers, conducting code reviews, and providing technical leadership while remaining hands-on. Experience with modern data platforms, workflow orchestration, cloud ecosystems, and insurance or financial services is preferred.

Design and build petabyte-scale data pipelines for genomic and imaging data, transforming public and proprietary datasets into AI-ready training data. Develop agent-based curation and quality-control systems, data cataloging tools, and reliable observable infrastructure. Collaborate with AI researchers and scientists to define data requirements and integrate diverse biological datasets while addressing performance, bandwidth, scalability, and scientific correctness.

Build and manage dbt data pipelines, canonical datasets, dashboards, reporting, and foundational data products. Partner with Product, Engineering, and cross-functional teams to define reliable metrics, data models, technical requirements, and analytics systems for consumer products. Establish data integrity standards and SLAs, enable self-service analytics, and influence Product roadmaps through data architecture expertise.

Designs trusted Snowflake data products and enterprise semantic models that standardize business concepts, metrics, relationships, and rules. Builds Cortex semantic views and agents, validates AI-generated results, develops regression tests, and maintains metadata, lineage, and documentation. Leads modeling initiatives from discovery through production while partnering with business and technical stakeholders to improve analytics, automation, and AI consistency.

Own Teal Health’s data warehouse, Fivetran ingestion, dbt transformation layer, and Omni semantic data models. Build reliable pipelines, integrations, tests, documentation, and layered warehouse models across marketing, product, operational, and healthcare data. Partner with Analytics on dashboards, establish data standards, troubleshoot source and schema issues, and manage ticket-driven priorities while protecting sensitive health data under HIPAA practices.

Design and maintain Snowflake-based data platforms, cloud-native data lakes, and scalable ETL/ELT pipelines for asset-backed finance analytics. Orchestrate workflows with Dagster and AWS services, ensure data reconciliation and integrity, and support migration from legacy systems. Contribute to project planning, Jira backlog management, testing, cutover planning, and cross-functional go-live readiness.

Design and maintain Snowflake data pipelines and data lake architectures for asset-based finance analytics. Develop ETL/ELT workflows, validate and reconcile financial data, migrate legacy systems, and translate quantitative finance logic into scalable architectures. Support project execution through requirements breakdown, planning, testing, cutover, and go-live readiness. Collaborate across engineering, quantitative analytics, and stakeholder teams.

Owns the data transformation layer powering agency reporting. Builds reliable analytical models, dimensional schemas, reporting architecture, monitoring, and alerting. Translates government and customer reporting requirements into reusable data models and tools, supports embedded analytics and recurring imports, and ensures authorization scoping prevents cross-organization data exposure. Partners across product and implementation teams while establishing data engineering practices and documentation.

Design, build, and maintain scalable batch and streaming data pipelines using Databricks, PySpark, Python, SQL, Spark, and Delta Lake. Develop transformations, orchestration workflows, data models, validation processes, and curated data products. Troubleshoot performance and quality issues, apply governance and security practices, document solutions, and collaborate with architects, data scientists, DevOps teams, analysts, and clients.

Build Reducto’s data infrastructure from the ground up, owning warehouses, databases, pipelines, orchestration, transformations, data modeling, reliability, testing, freshness, and lineage. Integrate product, sales, and finance data into trusted company-wide datasets. Partner with Engineering, Finance, Operations, and GTM leaders to define business metrics and priorities. As the founding data hire, establish scalable architecture and standards while supporting analytics and business decision-making.

Build and operate enterprise integrations and production data pipelines across Salesforce, NetSuite, Workday, Snowflake, and other business systems. Responsibilities include API-led integrations, SQL and Python transformations, data modeling, order-to-revenue workflows, testing, monitoring, data quality, security controls, and production support. Own projects end to end, partner with stakeholders, mentor engineers, and establish reusable engineering practices.

Leads the design, development, and optimization of scalable data pipelines and analytics platforms. Builds ETL/ELT workflows using Databricks and Spark, integrates and transforms source data into curated models, manages SQL Server data, and maintains documentation and lineage. Collaborates with product, analytics, and engineering teams to deliver reliable data solutions. Applies responsible AI practices and delivers projects independently within established timelines.

Designs, builds, and optimizes scalable Databricks-based data platforms, pipelines, integrations, and reusable engineering tools. The role focuses on platform performance, reliability, cost efficiency, security, governance, automation, and cloud infrastructure. It collaborates with engineering, analytics, security, and other stakeholders on architecture, technical planning, production support, and continuous improvement. The position requires hands-on data engineering experience with Databricks, cloud services, Python, SQL, Spark, infrastructure-as-code, and CI/CD practices.

Build and operate a governed Snowflake enterprise data warehouse, including ingestion, orchestration, dbt transformations, infrastructure as code, security controls, reliability checks, and CI/CD. Partner with data owners and BI teams during migration from legacy systems, ensure compliance for sensitive financial data, support reconciliation and incident resolution, and mentor engineers. The role requires hands-on ownership of pipelines and platform components across development and production environments.

Plans and executes enterprise data migrations, conversions, transformations, archival, and cloud data platform operations. Builds ETL/ELT pipelines and migration scripts, performs profiling and source-to-target mapping, validates data, manages cutovers and rollback procedures, and troubleshoots migration issues. Optimizes cloud platforms, databases, workloads, security, access, cost, and performance while collaborating with engineering, governance, architecture, and business teams.

Lead the architecture and development of Penn Foster Group’s Databricks-based cloud data platform. Build scalable batch, streaming, CDC, and API pipelines using Python, SQL, Spark, and Azure; establish engineering standards, governance, observability, and CI/CD practices; develop AI-ready semantic data products; and mentor data engineers. Partner with BI, product, security, governance, and MLOps teams to support analytics, machine learning, generative AI, and self-service reporting.

Own end-to-end client data migrations: reverse-engineer legacy ETL, design and build Airflow DAGs and Spark jobs, validate and reconcile data at scale, document architecture and decisions, and coordinate cross-functional teams through go-live and hypercare.

Designs, builds, tests, and maintains scalable AWS and Databricks data platforms, pipelines, ETL/ELT processes, data marts, integrations, and governance solutions. Develops batch and near-real-time data ingestion, ensures quality, security, performance, and audit readiness, and supports migration, archiving, disaster recovery, CI/CD, and reporting. Collaborates with architecture, governance, migration, and delivery teams in an Agile environment while mentoring junior engineers and supporting federal modernization objectives.

Lead production of marine and anadromous species range maps under FWS standards. Aggregate, normalize, transform, and validate spatial data; analyze habitat intersections; coordinate review cycles with FWS and NOAA Fisheries subject matter experts; document quality assurance; and obtain government sign-off for deliverables.

Designs and leads enterprise data pipelines and integrations across Snowflake, Oracle Fusion, on-premises databases, and cloud applications. Ensures data quality, governance, security, scalability, and performance while supporting Power BI, AI/ML, and self-service analytics. Partners with business and technical stakeholders, leads migrations, mentors engineers, optimizes data platforms, and establishes engineering best practices.

Designs, builds, deploys, and maintains scalable data pipelines, data lake systems, batch and streaming processing, MLOps frameworks, data documentation tooling, and an agentic data exploration platform. The role supports customer-facing products and enables engineering teams through reliable infrastructure, observability, data discovery, and AI capabilities.

Build and maintain production data ingestion, transformation, and pipeline systems for a governed federal data and AI platform. Develop data products, infrastructure-as-code, CI/CD workflows, monitoring, quality controls, cataloging, and access controls within FedRAMP, NIST 800-171, and CUI constraints. Support data science and machine learning teams, troubleshoot reliability and performance issues, and document engineering processes. The role is fully remote and requires U.S. citizenship and the ability to obtain a T5/SSBI adjudicated clearance.