Top Data Engineer Jobs in San Francisco Bay Area, CA

Build reliable ETL infrastructure, scalable data pipelines, and trusted datasets serving product and business teams. Partner with data scientists and stakeholders to model data, monitor pipelines, define delivery SLAs, automate workflows, and enable self-service analytics and AI-assisted data exploration. The role requires strong SQL, cloud data-platform expertise, object-oriented programming, data modeling, and experience delivering production-scale data solutions.

Leads enterprise-scale data architecture across batch, streaming, warehouse, lakehouse, transactional, AI, and vector-store use cases. Establishes data quality, governance, lineage, observability, access-control, and reliability standards. Defines AI-enabled data engineering patterns, mentors engineers, resolves complex architecture issues, and influences cross-functional roadmaps. Requires hands-on technical leadership and 15+ years of software, data engineering, or architecture experience.

Build and maintain Databricks data pipelines and reusable models for Go product analytics. Define metrics, improve event logging, connect product usage with customer data, and support adoption and growth analysis. Ensure data quality through testing, monitoring, troubleshooting, and documentation. Optimize pipeline performance and costs while collaborating with Product, Engineering, and Data Science. Contribute dashboards and analyses that support product and business decisions.

Designs and operates enterprise-scale data platforms, including batch and streaming pipelines, transformation and orchestration workflows, analytical data warehouses, telemetry systems, data catalogs, semantic layers, and AI-supporting data interfaces. The role requires production monitoring, incident support, architectural decision-making, and collaboration with engineering, analytics, data science, and business stakeholders.

Build and deploy scalable data pipelines for ingestion, transformation, quality assurance, migration, and automated data consumption. Apply cloud, big data, DevOps, and CI/CD practices while developing data governance processes, quality rules, compliance policies, and stewardship workflows. Collaborate with business and BI teams to define requirements, document technical designs, and support finance-related data solutions. The role may require significant client travel.

Architects, deploys, and operates scalable data platform infrastructure supporting analytics. Responsibilities include administering Airflow and Superset, managing Terraform infrastructure, building CI/CD pipelines, implementing security and compliance controls, developing dbt data models, cataloging assets in OpenMetadata, automating with Python, and mentoring engineers. The role requires extensive infrastructure and data engineering experience, cloud expertise, container orchestration, and eligibility for CJIS clearance.

Build and maintain analytical datasets, dimensional models, Power BI semantic models, and governed data transformations in Microsoft Fabric. Translate business requirements into reliable data products, perform testing and reconciliation, troubleshoot refreshes and deployment issues, and support Git-based development and CI/CD practices. The role requires independent ownership of low- to moderate-complexity engineering work, technical documentation, and collaboration with senior engineers, DataOps, governance teams, consultants, and business stakeholders.

Build and operate production data pipelines, models, and data products supporting finance, accounting, treasury, compliance, and regulatory reporting. Partner cross-functionally to define requirements, model on-chain and internal data, and ensure data reliability through reconciliation, quality checks, monitoring, and alerting. Use SQL, Python, cloud data warehouse, Airflow, dbt, and AI tooling. Mentor engineers, review technical work, document systems, and communicate data concepts to technical and non-technical audiences.

Build and maintain scalable data pipelines, integrations, and data models connecting product, customer, sales, billing, and revenue data. Partner with Sales, RevOps, Finance, and Product to support forecasting, reporting, customer intelligence, segmentation, retention, and expansion analysis. Implement data validation, monitoring, documentation, and governance while improving infrastructure and automation for the company’s growing B2B operations.

Build and maintain production data pipelines, dbt models, Airflow workflows, and governed datasets supporting People Analytics. Integrate Workday and other HR systems into reliable Snowflake data products, implement data quality monitoring, row-level security, semantic models, and AI agents. Partner with engineers, analysts, data scientists, and business stakeholders to deliver trusted data solutions, communicate technical trade-offs, enforce governance, and mentor other engineers.

Develop and maintain high-throughput backend and data systems for PlayStation’s cloud streaming platform. Build JVM applications, Flink stream-processing jobs, scalable ETL pipelines, APIs, and real-time data solutions. Optimize reliability, monitoring, and performance; support Kafka-to-cloud migrations; analyze requirements; contribute to architecture; and collaborate with Business Intelligence and product teams to convert data into actionable insights.

Designs, builds, and operates scalable batch and streaming lakehouse pipelines using Databricks, Spark, PySpark, Delta Lake, and Unity Catalog. Responsibilities include data ingestion, transformation, governance, orchestration, performance optimization, CI/CD integration, monitoring, data quality, and documentation. The role supports analytics, machine learning, anomaly detection, fraud analytics, and risk scoring while collaborating with technical stakeholders and Agile teams. A Public Trust determination is required.

Lead the architecture and standards for Adyen’s data foundations. Design shared data models, semantic layers, data contracts, lineage, and data products; establish metric definitions and quality standards across domains. Build prototypes, solve complex data-stack challenges, influence teams through technical leadership, partner with cross-functional stakeholders, and mentor engineers while remaining hands-on.

Build and optimize scalable ELT/ETL pipelines, data models, schemas, and BigQuery solutions. Ensure data quality, governance, compliance, monitoring, and pipeline reliability. Collaborate with data scientists, engineers, analysts, BI teams, and business partners while developing generative AI data products and documenting engineering standards.

Leads a small data engineering team while contributing hands-on development. Responsibilities include designing and maintaining dbt and Python ETL pipelines, managing data quality and UAT, coordinating implementation timelines, conducting code reviews, mentoring engineers, documenting systems, and supporting project delivery. The role also involves building and deploying AI agents for data quality, pipeline monitoring, and analytics workflows, with a focus on healthcare claims data and large-scale datasets.

Develop and operate cloud-based data pipelines, dimensional data models, data warehouse solutions, CI/CD automation, and platform governance controls. The role integrates APIs and diverse data sources while supporting security, monitoring, incident response, and reliable data product delivery. Responsibilities include collaborating with product owners and engineering teams, participating in design and code reviews, troubleshooting technical issues, and applying repeatable DevOps practices.

Build and own AAG’s consolidated data warehouse, including production-grade infrastructure, ELT pipelines, dimensional models, and reporting layers. Integrate data from ERP, CRM, and manufacturing systems into Databricks on Azure, maintain reliable data flows, and support business decision-making across the organization. Collaborate with leadership to translate requirements into code, while managing system development, maintenance, documentation, and continuous improvement.

Data Engineer Interns support the development and maintenance of data pipelines, ETL/ELT processes, data infrastructure, dashboards, and analytics tools. Responsibilities include writing Python and SQL, troubleshooting pipeline issues, improving data quality and performance, documenting workflows, and collaborating with engineering and analytics teams. The role provides hands-on exposure to cloud data infrastructure, warehouses, modern processing technologies, code reviews, and engineering best practices.

Build and operate production data pipelines, models, identity resolution, canonical data layers, and GTM data products. Integrate internal and external sources, establish data quality and governance, generate intelligence and AI-derived attributes, and deliver trusted data to CRM systems. Partner with Sales Operations, Finance, Data Science, and engineering teams on definitions, planning, segmentation, routing, observability, security, and platform scalability.

Designs and delivers large-scale, governed data pipelines and infrastructure across the full data lifecycle. Responsibilities include coding in Python and SQL, optimizing Airflow, dbt, AWS, Kafka, and Iceberg/Parquet workflows, implementing data quality and observability, contributing to architecture and tool evaluations, mentoring engineers, collaborating with stakeholders, and participating in on-call support.

Designs and maintains scalable data pipelines and infrastructure, establishes data governance and quality standards, and drives technical decisions across Headway’s data platform. Partners with Product, Engineering, Analytics, Data Science, Infrastructure, Finance, and Insurance Systems to improve architecture, performance, reliability, security, and AI-readiness. Ensures data is complete, traceable, trustworthy, and suitable for powering internal tools and AI-driven insights.

Design and modify data solutions using Databricks and Palantir Foundry; build data models and reports from ERP and legacy-system datasets; define and execute tests; and translate stakeholder reporting needs into technical specifications. The role requires SQL, Python, PySpark, database architecture, cloud architecture, software engineering practices, independent execution, documentation, and collaboration. It supports an Army client in a hybrid Arlington, Virginia environment and requires an active Secret clearance.

Build and maintain Nexus, an enterprise AI implementation platform. Develop backend services, APIs, integrations, orchestration workflows, automated data pipelines, transformations, semantic models, validation processes, and governed lakehouse assets. Partner with AI engineers and enterprise customers to deploy solutions, resolve complex data and system requirements, and create reusable integrations. Ensure secure, monitored, recoverable, and repeatable deployments across customer environments.

Build and maintain reliable data pipelines, computed tables, and scalable data models using SparkSQL and PySpark. Integrate IoT, product, customer, external, unstructured, and ML-related data into a central data lake for analytics, causal inference, model training, and dashboards. Ensure high data quality and uptime while collaborating with data science, analytics, AI/ML, and engineering teams.

Lead the reverse-engineering and migration of legacy ETL pipelines from SAP BusinessObjects Data Services and JD Edwards into dbt and Databricks architectures. Build automated data validation, maintain SQL Server write-back mechanisms, collaborate with client stakeholders, and lead and mentor a small engineering team. The role requires hands-on expertise in SQL, Python, Spark, orchestration, cloud platforms, and legacy ERP and ETL migrations.