Top Data Engineer Jobs in San Francisco Bay Area, CA

The Principal Data Engineer will lead data architecture initiatives, design scalable data systems, and oversee data standards across teams while ensuring high-quality data infrastructure and guiding engineering teams.

Design, build, and maintain an internal data platform and pipelines. Ensure secure, governed, resilient data access for engineering and BI teams. Partner with architects, implement automations, support platform consumption, mentor users, and drive continuous improvement and compliance.

Lead architecture and implementation of a large-scale analytics stack: design foundational pipelines, establish modeling/orchestration standards, integrate AI-assisted engineering tools, translate business questions into version-controlled data models, partner with stakeholders to define metrics and data contracts, migrate logic into maintainable pipelines, and optimize cloud warehouse performance for scale.

Build and maintain large-scale data processing pipelines using Apache Spark (PySpark/Scala) and Python. Design ETL workflows, implement distributed data processing, use version control (Git), and work with cloud platforms (GCP/AWS/Azure).

Build, maintain, and monitor ETL/ELT pipelines in the Microsoft Fabric ecosystem. Implement medallion architecture, transform raw data into analytics-ready models, ensure pipeline reliability and data quality, troubleshoot failures, optimize performance, and support CI/CD and version control for data artifacts.

Design, build, and maintain scalable data pipelines, data lakes, and databases; ingest and map customer financial datasets; monitor pipeline reliability; translate business requirements into data flows and analytical insights to ensure high-quality, usable data.

Build and maintain scalable, governed data pipelines from ingestion through transformation and integration. Implement schema validation, data quality checks, job monitoring, and production-grade APIs. Work with Python, SQL, Airflow, dbt, AWS (EMR, Athena, S3), Iceberg/Parquet, and contribute to CI/CD, observability, and on-call support while collaborating with analytics and product teams.

Lead design and operation of large-scale data pipelines and data APIs. Build Spark/PySpark workflows on Databricks, optimize job performance, manage data quality and observability, and develop MCP servers and AI-agent integrations. Mentor engineers, define standards, support production incidents and on-call rotations, and collaborate with stakeholders to deliver scalable data platform products.

Own and scale end-to-end data infrastructure: build production ETL pipelines, design scalable schemas, enforce data quality/governance/security, create self-serve data models for analytics and ML, instrument observability, and partner cross-functionally while participating in on-call incident response.

Lead design and implementation of enterprise data platforms, real-time pipelines, feature stores, and agentic data infrastructure. Ensure data governance, quality, lineage, and reliability while partnering with scientists, engineers, and product leaders. Own technical roadmap, mentor engineers, and drive platform scalability and engineering best practices.

Own product telemetry, maintain dbt models in Snowflake, support self-service reporting and NLP queries, drive product insights from user and retention data, and monitor Airbyte syncs to ensure data quality.

Build and operate ingestion and transformation pipelines to turn messy insurance data into clean datasets. Design bronze/silver/fact layers, model and tune Postgres and analytical stores, define KPIs and reconciliation checks, orchestrate observable data jobs, and ensure end-to-end data quality for analytics and AI products.

Build and maintain data marts, materialized views, and ETL/ELT pipelines in an AWS-hosted data lake. Design and optimize data workflows, monitor infrastructure health and data quality, support PostgreSQL migrations, collaborate with cross-functional teams, and ensure compliance with federal data security (FISMA/NIST 800-53).

Design, build, and maintain Keboola-based data pipelines integrating multiple business systems to deliver accurate, scalable data for supply chain, logistics, and transportation analytics; collaborate with analysts and engineers; support documentation, governance, and pipeline automation.

Own and scale the data platform powering ReadyOn's real-time labor OS. Design and operate batch and real-time pipelines, transformations, and data services (Python, PySpark, Airflow, Glue, Snowflake, Postgres). Build connectors, observability, data quality, and feature-store infrastructure while partnering with AI, backend, and product teams and providing technical leadership and mentorship.

Lead architecture, systems, and roadmap for Coinbase's Compliance Reporting Data Platform. Design and build production-grade data models, pipelines, quality frameworks, validations, and certification processes. Establish data contracts, monitoring, reconciliation, and audit-ready processes. Partner with engineering and compliance stakeholders, respond to regulatory exams, and mentor engineers while shaping standards and documentation.

The Data Engineer will design and maintain data pipelines, enhance the data warehouse, support machine learning applications, and collaborate with various teams to leverage data effectively.

As a Data Engineer, you will build and maintain data pipelines, convert data into metrics, and operate the platform on AWS. Responsibilities include writing production code, processing data from APIs, and enhancing the orchestration layer using Dagster, while ensuring compliance in a HIPAA-regulated environment.

As a Data Engineer, you will build and maintain data pipelines, ensure data quality, support ML workflows, and optimize query performance using various technologies.

Build and maintain scalable data pipelines and integrations using Databricks and Azure. Transform and model large structured and unstructured datasets to support reporting, analytics, and operational needs. Implement data quality, monitoring, and governance; troubleshoot pipeline performance; collaborate with stakeholders and mentor junior engineers.

The Data Engineer is responsible for designing data workflows and pipelines, implementing systems engineering principles, and deploying data streaming technologies while ensuring compliance with industry regulations and best practices.

Forward-deployed senior data engineer who deploys, hardens, integrates, and operates a data platform (lakehouse, streaming, governance) in DoD environments. Build pipelines across classification boundaries, produce accreditation evidence (STIG/cATO/FIPS), run day-2 operations, train mission teams, and mentor junior FDEs while feeding field feedback into product improvements.

The Forward Deployed Data Engineer will manage AI-automated data platform migration projects, ensuring timely delivery and communication with clients while utilizing proprietary tools.

The Data Engineer I at Storable supports the design and maintenance of data pipelines, works with ETL processes, optimizes workflows, and collaborates with various teams to enhance data accessibility and reliability.

Design, build, and maintain scalable batch and near-real-time data pipelines and data products using Databricks and Spark. Integrate internal/external sources, implement data models, ensure data quality, support BI (Sisense), CI/CD, and collaborate with analysts, scientists, and stakeholders to enable analytics and decision-making.