Top Lead Data Engineer Jobs in San Francisco, CA

Leads the design, development, optimization, and migration of modern data platforms in Snowflake. Builds pipelines with dbt, Python, and cloud services; gathers client requirements; creates solution designs and project plans; advises on Snowflake performance, cost, security, and workload management; integrates BI, governance, and AI/ML tools; contributes to accelerators and mentors junior engineers. The role is remote, client-facing, consultative, and focused primarily on technical delivery rather than sales.

Lead data engineering and machine learning initiatives by designing, architecting, developing, and supporting GCP data pipelines. Create information architecture documentation, estimate and plan technical work, collaborate with project teams and business analysts, and translate requirements into technical designs. Develop Python-based ELT pipelines, optimize BigQuery SQL, support CI/CD, and provide technical leadership to projects and teams.

Lead Data Engineer responsible for designing, maintaining, and improving data pipelines and platform infrastructure across ingestion, streaming, transformation, and delivery. The role uses Airflow, Spark, Iceberg, Kafka, Flink, dbt, Terraform, and GCP. Responsibilities also include defining data contracts, documenting systems, optimizing performance, reviewing code, supporting access controls and compliance, and collaborating with stakeholders and teammates.

Leads and mentors the Data Engineering team while remaining hands-on in designing, building, and maintaining scalable data pipelines, warehouses, integrations, and infrastructure. Establishes architecture, reliability, quality, and performance standards; guides technical delivery; and resolves complex issues. Partners with Finance and business stakeholders to translate data needs into practical solutions, sets technical direction, evaluates technologies, and communicates strategy, risks, and recommendations to leadership.

Lead the architecture, development, and optimization of Snowflake and enterprise data platforms. Design scalable ETL/ELT pipelines, dimensional data models, and medallion architectures while establishing engineering standards for testing, deployment, monitoring, security, and data quality. Partner with stakeholders on technical roadmaps, mentor engineers and analysts, conduct design and code reviews, modernize legacy pipelines, and deliver reliable, reusable data solutions.

Leads the design, development, and maintenance of scalable AWS data platforms, pipelines, ETL/ELT workflows, data models, and orchestration systems. Uses Python, PySpark, SQL, dbt, AWS services, modern lakehouse technologies, and CI/CD practices to deliver secure federal data solutions. Responsibilities include platform modernization, performance optimization, AI-enabled data integration, compliance support, technical mentoring, architecture discussions, and cross-functional Agile collaboration.

Lead the development of healthcare applications and data systems, focusing on audience intelligence and compliance. Design scalable data pipelines, APIs, and reporting systems while ensuring high data quality and security in regulated environments.

Lead end-to-end development and deployment of enterprise generative AI and LLM applications, including agentic workflows, MCP-based data integrations, secure tool usage, AI gateways, identity propagation, access controls, and observability. The role partners with Information Security and corporate functions to deliver governed AI automation across procurement, supply chain, legal, finance, HR, and manufacturing.

Lead the end-to-end data platform for large-scale physical-labor video, from on-device capture and reliable uploads over poor networks through object storage, compression, training-shard creation, dataset versioning, and delivery to research customers. Own petabyte-scale media infrastructure, cost and throughput optimization, observability, and ingestion contracts across firmware and international operations. The role requires strong software engineering and production experience with media or sensor data at significant scale.

Leads architecture and hands-on development of scalable data pipelines, governed data models, semantic layers, analytics products, and AI workflows. Writes production SQL and Python, establishes engineering standards, mentors engineers, and drives design reviews. Builds agentic and LLM solutions using retrieval and managed AI services while ensuring data quality, observability, reliability, privacy, and regulatory compliance for PHI and genomic data. Partners with stakeholders, platform teams, and governance forums to deliver trustworthy business solutions.

Design and support ETL, data migration, transformation, import, export, and validation workflows for moving legacy EHR data into athenahealth systems. Build Python and SQL utilities, define source-to-target mappings, support HL7 and FHIR interoperability, validate EHI exports, investigate data quality issues, document processes, and collaborate with product, implementation, operations, sales, and engineering teams in a remote environment.

Leads architecture, optimization, scaling, and ownership of Zoox’s enterprise and AI data platform. Builds self-healing pipelines, governance frameworks, data quality and observability systems, secure access controls, lifecycle policies, and automated CI/CD deployment infrastructure. Optimizes Databricks and AWS EMR performance and costs while enabling low-latency, AI-ready data foundations. The role also drives enterprise security, compliance, schema evolution, and developer velocity, with opportunities to apply GenAI, Terraform, and EKS.

The Lead Engineer for BI will design data solutions, lead project teams, mentor, and drive BI strategy aligning with business needs.

Lead Data Engineer responsible for defining architecture, engineering standards, and best practices for the enterprise data platform, collaborating with distributed teams for modernization and scalability.

Lead design and implement scalable data infrastructure and transformations. Build data repositories, convert data science algorithms into robust pipelines, mentor a small data engineering team, ensure delivery quality, and collaborate with Product, Data Analysts, and Data Scientists.

Own SQL and dbt-based data modeling and data quality for clinical and claims datasets. Author and refactor dbt models/tests, triage DQ alerts, design monitors, optimize SQL performance, and collaborate with Data Engineering and stakeholders to ensure trusted production datasets.

Lead a team of data engineers to design, build, and maintain scalable data pipelines, a Data Vault, data warehouses and marts. Own backlog, sprint planning, mentoring, stakeholder collaboration, data validation/monitoring, and platform improvements to deliver key metrics and data infrastructure across the business.

Lead HUD's data quality strategy for RL training data: build QC systems, validate synthetic data at scale, design evaluation experiments, partner with researchers and vendors, convert research insights into production validation pipelines and tools, and mentor research engineers.

Lead the analytics data layer: design and operate a ClickHouse-backed warehouse, define and validate trusted metrics, build customer-facing React dashboards and internal Grafana/self-serve tooling, and ensure data quality, lineage, and documentation for telemetry-derived insights.

Own the architecture, technical direction, reliability, and evolution of a shared cloud data platform. Lead hands-on development of streaming and batch pipelines using infrastructure-as-code, build governed analytical data products, and establish data contracts, quality, security, lineage, and observability standards. Serve as the final technical escalation point, mentor engineers, lead design and code reviews, promote responsible AI-assisted coding, respond to incidents, and communicate technical tradeoffs across stakeholders.

Lead a team of 5-6 data engineers to design and implement scalable data pipelines (Airflow), manage Snowflake data architecture, build real-time streaming solutions with Kafka/Spark Streaming, develop and integrate APIs, enforce code quality, and drive technical decisions and best practices.

Lead data engineering teams to design, build, and maintain scalable cloud-native data pipelines, data platforms, and data products using Snowflake, Databricks, Airflow, and modern architectures. Optimize ETL/ELT, SQL, and transformations for analytics, reporting, and AI use cases; ensure data quality, governance, observability, and stakeholder alignment. Produce documentation, runbooks, and present technical solutions to business and technical audiences.

Lead design of data center power architecture (up to ~50MW) from utility interconnection to IT load. Own reliability philosophy, substations, MV/LV distribution, DER/BESS/PV integration, protection/control, FMEA, studies, commissioning oversight, utility interconnection, and lifecycle support across projects and stakeholders.

Leads data and analytics engineering for Zero Trust maturity assessment and implementation measurement. Designs common data models, integrates and validates enterprise data, develops maturity and risk metrics, supports prioritization and gap analysis, and establishes data quality, lineage, traceability, and documentation practices. Partners with dashboard and automation specialists to streamline reporting and data refresh workflows.

Lead the design and implementation of a unified semantic layer and enterprise data models, transforming SAP, Workday, Salesforce, Anaplan and other sources into AI-ready datasets. Establish modeling standards, testing, observability, and automation; collaborate with engineers and business stakeholders to deliver scalable, high-performance data solutions for AI agents and analytics.