Top Lead Data Engineer Jobs in San Francisco, CA

As the first in-house Data Engineer, you will own the data infrastructure, build and maintain data pipelines, and collaborate with teams to ensure quality data access and insights.

Lead the development of healthcare applications and data systems, focusing on audience intelligence and compliance. Design scalable data pipelines, APIs, and reporting systems while ensuring high data quality and security in regulated environments.

Develop and maintain data ingestion pipelines, relational databases, ETL/ELT workflows, cloud resources, data reliability monitoring, and AI-enhanced applications. Integrate LLM APIs and MCP connectors, write Python and R automation code, migrate siloed processes into centralized systems, and collaborate with technical and non-technical stakeholders. The role also supports Azure deployments, enterprise data integrations, dashboards, reporting, and cloud data security.

Lead the Data Platform Engineering team owning ETL/reverse-ETL pipelines, replication, and production datastores. Define architecture, roadmap, standards (IaC, observability, schema contracts), mentor engineers, drive incident response and durable fixes, and partner on cross-team technical decisions and operational reliability for data correctness, completeness, and timeliness.

Design and support ETL, data migration, transformation, import, export, and validation workflows for moving legacy EHR data into athenahealth systems. Build Python and SQL utilities, define source-to-target mappings, support HL7 and FHIR interoperability, validate EHI exports, investigate data quality issues, document processes, and collaborate with product, implementation, operations, sales, and engineering teams in a remote environment.

Lead data engineering and machine learning initiatives by designing, architecting, developing, and supporting GCP data pipelines. Create information architecture documentation, estimate and plan technical work, collaborate with project teams and business analysts, and translate requirements into technical designs. Develop Python-based ELT pipelines, optimize BigQuery SQL, support CI/CD, and provide technical leadership to projects and teams.

Lead the modernization and operation of the State Bar's Azure-based data platform: design and build scalable data pipelines, enforce governance and security (PII/PCI/HIPAA/CPRA), optimize data for Power BI, manage migrations from legacy systems, and lead cross-functional teams to deliver analytics, automation, and compliant data solutions.

Lead the modernization of the State Bar's data platform using Microsoft cloud technologies. Design, build, and maintain scalable data pipelines, enforce data governance and security/compliance, optimize datasets for Power BI, manage migrations from legacy systems, troubleshoot and tune performance, and lead cross-functional teams while documenting and transferring knowledge to ensure long-term maintainability and observability.

Lead Data Engineer responsible for defining architecture, engineering standards, and best practices for the enterprise data platform, collaborating with distributed teams for modernization and scalability.

Lead design and implement scalable data infrastructure and transformations. Build data repositories, convert data science algorithms into robust pipelines, mentor a small data engineering team, ensure delivery quality, and collaborate with Product, Data Analysts, and Data Scientists.

Own SQL and dbt-based data modeling and data quality for clinical and claims datasets. Author and refactor dbt models/tests, triage DQ alerts, design monitors, optimize SQL performance, and collaborate with Data Engineering and stakeholders to ensure trusted production datasets.

Lead diverse technology projects and a team to create solutions for regulatory needs, leveraging distributed microservices and collaboration with product managers.

Lead Data Engineer builds and maintains scalable data pipelines and lakehouse/warehouse platforms (Databricks) to support analytics, BI, reporting, and AI. Responsibilities include data modeling, ETL/ELT, medallion architecture, orchestration and monitoring (Airflow/Cron), performance tuning, CDC/incremental loads, data quality, and leading/mentoring a team while collaborating with stakeholders and maintaining documentation.

Lead a team of data engineers to design, build, and maintain scalable data pipelines, a Data Vault, data warehouses and marts. Own backlog, sprint planning, mentoring, stakeholder collaboration, data validation/monitoring, and platform improvements to deliver key metrics and data infrastructure across the business.

Own and build CrewAI's end-to-end data foundation: consolidate telemetry and application data, design pipelines and warehouses, define trusted metrics and semantic models, improve instrumentation and data quality, enable self-serve analytics, and support product and GTM decisions with reliable dashboards and analyses.

Lead HUD's data quality strategy for RL training data: build QC systems, validate synthetic data at scale, design evaluation experiments, partner with researchers and vendors, convert research insights into production validation pipelines and tools, and mentor research engineers.

Lead the analytics data layer: design and operate a ClickHouse-backed warehouse, define and validate trusted metrics, build customer-facing React dashboards and internal Grafana/self-serve tooling, and ensure data quality, lineage, and documentation for telemetry-derived insights.

Own the architecture, technical direction, reliability, and evolution of a shared cloud data platform. Lead hands-on development of streaming and batch pipelines using infrastructure-as-code, build governed analytical data products, and establish data contracts, quality, security, lineage, and observability standards. Serve as the final technical escalation point, mentor engineers, lead design and code reviews, promote responsible AI-assisted coding, respond to incidents, and communicate technical tradeoffs across stakeholders.

Lead a team of 5-6 data engineers to design and implement scalable data pipelines (Airflow), manage Snowflake data architecture, build real-time streaming solutions with Kafka/Spark Streaming, develop and integrate APIs, enforce code quality, and drive technical decisions and best practices.

Leads engineering support and architecture for enterprise-scale healthcare data platforms on Azure. Designs Databricks, Synapse, Data Factory, and Data Lake solutions; builds batch and real-time pipelines, analytical data models, BI solutions, and automated data quality frameworks. Implements machine learning and AI-assisted development, integrates low-code systems, ensures HIPAA compliance, and translates business needs into technical solutions. Mentors engineers, resolves technical issues, supports Agile delivery, and partners with healthcare business leaders on Medicare and Medicaid analytics initiatives.

Lead data engineering teams to design, build, and maintain scalable cloud-native data pipelines, data platforms, and data products using Snowflake, Databricks, Airflow, and modern architectures. Optimize ETL/ELT, SQL, and transformations for analytics, reporting, and AI use cases; ensure data quality, governance, observability, and stakeholder alignment. Produce documentation, runbooks, and present technical solutions to business and technical audiences.

Lead a team to design, build, and maintain cloud data engineering solutions using Microsoft Fabric and Azure Synapse. Responsibilities include architecture, data pipelines, lakehouse and warehousing implementation, governance, performance optimization, mentoring engineers, cross-functional collaboration, support/on-call, documentation, and ensuring reliability and compliance.

The Lead Data Platform Engineer will design and build Virtuous's data platform, ensuring it is secure, scalable, and AI-ready. Responsibilities include architecting access controls, optimizing performance, and enabling self-service analytics while maintaining governance and compliance standards.

Lead design of data center power architecture (up to ~50MW) from utility interconnection to IT load. Own reliability philosophy, substations, MV/LV distribution, DER/BESS/PV integration, protection/control, FMEA, studies, commissioning oversight, utility interconnection, and lifecycle support across projects and stakeholders.

Leads data and analytics engineering for Zero Trust maturity assessment and implementation measurement. Designs common data models, integrates and validates enterprise data, develops maturity and risk metrics, supports prioritization and gap analysis, and establishes data quality, lineage, traceability, and documentation practices. Partners with dashboard and automation specialists to streamline reporting and data refresh workflows.