Top Senior Data Engineer Jobs in San Francisco Bay Area, CA

Design, build, and optimize enterprise-scale data models, pipelines, and cloud-native architectures to support analytics, AI/ML, and reporting. Lead data governance, metadata management, lineage, access controls, and data quality. Partner with stakeholders to translate business needs into scalable, secure, and high-performance data infrastructure, and evaluate emerging technologies to evolve the data platform.

Build and maintain low-latency, highly available distributed data systems and services to deliver high-quality data for product, analytics, and ML. Partner with data scientists and product teams to design scalable, observable, multi-tenant data products using batch and real-time processing frameworks. Ensure 24x7 reliability, monitoring, and operational excellence while owning end-to-end delivery.

Design, build, and own large-scale batch and streaming data pipelines and core Ads data products. Collaborate with analytics, data science, and engineering teams to model, transform, govern, and deliver high-quality ad datasets for analysis, ML, and reporting while ensuring privacy compliance.

Build and maintain scalable batch and real-time data pipelines, model data to power analytics and ML, and operate distributed data systems. Collaborate with business, engineering, and data science teams to ensure data quality, documentation, and debugging while adopting new technologies and working in an ambiguous, fast-changing environment.

Lead customer integrations by building warehouse-native or managed ClickHouse pipelines, construct semantic metadata layers, canonicalize source data into unified growth models, and act as the technical lead for enterprise deployments while partnering with AI discovery agents to accelerate onboarding and mapping.

Build and maintain centralized pricing data infrastructure, scalable workflows, and data pipelines; integrate AI capabilities; deliver pricing tools, reporting, and governance to support data-driven pricing and budgeting across the firm.

Build and scale core data infrastructure and foundational datasets. Design, develop, and maintain scalable data pipelines for analytics, ML, experimentation, and product use cases. Implement data quality, audits, alerting, and anomaly detection. Partner with product, engineering, and business stakeholders and help establish long-term, scalable data architecture and tooling.

Design, build, and optimize scalable Azure data platforms and lakehouse architectures. Develop ETL/ELT pipelines (PySpark/Spark SQL), enable Power BI datasets, implement CI/CD and IaC, ensure data governance/quality with Purview, support cloud migrations and production operations while collaborating with stakeholders and agile teams.

Design and implement end-to-end data integration and warehousing solutions for large volumes. Implement Hadoop/Hive data movement, script in Linux shell and Python, develop SQL for integration and analysis, and perform high-volume DW performance tuning. Vertica experience is a plus. Requires technical independence and a BS in CS/EE.

Design, build, and optimize scalable Azure data platforms and lakehouse architectures. Develop ETL/ELT pipelines (PySpark, ADF, Fabric), enforce data governance and quality, enable Power BI reporting, lead cloud migration and CI/CD, and provide production support and performance tuning across enterprise data systems.

Design, build, and maintain scalable data pipelines and a customer-facing recommendation engine (including LLM logic). Manage data warehouse, data quality monitoring, CI/CD for pipelines, and partner with product, analytics, engineering, and marketing to ensure consistent metrics and observability.

Design, build, and maintain scalable data ingestion and ETL pipelines across a growing data lake. Define and enforce data quality standards, SLOs, and validation frameworks. Optimize pipelines for performance and cost, expand monitoring and alerting, and enable stakeholders to self-serve on data infrastructure.

Build and maintain scalable data pipelines, trusted datasets, and reporting systems to support infrastructure deployment, hardware operations, capacity planning, and supply chain analytics. Partner with stakeholders, improve data quality, lineage, and observability, and enable executive reporting, forecasting, and operational reviews. Integrate new telemetry sources and document systems for long-term maintainability.

As a Data Engineer, you will design and maintain data pipelines, support DoD systems, and ensure data quality and compliance.

As a Senior Data Engineer, you will design and maintain data systems for Aircapture, focusing on building data pipelines and making architectural decisions to support climate technology.

This role involves leading data strategy and managing a team to build and maintain multimodal datasets, ensuring data quality and pipeline efficiency.

Own and build CrewAI's end-to-end data foundation: consolidate telemetry and application data, design pipelines and warehouses, define trusted metrics and semantic models, improve instrumentation and data quality, enable self-serve analytics, and support product and GTM decisions with reliable dashboards and analyses.

Design, build, and operate large-scale, fault-tolerant real-time and batch data pipelines powering an ad-tech DSP. Collaborate with engineers and data scientists to define data models and SLAs, optimize performance and capacity, enforce data quality and monitoring, mentor engineers, and evaluate new data technologies to scale analytics and ML workloads.

Lead design and hands-on delivery of a next-generation data platform and pipelines (batch, ELT, near-real-time) powering BI, experimentation, and ML. Own platform architecture, data modeling, governance, observability, and mentor engineers while modernizing legacy systems and driving AI-first data engineering practices across cross-functional teams.

Lead architecture and delivery of a cloud-native, petabyte-scale data platform: define technical roadmap, mentor engineers, build ETL/ELT pipelines, enable real-time streaming, implement Iceberg table format, optimize Snowflake/Redshift/Athena compute, and integrate observability with Splunk.

Design, build, and optimize scalable AWS-based data pipelines, data lakes, and warehouses. Migrate legacy systems, enforce data quality/security/governance, implement ETL/ELT and big-data solutions, adopt CI/CD and IaC, collaborate with stakeholders, and provide technical leadership and troubleshooting.

Lead end-to-end data engineering on GCP: design scalable data platforms and architectures (lake/lakehouse/warehouse), build and optimize BigQuery/Spark pipelines, ensure data quality/security, enable ML workflows with Data Science, mentor engineers, and own delivery and performance.

Design, build, modernize, and maintain operational and analytical data capabilities for Wealthscape Reporting, Analytics and Insights. Perform solution design, data analysis, ETL development, and production rollouts using Snowflake, AWS, cloud databases, and CI/CD pipelines. Collaborate across teams in a fast-paced financial services environment.

Design, build, and maintain ETL/ELT data pipelines and orchestration using Airflow, Python, and PySpark. Develop custom connectors, implement DataOps and CI/CD for data workflows, monitor and troubleshoot pipelines, validate data quality, and document processes while collaborating with cross-functional teams and supporting large-scale distributed data systems.

Design, build, and maintain scalable data pipelines on AWS using EMR, Spark, and PySpark. Optimize EMR clusters and Spark jobs, implement ETL/ELT processes, ensure data quality, automate ingestion, and collaborate with data scientists and analysts. Document infrastructure, participate in code reviews, and monitor performance and costs.