Top Staff Data Engineer Jobs in San Francisco, CA

Staff Engineer responsible for database architecture, performance, reliability, migrations, and operational standards across PostgreSQL, graph, and time-series data stores. The role involves writing production code, analyzing query performance, designing failover and migration systems, mentoring engineers, and guiding database practices across teams. Candidates need deep PostgreSQL expertise, experience with graph and analytical databases, cloud or Kubernetes operations, and proficiency in Go or Python.

Own the architecture, reliability, production operations, and evolution of Trustly’s Account Data Services. Lead distributed systems design for high-volume account data retrieval and storage with strict latency SLAs. Manage on-call operations, incident response, observability, documentation, AI-assisted implementation, integration testing, and cross-regional knowledge transfer. Help hire and lead the team responsible for this business-critical domain.

Design, build, and operate secure, scalable data infrastructure for Slack’s analytics, machine learning, and data workflows. Own core services such as Pinot, Trino, Airflow, Spark on Kubernetes, EMR, and Hive Metastore, ensuring reliability, observability, performance, and cost efficiency. Develop automation and self-service tools, collaborate across engineering and security teams, establish technical standards, and mentor engineers. Support AI-forward data platforms and agentic workflows at scale.

Lead architecture and hands-on development for cloud data optimization products and Stratusphere’s data platform. Diagnose and improve database, query, storage, networking, and workload performance across AWS, Azure, relational, analytical, search, and lakehouse systems. Build production services, integrations, testing, observability, and infrastructure automation while safely overseeing AI-assisted development. Partner with product teams, forward-deployed engineers, and customers to validate optimizations, document tradeoffs, and convert repeatable solutions into scalable platform capabilities.

Own the architecture, roadmap, and operation of Clarium’s AWS-based data platform. Build reliable batch and near-real-time pipelines, model healthcare supply chain data, optimize Postgres and Snowflake performance, and establish standards for testing, deployment, quality, and governance. Partner cross-functionally to create durable data models, lead multi-quarter initiatives, manage sensitive data controls, and mentor engineers.

Leads complex enterprise data migration and customer conversion programs for digital banking implementations. Responsibilities include migration strategy, governance, reconciliation, risk mitigation, operational transformation, portfolio oversight, executive stakeholder communication, and cross-functional coordination across implementation, engineering, product, operations, vendors, and customers. The role also mentors senior teams, develops scalable migration frameworks, drives automation, and improves conversion reliability and onboarding outcomes.

Own and scale a multi-cloud data platform across AWS and Azure. Manage Apache Airflow, Kubernetes, Terraform, Helm, ArgoCD, CI/CD, observability, big data infrastructure, governance tooling, and FinOps initiatives. Provide end-to-end architectural and operational ownership, improve developer tooling, define platform roadmaps, manage incidents, enforce engineering standards, and mentor data platform engineers.

As the first in-house Data Engineer, you will own the data infrastructure, build and maintain data pipelines, and collaborate with teams to ensure quality data access and insights.

Design and build PerfectServe’s unified cloud data platform, including ingestion pipelines, integrations, data models, metrics, and consumer-facing data layers. Establish architecture, engineering, quality, and testing standards; enable safe extraction from production systems; partner with data users; and mentor data engineers. The role requires substantial cloud data platform experience, deep SQL, warehouse expertise, ELT, change-data-capture, AWS, modeling, orchestration, and data quality skills.

Build and lead data catalog infrastructure supporting data discovery, metadata management, governance, quality, lineage, privacy, access control, and retention. Develop scalable metadata integrations, onboarding tools, data policies, and end-to-end lineage products across online, offline, machine learning, metrics, visualization, and third-party datasets. Collaborate with cross-functional infrastructure teams to ensure Airbnb data assets are cataloged, discoverable, governed, and compliant with regulatory requirements.

Lead the architecture, development, operation, and evolution of PlayStation’s real-time analytics platform. Build scalable distributed data systems using streaming and OLAP technologies, integrate batch and streaming workloads, improve reliability and performance, and develop AI-powered engineering capabilities. The role provides technical leadership across teams, mentors engineers, drives architecture and standards, resolves production incidents, and participates in on-call support.

Lead architecture and standards for CoreWeave's enterprise data ecosystem: design lakehouse/streamhouse architectures, modeling and semantic layers, governance, data quality, lineage, observability, and reusable data products. Drive cross-domain solutions, resolve performance and scalability constraints, evaluate core data technologies, conduct architecture reviews, and mentor senior engineers.

Lead development of Hercules's AI-focused platform-as-a-service, including its database-like object model, backend services, APIs, SDKs, hosting infrastructure, and distributed systems. Build reliable, scalable, multi-tenant services with strong isolation, observability, resilience, low latency, and high throughput. Apply advanced Rust, data systems, architecture, API design, performance optimization, and AI-assisted development skills.

Build and maintain dbt and semantic data models in Snowflake, develop ingestion and transformation pipelines, define trusted product metrics, and improve data collection. Partner with Product and Engineering to support experimentation and feature analysis. Establish data quality testing, documentation, HIPAA-aware access controls, and compliance practices. Use and validate AI-generated code and Snowflake Cortex outputs while raising technical standards across the team.

Lead architecture and strategic direction for Pinterest's data warehouse, analytics tools, and data governance. Drive cross-functional initiatives to build scalable data platforms, AI-assisted pipeline tooling, and analytics capabilities. Mentor engineers, define policies and tooling, and deliver measurable adoption and business impact across the company.

Own the technical direction and roadmap for Faire’s data infrastructure. Build CDC and streaming ingestion from production databases into Kafka, Iceberg, and S3; manage data quality, contracts, observability, SLAs, and reliability. Operate Airflow and Fivetran, lead platform migrations, establish governance, mentor engineers, and collaborate with data scientists, analysts, and senior engineering teams. The role requires hands-on development, architecture leadership, incident ownership, and cross-team influence.

Architect, build, and operate low-latency market data systems, including feed handlers, normalization, distribution, and exchange connectivity. Design high-throughput infrastructure for real-time trading data, improve reliability and performance, participate in observability, on-call, and incident response, and deliver well-tested features end to end. Partner with engineering, product, and cross-functional stakeholders while leading a lean, high-impact team.

Own and extend Airbnb’s Minerva semantic layer and data-authoring platform. Build core services, including query rewriting, and deliver reliable systems that help internal users define and consume business metrics. Scope ambiguous, cross-team initiatives, partner with users and product teams, evaluate data and database technologies, and manage design, implementation, testing, deployment, and on-call responsibilities.

Build and maintain large-scale healthcare data platform software using HL7/FHIR, Apache Spark, Python, Scala, Databricks, and cloud technologies. Shape technical roadmaps, architecture, and implementation approaches; collaborate with product and business stakeholders; maintain CI/CD and staging environments; and mentor junior engineers. The role requires deep healthcare data expertise, technical leadership, and experience with modern software engineering and integration practices.

Design and implement scalable data integrations and transformation layers for cybersecurity products. Build data contracts, dimensional/incremental models, observability, and lineage tooling. Collaborate with product engineering to integrate SIEMs (Sentinel, Splunk) and deploy cloud/on-prem solutions on Kubernetes/Docker, processing medium-to-large datasets and stream/message systems.

Design and maintain data pipelines, data contracts, and transformation layers for analytics. Model complex sources, enable observability and lineage, collaborate with product teams, and optimize data structures for scalable cybersecurity and industrial datasets.

Lead architecture and hands-on engineering to design, scale, and govern a cloud-native lakehouse using Databricks and Microsoft Fabric. Build batch/streaming pipelines, ensure HIPAA-compliant PHI handling, produce analytics-ready datasets, mentor engineers, and drive platform strategy, governance, and reliability.

Lead architecture and roadmap for a company-wide data lakehouse platform. Build platform-as-product components (SDKs, CLIs, control planes), optimize distributed compute/storage, prototype performance R&D, ensure compliance-by-design (lineage, PII masking, access control), enable real-time AI data ingestion, drive engineering standards, and mentor cross-functional teams.

Lead the data engineering strategy and architecture for distributed pharmaceutical data systems. Design and scale pipelines, APIs, and data platforms while ensuring reliability, governance, quality, and PII/PHI protection. Drive technical decisions, improve developer efficiency, deliver business impact, influence cross-functional roadmaps, mentor engineers, communicate with executives, and promote organizational adoption of new technologies and processes.

Build and maintain automated ETL/ELT data pipelines, improve data quality and monitoring, optimize Postgres and BigQuery schemas and queries, develop internal tooling, support data governance (HIPAA), participate in on-call rotation, and collaborate across engineering teams to scale data systems.