Top Staff Data Engineer Jobs in San Francisco, CA

Own end-to-end client data migrations: reverse-engineer legacy ETL, design and build Airflow DAGs and Spark jobs, validate and reconcile data at scale, document architecture and decisions, and coordinate cross-functional teams through go-live and hypercare.

Lead the strategy, technical direction, and long-term roadmap for Pinterest’s big data and AI infrastructure. Build and scale frameworks for compute, job management, resource management, scheduling, and remote shuffling across petabyte-scale datasets. Partner with internal customers on critical big data use cases and provide company-wide technical leadership on reliable, efficient data processing and storage.

Own the architecture and technical direction of Harvey’s central data platform. Build ingestion, CDC, streaming and batch pipelines into Snowflake; operate orchestration and transformation frameworks; create self-service tooling; and develop real-time processing, quality, observability, lineage, governance, PII protection, and multi-region residency capabilities. Establish technical standards, mentor engineers, and collaborate across Analytics, Data Engineering, Product, and Infrastructure.

Design and lead backend services and data pipelines for Haus’s data platform, including high-throughput ingestion, distributed systems, normalization, orchestration, data contracts, warehouse and lakehouse modeling, dbt architecture, data quality, lineage, observability, and BigQuery optimization. Set technical direction, establish engineering standards, mentor senior engineers, and collaborate with engineering, product, and data science leadership.

Lead architecture, systems, and roadmap for Coinbase's Compliance Reporting Data Platform. Design and build production-grade data models, pipelines, quality frameworks, validations, and certification processes. Establish data contracts, monitoring, reconciliation, and audit-ready processes. Partner with engineering and compliance stakeholders, respond to regulatory exams, and mentor engineers while shaping standards and documentation.

Design, build, and lead a scalable, configuration-driven data integration platform. Improve performance, automate monitoring and deployments, provide escalated support, and collaborate across product and engineering to deliver reliable, secure data integrations.

Build and operate reusable cloud data-platform infrastructure across engineering teams. Design, secure, monitor, optimize, and maintain relational, NoSQL, columnar, streaming, and analytical data systems. Establish database, schema, access, backup, recovery, privacy, and observability standards; improve infrastructure-as-code and operational tooling; and guide datastore selection and architecture. Lead cross-team initiatives, reduce technical debt, mentor senior engineers, document best practices, and influence technical strategy without direct management authority.

Lead design and implementation of identity resolution and data governance systems. Build batch and streaming pipelines, APIs, and services to ingest, normalize, link, and version identity data. Ensure auditable matching, data lineage, quality checks, schema enforcement, access controls, and privacy-by-design. Collaborate cross-functionally with product, analytics, legal, privacy, and security teams and create documentation, monitoring, and runbooks.

Lead design and operation of identity resolution and data governance systems. Build batch and streaming pipelines, APIs, and tooling for identity ingestion, matching, lineage, quality, access controls, and privacy compliance (GDPR/CCPA). Collaborate with product, analytics, legal, and security to ensure privacy-by-design, monitoring, and documented runbooks.

Design, build, and operate highly scalable distributed storage systems for Temporal Cloud. Responsibilities include developing persistence APIs, replication paths, caches, tiered storage, and observability; improving reliability and performance through SLO ownership, chaos testing, profiling, and incident reviews; and providing technical leadership through roadmap planning, RFCs, mentoring, and cross-team delivery.

As a Staff Software Engineer, you will design high-performance data infrastructure, develop scalable backend systems, and collaborate on advanced engineering solutions.

As a Staff Software Engineer, you'll lead the architecture and implementation of core systems and data platforms, ensuring adaptability and planning technical strategies.

Leads the architecture, development, operation, and evolution of PlayStation’s real-time analytics platform. Builds scalable distributed data systems using stream processing and high-performance analytics technologies, integrates batch and streaming workloads, applies AI/ML to improve platform operations, and drives reliability and performance. The role owns critical platform capabilities, establishes technical standards, mentors engineers, influences architecture across teams, supports production incidents, and shapes long-term data platform strategy.

Technical leader for Data Engineering responsible for designing, operating, and optimizing Snowflake-based data platforms, self-hosted orchestration (Prefect), dbt practices, AWS infrastructure, and data pipelines. Drive architecture, mentor engineers, partner cross-functionally, ensure observability and production-grade practices, and enable model-ready datasets and AI/LLM workflows.

Design, build, and scale reliable batch and real-time data pipelines and warehouses. Partner with engineering, data science, and business teams to define data models, improve observability, optimize storage and querying, and evangelize data engineering best practices across the company.

Build and scale AI data and ML infrastructure: evaluation systems, fine-tuning pipelines, agent-first product surfaces, high-throughput data workflows, and integrations of new models into production.

The Staff Engineer will lead technical direction on data infrastructure, focusing on ingestion frameworks, storage architecture, and orchestration to support scientific workflows. Responsibilities include mentoring, ensuring reliability, and collaborating with cross-functional teams to establish coding standards and data integrity.

Lead design and optimization of data-intensive distributed systems and query infrastructure. Improve query plans, indexing, caching, and schema/access patterns to reduce P95/P99 latency, ensure scalability, and support AI-powered data products. Mentor through design reviews and operate backend systems in the JVM ecosystem.

Design, implement, and maintain scalable distributed systems and storage for petabyte-scale 3D geospatial data. Build high-performance APIs, data ingestion and processing pipelines, monitoring/operational tooling, and collaborate with cross-functional teams to define requirements and drive the backend roadmap.

Lead the migration of Vanta’s resource data platform from MongoDB to schema-aware PostgreSQL while maintaining API compatibility and data integrity. Design high-throughput Kafka and S3 ingestion pipelines, eventual-consistency strategies, query infrastructure, tenant isolation, and reliable distributed systems. Set architectural direction across storage, caching, and streaming technologies; mentor engineers; drive cross-team technical roadmaps; and improve reliability, observability, and operational excellence.

Lead the vision and execution for scalable data governance and quality across Airbnb. Architect future-proof data ecosystems, define best practices and tooling, drive cross-functional policy implementation (privacy, legal, infosec), mentor senior engineers, and influence executive strategy to embed stewardship into the data product lifecycle.

Lead architecture and technical direction for data integrations moving clinical and financial data at scale. Design orchestration, pipelines, data stores, observability, security/compliance, and mentor engineers while owning end-to-end cross-system initiatives and reliability.

Lead architecture and implementation of agentic AI systems powering Zuma's platform. Rebuild onboarding and integration frameworks, implement analytics and observability for LLM performance, translate product needs into technical solutions, mentor engineers, establish engineering standards, and work directly with customers to enable scalable, production-grade AI experiences.

The role involves designing and scaling the data platform for Actively AI, including integrating data from various sources, ensuring data quality, and supporting real-time decision-making for automated agents.

Lead the design and development of core data storage, streaming, and indexing platforms while collaborating with cross-functional teams to deliver new features and improve processes.