Own and evolve end-to-end data pipeline architecture across ingestion, transformation, modeling, and serving. Lead platform improvements (cost, compute efficiency, access control), CI/CD, monitoring, and incident response. Drive cross-team technical initiatives, mentor engineers, and integrate AI tooling to improve workflows while remaining hands-on building and debugging production pipelines.
About the Role
Here's what you'll do as part of the team
Here are the skills and experience you'll need to be successful
Here's the Pay Range:
As Apartment List's data platform grows across more domains, more pipelines, and more stakeholders, the architectural decisions made today determine how much technical debt we're paying down in a year. We're looking for a Senior Data Engineer II (IC4) to own that architecture; end-to-end pipeline design, platform investment tradeoffs, and the technical judgment calls that keep the system reliable as it scales.
This is not a design-from-the-whiteboard role. You'll still be hands-on-keyboard: writing pipelines, debugging production issues, and shipping code alongside the team. What sets this role apart is scope; you'll make architectural decisions independently, influence how Analytics Engineering, Data Science, and Engineering partners build on the platform, and be the person other data engineers come to when a design decision needs a second opinion. You'll work closely with Analytics Engineering, Data Science, and Engineering partners to shape how the data platform evolves.
- Own and evolve data pipeline architecture across core domains — ingestion, transformation, modeling, and serving — making project-level architectural decisions independently and evaluating tradeoffs between freshness, cost, scalability, and simplicity.
- Lead the design and implementation of platform-level improvements: warehouse cost management, compute efficiency, and access control architecture, treating reliability, observability, and cost efficiency as core design constraints rather than afterthoughts.
- Identify and lead technical initiatives that improve the platform's long-term health — proactively surfacing investments (orchestration, CI/CD, data access, developer experience) before they become blockers, and making the case for them.
- Drive large, technically complex projects or multiple concurrent medium-sized initiatives that span teams, taking responsibility for outcomes rather than just execution.
- Lead monitoring and testing strategy for your domain: proactively close observability gaps across the org, build alerting ahead of failures, and serve as the go-to engineer for the hardest production issues.
- Influence technical decisions and architectural direction beyond the Data & Analytics team, partnering directly with EPD stakeholders on infrastructure decisions that affect their roadmaps.
- Actively mentor other data engineers and analytics engineers, reviewing architectural and modeling decisions, and operate as a technical peer to senior engineers across teams.
- Integrate AI meaningfully into data engineering workflows — building tooling and automation that creates leverage for the whole team, not just individual output, and coaching others on effective, validated use.
Must-haves:
- 7+ years of data engineering experience, including a demonstrated track record of owning end-to-end pipeline architecture, not just implementing to spec.
- Deep experience designing orchestration workflows in Apache Airflow, including making architectural tradeoffs across ingestion, transformation, modeling, and serving layers.
- Experience working with containerized data infrastructure in production, including deploying services, diagnosing operational issues, and contributing to platform reliability and scalability.
- Demonstrated ability to evaluate and communicate architectural tradeoffs (freshness vs. cost, scalability vs. simplicity) to both technical and non-technical stakeholders.
- Experience building or significantly improving CI/CD practices for data pipelines, including automated testing, validation, and deployment.
- A track record of leading incident response and monitoring strategy for a domain, including building alerting and observability ahead of failures rather than reacting to them.
- Experience influencing technical decisions across multiple teams or functions, including partnering with engineering, product, or data science stakeholders outside your immediate team.
- Experience mentoring other data engineers, including reviewing architectural and modeling decisions.
Nice-to-haves:
- Experience with Kubernetes-based data infrastructure
- Experience leading a legacy ETL-to-modern-orchestration migration end-to-end, not just contributing to one.
- Familiarity with observability and monitoring tooling such as Datadog at a platform-wide scale.
- Experience building internal tooling or automation (including AI-assisted) that other engineers rely on.
At Apartment List, we carefully consider a variety of factors to determine compensation for each position, including the role, level, and work. The US Total Target Compensation (TTC) for this position is:
- Zone 1: $171,000 - $207,000 TTC (including $154,000 - $182,000 base salary) + equity
- Zone 2: $158,000 - $191,000 TTC (including $142,000 - $168,000 base salary) + equity
- Zone 3: $145,000 - $176,000 TTC (including $130,000 - $155,000 base salary) + equity
This reflects the compensation target for new hire salaries for the position across all US locations. Please note, the compensation details provided do not include benefits and perks that we offer.
We also rely on market indicators along with considering your work location, job related skills, experience and relevant education and training, to determine compensation that is fair and competitive for you. Apartment List will consider paying compensation near the higher of the range in exceptional circumstances, where candidates have the experience, credentials or expertise that would warrant such consideration. It is always our goal to hire exceptional talent and we would be happy to share more about compensation during the hiring process.
Apartment List San Francisco, California, USA Office
Although we are Virtual First, we have an office space in downtown San Francisco to host off-sites. With Oracle Park and Chase Center just a few blocks away, we're in the perfect location for an easy commute into the office and happy hour at a trendy bar or restaurant after work.
Similar Jobs
Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Lead design and operation of large-scale data pipelines and data APIs. Build Spark/PySpark workflows on Databricks, optimize job performance, manage data quality and observability, and develop MCP servers and AI-agent integrations. Mentor engineers, define standards, support production incidents and on-call rotations, and collaborate with stakeholders to deliver scalable data platform products.
Top Skills:
Apache IcebergApi GatewayAWSAws LambdaAws Rds/AuroraAzureDatabricksDatadogDbtFastapiFivetranGCPGoogle BigqueryLlms/Ai AgentsMcp ServersMs Sql ServerMySQLOraclePostgresPysparkPythonS3SecretsmanagerSnowflakeSnsSparkSplunkSQLSqs
Kids + Family • Mobile
Design, build, and maintain scalable distributed data pipelines and a secure data lakehouse for streaming and batch processing to support real-time analytics, ML, and experimentation. Automate, test, and harden workflows, architect logical and physical data models, build ML model features, and collaborate with product, analytics, and data science teams to turn data into value.
Top Skills:
AirflowAWSAzureBigQueryDabsData LakehouseDatabricksDatabricks WorkflowsDbtGCPGithub ActionsLlmsPrestoPythonSnowflakeSparkSQLTerraformTrino
Insurance
Design, build, and operate scalable batch and streaming data pipelines using Apache Spark and cloud lakehouse platforms. Develop ETL/ELT workflows, data models, CI/CD for data workloads, ensure data quality and performance, monitor and troubleshoot jobs, and collaborate with analytics, platform, security, and product teams to deliver analytics-ready datasets.
Top Skills:
SparkCi/CdData LakeEtl/EltInfrastructure-As-CodeLakehouseMicrosoft FabricOnelakePythonSQLVersion Control
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
