We don't just use AI. We build with it, ship with it, and think with it. We're in the top 1% of companies applying AI effectively, not just to our products, but to how we build them. With an unlimited budget for Anthropic tokens, our engineers use AI agents to write, review, and ship production code every day. We're building toward a world where humans design systems and AI builds them, and we're already further along that path than almost anyone else.
As a Sr. Data Engineer, you'll be a technical leader at the center of this transformation. You'll shape platform architecture, drive AI-powered product delivery, and raise the bar for what a small, AI-augmented engineering team can accomplish. This is a role for someone who already lives in their terminal with AI, who has strong opinions about how AI changes system design, and who wants to help define what a next-generation engineering organization looks like.
What You'll Do
Build and scale high-throughput streaming pipelines. Design, implement, and operate pipelines ingesting 400M+ monthly MLS updates across 350+ integrations using Airflow, Spark Streaming, Kafka, and Iceberg—ensuring reliability, performance, and data correctness.
Model and deliver high-quality, production-grade real estate datasets. Develop and maintain datasets that power core product experiences, with a focus on data modeling, transformation logic, and balancing freshness, accuracy, and cost.
Strengthen data quality and observability. Implement and improve data quality checks, monitoring, and alerting to detect issues early and reduce downstream impact.
Leverage AI to improve data operations. Contribute to AI-driven tooling that helps triage, debug, and resolve data quality issues, increasing team efficiency and reducing manual intervention.
What We're Looking ForAttributes
You already build with AI daily. You use Claude Code as a core part of your workflow, not as a novelty
You have strong opinions, loosely held, about how AI changes software architecture, team structure, and engineering culture
You think in systems. You connect technical decisions to customer outcomes and long-term business value
You communicate clearly and directly. You can explain complex tradeoffs to product, design, and executive stakeholders
You're energized by ambiguity and speed. You thrive in a fast-growing company where the roadmap evolves and ownership is real
You like to have fun at work. We take our craft seriously, but we don't take ourselves too seriously. We celebrate wins, crack jokes, and genuinely enjoy building together
Skills and Experience
6+ years of professional data engineering or software engineering experience
Strong experience with distributed data processing and streaming systems (Spark / PySpark, Kafka)
Proficiency in Python (Pydantic preferred) and familiarity with Node/TypeScript is a plus
Experience building and maintaining data pipelines on AWS using tools like Airflow, Spark Streaming, and Iceberg
Solid understanding of data modeling and working with large-scale datasets
Familiarity with event-driven systems and ingestion patterns (Kafka, SQS)
Experience implementing data quality checks, monitoring, and debugging data issues
Interest in applying AI/ML or automation to improve data workflows is a plus
Proven track record leading high-impact initiatives from concept through production in a SaaS environment
Expert-level grasp of software design principles and experience with multi-tenant platform architectures
Tech Stack
Backend: Python, PySpark, Pydantic, Node/TypeScript
Data: Iceberg, Postgres
Infrastructure: AWS, Kubernetes, Airflow, Spark Streaming
Messaging: Kafka, SQS
Top Skills
Similar Jobs at Luxury Presence
What you need to know about the San Francisco Tech Scene
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
