The Principal Data Engineer will design and implement scalable data platforms, lead architecture discussions, and mentor teams. Requires expertise in Azure and Databricks.
About BayRock Labs
At BayRock Labs, we pioneer innovative tech solutions that drive business transformation. As a leading product engineering firm based in Silicon Valley, we provide full-cycle product development, leveraging cutting-edge technologies in AI, ML, and data analytics. Our collaborative, inclusive culture fosters professional growth and work-life balance. Join us to work on ground-breaking projects and be part of a team that values excellence, integrity, and innovation. Together, let's redefine what's possible in technology.
We are looking for a highly skilled Senior Data Engineer / Data Architect with deep hands‑on expertise in Azure, Databricks, PySpark, Python, SQL, and modern data warehousing. This role is ideal for someone who has built large‑scale data platforms, designed enterprise‑grade architectures, and delivered high‑quality data products in cloud environments.
Core Technical ExpertiseAzure Cloud Platform- ADLS Gen2, ADF, Synapse, Azure SQL, Key Vault, App Services
- Cloud‑native architecture design, scalability, and cost optimization
- Security & governance: RBAC, Managed Identity, Private Endpoints
- End‑to‑end development on Azure Databricks
- Lakehouse architecture implementation
- Delta Live Tables (DLT), Unity Catalog
- Performance tuning, cluster optimization
- CI/CD for Databricks workloads
- Apache Spark with advanced PySpark transformations
- Structured Streaming & batch data pipelines
- Delta Lake architecture & optimization
- Python for automation, data engineering, and orchestration
- PySpark for distributed data processing
- SQL for complex query optimization and analytics engineering
- Enterprise Data Warehouse (EDW) architecture
- Dimensional modeling (Star/Snowflake)
- Data Vault 2.0 modeling
- Metadata‑driven ingestion frameworks
- Change Data Capture (CDC) implementation
- Medallion architecture (Bronze/Silver/Gold)
- Data lineage, governance, cataloging
- Master Data Management (MDM)
- Design and delivery of scalable data products
- Business‑aligned semantic layer design
- KPI frameworks & enterprise reporting enablement
- Integration of ERP, SaaS, and operational systems
- Hybrid Lakehouse + EDW architectures
- CI/CD pipelines (Azure DevOps, GitHub Actions, Bitbucket)
- Infrastructure as Code (Terraform, ARM templates)
- Automated testing (unit, integration, data quality)
- Monitoring, logging, and observability
- Agile/Scrum delivery
- Principal‑level solution architecture design
- Cross‑functional stakeholder collaboration
- Technical roadmap planning and execution
- Guidance on data governance frameworks (GDPR, SOX)
- Cloud cost optimization strategies
- Migration from legacy EDWs (Teradata, Oracle, SQL Server) to Lakehouse
- 5–10+ years of hands‑on experience in data engineering or data architecture
- Strong expertise in Azure + Databricks ecosystem
- Proven experience building scalable, secure, high‑performance data platforms
- Ability to lead design discussions, influence architecture decisions, and mentor teams
- Strong problem‑solving mindset and passion for modern data engineering
Top Skills
Azure
Databricks
Pyspark
Python
SQL
BayRock Labs San Jose, California, USA Office
San Jose, California, United States, 94089
Similar Jobs
Cloud • Information Technology • Security • Software
The Principal Engineer will lead the architecture and implementation of AI-powered systems for data analytics, collaborating with teams to create closed-loop systems and driving engineering excellence while measuring business outcomes.
Top Skills:
AIAthenaCi/CdClickhouseCloudFormationDatabricksGoIacJavaKubernetesMlPrestoPythonRedshiftSnowflakeTerraformTrino
Digital Media • Fintech • Information Technology • Machine Learning • Financial Services • Cybersecurity • Automation
The Principal Data Engineer will design and implement enterprise data frameworks, mentor teams, and ensure data quality while collaborating with stakeholders.
Top Skills:
AngularApache KafkaSparkAWSBeamDatastageEtl ToolsFlinkGlueJavaLake FormationPythonReactRedshiftS3ScalaSpring BootSQLTalend
Fintech
Lead the integration between data engineering and analytics by building and managing ETL processes, designing data pipelines, and implementing best data practices.
Top Skills:
AirflowAws LambdaBitbucketDbtDockerGitGitPythonS3SQL
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
