- As a Databricks Data Engineer, you will work directly with clients across multiple industries to design, implement, and optimize Databricks-based data solutions
- You will be a key member of our Professional Services delivery teams, delivering high-quality projects on time and within scope while building strong client relationships
This is a client-facing role that combines hands-on technical delivery with consulting best practices
Key Responsibilities
- Lead and contribute to end-to-end Databricks implementations for clients, including data migration, Lakehouse architecture, and pipeline development
- Gather technical requirements, design solutions, and present recommendations to client stakeholders (technical and business)
- Build scalable ETL/ELT pipelines using PySpark, Delta Lake, Delta Live Tables (DLT), and Databricks Workflows
- Design and implement Databricks Genie
- Design and implement semantic layers
- Use Databricks AI features to accelerate development, debugging, and code optimization
- Design and implement Lakebase architectures for operational and analytical workloads, including transactional data use cases
- Develop solutions using SDLC best practices, including modular code design, testing, and documentation
- Use Git based version control with proper branching strategies
- Implement CI/CD pipelines for Databricks asset
- Implement data quality checks, validations, and expectations within workflows
- Design and implement Unity Catalog governance, security, and lineage solutions
- Optimize Databricks workloads for performance, cost, and reliability (Photon, cluster policies, Liquid Clustering, Auto Loader, etc.)
- Integrate Databricks with client ecosystems (Azure, AWS, GCP, Snowflake, Kafka, legacy systems, etc.)
- Support client workshops, proof-of-concepts (POCs), and knowledge transfer sessions
- Collaborate with client data teams to ensure successful adoption and handover of solutions
- Deliver projects following consulting methodologies while meeting quality, timeline, and budget expectations
- Document architectures, runbooks, and best practices for client use
Participate in solutioning activities (scoping, estimation, technical demos) as needed
Requirements (Must-Have)
- 3 -5 years of hands-on Databricks experience (or strong Spark experience with significant recent Databricks work)
- Proven experience delivering Databricks projects in a consulting or professional services environment (preferred) or equivalent client-facing project delivery
- Strong proficiency in PySpark, Spark SQL, Python, and SQL
- Deep experience with Delta Lake, Unity Catalog, Delta Live Tables, and Databricks Jobs
- Hands-on experience with Git version control, pull requests, code reviews, and collaborative development workflows
- Cloud platform experience (Azure Databricks, AWS, or GCP - at least one)
- Excellent client-facing and communication skills - able to explain complex concepts to both technical and non-technical audiences
- Solid understanding of data governance, security, and Lakehouse best practices
Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience)
Nice-to-Haves
- Databricks certifications (Data Engineer Associate / Professional, Lakehouse, etc.)
- Experience with dbt, Airflow, Terraform, Databricks Asset Bundles (DABs), or MLflow
- Background in specific industries (Financial Services, Healthcare, Retail, Manufacturing, etc.)
- Experience with large-scale data migrations or legacy system modernization
Knowledge of streaming (Spark Structured Streaming / Kafka) and real-time analytics
Skills & Qualities
- Strong consulting mindset: ownership, adaptability, and client success focus
- Excellent problem-solving, analytical, and presentation skills
- Ability to work independently and as part of a delivery team
- High emotional intelligence and stakeholder management ability
Commitment to delivering exceptional client outcomes
What We Offer
Datavail is one of the largest data focused services company in North America and provides both professional and managed services and expertise in Database Management, Application Development and Management, Cloud & Infrastructure Management, Packaged Applications and BI/Analytics
As an Oracle Platinum Partner with 17 specializations, we have extensive experience with everything Oracle. Our experts have an average of 16 years of experience. They’ve overcome every obstacle in helping clients manage everything from databases, BI analytics, reporting, migrations, and upgrades to monitoring and overall data management.
You can free up your IT resources to focus on growing your business rather than fighting fires. Our Oracle experts can guide you through strategic initiatives or support routine database management.
Datavail’s Comprehensive Oracle Database Services
Datavail offers Oracle consulting services that allow you to take advantage of all the features of the Oracle database. We can also assist you in designing, implementing, and managing a wide range of Oracle applications.
Datavail’s business focuses on helping you use your data to drive business results through cost-saving services. The success of your business depends on how well you understand and manage your data. Our Oracle managed cloud services give you the power to unleash your organization’s potential. We provide comprehensive and technically advanced support for Oracle installations to ensure that your databases are safe, secure, and managed with the utmost level of care.
Our delivery performance in data management leads the industry. We offer highly trained Oracle database administrators via a 24×7, always on, always available, global delivery model. Datavail’s flexible and client focused services always add value to your organization. Our Oracle database managed services and products include:
Similar Jobs
What you need to know about the San Francisco Tech Scene
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
