Databricks Data Engineer

EXL Service is seeking an accomplished Senior Databricks Data Engineer with 10–12 years of experience designing, developing, and optimizing enterprise data platforms and large-scale ETL solutions. The ideal candidate brings deep expertise in Databricks, Spark, Delta Lake, Azure Data Platform, and modern data engineering practices.

This role is responsible for building scalable data pipelines, implementing cloud-native data solutions, improving platform performance, and delivering reliable data products for analytics, reporting, and AI/ML initiatives.

Base Compensation Range: 65,000 – 87,000

The posted range is the hiring range for this role — a subset of the broader range available to employees over time — and reflects base salary across our national hiring scale. Final offers are based on several factors, including the candidate's skills and experience, internal pay equity, work location, market conditions for the role, and the specific scope and responsibilities of the position. The top of the range is reserved for candidates who notably exceed the requirements; the lower end applies to those with less experience or fewer preferred qualifications. For positions based in higher-cost zones (e.g., California, New York, New Jersey), actual compensation may exceed the posted range; your recruiter will share specifics during the process.

For more information on benefits and what we offer please visit us at https://www.exlservice.com/us-careers-and-benefits

Key Responsibilities

Data Engineering & Platform Development

• Design, develop, and optimize enterprise-scale data pipelines using Databricks, Spark, Delta Lake, and Azure Data Lake Storage.
• Build batch and incremental ETL/ELT pipelines, real-time streaming architectures, and reusable data integration frameworks. 
• Develop scalable ingestion frameworks supporting structured, semi-structured, and API-based data sources.
• Build reusable data engineering frameworks for ingestion, transformation, validation, reconciliation, and publishing.
• Develop Delta Lake solutions using partitioning, optimization, Z-Ordering, Liquid Clustering, and performance tuning techniques.
• Implement CDC, incremental processing, merge strategies, and data synchronization across enterprise platforms.
• Develop Databricks Workflows, notebooks, SQL Jobs, and automation for production workloads.
• Integrate external REST APIs and process JSON/XML data into analytics-ready datasets.
• Implement robust data quality checks, audit frameworks, monitoring, and error handling.
• Optimize Spark jobs for performance, scalability, and cost efficiency.

Databricks & Azure Development

• Develop solutions using Azure Data Lake Storage Gen2, Databricks, Unity Catalog, Azure Key Vault, Azure DevOps, and Azure Synapse.
• Build secure data pipelines using Unity Catalog, RBAC, service principals, and managed identities.
• Develop reusable notebook frameworks using PySpark and Spark SQL.
• Implement CI/CD deployment pipelines using Azure DevOps.
• Manage environment promotion across Development, QA, UAT, and Production.
• Troubleshoot production issues and optimize workloads for reliability and scalability.

Data Integration & Analytics

• Design enterprise data models supporting reporting, analytics, and downstream applications.
• Develop healthcare and financial data integration pipelines supporting multiple source systems.
• Build reusable metadata-driven ETL frameworks.
• Integrate third-party APIs including NLP, terminology normalization, and identity resolution services.
• Support data governance, lineage, and metadata management initiatives.

Required Skills & Experience

Technical Expertise

• 10-12 years of experience in Data Engineering, ETL Development, and Data Warehousing.
• Hands-on Databricks development experience.
• Strong experience with Spark, Delta Lake, Unity Catalog, Databricks Workflows, and SQL Warehouses.
• Strong experience with PySpark, Spark SQL, SQL, and Python.
• Experience building enterprise ETL/ELT pipelines using Databricks and Azure Data Platform.
• Experience with Azure Data Lake Storage (ADLS Gen2), Azure Synapse, Azure Key Vault, and Azure DevOps.
• Experience implementing CDC, SCD, incremental loading, and data quality frameworks.
• Experience integrating REST APIs and processing JSON/XML datasets.
• Experience with Git, CI/CD, release management, and deployment automation.
• Strong knowledge of performance tuning, partitioning, caching, broadcast joins, and Spark optimization.
• Experience working with healthcare data platforms is preferred.
• Microsoft Azure certifications (e.g., DP-203 Azure Data Engineer Associate) or Databricks certifications (Data Engineer Associate/Professional).
• Experience with Ab Initio suite of products is also preferred.