Data Engineering Lead

Location: United States

About MediaRadar

MediaRadar, An Industry Leader in Marketing Intelligence now including the data and capabilities of Vivvix, powers the mission-critical marketing and sales decisions that drive competitive advantage. Our next-generation marketing intelligence platform enables clients to achieve peak performance with always-on insights that span the media, creative, and business strategies of 5 million brands across 30+ media channels and 275 billion in media spend. By bringing the advertising past, present, and future into focus, our clients rapidly act on the competitive moves and emerging advertising trends impacting their business.
About the Role:
We are seeking a visionary and hands-on Data Engineering Lead to spearhead the design, development, and optimization of our next-generation data platform. As a Lead, you will balance technical excellence with people leadership, ensuring our data architecture is scalable, resilient, and perfectly aligned with our business goals - while collaborating cross-functionally to support analytics, reporting, and operational data needs..The ideal candidate should be a PySpark expert who thrives in the Azure ecosystem and has a deep appreciation for clean, modular code and robust ETL patterns.This is an exciting opportunity to work along with a great team of data engineers, demanding technologies and an engaging work environment to help shape our data engineering best practices.

Key Responsibilities:
Technical Leadership: Design and supervise the implementation of comprehensive data pipelines utilizing Azure Databricks and PySpark.

Team Mentorship: Direct a team of data engineers, performing code reviews, offering technical expertise, and cultivating a culture of ongoing learning.

Data Modeling & Optimization: Develop high-performance schemas in PostgreSQL and refine complex SQL queries for large datasets.

ETL Strategy: Establish and apply optimal practices for data ingestion, transformation, and storage (Delta Lake/Lakehouse patterns).

Strategic Collaboration: Collaborate closely with Data Analysts, Architects, and Product Managers to convert business requirements into technical specifications.

Process Improvement: Promote the implementation of CI/CD, unit testing, and automated monitoring to achieve 99.9% data reliability.

Ensure data quality, governance, and compliance through validation, documentation, and secure practices.

Continuously improve data systems for enhanced performance, reliability, and scalability.

Effectively engage within an agile, cross-functional project team.

Mandatory Skillset

● Azure Databricks:

• Expert-level experience managing workspaces, clusters, and job scheduling.
• Solid understanding of data lakehouse architectures and Delta Lake.
• Proven experience in Performance Tuning, Spark Optimization and Cost Reduction.

● PySpark: Advanced proficiency in Spark DataFrame APIs and Spark SQL for large-scale data processing involving various data formats.

● SQL Mastery: Exceptional ability to write, tune, and troubleshoot complex queries.

● PostgreSQL: Hands-on experience with relational database design, indexing, and performance optimization.

● ETL/ELT Frameworks: Proven track record of building scalable data pipelines from scratch. Desired Skills

● Workflow Orchestration: Experience with Apache Airflow for managing complex task dependencies.

● Containerization: Familiarity with Azure Kubernetes Service (AKS) for deploying containerized data services.

● Infrastructure as Code (IaC): Knowledge of Terraform or Bicep for managing Azure resources.

Qualifications

● 10+ years of experience in Data Engineering or Software Engineering.

● 3+ years as a formal technical Lead managing an agile team and implementing E2E solutions.

● Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

● Strong communication skills with the ability to explain complex technical concepts to non-technical stakeholders.

● Strong problem-solving skills and attention to detail.

The Modern Data Stack Architecture :

Why join us? You won't just be "maintaining" pipelines; you'll be the primary architect of a data ecosystem that powers real-time decision-making across the entire organization.

In addition to career progression, training and development, and an excellent work/life balance, future Radarians can expect a great benefits package that includes:

• Medical, Dental & Vision Insurance
• 401k with Company Match
• Flexible PTO
• Commuter Benefits
• Gym Discounts
• Summer Fridays

At MediaRadar, we are committed to creating an inclusive and accessible workplace where everyone can thrive. We believe that diversity of backgrounds, perspectives, and experiences makes us stronger and more innovative. We are proud to be an Equal Opportunity Employer and make employment decisions without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, disability, genetic information, or any other legally protected status.

In accordance with the EEO-1 reporting requirements, we collect demographic data as part of our efforts to ensure fair and equitable hiring practices across all levels of our organization.

We are committed to ensuring our recruitment process is accessible to all applicants. If you need a reasonable accommodation during the application or interview process, please contact us at [email protected] .

We’re excited to meet people who share our values and want to build the future with us!