Senior Data Engineer

Position Summary

Penn Foster Group is seeking an experienced Senior Data Engineer to help shape the future of our enterprise data platform and AI strategy. This is a highly visible technical Leadership role responsible for architecting, building, and evolving our modern cloud data platform while establishing engineering standards, mentoring a growing team, and driving innovation across the organization.

As a Senior Data Engineer, you will serve as a Senior Technical expert for our Databricks Lakehouse platform, providing hands-on Leadership across data ingestion, transformation, governance, analytics enablement, and AI-ready data products. You will partner closely with Business Intelligence, Product, Platform Engineering, Security, Data Governance, and MLOps teams to build trusted, scalable, and governed data solutions that power enterprise analytics, machine learning, and generative AI.

This role combines deep technical expertise with Leadership, mentoring, and strategic influence. You'll help establish the long-term technical direction of Penn Foster Group's data ecosystem while coaching and developing engineers, driving engineering excellence, and implementing modern data engineering best practices.

Why Join Penn Foster Group?

Penn Foster Group is in the midst of an exciting enterprise-wide data platform modernization journey. We are investing in a modern cloud-native architecture built on Databricks, Azure, and AI-enabled technologies that will become the foundation for analytics, operational reporting, machine learning, and generative AI across the company.

This is a unique opportunity to join at a transformational point in our technology journey. Rather than simply maintaining existing systems, you'll help define the future of our data platform—shaping architecture, engineering standards, governance, and the adoption of emerging AI technologies that will influence how data is used across the organization for years to come.  You will also be a key player in our drive for Self Service Analytics at PFG.

As our Senior Data Engineer, you will have the opportunity to:

• Help define the long-term strategy for Penn Foster Group's modern Data & AI platform.
• Influence enterprise architecture decisions around Databricks, Azure, Unity Catalog, semantic data products, and AI-powered analytics.
• Build modern, scalable data products that enable self-service analytics and generative AI.
• Lead adoption of emerging Databricks capabilities, including Genie, AI/BI, Unity Catalog, Lakeflow, and other evolving platform innovations.
• Mentor and develop a growing Data Engineering team while establishing engineering standards that scale across the organization.
• Partner directly with technology and business Leaders to solve meaningful business problems using modern cloud and AI technologies.

If you're passionate about building modern data platforms, mentoring engineers, and helping shape the future of AI-enabled analytics, this is an opportunity to make a lasting impact.

Key Responsibilities

Technical Leadership

• Provide technical Leadership for the Data Engineering team through architecture guidance, code reviews, mentoring, and engineering best practices.
• Mentor and develop junior and mid-level Data Engineers, fostering technical growth, knowledge sharing, and continuous learning.
• Establish engineering standards for software development, testing, CI/CD, documentation, observability, and operational excellence.
• Senior technical design discussions, evaluate architectural tradeoffs, and drive adoption of modern engineering practices.
• Champion a culture of quality, collaboration, innovation, and continuous improvement.

Data Platform & Databricks Leadership

• Serve as the technical Senior Engineer for Penn Foster Group's Databricks Lakehouse platform.
• Design, build, and support scalable enterprise data pipelines using SQL, Python, Apache Spark, and Databricks.
• Define best practices for Databricks development, including Workflows, Repos, Jobs, notebooks, reusable Python libraries, Git integration, cluster policies, SQL Warehouses, and deployment automation.
• Design and optimize Delta Lake architectures using Medallion patterns, Delta optimization, Liquid Clustering, and Photon.
• Build reusable ingestion frameworks supporting batch, streaming, CDC, and API-based integration patterns.
• Optimize Spark workloads for performance, scalability, reliability, and cloud cost efficiency.

AI & Databricks Genie Enablement

• Partner with Analytics and business stakeholders to develop trusted semantic data products that power Databricks Genie.
• Design and maintain Genie Spaces and Genie Ontologies that accurately represent business entities, relationships, metrics, and terminology.
• Establish best practices for semantic modeling, governed metrics, business metadata, and AI-ready datasets.
• Evaluate and implement emerging Databricks AI capabilities to improve self-service analytics and business productivity.
• Collaborate with Data Science and MLOps teams to enable machine learning, generative AI, and advanced analytics initiatives.

Data Architecture, Governance & Operational Excellence

• Design scalable Lakehouse architectures supporting analytics, reporting, AI, and operational data products.
• Partner with Data Governance teams to implement enterprise governance using Unity Catalog, including data lineage, fine-grained security, metadata management, and access controls.
• Champion automated testing, monitoring, observability, data quality, and production reliability.
• Serve as the technical escalation point for complex production issues and Senior root cause analysis and continuous improvement efforts.
• Drive platform modernization initiatives while balancing delivery, scalability, maintainability, and operational excellence.

Cross-Functional Partnership

• Collaborate with Business Intelligence, Product, Platform Engineering, Security, and MLOps teams to deliver trusted, scalable data products.
• Translate business requirements into modern technical solutions that support enterprise reporting, analytics, AI, and strategic decision-making.
• Contribute to technical roadmaps, platform strategy, and the continued evolution of Penn Foster Group's Data & Analytics capabilities.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience.
• 6+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Engineering.
• 2+ years of experience leading technical initiatives and mentoring engineering teams.
• Demonstrated success developing engineers while driving engineering excellence and technical standards.
• Strong communication and collaboration skills with the ability to influence both technical and business stakeholders.

Technical Qualifications

Required

• Expert knowledge of the Databricks Lakehouse Platform, including:
  • Apache Spark (PySpark)
  • Delta Lake
  • Unity Catalog
  • Databricks Workflows & Jobs
  • Repos
  • SQL Warehouses
  • Cluster Policies
  • Serverless Compute
  • MLflow
  • Lakehouse Monitoring
  • Auto Loader
  • Delta Live Tables / Lakeflow
• Expert Knowledge in SQL and Python.
• Strong Experience working with BI tools such as PowerBI, Tableau or Business Objects
• Strong Microsoft Azure experience, including ADLS Gen2, Microsoft Entra ID (Azure AD), RBAC, networking, and cloud security.
• Experience implementing Medallion Architecture, dimensional modeling, and domain-oriented data products.
• Deep understanding of Spark optimization techniques, including Adaptive Query Execution, partitioning, caching, Photon, Liquid Clustering, and Delta optimization.
• Experience implementing CI/CD pipelines, Git-based development workflows, Infrastructure as Code, and automated testing.
• Strong understanding of data governance, metadata management, data quality, observability, security, and compliance.

Preferred

• Experience implementing Databricks Genie, Genie Spaces, and Genie Ontologies.
• Experience designing semantic models and AI-ready data products.
• Experience supporting Power BI, Tableau, or other enterprise BI platforms.
• Experience with machine learning platforms, MLOps, or Generative AI applications.
• Experience with dbt, Great Expectations, or similar modern data engineering tools.
• Databricks Certified Data Engineer Professional and/or Microsoft Azure certifications.

• U.S. Work Location Requirement: All employees must reside and perform work within the United States. International remote work arrangements are not supported for this role.
• Travel Expectations: This is a remote position, though occasional travel may be required for team meetings, company events, conferences, summits, and similar activities. Candidates must be willing and able to travel as needed.
• On-Camera Work Environment: Penn Foster Group operates in a highly collaborative, remote, on-camera culture. Employees are expected to be on camera during meetings unless extenuating circumstances apply.
• Recorded Video Interviews: Video interviews conducted during the hiring process will be recorded to support internal hiring consistency and process improvement.
• Background Checks: Employment with Penn Foster Group is contingent upon successfully completing applicable pre-employment screening requirements. These may include verification of employment history, education, and criminal background, where permitted by law.
• Identity Verification (Form I-9): As part of the onboarding process, all hires must complete employment eligibility verification in compliance with federal law. This includes remote Form I-9 verification, which may require an in-person identity verification step with an authorized representative.

About Us: At Penn Foster Group, we are transforming online learning to help learners by bringing together Penn Foster, CareerStep, Ashworth College, James Madison High School, the New York Institute of Photography, the New York Institute of Art and Design, and other education platforms. Together, we create an accelerated path to greater economic mobility through real-world skills and knowledge that enable learners to achieve long-term success in the workplaces of the future.  Our history dates back to 1890 when our founder, Thomas Foster, pioneered distance education by offering training by mail for coal miners to get the necessary skills for safer jobs. Today, with the partners who use our education and training programs, we continue that mission of providing accessible training and education for in-demand skills and are building a workforce that’s prepared for the future job market.

Equal Employment Opportunity: We strive toward Diversity, Equity, and Inclusion at Penn Foster Group by intentionally building diverse teams – in identities, lived experiences, and ideas to create a culture where people feel connected to each other and have a sense of belonging. We value diversity, equity, and inclusion because it is the foundation that enables us to achieve what we set out to do as an organization – from maximizing the number of learners who can reach their goals while giving them the kinds of experiences we want them to have, to becoming the type of company we want to work in.  

What We Offer: We offer a robust benefits package that includes medical, dental, vision, flexible spending, generous paid time off, sponsored volunteer opportunities, a 401K with a company match, and free access to our online programs.