Leads a data engineering team and oversees the design, development, reliability, security, and scalability of data pipelines, warehouses, platforms, and integrations. Translates business needs into technical solutions, establishes data quality and governance standards, reviews architecture and code, manages projects and resources, resolves infrastructure issues, mentors engineers, and communicates progress and risks to leadership.
The Data Engineering Manager leads the design, development, and maintenance of data infrastructure and pipelines that support analytics, reporting, machine learning, and business operations. This role combines technical leadership with people management, ensuring that data systems are reliable, scalable, secure, and aligned with business requirements.
Key Responsibilities
- Lead and manage a team of data engineers.
- Design and oversee scalable data pipelines, data warehouses, and data platforms.
- Establish standards for data quality, reliability, security, and governance.
- Collaborate with data scientists, analysts, software engineers, and business stakeholders.
- Translate business requirements into effective data engineering solutions.
- Monitor data infrastructure and resolve performance or reliability issues.
- Oversee data integrations from internal and external sources.
- Review technical designs and code to maintain engineering quality.
- Plan projects, prioritize workloads, and manage engineering resources.
- Mentor team members and support their professional development.
- Evaluate and introduce appropriate data technologies and tools.
- Ensure data systems comply with organizational security and privacy requirements.
- Monitor project timelines, budgets, and deliverables.
- Communicate technical progress, risks, and recommendations to senior leadership.
Qualifications
- Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related field.
- 5+ years of experience in data engineering or a related technical field.
- Previous experience leading or managing data engineering teams.
- Strong knowledge of SQL and programming languages such as Python or Java.
- Experience with data warehouses, databases, ETL/ELT processes, and data pipelines.
- Experience with cloud platforms such as AWS, Microsoft Azure, or Google Cloud.
- Strong understanding of data architecture and data modeling.
- Excellent leadership, communication, and problem-solving skills.
Preferred Skills
- Experience with platforms such as Snowflake, Databricks, BigQuery, or Redshift.
- Familiarity with Apache Spark, Kafka, Airflow, or similar data technologies.
- Experience implementing CI/CD and automated testing for data pipelines.
- Knowledge of data governance, security, and compliance practices.
- Experience working in Agile/Scrum environments.
- Strong project management and stakeholder-management skills.
Similar Jobs
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Lead Atlassian’s TDP SQL engineering team, owning people leadership, platform strategy, roadmap execution, reliability, security, compliance, incident response, technical direction, cross-functional dependencies, infrastructure economics, and operational outcomes. Build a high-performing organization and evolve a scalable managed relational database platform across multiple cloud environments while supporting enterprise requirements such as data residency, encryption, disaster recovery, and tenant isolation.
Top Skills:
Backup And RecoveryCloud InfrastructureDisaster RecoveryDistributed Relational DatabasesEncryptionMulti-Cloud InfrastructureSQLTenant Isolation
Fintech • Financial Services
Lead and mentor data engineering and DBA teams while driving GCP-based streaming and batch data platform initiatives. Oversee BigQuery, dbt, Apache Flink or Dataflow, data quality, testing, observability, governance, reliability, and cloud cost optimization. Remain hands-on with coding, code reviews, architecture, and complex system problems while aligning data infrastructure with business and product goals. Champion AI-assisted engineering and automation across the platform.
Top Skills:
Apache FlinkBigQueryDataflowDbtGoogle Cloud PlatformMySQLPostgres
Fintech • Financial Services
Lead the data engineering squad and oversee the DBA team while driving GCP-based streaming and batch infrastructure projects. Build and review code, establish standards for data quality, testing, observability, governance, and cost optimization, and align data platforms with business goals. The role requires expertise in GCP, BigQuery, dbt, Apache Flink or Dataflow, relational databases, AI-assisted engineering, and team leadership.
Top Skills:
Ai Coding AssistantsApache FlinkBigQueryDataflowDbtGoogle Cloud Platform (Gcp)MySQLPostgres
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
