crewAI Logo

crewAI

Lead Data Engineer, Data Platform

Posted 12 Days Ago
Be an Early Applicant
In-Office
San Francisco, CA, USA
Senior level
In-Office
San Francisco, CA, USA
Senior level
Own and build CrewAI's end-to-end data foundation: consolidate telemetry and application data, design pipelines and warehouses, define trusted metrics and semantic models, improve instrumentation and data quality, enable self-serve analytics, and support product and GTM decisions with reliable dashboards and analyses.
The summary above was generated by AI
About CrewAI

CrewAI is the leading framework and enterprise platform for building and orchestrating multi-agent AI systems, powering 300M+ agent executions per month across thousands of companies. As the product, platform, and customer base scale, data is becoming one of the most important systems in the company: how we understand usage, reliability, activation, customer health, cost, governance, and where to invest next.

Today, we have meaningful data already, but it is spread across product telemetry, trace data, application databases, analytics tables, Cube models, Metabase dashboards, and team-specific queries. We need someone to turn that into a coherent, trusted, useful data foundation.

The Role

You’ll be CrewAI’s first dedicated data engineering hire. Your job is to own the data foundation end to end: rationalize what exists, improve the infrastructure, define trusted metrics, close instrumentation gaps, and make data accessible enough that product, growth, engineering, customer success, and leadership can actually use it.

This is a foundational role with real range. The center of gravity is data infrastructure and analytics engineering: pipelines, warehouse/lake design, semantic modeling, metric definitions, data quality, and self-serve access. You’ll also be the person who turns messy questions into clear analysis, reliable dashboards, and better product decisions.

This is not a maintenance role. It is a “make data legible and useful for the company” role.

What You’ll Do
  • Own and evolve CrewAI’s data platform across ingestion, transformation, storage, semantic modeling, BI, and operational data quality.
  • Rationalize the existing data estate: product events, execution telemetry, OpenTelemetry-derived traces, application tables, Cube models, Redshift/data-lake tables, Metabase dashboards, and team-specific reporting.
  • Establish trusted source-of-truth metrics for the business and product, including executions, active builders/users, activation, deployment health, token and cost usage, customer health, governance adoption, retention, and feature usage.
  • Build and maintain the models, pipelines, and metric layers that make those numbers consistent across teams.
  • Partner with product and engineering to improve instrumentation, event taxonomy, data contracts, and telemetry coverage for new features.
  • Make data self-serve through clear dashboards, documented datasets, reusable metric definitions, and sensible access patterns.
  • Improve reliability and trust in the stack through data quality checks, freshness monitoring, lineage, alerting, backfills, and incident/debug workflows.
  • Partner with Discovery, product, and go-to-market teams on analysis behind recommendations, customer signals, usage patterns, and roadmap decisions.
  • Keep the stack secure and cost-aware, including access control, PII handling, retention, and warehouse/query efficiency.
  • Help define how CrewAI uses data internally as the company scales.

RequirementsWhat We’re Looking For
  • Strong data engineering or analytics engineering experience, especially building data foundations in fast-moving product companies.
  • Excellent SQL and data modeling skills, with experience designing reliable datasets, fact/dimension models, and metric definitions.
  • Experience operating a warehouse or analytics store such as Redshift, Snowflake, BigQuery, Postgres, or similar.
  • Familiarity with transformation and modeling tools such as dbt, Cube, semantic layers, or equivalent systems.
  • Experience with event pipelines, product telemetry, application data, and BI tools such as Metabase, Looker, Mode, or similar.
  • Strong Python for data work, automation, validation, and operational workflows.
  • Product sense: you can turn ambiguous questions into useful metrics, and you care whether the numbers are understood correctly.
  • Pragmatism: you are comfortable inheriting messy systems, improving them incrementally, and choosing boring reliable solutions when they are right.
  • Strong communication and documentation habits. You make data easier for other people to use.
  • Comfort being the first dedicated owner in an early-stage, high-growth environment.
Bonus
  • Experience with LLM, agent, observability, trace, usage, or cost analytics.
  • Experience with OpenTelemetry, high-volume event data, or operational telemetry.
  • Experience with experimentation, causal analysis, activation/retention modeling, or customer health scoring.
  • Experience defining event taxonomies and instrumentation standards for SaaS products.
  • Familiarity with Rails/Postgres application data, background jobs, and product analytics in B2B SaaS.
  • Lightweight ML or recommendation experience, especially where it supports product or customer workflows.

Similar Jobs

3 Days Ago
In-Office
148K-219K Annually
Senior level
148K-219K Annually
Senior level
Digital Media • Gaming • News + Entertainment • Sports
Lead development teams in creating scalable data platforms, focus on ETL/ELT pipelines, data architectures, mentoring junior engineers, and ensuring project alignment with business goals.
Top Skills: AirflowAWSAzureDbtGCPJavaPythonScalaSnowflakeSQL
24 Days Ago
In-Office or Remote
United States
Junior
Junior
Artificial Intelligence • Cloud • Information Technology • Machine Learning • Software • Sports • Analytics
The Data Engineer I will build and maintain data pipelines, contribute to platform code, and integrate data sources while growing alongside senior engineers in a production-focused role.
Top Skills: AirflowAWSBigQueryDatabricksDbtDelta LakeIcebergPythonRedshiftSnowflakeSparkSQLTerraformTrino
20 Days Ago
Remote or Hybrid
United States
100K-140K Annually
Senior level
100K-140K Annually
Senior level
Software
The Data Engineer builds and maintains data infrastructure, develops data pipelines, ensures data quality, and collaborates with teams for insights.
Top Skills: Apache AirflowAWSAzureGCPGitJavaMongoDBMySQLOraclePostgresPythonSQL

What you need to know about the San Francisco Tech Scene

San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.

Key Facts About San Francisco Tech

  • Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Google, Apple, Salesforce, Meta
  • Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
  • Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
  • Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account