Design and scale end-to-end data pipelines and RAG architectures for LLM consumption: ETL/ELT, vector databases, chunking/embeddings, data cleaning/PII removal, metadata engineering, low-latency streaming, and evaluation/versioning to support agentic AI.
We are seeking a Lead Data Engineer to build and scale the data infrastructure powering our Agentic AI products. You will be responsible for the "Ingestion-to-Insight" pipeline that allows autonomous agents to access, search, and reason over vast amounts of proprietary and public data.
Your role is critical: you will design the RAG (Retrieval-Augmented Generation) architectures and data pipelines that ensure our agents have the right context at the right time to make accurate decisions.
Key Responsibilities
- AI-Ready Data Pipelines: Design and implement scalable ETL/ELT pipelines that process both structured (SQL, logs) and unstructured (PDFs, emails, docs) data specifically for LLM consumption.
- Vector Database Management: Architect and optimize Vector Databases (e.g., Pinecone, Weaviate, Milvus, or Qdrant) to ensure high-speed, relevant similarity searches for agentic retrieval.
- Chunking & Embedding Strategies: Collaborate with AI Engineers to optimize data chunking strategies and embedding models to improve the "recall" and "precision" of the agent's knowledge retrieval.
- Data Quality for AI: Develop automated "Data Cleaning" workflows to remove noise, PII (Personally Identifiable Information), and toxicity from training/context datasets.
- Metadata Engineering: Enrich raw data with advanced metadata tagging to help agents filter and prioritize information during multi-step reasoning tasks.
- Real-time Data Streaming: Build low-latency data streams (using Kafka or Flink) to provide agents with "fresh" data, enabling them to act on real-time market or operational changes.
- Evaluation Frameworks: Construct "Gold Datasets" and versioned data snapshots to help the team benchmark agent performance over time.
Required Skills & Qualifications
- Experience: 10+ years in Data Engineering, with at least 2 years focusing on data for LLMs or AI/ML applications.
- Python Mastery: Deep expertise in Python (Pandas, Pydantic, FastAPI) for data manipulation and API integration.
- Data Tooling: Strong experience with modern data stack tools (e.g., dbt, Airflow, Dagster, Snowflake, or Databricks).
- Vector Expertise: Hands-on experience with at least one major Vector Database and knowledge of similarity search algorithms (HNSW, Cosine Similarity).
- Search Knowledge: Familiarity with hybrid search techniques (combining semantic search with traditional keyword search like Elasticsearch/BM25).
- Cloud Infrastructure: Proficiency in managing data workloads on AWS, Azure, or GCP.
Preferred Qualifications
- Experience with LlamaIndex or LangChain for data ingestion.
- Knowledge of Graph Databases (e.g., Neo4j) to help agents understand complex relationships between data points.
- Familiarity with "Data-Centric AI" principles—prioritizing data quality over model size.
Compensation, Benefits and Duration
Minimum Compensation: USD 46,000
Maximum Compensation: USD 162,000
Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.
Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.
This position is not available for independent contractors
No applications will be considered if received more than 120 days after the date of this post
Photon San Francisco, California, USA Office
San Francisco, United States
Similar Jobs
Cloud • Fintech • Food • Information Technology • Software • Hospitality
Lead development of Toast’s consumer-facing web experiences, building scalable guest and restaurant features across the stack. Collaborate with frontend, backend, iOS, and Android engineers using React, TypeScript, Next.js, APIs, and backend services. Own projects from development through launch, monitoring, and analysis while experimenting with users and influencing consumer product strategy.
Top Skills:
AndroidGraphQLiOSJavaKotlinNext.JsReactRestTypescript
Professional Services
Coordinates multiple client engagements, including timelines, deliverables, meetings, documents, data, and client communications. Supports CME/CE activities by managing disclosures, accreditation documentation, grant submissions, evaluations, attendance, and credit records. Maintains organized files and systems, prepares client-facing materials, tracks action items and deadlines, and helps improve internal processes. Occasional travel of one to two trips annually may be required.
Top Skills:
Accme ParsAssociation Management SystemsLearning Management SystemsMS OfficeShared Drives
Big Data • Information Technology • Software • Analytics • Energy
Drive new business and revenue growth across Texas utilities by selling Enverus’s power portfolio. Manage complex sales cycles from initial engagement through technical evaluation, cybersecurity review, legal negotiation, procurement, contracting, and renewal. Build executive and technical relationships, develop strategic account plans, and partner with Product, Solutions Consulting, Customer Success, and leadership to deliver long-term customer value.
Top Skills:
ErcotGenerative AiLoad ForecastingMidcontinent Independent System Operator (Miso)Power Markets And Trading SolutionsRenewable Energy ForecastingSaaSSouthwest Power Pool (Spp)Transmission And Distribution Engineering Software
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
