Location: Remote
Job Description:We are hiring a Data Engineer with strong hands-on experience in PySpark, Scala, and Python. You must have solid expertise in Apache Spark, as it will be the core technology used for building and managing large-scale data processing pipelines.
Experience with cloud platforms like Google Cloud Platform (GCP), Microsoft Azure, or AWS is a plus.
Required Skills:Strong hands-on experience with Apache Spark
Proficient in PySpark
Experience in Scala and Python
Knowledge of ETL processes and data pipeline design
Understanding of distributed data processing
Familiarity with version control tools like Git
Basic knowledge of cloud platforms (GCP, AWS, or Azure)
Experience with cloud-native data tools (e.g., Dataproc, Glue, EMR, BigQuery)
Familiarity with workflow/orchestration tools like Airflow or Cloud Composer
Experience with CI/CD for data engineering
Exposure to both structured and unstructured data
Similar Jobs
.png)
What you need to know about the San Francisco Tech Scene
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
