Senior Data Engineer (AI Systems)

Join our AI Systems team as a Senior Data Engineer, where you will own the Spark-based data pipelines and data infrastructure that powers our core recommendations engine and ML platform. You will work alongside ML engineers and scientists in a collaborative environment, contributing to the development of our core recommender systems and DaVinci Personalization product. This is an opportunity to work end-to-end on large-scale data systems that impact millions of customers.

• Construire, maintenir et optimiser des pipelines de données de production qui alimentent la personnalisation basée sur l'IA à grande échelle.
• Posséder et faire évoluer des pipelines batch basés sur Spark, y compris la configuration du cluster, le réglage et l'optimisation des performances.
• Collaborer avec des ingénieurs et des scientifiques ML pour répondre aux besoins en données pour le développement, l'entraînement et l'évaluation des modèles.
• Enjoys working in a fast-paced, goal-driven environment
• Experience with advanced query optimization
• Familiar with Software Development Lifecycle practices, such as continuous integration/continuous delivery and automated deployment (we use Docker, Kubernetes, and GitHub Actions)
• Deep expertise with Apache Spark, including the PySpark DataFrame API and experience solving challenging scaling problems
• 5+ years of data engineering experience
• Strong software development skills in Python (unit testing, git, code review, CI/CD)
• Experience with data storage formats (we use Parquet, Delta Lake)
• Experience with large-scale data processing, cluster configuration, optimization, and tuning (we use GCP Dataproc)
• Experience with event streaming data (we use Kafka)
• Ability to collaborate with technical partners - you'll be working closely with ML engineers, scientists, and other teams to determine requirements and make design decisions
• Experience with cloud computing platforms (we use Google Cloud Platform)
• If you’re excited about the role but don’t meet all of the abovementioned qualifications, we encourage you to apply. Our differences bring a breadth of knowledge and perspectives that makes us collectively stronger