Senior Data Engineer (Databricks)

Join SugarCRM as a Senior Data Engineer and play a key role in powering revenue intelligence for mid-market enterprises. You will own the Databricks pipelines, ensuring production reliability, cost efficiency, and platform growth. Collaborate with ML engineers, product teams, and the Enterprise Architecture team to maintain a fast, clean, and scalable data backbone. Enjoy a comprehensive benefits package, including a 401(k) match, excellent healthcare, unlimited paid time off, and opportunities for career growth.

• Posséder les pipelines Databricks qui alimentent la plateforme Sugar Predict, en garantissant la fiabilité de la production, l'efficacité des coûts et la croissance de la plateforme.
• Travailler en étroite collaboration avec les ingénieurs ML, les équipes produit et l'équipe d'architecture d'entreprise pour garantir que l'infrastructure de données derrière Sugar Predict est toujours rapide, propre et prête à livrer à l'échelle mondiale.
• Identifier et mettre en œuvre des optimisations de pipeline qui réduisent les coûts de calcul Databricks, améliorent le débit et réduisent les fenêtres de traitement tout en suivant les impacts à travers des KPI mesurables.
• Hands-on experience with Delta Lake including schema evolution, ACID transactions, optimize/vacuum lifecycle, and both incremental and streaming processing patterns
• Experience supporting and maintaining legacy ETL tooling (SSIS, Informatica, custom Python/SQL pipelines, or similar) in production
• Hands-on experience with pipeline performance tuning and compute optimization in production Databricks environments
• 4+ years of data engineering experience
• Experience supporting large-scale multi-tenant architectures with a focus on tenant isolation, per-tenant performance, and data privacy, including navigating tools and platforms that default to single-tenant assumptions
• Proficiency in PySpark, SQL, and Python with a strong track record building and operating production-grade pipelines under SLA constraints
• Proven ability to work collaboratively across data science, product, and infrastructure teams, owning end-to-end delivery in a cross-functional environment
• Strong understanding of data governance, security, and compliance principles, including access control, data privacy, and protection of sensitive enterprise data across multi-tenant environments
• Solid working knowledge of PostgreSQL including query optimization, schema design, and use as a source or sink in production data pipelines
• At least 2 years on Databricks or the Apache Spark ecosystem across Azure and/or AWS
• Exposure to ML feature engineering or feature stores (Databricks Feature Store, Feast, or similar) supporting predictive analytics
• Experience operating Databricks workspaces across both Azure and AWS, including cost governance, cluster management, and cross-cloud data access
• Databricks Certified Data Engineer Associate or Professional certification
• Experience with customer onboarding automation or IaC patterns for provisioning tenant data pipelines at scale
• Experience with Microsoft SQL Server in a data engineering or ETL context
• Experience optimizing Databricks workloads in a Serverless environment, including compute cost governance and performance tuning for serverless compute
• We want you to learn new things in this role, and we encourage you to apply if your experience is close to what we’re looking for. We also know that diversity of background and thought makes for better problem solving and more creative thinking, which is why we're dedicated to adding new perspectives to the team