Fullstack Lead- ITO Transition

Role Overview

We are seeking a Senior Data Engineer with 6–9 years of experience to design, build, and optimize scalable data platforms and pipelines using modern cloud-based data technologies. The ideal candidate will have strong expertise in Azure Databricks, PySpark, SQL, Unity Catalog, and workflow orchestration tools, along with experience in data governance, security, and performance optimization.

This role involves leading data engineering initiatives, enabling analytics and data science use cases, implementing governance frameworks, and ensuring high-performance, reliable data systems across enterprise environments.

Key Responsibilities

• Design and implement scalable data architectures using Azure Databricks and modern lakehouse principles.
• Architect batch and streaming data pipelines for large-scale structured and unstructured datasets.
• Implement Databricks Delta Lake patterns, including Medallion Architecture.
• Design and manage Delta Live Tables pipelines and Structured Streaming solutions.
• Develop complex ETL/ELT pipelines using PySpark and SQL.
• Optimize Spark workloads, SQL query performance, cluster configuration, resource utilization, and partitioning strategies.
• Administer Databricks Unity Catalog, including access control, metadata management, lineage, and audit mechanisms.
• Implement Delta Sharing protocols and data governance, security, and compliance standards.
• Automate workspace configuration, cluster provisioning, and deployments using Databricks CLI, Python scripting, and CI/CD pipelines.
• Manage orchestration using Databricks Workflows, Azure Data Factory, or Apache Airflow.
• Implement data validation, monitoring, quality checks, alerting, and automated recovery mechanisms.
• Partner with data analysts, data scientists, architects, and business stakeholders to enable analytics and machine learning workflows.
• Mentor junior data engineers and provide technical leadership.

Required Skills

• Strong expertise in Azure Databricks, PySpark, SQL, Unity Catalog, and Delta Lake architecture.
• Hands-on experience with Delta Live Tables, Structured Streaming, and Databricks Workflows.
• Experience with Azure Data Factory or Apache Airflow.
• Knowledge of distributed data processing and Spark performance tuning.
• Understanding of data modeling, data warehousing, CI/CD, and data pipeline automation.
• Strong debugging, analytical, and problem-solving skills.

Preferred Qualifications

• Experience with Kafka or Event Hubs.
• Exposure to Synapse, Snowflake, or Redshift.
• Knowledge of Python beyond PySpark.
• Familiarity with Power BI, Tableau, machine learning pipelines, or MLOps.
• Azure or Databricks certifications are a plus.

Education and Experience

A Bachelor’s Degree in Engineering, Computer Science, or Information Technology is required. Candidates from other educational backgrounds may be considered with 100% relevant professional experience. The role requires 6 to 9 years of experience in data engineering, big data development, or related roles, preferably on enterprise Azure cloud data platforms.

Location and Contact

Mercedes-Benz Research and Development India Private Limited, Brigade Tech Gardens, Bengaluru.

Contact: Jacob Cyril, jacob.cyril@mercedes-benz.com