Sr Databricks Developer - TechDigital

Azure Databricks

Apache Spark

Data Modelling

Azure Data Lake Creation

Python Programming

Preferred /Desired Skills:

predictive analytics

Experience in Client Libraries

Responsibilities:

• Develop and maintain ETL (Extract, Transform, Load) pipelines using Databricks to process and transform large datasets.
• Collaborate with data engineers and data scientists to design and implement scalable and efficient data processing workflows.
• Build and optimize Apache Spark jobs and clusters on the Databricks platform.
• Develop and maintain data ingestion processes to acquire data from various sources and systems.
• Implement data quality checks and validation procedures to ensure accuracy and integrity of data.
• Perform data analysis and exploratory data mining to derive insights from complex datasets.
• Design and implement machine learning workflows using Databricks for predictive analytics and model training.
• Troubleshoot and debug issues related to data processing, performance, and job failures.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Stay updated with the latest advancements in big data technologies and contribute to the improvement of existing systems and processes.

• Solid experience in developing data processing workflows using Apache Spark and Databricks.
• Proficiency in programming languages like Python, Scala, or SQL for data manipulation and analytics.
• Strong understanding of distributed computing principles and experience with large-scale data processing frameworks.
• Familiarity with cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP).
• Experience with data modeling, database systems, and SQL.
• Knowledge of machine learning concepts and experience with Client libraries and frameworks.
• Excellent problem-solving skills and ability to work independently and in a team.
• Strong communication skills to collaborate with stakeholders from different technical backgrounds.