Azure Data Engineer - Diverse Linx
Yorkville, IL
About the Job
Job Description: As a Microsoft Azure Data Engineer, you will be responsible for designing, implementing, and managing data pipelines and architectures on the Azure Synapse platform. You will work with a variety of data sources and ensure data is processed and stored efficiently using the Medallion Architecture (Bronze, Silver, Gold layers). Your expertise in PySpark, Synapse pipelines, API integrations, Azure Synaspse Analytics and integration with Master Data Management tools like Profisee will be crucial in developing robust data solutions that meet our business needs.
Key Responsibilities:
Design and implement data pipelines using Pyspark Notebooks.
Manage and optimize data storage and processing through the Medallion Architecture (Bronze, Silver, Gold layers).
Develop and maintain ETL processes using notebooks to ingest, transform, and load data from various sources.
Integrate APIs to facilitate data exchange between systems.
Create and manage table views to surface data to end users.
Ensure data quality, consistency, and security across all layers.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
Monitor and troubleshoot data pipelines to ensure reliability and performance.
Stay up to date with the latest Azure technologies and best practices.
Qualifications:
Bachelor s degree in computer science, Information Technology, or a related field.
Proven experience as a Data Engineer with a focus on Azure technologies.
Strong proficiency in PySpark notebooks and Synapse pipelines with at least 2 years of demonstrable experience with notebooks
Experience with Medallion Architecture and managing data through bronze, silver, and gold layers.
Familiarity with API integration and data exchange.
Experience with master data management tools such as Profisee.
Ability to create and manage table views for data visualization.
Excellent problem-solving skills and attention to detail.
Strong communication and collaboration skills.
Ability to work in a fast-paced, dynamic environment.
Preferred Qualifications:
Azure certifications (e.g., Azure Data Engineer Associate).
Experience with other Azure services such as Azure Databricks, Azure Synapse Analytics, and Azure SQL Database.
Knowledge of data governance and security best practices.
Experience with additional master data management tools and techniques
Competencies: Digital : PySpark
Experience (Years): 6-8
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.