Data Engineer - TrekRecruit LLC

Job Title: Data Engineer

Location: Chicago, IL

Duration: 12+ Months

Tax Terms: W2

Responsibilities:

• Design, develop, and deploy scalable and high-performance data pipelines using Scala, Spark, and AWS services (e.g., AWS Lambda, S3, EMR, Glue).
• Extract, transform, and load (ETL) data from various sources, including databases, APIs, and streaming platforms.
• Develop and maintain data quality checks and monitoring systems to ensure data accuracy and integrity.
• Collaborate with data scientists, analysts, and business stakeholders to understand their data needs and translate them into technical solutions.
• Optimize data pipelines for performance, cost-effectiveness, and maintainability.
• Troubleshoot and resolve data-related issues effectively.
• Stay abreast of the latest advancements in data engineering technologies and best practices.
• Contribute to the development and improvement of data engineering standards and processes.

Qualifications:

• Education: Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Experience: 3+ years of experience as a Data Engineer or a similar role.
• Programming: Strong proficiency in Scala and Python. Experience with Java is a plus.
• Big Data Technologies: Expertise in Apache Spark (core, SQL, streaming) and related technologies.
• Cloud Computing: In-depth knowledge of AWS services, with a strong focus on AWS Lambda, S3, EMR, Glue.
• Data Pipelining: Experience with building and maintaining ETL/ELT pipelines using tools like Airflow or similar.
• Database Technologies: Experience with relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).