Bigdata Engineer - eTek IT Services, Inc.

Overview

The Big Data Engineer plays a crucial role in the organization by designing, developing, and maintaining scalable data processing systems. As companies increasingly rely on data-driven insights to fuel their operations and strategic decisions, the importance of robust data engineering cannot be overstated. This position is essential for transforming raw data into meaningful information that can drive business growth and innovation. The Big Data Engineer works closely with data scientists, analysts, and stakeholders to ensure that the architecture meets business needs and optimizes performance. They are responsible for implementing the latest tools and technologies in big data, helping organizations manage, analyze, and derive insights from large volumes of data across various sources.

Key Responsibilities

Design and implement scalable data processing systems.

Develop ETL (Extract, Transform, Load) pipelines for data ingestion.

Collaborate with data scientists to optimize data workflows.

Maintain and enhance big data architectures.

Ensure data quality and integrity through robust validation processes.

Monitor data systems for performance issues and optimize appropriately.

Integrate new data sources into existing systems.

Utilize Hadoop, Spark, and other big data frameworks.

Develop and maintain data modeling solutions.

Create documentation for data architecture and processes.

Implement data governance and compliance standards.

Participate in code reviews and best practices discussions.

Collaborate with cross-functional teams to align data initiatives with business goals.

Perform troubleshooting and debugging for data solutions.

Stay updated with emerging technologies in big data.

Required Qualifications

Bachelor's degree in Computer Science, Engineering, or related field.

3+ years of experience in data engineering or related roles.

Proficiency in programming languages such as Java, Python, or Scala.

Extensive experience with big data technologies, especially Hadoop and Spark.

Familiarity with data warehousing solutions like Redshift or Snowflake.

Experience with ETL tools such as Apache NiFi, Talend, or similar.

Strong understanding of SQL and NoSQL databases.

Knowledge of cloud platforms such as AWS, Azure, or Google Cloud.

Experience integrating data from a variety of sources.

Understanding of data modeling concepts and techniques.

Ability to work with large, complex datasets and derive meaningful insights.

Knowledge of data governance frameworks and compliance regulations.

Strong analytical and problem-solving skills.

Excellent communication skills, both written and verbal.

Adept at working in a fast-paced and dynamic environment.