Data Engineer

Key Responsibilities 1. Data Pipeline Development • Design, build, and optimize scalable data pipelines using Scala and Hadoop frameworks. • Implement ETL processes for ingesting, transforming, and storing data from various sources. 2. Data Analysis and Query Optimization • Write and optimize complex SQL queries for efficient data retrieval and transformation. • Troubleshoot and resolve performance issues in queries and data workflows. 3. Big Data Ecosystem Management • Manage Hadoop-based data infrastructure, including HDFS, Hive, and related components. • Monitor system performance and optimize resource utilization in a distributed environment. 4. Collaboration and Problem Solving • Work closely with data analysts, data scientists, and business stakeholders to understand requirements and translate them into technical solutions. • Provide technical guidance and best practices for data engineering within the team. 5. Automation and Monitoring • Develop scripts and tools to automate repetitive tasks and enhance system monitoring. • Ensure data quality and consistency through validation and monitoring mechanisms. 6. Documentation and Compliance • Document technical solutions, workflows, and processes to ensure knowledge sharing and reproducibility. • Ensure compliance with data governance and security policies. Skills and Qualifications · Technical Skills: • Proficiency in Scala programming and functional programming principles. • Strong SQL skills, including query optimization and database design. • Hands-on experience with Hadoop ecosystem tools (HDFS, Hive, YARN, MapReduce). • Familiarity with other big data tools like Spark, Kafka, or Flink is a plus. · Problem-Solving: • Strong analytical skills with the ability to troubleshoot and resolve complex data issues. · Communication: • Excellentcommunication skills to effectively collaborate with technical and non-technical stakeholders. · Experience: • 6+ years of experience in data engineering,big data technologies, or a similar role.