Data Engineer

Job Summary Design, build, and maintain scalable, fault-tolerant data pipelines using Python, SQL, and orchestration tools. Manage package dependencies, ensure security compliance, and optimize code for production data workflows. Responsibilities • Develop modular Python code with robust error handling and logging for data pipelines • Write advanced SQL queries using joins, window functions, and optimization techniques to support data processing • Process and transform data using Pandas and integrate streaming data with Kafka producers and consumers • Design, build, and maintain Directed Acyclic Graphs (DAGs) and flows using orchestration tools such as Apache Airflow and Prefect • Ensure pipelines are idempotent, scalable, and fault-tolerant to support reliable data workflows • Implement logging, monitoring, and alerting mechanisms to maintain pipeline observability and operational health • Manage Python package installations, upgrades, and dependency resolution across development, UAT, and production environments • Maintain dependency manifests (e.g., requirements.txt) with version pinning to ensure environment consistency • Support deployments in restricted or air-gapped environments by managing package and dependency constraints • Analyze vulnerability reports from security scanning tools and remediate security issues by upgrading or replacing vulnerable libraries • Fix broken imports, deprecated APIs, and compatibility issues arising from library updates while maintaining pipeline stability • Collaborate with security teams to ensure compliance with organizational security standards and secure coding practices • Refactor legacy code in data ingestion APIs, data transformation (Pandas/SQL), model training/inference pipelines, and orchestration workflows to improve modularity, readability, and performance • Ensure backward compatibility and minimize disruption to production systems during code changes • Perform data validation and ensure schema consistency and data quality across pipeline stages • Implement unit and integration tests for data pipelines to ensure reliability before deployment • Troubleshoot pipeline failures, perform root cause analysis, and provide production support for continuous workflow improvement • Handle Kafka schema evolution and message serialization/deserialization to maintain streaming data integrity • Work effectively in regulated or high-security environments, applying security and reliability best practices Preferred competencies and qualifications • Preferably 2-3 or more years of experience in data engineering • Prior experience working with production data pipelines • Experience handling dependency conflicts, library upgrades, and refactoring in live systems • Ability to work across multiple layers including API, data processing, orchestration, and machine learning pipelines