Automation Engineer

Experience Requirements ● Data Testing Experience: 3+ years specifically in Big Data, Hadoop, or Cloud Data Warehouse environments. ● Good to have : Databricks Experience: 1+ years of experience testing pipelines within a Databricks environment. ● Automation Focus: Proven track record of moving from manual SQL checks to automated Python-based testing frameworks. ● Migration automation experience using Python Required Certifications ● Good to have: Databricks Certified Data Engineer Associate (at minimum). Core Technical Skills Data Validation & Frameworks ● Great Expectations / Pandera: Proficiency in using Python-based libraries to define data "contracts" and automated validation suites. ● DLT Expectations: Deep understanding of Delta Live Tables (DLT) expectations (Fail, Drop, Quarantining bad records). ● Advanced SQL: Expert-level SQL for complex data reconciliation, identifying duplicates, and null-value analysis across billions of records. 2. Python for QA (PySpark) ● Pytest-Spark: Experience using pytest to write unit tests for PySpark transformations and logic. ● Notebook Testing: Ability to write automated test notebooks that validate Medallion Architecture transitions (Bronze to Silver, Silver to Gold). ● Data Reconciliation: Building Python scripts to perform "source-to-target" counts and checksums across distributed file systems. 3. Performance & Integration Testing ● Scalability Testing: Ability to validate that data pipelines meet performance SLAs when data volume spikes. ● End-to-End Orchestration Testing: Testing the reliability of Databricks Workflows and handling of job failures/retries. ● Schema Evolution: Testing how pipelines handle upstream schema changes without breaking downstream Gold tables. 4. Governance & Security Testing ● Unity Catalog Validation: Testing Row-Level Security (RLS) and Column-Level Masking to ensure unauthorized users cannot see sensitive data. ● Data Lineage: Validating that data lineage in Unity Catalog correctly reflects the movement of data across the Lakehouse. Preferred Candidate Background ● "Data-First" Mindset: Understanding that testing a Lakehouse is about testing the data and its behavior, not just the "UI" or "API." ● Software Engineering Foundation: Candidates who know how to use Git (Branching/Merging) to manage their test code alongside the engineering team. ● Distributed Systems Knowledge: Basic understanding of Spark (shuffling, partitioning) to understand why data might be missing or duplicated in a distributed environment. Disclaimer: The company is committed to ensuring theprivacy and security of your information. By submitting this form, you consentto the collection, processing, and retention of the information you provide.The data collected (which may include your contact details, educationalbackground, work experience and skills) will be used solely for the purpose ofevaluating your qualifications for the position you're applying for. Your datawill be stored securely and retained for the duration necessary to fulfil ourhiring process. If you are not selected for the position, your data will bekept on file for a limited period in case future opportunities arise. You havethe right to access, correct, or delete your data at any time by contacting usat Quess Singapore | A Leading Staffing Services Provider in Singapore(quesscorp.sg) “This is in partnership with the Employment andEmployability Institute Pte Ltd (“e2i”). e2i is the empowering network for workers andemployers seeking employment and employability solutions. e2i serves as abridge between workers and employers, connecting with workers to offer jobsecurity through job-matching, career guidance and skills upgrading services,and partnering employers to address their manpower needs through recruitment,training, and job redesign solutions. e2i is a tripartite initiative of theNational Trades Union Congress set up to support nation-wide manpower andskills upgrading initiatives. By applying for this role, you consentto Quesscorp Singapore’s PDPA and e2i’s PDPA.”