Data Engineer

Design, build, and maintain robust ETL/ELT data pipelines, reusable components, frameworks, and libraries to process data from a variety of data sources ensuring data quality and consistency. Implement pipelines using spark on Databricks to perform task cleaning, data validation, standardization and then apply transformations as per the use cases. Ingest data from various source systems like Teradata, Snowflake and Hive. Process the ingested data using Hadoop and AWS technologies such as Pyspark, Data Bricks, Glue, RedShift and lambda functions. Use Airflow as jobs orchestrate tool. Monitor and troubleshoot data pipelines, ensuring high availability and performance. Collaborate with other data engineers, analysts, product managers and business stakeholders to understand data requirements, translate them into technical specifications and deliver data solutions that drive decision-making. Implement CI/CD pipelines to automate deployment and testing of data engineering workflows with Git Hub and Jenkins. Work under supervision. Travel and/or Relocation to various unanticipated client sites throughout USA is required.

Experience of Six (6) months working with any one of Teradata, Snowflake, Hive, Hadoop, Pyspark, Data Bricks, Glue, RedShift, Lambda, Git Hub or Jenkins is required. Travel and/or relocation is required to unanticipated client sites within USA. International travel is not required. The frequency of travel is currently not known exactly but it could reach as high as 100% as it depends on the client and project requirement that cannot be currently anticipated. Employer provides Information technology services to various clients in USA and hence implementing projects will require such travel which could be as high as 100%.