Data Engineer (Python & SQL)

Job Description

Duties & Responsibilities

ENVIRONMENT:

BUILD and maintain custom integrations with client systems as the next Data Engineer sought by a provider of Data-driven solutions with a global operational footprint. The team follows the Scrum methodology. You will work closely with the Project Teams to solve client-specific problems and deliver exceptional solutions. The successful candidate will need to have at least 3+ years’ experience working with data pipelines, Python and SQL. You will also require knowledge of Apache Airflow, the AWS ecosystem and the ability to troubleshoot issues and determine and resolve the root cause of an issue. Any experience with Docker/Kubernetes, OOP, NumPy, Pandas, PyJanitor; Great Expectations; Pandera or WhyLogs in addition will prove beneficial.

DUTIES:

Architect, build, monitor and maintain data integrations and ETL pipelines.
Ensure data quality and implement tools and frameworks for automating the identification of data quality issues in data source pipelines.
Investigate, document and resolve issues reported internally or by clients.
Write custom scripts to fix or import data for clients with specific needs.
Implement sustainable data pipeline solutions, viz. create admin tools and scripts to address repeat support issues or requests.
Communicate problems and workarounds to our internal support team in a simple, concise manner they can use with customers.
Assist with answering Tier 2 Support Questions.

REQUIREMENTS:

3+ Years experience with integrating data and working with data sources.
3+ Years programming experience in Python.
Proficient in SQL.
Experience with data orchestration tools such as Apache Airflow.
Experience working with AWS. Alternatively, experience with GCP or Azure would be beneficial.
Ability to troubleshoot issues and determine and resolve the root cause of an issue.

Advantageous –

Experience with Docker or Kubernetes.
Understanding of best practices with regards to web applications and OOP.
Understanding of system integration design patterns.
Mathematical proficiency.
Exposure to these libraries or frameworks: NumPy; Pandas; PyJanitor; Great Expectations; Pandera; WhyLogs.

ATTRIBUTES:

Professional communication skills with the ability to engage directly with clients.
A client-focused mindset, as finding solutions that make clients happy should be your number one priority.
The ability to work independently.
While we would really like to respond to every application, should you not be contacted for this position within 10 working days please consider your application unsuccessful.