This Job Vacancy has Expired!

Data Processing Senior Developer, 27.73 per hour Inside

SmartSourcing Ltd

Posted on May 4, 2021 by SmartSourcing Ltd

Newport, Gwent, United Kingdom
IT
Immediate Start
£27.7 - £27.7 Hourly
Contract/Project


*Data Processing Pipeline Senior Developer*UNTIL END OF AUGUST*REMOTEDUE TO COVID THEN NEWPORT*SC CLEARANCE*27.73 PH INSIDE IR35

Responsibilities -

Work with others, through pair programming, to implement a data engineering pipelines from the ETL of the data at ingest through to the creation of the final analytical dataset.

As far as is possible, ensure consistency between pipelines ensuring that any associated standards are followed.

Ensure that pipelines can be adapted and re-run should data quality issues be identified.

Liaise with analysis teams to ensure that requirements for outputs are understood and incorporated into the pipeline.

Address technical blockers, actively seeking solutions to remove them and proposing alternative routes to delivery

Routinely test the developed pipelines to ensure that they are fit for purpose.

Peer review work of others in the team, to ensure quality and consistency.

Skills -

Data engineering functional programming in Python (including PySpark), multicore/distributed processing (hadoop), relational and non-relational (Hive/Impala) SQL database systems

Development of ETL routines for slowly changing dimensions

Writing unit tests for PySpark routines using pytest

Applying Agile software development practices

Applying DevOps practices, including continuous integration and continuous deployment

- SmartSourcing provides services as an Employment Agency and welcomes applications from all suitably qualified people regardless of age, race, religion, disability, age, gender or sexual orientation.




Reference: 1187812552

Set up alerts to get notified of new vacancies.