Data Processing Senior Developer, 27.73 per hour Inside
Posted on May 4, 2021 by SmartSourcing Ltd
*Data Processing Pipeline Senior Developer*UNTIL END OF AUGUST*REMOTEDUE TO COVID THEN NEWPORT*SC CLEARANCE*27.73 PH INSIDE IR35
Work with others, through pair programming, to implement a data engineering pipelines from the ETL of the data at ingest through to the creation of the final analytical dataset.
As far as is possible, ensure consistency between pipelines ensuring that any associated standards are followed.
Ensure that pipelines can be adapted and re-run should data quality issues be identified.
Liaise with analysis teams to ensure that requirements for outputs are understood and incorporated into the pipeline.
Address technical blockers, actively seeking solutions to remove them and proposing alternative routes to delivery
Routinely test the developed pipelines to ensure that they are fit for purpose.
Peer review work of others in the team, to ensure quality and consistency.
Data engineering functional programming in Python (including PySpark), multicore/distributed processing (hadoop), relational and non-relational (Hive/Impala) SQL database systems
Development of ETL routines for slowly changing dimensions
Writing unit tests for PySpark routines using pytest
Applying Agile software development practices
Applying DevOps practices, including continuous integration and continuous deployment
- SmartSourcing provides services as an Employment Agency and welcomes applications from all suitably qualified people regardless of age, race, religion, disability, age, gender or sexual orientation.