GB DATAWORKS

DATA ENGINEER

ProArch • Período integral

Aug 2024 - May 2026 • 1 yr 9 mos

Project Overview : Clinician Insights is a healthcare data engineering project designed to build a scalable and secure ETL pipeline for processing clinical data. The system ingests raw data from multiple healthcare source systems such as patient registration, clinician management, visit encounters, and diagnosis platforms.The project follows a Bronze–Silver–Gold architecture, where raw data is ingested into Amazon S3, cleaned and standardized using AWS Glue and PySpark, and finally transformed into analytics-ready datasets. The curated data supports clinician performance analysis, patient visit summaries, and operational reporting. Roles and Responsibility • Ingested raw healthcare data into Amazon S3 (Bronze layer) • Developed and enhanced AWS Glue ETL jobs using PySpark • Performed data cleaning, validation, and standardization • Removed duplicate patient and visit records • Handled missing and null values in critical columns • Standardized categorical fields such as gender and diagnosis codes • Converted date and timestamp fields into consistent formats • Joined related datasets like patient, visit, and diagnosis data • Performed post-transformation data validation using Athena • Assisted in debugging Glue job failures using CloudWatch logs Challenges • Poor Data Quality: • Raw data contained duplicates, missing values, and inconsistent formats, which required robust validation and cleansing logic in PySpark. • Glue Job Failures: • Schema mismatches and unexpected null values caused job failures, resolved through CloudWatch log analysis and code optimization. • Performance Issues: • Large healthcare datasets led to performance bottlenecks, which were mitigated by optimizing Spark transformations and reducing unnecessary data shuffles.