Shivam Pandey

Senior Software Engineer

Not Found • Período integral

Jan 2023 - Present • 3 yrs 7 mos

Work History Senior Data Engineer & Team Lead | Leading IT Services Company | 2023 – Present Data Lake & Analytics — BFSI (Banking & Financial Services) Client Lead a data engineering team building and maintaining a multi-tier AWS data lake (bronze → stage → standard model → marts → analytics) for a major banking/mutual fund client Own end-to-end SQL/Spark pipeline architecture handling terabytes of financial and institutional reporting data Act as primary technical point of contact for data pipeline issues, architecture decisions, and cross-team coordination Key Achievements - Led Data Lake architecture project — architected and delivered a large-scale AWS data lake serving mutual fund and institutional reporting needs, now a core production system for the client Large-scale data refresh design — built a historical RM-level attribution refresh pipeline handling 100+ GB of partitioned S3/Parquet data, using EMR with broadcast joins to keep fortnightly refreshes performant Legacy SQL modernization — converted a large, complex SQL Server stored procedure into Athena-compatible, CTE-based queries — improved both maintainability and query performance Cross-account infrastructure debugging — diagnosed and resolved Glue Catalog access issues between EMR Spark SQL and Athena in a cross-account AWS setup, unblocking a critical reporting workflow Automated financial reporting — built AWS Lambda-based pipelines generating multi-sheet Excel sales reports and institutional AUM/AAUM summaries, replacing manual reporting processes Pipeline consolidation — simplified a multi-step Scala/Spark channel-derivation pipeline into a clean, lookup-table-driven design, reducing complexity and maintenance overhead