v
vivekpandey600

Vivek Pandey

@vivekpandey600

Senior Data Engineer

Índia
Hindi, Inglês
Algumas informações são exibidas no idioma inglês.
Sobre mim
I am a Senior Data Engineer with over 6 years of experience designing and optimizing end-to-end pipelines on Azure Databricks and PySpark. I specialize in Medallion Architecture, real-time streaming with Kafka, and data governance using Unity Catalog for global brands in retail and energy.... Saiba mais

Habilidades

v
vivekpandey600
Vivek Pandey
offline • 

Conheça meus serviços

ETLs de dados
I will be your databricks and pyspark data engineer for etl and delta lake pipelines
A partir deUS$ 75 /projetoMais detalhes

Experiência profissional

Infosys

Senior Associate Consultant

Infosys • Período integral

Feb 2025 - Present • 1 yr 8 mos

Building Databricks Lakehouse pipelines for global retail finance and consumer-service data. Designing end-to-end batch ETL pipelines using Medallion Architecture with Delta Lake and Unity Catalog. Ingesting data from 1,200+ global stores into AWS S3. Implemented CDC for incremental loads and optimized PySpark transformations, reducing batch runtime by 35%. Managing Amazon Connect call-transcript pipelines processing 200K JSON files daily.

Tata_Consultancy Services

Azure Data Engineer

Tata Consultancy Services • Período integral

Feb 2020 - Jan 2025 • 4 yrs 11 mos

Data Engineer for TotalEnergies' digital drilling operations. Built real-time streaming pipelines using Kafka and PySpark to ingest sensor and rig performance data. Processed 1 TB/day of sensor data into Azure Data Lake and Cosmos DB. Reduced pipeline execution time by 50% through performance optimization. Developed batch pipelines with Azure Function Apps and automated job monitoring, saving 2 hours of manual effort daily.