Ver categorias
Explorar
Fiverr Pro
Português
$
USD
I will build a real time data lakehouse pipeline
Python Developer, FastAPI , Web Scraping , AI Automation, Data Engineering
I'm a Data Engineer with 3+ years of industry experience building RESTful APIs, web scraping systems, and AI-powered applications. I specialize in FastAPI, Streamlit, and LangGraph, and work with lead...
Sobre este Serviço
Looking to build a real-time data pipeline that keeps your data warehouse always up to date without manual ETL jobs?
I will design and deliver a fully automated, end-to-end data lakehouse pipeline that captures every change in your database the moment it happens, streams it through Kafka, and lands it as queryable Delta Lake tables all orchestrated and monitored by Apache Airflow.
What you get:
- Live CDC from your MySQL database (no downtime, no manual exports)
- Scalable stream processing with Apache Spark
- S3-compatible Delta Lake storage (MinIO) query with Trino or Spark SQL
- Airflow DAG for automated health checks and pipeline monitoring
- Fully Dockerized runs on your server or cloud VM
- Setup guide and documentation included
Perfect for startups, data teams, and businesses that need reliable, real-time data availability without managing complex infrastructure from scratch.
Meu portfólio
Perguntas frequentes
What information do you need to get started?
I need details about your source database (type, version, size), your preferred storage destination, and your server/cloud environment. If you're unsure, a free discovery call can help scope it out.
Can you connect to my existing database without downtime?
Yes. Using CDC (Change Data Capture) via Debezium, the pipeline reads your MySQL binary log — no locks, no downtime, no impact on your running application.
What does the pipeline deliver in real time?
Every INSERT, UPDATE, and DELETE in your source database is captured instantly and lands in Delta Lake tables on MinIO (S3-compatible) within seconds — queryable via Spark SQL or Trino.
Do I need cloud infrastructure or does this run on-premise?
Both. The entire stack runs on Docker Compose — deploy it on your local server, a cloud VM (AWS EC2, GCP, Azure), or any Linux machine with 8GB+ RAM.
Can you handle schema changes in my source database?
Yes. The pipeline is built with schema evolution in mind. I configure Debezium and Spark to handle new columns and type changes gracefully without breaking the pipeline.
Will you sign an NDA if my data is sensitive?
Absolutely. I am happy to sign an NDA before the project starts.
Do you offer post-delivery support?
Yes — 7 days (Basic), 14 days (Standard), 30 days (Premium) for bug fixes and deployment issues.
