Md Ariful Islam

Data Acquisition and Automation Engineer

Fixter • Meio período

Apr 2024 - Present • 2 yrs 4 mos

● Production data acquisition systems: Design and maintain specification-driven Python ingestion pipelines using YAML-based configuration and declarative field schemas, powering Playwright and Selenium crawlers across hundreds of dynamic, JavaScript-heavy source websites at scale. ● Reliability & data integrity engineering: Build resilient acquisition workflows with schema validation (Pydantic, JSON Schema), data quality enforcement, deduplication, and schema-drift detection, maintaining structured dataset integrity as upstream sources change over time. ● Anti-fragile system design: Implement proxy rotation, authenticated session management, and resilient automation patterns to sustain high-reliability data collection in adversarial, frequently-changing web environments. ● AI-assisted extraction: Integrate the OpenAI API into ingestion workflows to support LLM-assisted, semantic extraction and structured-data normalisation on complex or unstructured sources, improving output quality where traditional parsing falls short. ● Engineering collaboration & delivery: Develop and maintain reusable, modular pipeline components and shared internal tooling; ship and review changes through Git and Azure DevOps pull requests, track work via Jira, and collaborate directly with the engineering team on source onboarding, debugging, and