Data Engineer Logo
Project 04Research — Campus Faculty Data Gathering
Data Engineering & Web Scraping

Data Engineer

Automated crawling and data scraping infrastructure extracting 10,000+ structured records for academic research.

Data Engineer - Architecture Graph 1
Click to expand
Pipeline 1: Indonesian Food Crawl & Scrap ArchitectureView Graph →
Data Engineer - Architecture Graph 2
Click to expand
Pipeline 2: News Aggregation & Parsing ArchitectureView Graph →
Architecture DiagramsData Engineering Pipeline: Crawling & Scraping Indonesian Food & News Portals (10k+ Records)
Role & Discipline

Data Engineer

Data Pipelines × Web Scraping × Automated Crawling

Contribution

Architecting and executing automated crawling and scraping data pipelines to extract 10,000+ multi-source records for campus academic research.

Timeline & Context

2024

Research — Campus Faculty Data Gathering

Tech Stack & Tools
PythonWeb ScrapingData PipelineBeautifulSoupSeleniumPandasJupyterGitRegexPostman
Overview & Context

In academic research, the availability of large-scale structured datasets is fundamental to producing high-impact findings. Serving as Data Engineer for campus faculty research, I architected and deployed automated web crawling and data scraping pipelines. The system systematically extracted over 10,000+ records across diverse target platforms (including Indonesian culinary archives and news portals), automating data cleaning, schema validation, and structured ingestion into research-ready analytical datasets.

Next Project

Narabuna

Hackathon — Hackfest UC

View Next Project