Technical Case Study · Full-Stack & Cloud

automation_hub

FastAPI + Celery/Redis scraping and PDF-to-Gemini pipeline, Docker and Kubernetes, 68 pytest tests

Cloud-native data extraction and document intelligence platform combining a multi-mode web scraper, `pdfplumber` table/text extractor, Google Gemini 1.5 AI summarizer, FastAPI REST service, and Celery + Redis distributed worker queues—packaged with Docker Compose and Kubernetes (`k8s/`) manifests.

Automated Tests68 Passing Tests

Python Codebase95.1% Python 3.12

OrchestrationDocker + Kubernetes

Python 3.12FastAPICeleryRedisPlaywrightGemini APIDockerKubernetes68 Pytest Tests

Problem Statement

Synchronous web scraping and large PDF extraction block API event loops, suffer from memory spikes under concurrent requests, and fail silently when upstream rate limits hit.

Deterministic Architecture Advantage

Decoupled ingestion into an asynchronous FastAPI gateway backed by Redis message brokering, Celery worker pools, Playwright headless browsers, and structured Gemini AI extraction—containerized with Docker and Kubernetes HPA manifests.

Execution Pipeline & System Topology

STEP 01

FastAPI & CLI Gateways

Accepts synchronous or async batch scrape + PDF ingestion jobs with API key protection.

FastAPI + UvicornSwagger / OpenAPI Docs

STEP 02

Celery + Redis Queue

Distributes scraping and PDF parsing across scalable container workers with sync fallback.

Celery + Redis + DockerZero-Downtime Fallback

STEP 03

PDFPlumber & Gemini 1.5

Extracts deterministic Markdown/CSV tables and synthesizes Gemini executive summaries.

pdfplumber + Gemini REST68 Passing Pytest Tests

Verification & Engineering Highlights

Unified orchestration pipeline (`automation_pipeline.py`) that crawls domains, discovers linked PDFs, extracts tables to CSV/Markdown, summarizes via Gemini AI, and writes `pipeline_manifest.json`.
Exposes 8 FastAPI REST endpoints (`/api/scrape`, `/api/pdf`, `/api/pipeline`, `/api/jobs/*`) with Swagger UI and SQLAlchemy job audit logs.
Verified by 68 automated Pytest tests (`test_pdf.py`, `test_pipeline.py`, `test_scraper.py`, `test_summarizer.py`) running in 2.12s.

tasks.py

python

@celery_app.task(bind=True, max_retries=3, default_retry_delay=5)
def execute_pipeline_job(self, job_id: str, target_url: str, follow_pdfs: bool = True):
    update_job_status(job_id, "RUNNING")
    manifest = run_unified_pipeline(
        url=target_url,
        scrape_mode="all",
        follow_pdfs=follow_pdfs,
    )
    update_job_status(job_id, "COMPLETED", result_manifest=manifest)
    return manifest