ETL v2 - Chain Feed Pipeline

Backend (.NET Workers)

About

A distributed ETL platform that moves grocery retailer feeds from S3 drops through extraction, transformation, and ingestion services, with each stage as its own independently deployable .NET worker host.

My Role

Designed a distributed ETL platform separating extract, transform, and ingest into independent worker services. Implemented S3-triggered extraction with chunking, cache-coordinated ingestion, and event-driven messaging. Leveraged ActionBlock and robust logging to handle high-volume, multi-chain retail feeds.

Core Capabilities

• S3 event-triggered extraction with CsvHelper parsing and configurable chunking
• Configuration-driven transformation with per-domain mappings and validations
• High-performance ActionBlock-based ingestion with backpressure management
• Distributed + in-memory cache coordination for extraction/ingestion state
• GK logging with structured EtlLogEntry payloads and Serilog lifecycle messages

Architecture

• Three independent .NET 6 worker hosts: Extractor, Transformer, Ingestor
• S3 for raw feeds and feed configuration files
• RabbitMQ-style producer/consumer abstractions between stages
• Redis distributed cache + in-memory cache for state coordination
• Dataflow ActionBlock for concurrent multi-chain ingestion

Tech Stack