Reporting 2 - Event Streaming & Core Data Sync

Backend (.NET + Kafka + AWS Lambda)

About

A Kafka-based event ingestion and data lake pipeline that captures retail events via a secure Events API, streams them into Redshift and S3, and keeps core data tables synchronized through scheduled consumers and AWS Lambda.

My Role

Developed a secure events API and Kafka consumers for streaming events into S3 and Redshift. Used S3-hosted JSON schemas for validation, enabling dynamic event-type management. Built a Lambda job to keep Redshift core tables in sync with transactional SQL systems. Implemented rate limiting, HMAC/AES token validation, and structured logging.

Core Capabilities

• Events API with token authorization, rate limiting, schema validation, and Kafka producers
• Kafka consumer with batch buffering, S3 upload, and Redshift COPY commands
• S3-driven configuration for event types, schemas, and chain database names
• HMAC/AES token security for person/chain ID extraction
• Lambda core data sync for Redshift alignment with Monolith

Architecture

• ASP.NET Core Events API publishing to Kafka topics
• Quartz-scheduled Kafka consumers with batch buffering
• S3 data lake with Redshift COPY for analytics tables
• AWS Lambda for delta-based core data synchronization

Tech Stack