• 🎓   B.E. CSE (Data Science) at DSCE, Bengaluru — SGPA 8.73/10
• 🔭   Currently building ChainBreaker — a graph-driven real-time intrusion detection platform
• 💡   Interested in streaming infrastructure, data platform engineering, and ETL/ELT systems
• 🤝   Open to data engineering internships and collaborations
• 📬   Reach me at kaushikkishor198@gmail.com

Core pipeline tools

Databases & storage

Languages & frameworks

• Ingestion: Kafka
• Processing: Spark Structured Streaming
• Storage: Medallion Lake (Bronze → Silver → Gold)
• Transformation: dbt
• Warehouse: PostgreSQL (Star Schema)
• Orchestration: Airflow DAGs
• Analytics: BI / ML Layer

Real-time streaming pipeline ingesting 15,000+ events/day across 3 Kafka topics with ~3s micro-batch latency. Medallion architecture on Azure Data Lake Gen2, dbt-modeled dimensional warehouse, and anomaly detection using Isolation Forest.

Kafka Spark Structured Streaming dbt Azure Data Lake Gen2 PostgreSQL scikit-learn Airflow

End-to-end ETL pipeline processing 3M+ NYC taxi records into a star schema warehouse. Airflow-orchestrated ingestion with 30+ dbt data quality tests and a fully Dockerized local setup.

Airflow PostgreSQL dbt Docker Python

Real-time network intrusion detection: Kafka ingestion → Spark processing → Neo4j attack graph → RL-based automated response. Modeled on MITRE ATT&CK kill chain stages.

Kafka Spark Neo4j FastAPI Python MaskablePPO