Set up event data pipeline with Kafka

Set up the foundational data pipeline infrastructure for the metrics dashboard project.

Technical Approach

  1. Deploy Kafka on our k8s cluster (3 brokers for HA)
  2. Create producer library with schema validation
  3. Set up Kafka Connect sink to ClickHouse
  4. Add Grafana dashboards for pipeline monitoring

Notes

  • Consider using Confluent Schema Registry for Avro schemas
  • Need to coordinate with infra team on k8s resources

Acceptance Criteria

  • Kafka cluster deployed and configured
  • Event producer library ready for backend integration
  • Consumer writes to data warehouse
  • Basic monitoring in place
  • Documentation for other teams

Updates

TU
Test UserFeb 4, 2026, 08:03 PM

Whatever update

0 characters