An open-source stream-processing engine that gives you Spark Structured Streaming's programming model and exactly-once guarantees — event-time windows, watermarks, stateful aggregations, replay — on infrastructure you already run: Kafka for the log, Redis for state, RabbitMQ for side-effects, and plain worker processes as the runtime. No JVM, no Spark cluster.
The ideaSpark's guarantees come from a durable log + checkpointed state + idempotent sinks — not from Spark itself. Rebuild those three on commodity infra and you get the same correctness at a fraction of the operational weight.
the engineKafka — logRedis — stateRabbitMQ — side-effects
Structured Streaming, without Spark — Kafka log · Redis state · RabbitMQ side-effects · worker runtime architecture by Blake Medulan · for more information contact blake@3halves-labs.com