Delivering Real-Time Personalization with Databricks and Redis

The integration of Databricks Real-Time Mode (RTM) and Redis addresses a core challenge in real-time systems: bridging the gap between analytics and action. Traditional batch pipelines refresh data on schedules that are too slow for use cases like personalization or fraud detection, where decisions must be made while a user is still interacting with a page. RTM processes events continuously within Structured Streaming, delivering p99 latencies in the tens to low hundreds of milliseconds. This eliminates the need for a separate stream-processing engine like Apache Flink, reducing operational complexity for teams already using Spark.

Redis complements RTM by handling the serving layer, where applications need to read data in well under a millisecond. Its sub-millisecond read/write performance, flexible data models, and built-in TTL/eviction mechanisms make it suitable for materializing the latest operational state computed by RTM. For teams already using Redis as a cache, this pattern extends its role without introducing new technologies, simplifying integration and reducing the operational burden on application teams.

The division of labor between RTM and Redis is critical. RTM focuses on continuous computation of streaming events, while Redis ensures instant access to the results. This separation allows each component to specialize in what it does best: RTM for processing and Redis for serving. The integration is particularly valuable for high-scale use cases, such as iFood’s machine learning platform, where millions of daily orders require real-time decisions without sacrificing operational simplicity or scalability.

For engineers, the key advantage is the elimination of dual-engine architectures. By using RTM within Databricks, teams can leverage existing Spark APIs to process streams continuously, avoiding the need to maintain separate codebases or logic for batch and real-time pipelines. Redis’s high-availability options, including 99.99% and 99.999% commitments for Multi-AZ and Active-Active deployments, ensure reliability in production environments. This integration simplifies the design of real-time pipelines while meeting the latency and throughput demands of modern applications.