Understanding Real-time Stream Processing

Real-time stream processing differs from batch processing by handling continuous, unbounded data streams with low latency. Unlike batch jobs that operate on complete datasets, stream processing applies computations to data as it arrives, often using incremental updates for efficiency.

Key Characteristics

• Unbounded Input: Data streams have no predefined end
• Continuous Output: Results are generated and updated continuously
• Low Latency: Processing occurs with minimal delay
• Incremental Computation: Only new data affects calculaitons

Common Use Cases

1. Log Analysis: Real-time user behavior tracking and metrics calculation
2. Dashboard Statistics: Live updates of business KPIs
3. Transportation Tracking: Real-time vehicle location updates
4. Content Scoring: Dynamic ranking of articles based on user engagement

Kafka Streams Implementation

Kafka Streams provides a lightweight library for stream processing with these advantages:

• Embedded direct in Java applications
• No external dependencies beyond Kafka
• Horizontal scalability using Kafka partitions
• Exactly-once processing semantics
• Millisecond-level latency

Core Concepts

Processing Topology

• Stream: An ordered, fault-tolerant sequence of key-value records
• Processor: A node that transforms stream data
• Source Processor: Ingests data from Kafka topics
• Sink Processor: Outputs results to Kafka topics

Data Structures

• KStream: Represents an immutable log of events (insert-only)
• KTable: Represents a mutable table with update semantics

Practical Implementation

Dependencies

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-streams</artifactId>
  <version>${kafka.version}</version>
</dependency>

Word Count Example

public class WordCountProcessor {
  public static void main(String[] args) {
    Properties config = new Properties();
    config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka:9092");
    config.put(StreamsConfig.APPLICATION_ID_CONFIG, "word-count-app");
    
    StreamsBuilder builder = new StreamsBuilder();
    
    builder.stream("input-topic")
      .flatMapValues(text -> Arrays.asList(text.split("\\s+")))
      .groupBy((key, word) -> word)
      .count()
      .toStream()
      .to("output-topic");
    
    new KafkaStreams(builder.build(), config).start();
  }
}

Hot Article Calculation System

Architecture

1. Batch Processing: Daily job calculates baseline scores
2. Real-time Updates: User actions trigger incremental score updates
3. Caching: Channel-specific rankings stored in Redis

Scoring Logic

• View: 1 point
• Like: 3 points
• Comment: 5 points
• Favorite: 8 points

Redis Integration

// Store hot articles
redisTemplate.opsForValue().set(
  "hot_articles:" + channelId, 
  JSON.toJSONString(sortedArticles)
);

// Retrieve for API
String cached = redisTemplate.opsForValue().get("hot_articles:all");
List<Article> articles = JSON.parseArray(cached, Article.class);

Kafka Stream Aggregation

@Component
public class ArticleScoreStream implements KafkaStreamListener<KStream<String,String>> {
  
  @Override
  public KStream<String,String> process(KStream<String,String> stream) {
    return stream
      .map((key, value) -> {
        ArticleEvent event = parseEvent(value);
        return new KeyValue<>(event.getArticleId(), event);
      })
      .groupByKey()
      .windowedBy(TimeWindows.of(Duration.ofMinutes(5)))
      .aggregate(
        ArticleStats::new,
        (articleId, event, stats) -> updateStats(event, stats)
      )
      .toStream()
      .map(this::formatOutput);
  }
}