ZooKeeper Watcher Mechanism: Principles and Practical Guide

This article explores the ZooKeeper Watcher mechanism, a core component for distributed coordination. It covers fundamental concepts, the publish/subscribe model, workflow, application scenarios, and practical command examples.

Background

We have three cloud servers (2C4G each) running a Hadoop learning environment:

• h121 (2C4G)
• h122 (2C4G)
• h123 (2C2G)

[Image: server setup]

Watcher Mechanism Overview

ZooKeeper uses a Watcher mechanism to implement distributed data publish/subscribe functionality. This is a key component for distributed coordination, allowing clients to monitor changes on ZooKeeper nodes (znodes).

Publish/Subscribe Model

The typical publish/subscribe model defines a one-to-many subscription relationship, enabling multiple subscribers to simultaneously monitor a single topic (a znode). Features include:

• Multi-client monitoring: multiple clients can register to watch the same znode.
• Event-driven: triggers notifications on events such as:
  • DataChanged (node data modification)
  • ChildrenChanged (child node list change)
  • NodeCreated / NodeDeleted
• One-time notification: By default, a Watcher fires only once, requiring re-registration after each event.

Workflow

1. Register: Client sets a Watcher via getData(), exists(), or getChildren().
2. Trigger: Server generates an event when the watched znode changes.
3. Notify: Server sends the event via Watcher callback to the client.
4. Handle: Client executes predefined logic upon receiving the event.

Application Scenarios

• Configuration Center: Multiple services listen to a config node; all services update automatically on changes.
• Cluster Management: Monitor node liveness for failure detection and recovery.
• Distributed Locks: Listen to lock node changes to notify lock release.
• Leader Election: Listen to leader node changes for failover.

Performance Characteristics

• Lightweight: Notification contains only event type and node path, not the data.
• Ordered: All Watcher notifications are sent in the order they occurred.
• Reliable: Notifications are guaranteed to be delivered, though network delays may occur.

[Image: Watcher mechanism architecture - client thread, client Watcher Manager, ZooKeeper server]

How Watcher Works

Registering a Watcher

Clients can register a Watcher when reading or fetching node data. ZooKeeper provides several API methods:

• getData(path, watch) - Monitor data changes of a specific node.
• getChildren(path, watch) - Monitor child node changes.
• exists(path, watch) - Monitor node creation/deletion.

Process:

1. Client calls one of these methods with watch=true, sending a registration request to the server.
2. Server records the Watcher in the corresponding node's Watcher list.
3. Server maintains a session ID for each Watcher to ensure only the registering client receives notifications.

Example: In a distributed config system, client calls getData("/config", true) to watch for config changes.

Triggering a Watcher

A Watcher is triggered when the monitored node undergoes:

1. Data modification (setData)
2. Node creation
3. Node deletion
4. Child node list change (add/remove)

Important properties:

• Watcher fires once, then becomes invalid.
• Client must re-register after receiving a notification to continue monitoring.
• Multiple clients can watch the same node.
• Watcher firings are ordered, ensuring consistent state changes.

This one-time mechanism prevents network congestion from excessive events, but clients must handle re-registration correctly.

Sending Notifications

When triggered, ZooKeeper sends a WatchedEvent containing:

1. Event type: NodeCreated, NodeDeleted, NodeDataChanged, NodeChildrenChanged
2. Node path: full path of the changed node

Client processing:

• Receive event, parse type and path.
• Take action: re-fetch data, rebuild watches, update local cache, trigger business logic.
• Re-register if needed (call getData/getChildren/exists again).

Typical use cases:

• Configuration Center: notify all clients on config changes.
• Cluster Management: notify menagement node on node up/down.
• Distributed Locks: notify waiting clients on lock release.
• Service Discovery: notify consumers on service registration changes.

Watcher Features

• One-time: Must be re-registered after each event.
• Asynchronous: Client does not block; events are handled via callbacks.
• Lightweight: Minimal performance impact on ZooKeeper server.

Usage Scenarios

• Configuration Management: Dynamically update configurations across services.
• Service Discovery: Detect service nodes joining or leaving.
• Distributed Locks: Monitor lock node status for lock acquisition.

Practical ZooKeeper Commands

Create Nodes

• Persistent node: create /wzk 123456
• Sequential node: create -s /wzk-order 654321
• Ephemeral node: create -e /wzk-temp 123123

[Image: command execution results]

Read Nodes

• List children: ls /
• Get data: get /wzk

[Image: ls / and get /wzk output]

Update Node

set /wzk-temp 111222

Delete Node

delete /wzk-temp

[Image: update and delete command results]