Apache Flink and Apache Hive are commonly integrated in modern data pipelines to enable efficient batch and stream processing with persistent, partitioned storage. One powerful capability is writing data from Flink SQL into Hive tables using dynamic partitioning—where partition values are derived from the data itself rather than being hardcoded.

Understanding Dynamic Partitioning in Hive

In Hive, tables can be partitioned to improve query performance and manageability. Static partitioning requires explicitly specifying partition column values during insertion. In contrast, dynamic partitioning infers partition values from the input dataset at runtime, allowing a single INSERT statement to populate multiple partitions automatically.

Setting Up a Hive Table in Flink

To write to a Hive table via Flink SQL, first register the Hive catalog and ensure the target table exists with appropriate partition columns. For example, assume a Hive table defined as:

CREATE TABLE sales_data (
  user_id BIGINT,
  product STRING
)
PARTITIONED BY (dt STRING, hr STRING)
STORED AS PARQUET;

This table is partitioned by dt (date) and hr (hour).

Writing Data with Dynamic Partitioning

Suppose you have a source table in Flink (e.g., from Kafka or a file system) that includes the partition columns:

CREATE TABLE event_stream (
  user_id BIGINT,
  product STRING,
  dt STRING,
  hr STRING
) WITH (
  'connector' = 'kafka',
  'topic' = 'sales-events',
  'properties.bootstrap.servers' = 'localhost:9092',
  'format' = 'json'
);

You can insert data into the Hive table with dynamic partitioning using a standard INSERT INTO statement:

INSERT INTO hive_catalog.my_db.sales_data
SELECT user_id, product, dt, hr
FROM event_stream;

Flink automatically routes each row to the correct Hive partition based on the values of dt and hr. No explicit PARTITION (...) clause is needed—the presence of partition columns in the SELECT list enibles dynamic behavior.

Configuration Requirements

To enable dynamic partitioning when writing to Hive:

• Ensure the Hive catalog is properly configured in Flink.
• The target Hive table must already exist with defined partition columns.
• The SELECT clause must include all partition columns in the same order as defined in the Hive table.
• Set the following Hive-compatible options in your Flink environment (if using older versions):

SET 'table.sql-dialect' = 'hive';

Note: In Flink 1.11+, the Hive dialect is often auto-detected when using a Hive catalog.

Benefits of Dynamic Partitioning

Dynamic partitioning reduces boilerplate code, minimizes human error in specifying partition paths, and scales efficiently across large datasets with many partition combinations. It also aligns well with event-time processing, where partition values naturally derive from timestamps in the data.

The diagram illustrates a conceptual relationship: both FlinkTable and HiveTable inherit from a base Table structure. The HiveTable extends it with partition columns (dt, hr), which are essential for dynamic partition writes.