Skip to content

Create keeper-dependent-features.md #165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
filimonov wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Create keeper-dependent-features.md #165

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions content/en/altinity-kb-setup-and-maintenance/keeper-dependent-features.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
title: "Keeper-Dependent Features in ClickHouse"
linkTitle: "Keeper-Dependent Features in ClickHouse"
weight: 100
description: >-
Keeper-Dependent Features in ClickHouse
---

# Keeper-Dependent Features in ClickHouse

This is a consolidated list of features that depend on ClickHouse Keeper (or ZooKeeper-compatible API).

`Keeper` below means either ClickHouse Keeper or Apache ZooKeeper, depending on deployment.

| # | Feature | How configured | Default path / example | Keeper structure (brief) |
|---|---|---|---|---|
| 1 | Replicated tables (`ReplicatedMergeTree` family) | Use `ENGINE = ReplicatedMergeTree(...)`, or omit args and rely on server defaults `default_replica_path`, `default_replica_name`. | Default path: `/clickhouse/tables/{uuid}/{shard}`; replica: `{replica}`. | `<table_path>/metadata`, `columns`, `log`, `blocks`, `async_blocks`, `deduplication_hashes`, `block_numbers`, `leader_election`, `replicas/<replica>/queue, parts, flags, ...`, `mutations`, `quorum`. |
| 2 | `S3Queue` | Table setting `keeper_path`; if omitted CH builds path from `s3queue_default_zookeeper_path` + DB UUID + table UUID. | Default prefix: `/clickhouse/s3queue/`. | `<keeper_path>/metadata`, `processed`, `failed`, `processing`, `persistent_processing`, `registry`; ordered mode may also use `buckets/...` subtrees. |
| 3 | `Kafka` Keeper-offset mode (`StorageKafka2`, experimental) | Enable setting `allow_experimental_kafka_offsets_storage_in_keeper=1` and set both `kafka_keeper_path`, `kafka_replica_name`. | No default keeper path (must be set), docs example: `/clickhouse/{database}/{uuid}`. | `<kafka_keeper_path>/topics/<topic>/partitions`, `topic_partition_locks`, `replicas/<replica>`, temporary `dropped` for drop coordination. |
| 4 | Distributed DDL queue (`ON CLUSTER`) | Server settings: `distributed_ddl.path`, `distributed_ddl.replicas_path`. | Defaults: `/clickhouse/task_queue/ddl/` and `/clickhouse/task_queue/replicas/`. | Queue entries as `query-XXXXXXXXXX` nodes. Each entry has status dirs: `active/<host_id>`, `finished/<host_id>`, optional `synced/<host_id>`, `shards/<shard>/...`. Replicas liveness is tracked under `<replicas_path>/<host_id>/active` (ephemeral). |
| 5 | `KeeperMap` table engine | Server config must define `keeper_map_path_prefix`; table uses engine arg `root_path`. | No built-in default (disabled if prefix is absent). Common example: `/keeper_map_tables`. | `<prefix>/<root_path>/metadata`, `metadata/tables/<table_unique_id>`, `data/<serialized_key>`, drop/cleanup coordination nodes under `metadata/...`. |
| 6 | Replicated databases (`ENGINE=Replicated`) | `ENGINE = Replicated(zoo_path, shard, replica)` or omit args and use DB-replicated defaults. | Default DB path: `/clickhouse/databases/{uuid}`. | `<db_path>/log/query-*`, `replicas/<full_replica_name>/log_ptr, digest, replica_group, ...`, `metadata/<table_name>`, `counter/cnt-*`, `max_log_ptr`, `logs_to_keep`. |
| 7 | Replicated access entities (users/roles/grants/quotas/policies) | Configure `user_directories` with `<replicated><zookeeper_path>...</zookeeper_path></replicated>`. | No mandatory default; common example: `/clickhouse/access`. | `<path>/uuid/<entity_uuid>` stores entity payload. Type maps: `U` (users), `R` (roles), `S` (settings profiles), `P` (row policies), `Q` (quotas), `M` (masking policies), each mapping name -> UUID. |
| 8 | Replicated SQL UDFs (`CREATE FUNCTION`) | Set server config `user_defined_zookeeper_path` (otherwise disk storage is used). | No default keeper path; common example: `/clickhouse/udf`. | Root node plus one znode per function, e.g. `function_<escaped_name>.sql` containing CREATE FUNCTION text. |
| 9 | Named collections in Keeper | Configure `named_collections_storage.type = keeper|zookeeper` (or encrypted variants) and set `named_collections_storage.path`. | Default storage type is `local`; no default keeper path when keeper mode is selected. | Root path with one znode per collection: `<path>/<escaped_collection_name>.sql` containing CREATE NAMED COLLECTION statement. |
| 10 | Workload scheduler definitions in Keeper (`CREATE WORKLOAD`, `CREATE RESOURCE`) | Set `workload_zookeeper_path` (if absent, disk `workload_path` is used). | No default keeper path; docs example: `/clickhouse/workload/definitions.sql`. | Single watched znode at the configured path, content is a serialized list of workload/resource CREATE statements. |
| 11 | Cluster discovery (experimental) | Enable `allow_experimental_cluster_discovery=1`; configure `<remote_servers>...<discovery><path>...`. | No default path; examples use `/clickhouse/discovery/<cluster>`. | Discovery root contains `shards/<server_uuid>` ephemeral nodes with JSON payload (`address`, `shard_id`, version). |
| 12 | `BACKUP/RESTORE ... ON CLUSTER` coordination | Server config `backups.zookeeper_path`. | Default: `/clickhouse/backups`. | Operation roots like `backup-<uuid>` / `restore-<uuid>` with coordination subtrees: stage sync, replicated objects acquisition, file mapping, keeper-map coordination, etc. |
| 13 | `AzureQueue` | Same object-storage queue keeper model as `S3Queue`, with `keeper_path` setting. | Uses the same queue metadata path logic (`s3queue_default_zookeeper_path` prefix if no explicit path). | Same pattern as `S3Queue`: `metadata`, `processed/failed/processing`, `registry`, optional `buckets` in ordered mode. |
| 14 | `generateSerialID()` function | Server setting `series_keeper_path`. | Default: `/clickhouse/series`. | One node per series: `<series_keeper_path>/<series_name>`, value is current counter. |
| 15 | Experimental transactions | Configure `transaction_log.zookeeper_path` (and enable related experimental transaction settings). | Default: `/clickhouse/txn`. | `<path>/tail_ptr` and `<path>/log/csn-*` sequential nodes storing commit sequence and transaction IDs. |
| 16 | `Shared` database engine (Cloud) | ClickHouse Cloud managed behavior (not typically user-configured in self-managed OSS). | Internal/cloud-managed. | Shared catalog is Keeper-backed; low-level path layout is internal and not documented as a stable public contract. |

## ON CLUSTER relation (important)

`ON CLUSTER` itself uses Keeper-backed distributed DDL queue (row 4).
Some features above already replicate through Keeper, so `ON CLUSTER` can be redundant:

- Replicated database DDLs (row 6).
- Replicated access entities (row 7).
- Replicated UDFs (row 8).
- Keeper-backed named collections (row 9).

ClickHouse has dedicated settings like `ignore_on_cluster_for_replicated_*` to control this behavior.

## Notes

- For many rows, path names can be redirected to auxiliary Keeper clusters using `<auxiliary_zookeepers>` and `cluster_name:/path` notation where supported.
- Node names shown above are the stable conceptual layout from current source tree; some minor subnodes are version-specific.