ClickHouse Hot-NVMe to Cold-ZFS/S3 Tiering: ClickHouseCalc 3-Year Compliance Dossier
In our deterministic ClickHouseCalc model, multi-volume MergeTree storage policies keep active 14-day partitions on fast local NVMe and automatically `RECOMPRESS CODEC(ZSTD(9))` aged compliance partitions onto bulk ZFS arrays or S3 object buckets.
ClickHouseCalc Compression, RAM & TCO Schedule for hot-nvme-cold-s3-tiering
| Ingestion Scale | Raw Volume | MergeTree NVMe Disk | Node RAM & vCPU | Modeled TCO Delta (2026-11-12) |
|---|---|---|---|---|
| 100M Logs/Day (3 Years · 148 GB/d) | 162.0 TB Raw | 600 GB Hot NVMe + 14.5 TB Cold ZSTD-9 | 64 GB RAM / 12–16 vCPU | ~$14,200/mo Cloud SIEM vs $109–$149/mo InterServer Storage |
| 500M Logs/Day (365d · 740 GB/d) | 270.1 TB Raw | 2.2 TB Hot NVMe + 23.0 TB Cold Tier | 128 GB RAM / 24 vCPU | ~$31,500/mo Cloud SIEM vs $149.00/mo Bare-Metal + Cold Array |
ClickHouseCalc Multi-Volume `storage_tiers.xml` & `RECOMPRESS` TTL Policy
<!-- /etc/clickhouse-server/config.d/storage_tiers.xml -->
<clickhouse>
<storage_configuration>
<disks>
<hot_nvme><path>/var/lib/clickhouse/hot_nvme/</path></hot_nvme>
<cold_bulk>
<type>s3</type>
<endpoint>https://s3-archive.internal/ch-cold/</endpoint>
<metadata_path>/var/lib/clickhouse/disks/cold_bulk/</metadata_path>
</cold_bulk>
</disks>
<policies>
<tiered_compliance>
<volumes>
<hot><disk>hot_nvme</disk></hot>
<cold><disk>cold_bulk</disk><prefer_not_to_merge>true</prefer_not_to_merge></cold>
</volumes>
</tiered_compliance>
</policies>
</storage_configuration>
</clickhouse>
-- Table TTL re-compressing after 14d & moving to cold volume after 30d:
ALTER TABLE audit.security_events
MODIFY TTL
event_time + INTERVAL 14 DAY RECOMPRESS CODEC(ZSTD(9)),
event_time + INTERVAL 30 DAY TO VOLUME 'cold',
event_time + INTERVAL 1095 DAY DELETE;
ClickHouseCalc MergeTree Production Checklist for hot-nvme-cold-s3-tiering
- Set `<cache_on_write_operations>0</cache_on_write_operations>` on cold S3 disks so background TTL partition transfers do not evict hot NVMe mark caches.
- Execute `RECOMPRESS CODEC(ZSTD(9))` at the 14-day mark before moving parts to the cold volume, cutting cold object footprint by ~21% compared to ZSTD(3).
- Enable `<prefer_not_to_merge>true</prefer_not_to_merge>` on remote cold volumes to prevent multi-gigabyte historical part re-downloads.
ClickHouseCalc Engineering FAQ: hot-nvme-cold-s3-tiering
- Does ClickHouseCalc multi-volume tiering require separate SQL table names for hot NVMe vs 2-year-old cold S3 data?
-
No—both hot NVMe parts and cold ZFS/S3 parts remain inside one logical MergeTree table, and ClickHouse prunes partitions automatically via the `WHERE` time predicate.
- How does ClickHouseCalc compare local multi-TB SATA/ZFS cold mounts against S3 object buckets for compliance archives?
-
On dedicated servers with 12TB–20TB local enterprise drives, mounting `/mnt/cold_array` avoids S3 API PUT/GET charges and object-store first-byte network latency.