AdTech Bidstream & Clickstream Architecture: ClickHouseCalc High-Cardinality RAM Sizing
In our deterministic ClickHouseCalc model, real-time bidding (RTB) pipelines carrying high-entropy `auction_id` UUIDs require explicit `max_bytes_before_external_group_by` NVMe spill limits and `uniqCombined64` sketches so wide attribution queries do not trigger Code 241 OOM exceptions.
ClickHouseCalc Compression, RAM & TCO Schedule for adtech-clickstream-mergetree
| Ingestion Scale | Raw Volume | MergeTree NVMe Disk | Node RAM & vCPU | Modeled TCO Delta (2026-11-12) |
|---|---|---|---|---|
| 250M Bids/Day (30d · 190 GB/d) | 5.70 TB Raw | 765 GB RAID-10 NVMe | 64 GB RAM / 16 vCPU | ~$2,150/mo BigQuery vs $58.00/mo UltaHost VDS Pro |
| 1B Bids/Day (90d · 760 GB/d) | 68.4 TB Raw | 9.15 TB Enterprise NVMe | 128 GB RAM / 24 vCPU | ~$9,800/mo BigQuery vs $109–$119/mo Dedicated Node |
| 2B Bids/Day (90d · 1.52 TB/d) | 136.8 TB Raw | 18.3 TB NVMe + Cold Array | 128–256 GB RAM / 32 vCPU | ~$19,400/mo BigQuery vs $149.00/mo InterServer Ryzen |
ClickHouseCalc External Spill-to-NVMe `GROUP BY` Guard & HyperLogLog SQL
-- 1. Prevent Code 241 OOM on high-cardinality UUID aggregations:
SET max_memory_usage = '40000000000',
max_bytes_before_external_group_by = '20000000000',
distributed_aggregation_memory_efficient = 1;
-- 2. Replace exact COUNT(DISTINCT) with HyperLogLog uniqCombined64:
SELECT
campaign_id,
count() AS impressions,
countIf(is_click = 1) AS clicks,
uniqCombined64(device_hash) AS unique_reach,
sum(win_price_micros) / 1e6 AS spend_usd
FROM adtech.rtb_impressions
WHERE event_date >= today() - 30
GROUP BY campaign_id
ORDER BY spend_usd DESC;
ClickHouseCalc MergeTree Production Checklist for adtech-clickstream-mergetree
- Encode UUIDs as binary 16-byte `UUID` or `UInt128` columns rather than 36-byte ASCII strings, cutting hash-table key memory by more than half.
- Configure `max_bytes_before_external_group_by` at 50% of `max_memory_usage` with `<tmp_path>` pointed at local RAID-10 PCIe Gen4 NVMe.
- Never lead `ORDER BY` with random `auction_id`; place low-cardinality filter dimensions `(advertiser_id, campaign_id, toStartOfHour(ts))` first.
ClickHouseCalc Engineering FAQ: adtech-clickstream-mergetree
- How does ClickHouseCalc model memory reduction when replacing COUNT(DISTINCT) with uniqCombined64() in AdTech queries?
-
Exact `COUNT(DISTINCT)` allocates an unbounded hash table storing every unique 128-bit ID in RAM, whereas `uniqCombined64()` uses a bounded 64-bit HyperLogLog state (<96 KB per group) with sub-1% estimation error.
- Why does ClickHouseCalc prioritize dedicated VDS or bare-metal Ryzen CPU cores for high-concurrency AdTech scans?
-
Vectorized SIMD column execution processes threads in parallel across CPU cores; avoiding noisy-neighbor hypervisor CPU steal keeps tail aggregation latency predictable.