Tidehunter: 低数据迁移的大值存储
Tidehunter: Large-Value Storage With Minimal Data Relocation
AI总结：
Tidehunter通过将WAL视为永久存储，消除值合并，实现高吞吐量的大值存储，提升查询性能。
摘要：
日志结构合并树（LSM-trees）主导着持久化的键值存储，但在随机工作负载下，由于重复的合并，写放大率从10倍到30倍，导致性能问题。这种开销对于具有均匀分布键的大值来说尤为严重，这在内容可寻址存储、去重系统和区块链验证器中很常见。我们提出了Tidehunter存储引擎，通过将预写日志（WAL）视为永久存储而非临时恢复缓冲区，从而消除值的合并。值永远不会被覆盖；小的、懒惰刷新的索引表将键映射到WAL位置。Tidehunter引入了（a）无锁写入，通过原子分配和并行复制饱和NVMe驱动器，（b）乐观索引结构，利用均匀的键分布实现单次往返查找，以及（c）基于纪元的修剪，无需阻塞写入即可回收空间。在1TB数据集上，Tidehunter实现了每秒830,000次写入，比RocksDB高8.4倍，比BlobDB高2.9倍，同时将点查询提高了1.7倍，存在检查提高了15.6倍。我们通过将Tidehunter集成到Sui区块链中验证了实际影响，其中它在导致RocksDB支持的验证器崩溃的负载下保持稳定的吞吐量和延迟。Tidehunter已经投入生产并在Sui中部署。
英文摘要：
Log-Structured Merge-Trees (LSM-trees) dominate persistent key-value storage but suffer from high write amplification from 10x to 30x under random workloads due to repeated compaction. This overhead becomes prohibitive for large values with uniformly distributed keys, a workload common in content-addressable storage, deduplication systems, and blockchain validators. We present Tidehunter, a storage engine that eliminates value compaction by treating the Write-Ahead Log (WAL) as permanent storage rather than a temporary recovery buffer. Values are never overwritten; and small, lazily-flushed index tables map keys to WAL positions. Tidehunter introduces (a) lock-free writes that saturate NVMe drives through atomic allocation and parallel copying, (b) an optimistic index structure that exploits uniform key distributions for single-roundtrip lookups, and (c) epoch-based pruning that reclaims space without blocking writes. On a 1 TB dataset with 1 KB values, Tidehunter achieves 830K writes per second, that is 8.4x higher than RocksDB and 2.9x higher than BlobDB, while improving point queries by 1.7x and existence checks by 15.6x. We validate real-world impact by integrating Tidehunter into Sui, a high-throughput blockchain, where it maintains stable throughput and latency under loads that cause RocksDB-backed validators to collapse. Tidehunter is production-ready and is being deployed in production within Sui.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
