一种近似最优的确定性更新数据流摘要结构

以$O(\frac{1}{\varepsilon} \log \frac{1}{\varepsilon})$单词表示的随机在线分位数总结

A randomized online quantile summary in $O(\frac{1}{\varepsilon} \log \frac{1}{\varepsilon})$ words

分位数摘要是一种数据结构，它近似于$\varepsilon$ -相对误差，即更大的基础数据集的顺序统计量。在本文中，我们开发了一个随机在线分位数摘要的收银机数据输入模型和比较数据领域模型，使用$O(\frac{1}{\varepsilon} \log \frac{1}{\varepsilon})$的记忆单词。这改进了Agarwal等人的$O(\frac{1}{\varepsilon} \log^{3/2} \frac{1}{\varepsilon})$的先前最佳上界。（PODS 2012）。此外，根据Hung和Ting（FAW 2010）的下限，比较模型的确定性总结在空间复杂度方面无法优于我们的随机总结。最后，我们的总结具有良好的性质，即$O(\frac{1}{\varepsilon} \log \frac{1}{\varepsilon})$字足以确保成功概率为$1 - e^{-\text{poly}(1/\varepsilon)}$。

Even Simpler Deterministic Matrix Sketching

This paper provides a one-line proof of Frequent Directions (FD) for sketching streams of matrices. The simpler proof arises from sketching the covariance of the stream of matrices rather than the stream itself.

确定性近优P2P流传输

Deterministic Near-Optimal P2P Streaming

我们认为流在一个对等网络中的同质节点，其中一个单一的源广播的数据流在系统中的所有用户。对等体被允许任意地（对抗性地）进入或离开系统。在这种情况下，以前的流式传输方法要么使用随机分布图，要么使用具有随机维护算法的结构化树。随机图处理对等体流失很好，但有穷人的连通性保证，而结构化树有良好的连通性，但已被证明难以维持对等体流失。我们提出了一种新的分布式结构，一个确定性和分布式算法的对等体流失下的维护改进这两种方法，我们的结果是最近的工作提出确定性算法的谣言传播图的启发。我们的方法的一个关键创新是在分销结构中有冗余链接。虽然这会导致最大流率可能减少，我们表明，所使用的冗余量，所提出的算法的延迟保证是接近最佳的。我们引入了一个宽容的参数，捕捉最坏情况下的瞬时流速率接收的同行在流失事件和速率，延迟和宽容之间的基本权衡的特点。确定性算法的自然推广实现了这种折衷接近最优。最后，所提出的确定性算法是足够强大的，以处理各种概括：能够处理异构节点容量的同行和更复杂的流模式，多个源传输存在。

通过ReliableSketch在流摘要中逼近100%置信度

Approaching 100% Confidence in Stream Summary through ReliableSketch

提出ReliableSketch，首个将键值流中所有键误差控制在$Λ$内且失败概率为$Δ$的算法，仅需低时空开销，在CPU、FPGA和可编程交换机上实现，实验证明其在小空间下误差受控且吞吐量接近最优。

流中伪确定性计数的下界

Lower Bounds for Pseudo-Deterministic Counting in a Stream

本研究证明流中伪确定性近似计数需要 $\Omega(\sqrt{\log n / \log\log n})$ 比特空间，否定了其匹配 Morris 计数器 $O(\log\log n)$ 空间复杂度的可能性，并引入“移位查找”问题作为关键技术工具。

Optimal Document Exchange and New Codes for Insertions and Deletions

We give the first communication-optimal document exchange protocol. For any $n$ and $k < n$ our randomized scheme takes any $n$-bit file $F$ and computes a $Θ(k \log \frac{n}{k})$-bit summary from which one can reconstruct $F$, with high probability, given a related file $F'$ with edit distance $ED(F,F') \leq k$. The size of our summary is information-theoretically order optimal for all values of $k$, giving a randomized solution to a longstanding open question of [Orlitsky; FOCS'91]. It also is the first non-trivial solution for the interesting setting where a small constant fraction of symbo…

A Deterministic Streaming Sketch for Ridge Regression

We provide a deterministic space-efficient algorithm for estimating ridge regression. For $n$ data points with $d$ features and a large enough regularization parameter, we provide a solution within $\varepsilon$ L$_2$ error using only $O(d/\varepsilon)$ space. This is the first $o(d^2)$ space deterministic streaming algorithm with guaranteed solution error and risk bound for this classic problem. The algorithm sketches the covariance matrix by variants of Frequent Directions, which implies it can operate in insertion-only streams and a variety of distributed data settings. In comparisons to ra…

Simple and Deterministic Matrix Sketching

We adapt a well known streaming algorithm for approximating item frequencies to the matrix sketching setting. The algorithm receives the rows of a large matrix $A \in \R^{n \times m}$ one after the other in a streaming fashion. It maintains a sketch matrix $B \in \R^ {1/\eps \times m}$ such that for any unit vector $x$ [\|Ax\|^2 \ge \|Bx\|^2 \ge \|Ax\|^2 - \eps \|A\|_{f}^2 \.] Sketch updates per row in $A$ require $O(m/\eps^2)$ operations in the worst case. A slight modification of the algorithm allows for an amortized update time of $O(m/\eps)$ operations per row. The presented algorithm stan…

The White-Box Adversarial Data Stream Model

We study streaming algorithms in the white-box adversarial model, where the stream is chosen adaptively by an adversary who observes the entire internal state of the algorithm at each time step. We show that nontrivial algorithms are still possible. We first give a randomized algorithm for the $L_1$-heavy hitters problem that outperforms the optimal deterministic Misra-Gries algorithm on long streams. If the white-box adversary is computationally bounded, we use cryptographic techniques to reduce the memory of our $L_1$-heavy hitters algorithm even further and to design a number of additional …

伪确定性近似计数的紧空间下界

Tight Space Lower Bound for Pseudo-Deterministic Approximate Counting

该研究针对流算法中的近似计数问题，分析其伪确定性复杂度，证明了紧的Ω(log N)空间下界，解决了Goldwasser等人提出的相关问题，补充了伪确定性流算法的空间复杂度理论。

Enabling Efficient and General Subpopulation Analytics in Multidimensional Data Streams

Today's large-scale services (e.g., video streaming platforms, data centers, sensor grids) need diverse real-time summary statistics across multiple subpopulations of multidimensional datasets. However, state-of-the-art frameworks do not offer general and accurate analytics in real time at reasonable costs. The root cause is the combinatorial explosion of data subpopulations and the diversity of summary statistics we need to monitor simultaneously. We present Hydra, an efficient framework for multidimensional analytics that presents a novel combination of using a ``sketch of sketches'' to avoi…

Nearly Linear Time Deterministic Algorithms for Submodular Maximization Under Knapsack Constraint and Beyond

In this work, we study the classic submodular maximization problem under knapsack constraints and beyond. We first present an $(7/16-\varepsilon)$-approximate algorithm for single knapsack constraint, which requires $O(n\cdot\max\{\varepsilon^{-1},\log\log n\})$ queries, and two passes in the streaming setting. This provides improvements in approximation ratio, query complexity and number of passes on the stream. We next show that there exists an $(1/2-\varepsilon)$-approximate deterministic algorithm for constant number of binary packing constraints, which achieves a query complexity of $O_{\…

Improved Algorithms for Time Decay Streams

In the time-decay model for data streams, elements of an underlying data set arrive sequentially with the recently arrived elements being more important. A common approach for handling large data sets is to maintain a \emph{coreset}, a succinct summary of the processed data that allows approximate recovery of a predetermined query. We provide a general framework that takes any offline-coreset and gives a time-decay coreset for polynomial time decay functions. We also consider the exponential time decay model for $k$-median clustering, where we provide a constant factor approximation algorithm …

基于比较的分位数摘要的紧下界

A Tight Lower Bound for Comparison-Based Quantile Summaries

本文提出了基于比较的分位数摘要的紧空间下界，证明了现有方法的最优性，并改进了偏倚分位数和其他相关任务的下界。

Deterministic Heavy Hitters with Sublinear Query Time

This paper studies the classic problem of finding heavy hitters in the turnstile streaming model. We give the first deterministic linear sketch that has $O(ε^{-2} \log n \cdot \log^*(ε^{-1}))$ rows and answers queries in sublinear time. The number of rows is only a factor of $\log^*(ε^{-1})$ more than that used by the state-of-the-art algorithm prior to our paper due to Nelson, Nguyen and Woodruff (RANDOM'12). Their algorithm runs in time at least linear in the universe size $n$, which is highly undesirable in streaming applications. Our approach is based on an iterative procedure, where most …

概率流上聚集性质的估计

Estimating Aggregate Properties on Probabilistic Streams

概率流模型由Jayram等人在\cite{JKV07}中引入。它是数据流模型的一般化，适用于处理“概率”数据，其中流的每个项表示一组可能事件的概率分布。因此，概率流确定潜在地在非常大数量的经典“确定性”流上的分布，其中每个项确定性地是域值之一。概率模型不仅适用于分析输入具有不确定性的流（例如测量物理过程的传感器数据流），而且适用于分析通过后处理（例如标记或协调不一致和质量差的数据）从输入数据导出的流。我们提出了流算法计算常用的聚合概率流。我们提出了第一个已知的，一个通过流算法估计的\AVG，改善结果在\cite{JKV07}。我们提出了第一个已知的流算法估计概率流上的\DISTINCT项目的数量。此外，我们提出了扩展到其他聚合，如重复率，分位数等。在所有情况下，我们的算法工作与可证明的准确性保证和数据流模型的空间约束内。

Streaming Algorithms for Ellipsoidal Approximation of Convex Polytopes

We give efficient deterministic one-pass streaming algorithms for finding an ellipsoidal approximation of a symmetric convex polytope. The algorithms are near-optimal in that their approximation factors differ from that of the optimal offline solution only by a factor sub-logarithmic in the aspect ratio of the polytope.

基于压缩计数的数据流熵估计

Estimating Entropy of Data Streams Using Compressed Counting

Shannon熵是一种广泛应用的概括统计量，例如，网络流量测量，异常检测，神经计算，尖峰序列等。已知Shannon熵可以用Reenyi熵或Tsallis熵来近似，它们都是p阶频率矩的函数，并且当p>1时接近Shannon熵。压缩计数是一种新的数据流p阶频率矩的近似计算方法。我们的贡献包括：1）证明了Renyi熵比Tsallis熵更好地逼近Shannon熵。2)我们提出了CC的最佳分位数估计，这大大改善了以前的估计。3)我们的实验表明，CC确实是非常有效的近似的时刻和熵。我们还证明了利用方差偏差权衡的至关重要性。

Efficient Similarity Search in Dynamic Data Streams

The Jaccard index is an important similarity measure for item sets and Boolean data. On large datasets, an exact similarity computation is often infeasible for all item pairs both due to time and space constraints, giving rise to faster approximate methods. The algorithm of choice used to quickly compute the Jaccard index $\frac{\vert A \cap B \vert}{\vert A\cup B\vert}$ of two item sets $A$ and $B$ is usually a form of min-hashing. Most min-hashing schemes are maintainable in data streams processing only additions, but none are known to work when facing item-wise deletions. In this paper, we …

Flattened Exponential Histogram for Sliding Window Queries over Data Streams

The Basic Counting problem [1] is one of the most fundamental and critical streaming problems of sliding window queries over data streams. Given a stream of 0's and 1's, the purpose of this problem is to estimate the number of 1's in the last N elements (or time units) seen from the stream. Its solution can be used as building blocks to solve numerous more complex problems such as heavy hitter, frequency estimation, distinct counting, etc. In this paper, we present the flattened exponential histogram (FEH) model for the Basic Counting problem. Our model improves over the exponential histogram …

Random Projections for k-Means: Maintaining Coresets Beyond Merge & Reduce

We give a new construction for a small space summary satisfying the coreset guarantee of a data set with respect to the $k$-means objective function. The number of points required in an offline construction is in $\tilde{O}(k ε^{-2}\min(d,kε^{-2}))$ which is minimal among all available constructions. Aside from two constructions with exponential dependence on the dimension, all known coresets are maintained in data streams via the merge and reduce framework, which incurs are large space dependency on $\log n$. Instead, our construction crucially relies on Johnson-Lindenstrauss type embeddings …

通用对称范数估计的私有数据流分析

Private Data Stream Analysis for Universal Symmetric Norm Estimation

本文针对差分隐私约束下的数据流对称范数估计问题，提出一个通用可参数化框架，通过按坐标幅值分层并发布充分统计量，实现任意数量对称范数的同时近似且无额外隐私损失，并支持亚线性空间实现。

Data stream fusion for accurate quantile tracking and analysis

UDDSKETCH is a recent algorithm for accurate tracking of quantiles in data streams, derived from the DDSKETCH algorithm. UDDSKETCH provides accuracy guarantees covering the full range of quantiles independently of the input distribution and greatly improves the accuracy with regard to DDSKETCH. In this paper we show how to compress and fuse data streams (or datasets) by using UDDSKETCH data summaries that are fused into a new summary related to the union of the streams (or datasets) processed by the input summaries whilst preserving both the error and size guarantees provided by UDDSKETCH. Thi…

数据流频率矩的斜投影近似

On Approximating Frequency Moments of Data Streams with Skewed Projections

本文提出了一种用于逼近动态数据流（0<p<=2）的p阶频率矩的斜稳定随机投影方法，该方法在理论计算机科学和数据库领域得到了广泛的研究.我们的方法显着（或当p->1时甚至无限）改进了以前基于（对称）稳定随机投影的方法。我们提出的方法适用于数据流是（a）插入只（收银机模型）;或（b）总是非负的（严格的旋转栅门模型），或（c）最终非负的检查点。这只是实际应用中的一个小限制。我们的方法在p = 1+/- Δ且Δ很小时效果特别好，这是一个实际上很重要的场景。例如，Δ可以是衰减率或利率，它们通常很小。当然，当Δ= 0时，可以计算第1个频繁矩（即，和）使用简单的计数器基本上无误差。我们的方法可以被看作是一个“广义计数器”，因为它可以计算未来的总价值，考虑到衰减或利息应计的影响。总之，我们的贡献是双重的。(A)这是斜稳定随机投影的第一个建议。(B)基于第一性原理，我们开发了各种偏态稳定分布的统计估计，包括它们的方差和误差（尾）概率界，以及样本复杂性界。