迈向观察湖仓:软件行为的活体、交互式档案库
Towards Observation Lakehouses: Living, Interactive Archives of Software Behavior
AI总结:
本文提出观察湖仓,通过持续SRCs实现软件行为的动态观察与分析,支持行为聚类和共识或acles,展示在509问题基准测试中高效处理能力。
摘要:
代码生成大语言模型主要在静态艺术制品(源代码、注释、规范)上进行训练,很少在运行时行为的材料化上进行训练。因此,它们容易内化有缺陷或标签错误的代码。由于非平凡语义属性在一般情况下是不可判定的,唯一实际获得真实功能的方法是通过动态观察执行。在先前的工作中,我们通过序列表、刺激-响应矩阵(SRMs)和刺激-响应立方体(SRCs)来捕捉和比较测试、实现和上下文中的行为。这些结构使观察数据可以离线分析和重用,但它们本身并不提供持久性、演变或大规模的交互式分析。因此,本文介绍观察湖仓,即持续SRCs的 operationalization:一个高而只允许追加的观察表,存储每一个执行(刺激、响应、上下文)和SQL查询,这些查询可以按需材料化SRC切片。基于Apache Parquet + Iceberg + DuckDB,湖仓从受控管道(LASSO)和CI管道(例如单元测试执行)中摄入数据,使n版本评估、行为聚类和共识或acles无需重新执行。在509问题基准测试中,我们摄入约860万观察行(<51MiB),并在笔记本电脑上在<100ms内重建SRM/SRC视图和聚类,证明持续行为挖掘在无需分布式机器集群的情况下是可行的。这使行为真实值与其它运行时数据同样成为第一类,并提供了一条通往行为感知评估和训练的基础设施路径。观察湖仓,与配套数据集一起,作为开源项目在GitHub上公开发布:https://github.com/SoftwareObservatorium/observation-lakehouse
英文摘要:
Code-generating LLMs are trained largely on static artifacts (source, comments, specifications) and rarely on materializations of run-time behavior. As a result, they readily internalize buggy or mislabeled code. Since non-trivial semantic properties are undecidable in general, the only practical way to obtain ground-truth functionality is by dynamic observation of executions. In prior work, we addressed representation with Sequence Sheets, Stimulus-Response Matrices (SRMs), and Stimulus-Response Cubes (SRCs) to capture and compare behavior across tests, implementations, and contexts. These structures make observation data analyzable offline and reusable, but they do not by themselves provide persistence, evolution, or interactive analytics at scale. In this paper, therefore, we introduce observation lakehouses that operationalize continual SRCs: a tall, append-only observations table storing every actuation (stimulus, response, context) and SQL queries that materialize SRC slices on demand. Built on Apache Parquet + Iceberg + DuckDB, the lakehouse ingests data from controlled pipelines (LASSO) and CI pipelines (e.g., unit test executions), enabling n-version assessment, behavioral clustering, and consensus oracles without re-execution. On a 509-problem benchmark, we ingest $\approx$8.6M observation rows ($<$51MiB) and reconstruct SRM/SRC views and clusters in $<$100ms on a laptop, demonstrating that continual behavior mining is practical without a distributed cluster of machines. This makes behavioral ground truth first-class alongside other run-time data and provides an infrastructure path toward behavior-aware evaluation and training. The Observation Lakehouse, together with the accompanying dataset, is publicly available as an open-source project on GitHub: https://github.com/SoftwareObservatorium/observation-lakehouse
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。