Kazuki Seshimo, Ota Akira, Nishio Daichi, Yamane Satoshi
We propose the online machine learning for big data analysis with heterogeneity. We performed an experiment to compare the accuracy of each iteration between batch one and online one. It is possible to converge quickly with the same accuracy as the batch one.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
Individual-heterogeneous sub-Gaussian Mixture Models
本文提出个体异质子高斯混合模型，通过为每个观测分配异质性参数，有效捕捉实际应用中的异质性。该模型基于谱方法，在弱分离条件下可精确恢复真实聚类标签，优于传统高斯混合模型。
In regression tasks, aleatoric uncertainty is commonly addressed by considering a parametric distribution of the output variable, which is based on strong assumptions such as symmetry, unimodality or by supposing a restricted shape. These assumptions are too limited in scenarios where complex shapes, strong skews or multiple modes are present. In this paper, we propose a generic deep learning framework that learns an Uncountable Mixture of Asymmetric Laplacians (UMAL), which will allow us to estimate heterogeneous distributions of the output variable and shows its connections to quantile regre…
Adaptive Heterogeneous Mixtures of Normalising Flows for Robust Variational Inference
针对单流变分推断在不同分布上表现不稳定的问题，提出由多种互补流组成的两阶段训练自适应混合模型AMF-VI，在六类标准后验上实现更优的似然与传输度量，以低开销提升了变分推断的鲁棒性。
A Heterogeneous Mixture of Experts Framework for Interpretable Machine Learning
该研究提出异构混合专家框架，将决策树、线性SVM等异构专家与概率门控结合，在保证可解释性的同时，在基准数据集上达到与同构MoDT、随机森林相当的预测性能。
Finite Mixture Regression (FMR) refers to the mixture modeling scheme which learns multiple regression models from the training data set. Each of them is in charge of a subset. FMR is an effective scheme for handling sample heterogeneity, where a single regression model is not enough for capturing the complexities of the conditional distribution of the observed samples given the features. In this paper, we propose an FMR model that 1) finds sample clusters and jointly models multiple incomplete mixed-type targets simultaneously, 2) achieves shared feature selection among tasks and cluster comp…
N-BEATS-MOE: N-BEATS with a Mixture-of-Experts Layer for Heterogeneous Time Series Forecasting
本文提出 N-BEATS-MOE，通过引入专家混合层和动态块加权策略，使模型更好地适应异构时间序列特征，在多个基准数据集上提升了预测性能并增强了可解释性。
Learning from Demonstration (LfD) approaches empower end-users to teach robots novel tasks via demonstrations of the desired behaviors, democratizing access to robotics. A key challenge in LfD research is that users tend to provide heterogeneous demonstrations for the same task due to various strategies and preferences. Therefore, it is essential to develop LfD algorithms that ensure \textit{flexibility} (the robot adapts to personalized strategies), \textit{efficiency} (the robot achieves sample-efficient adaptation), and \textit{scalability} (robot reuses a concise set of strategies to repre…
This work builds upon previous efforts in online incremental learning, namely the Incremental Gaussian Mixture Network (IGMN). The IGMN is capable of learning from data streams in a single-pass by improving its model after analyzing each data point and discarding it thereafter. Nevertheless, it suffers from the scalability point-of-view, due to its asymptotic time complexity of $\operatorname{O}\bigl(NKD^3\bigr)$ for $N$ data points, $K$ Gaussian components and $D$ dimensions, rendering it inadequate for high-dimensional data. In this paper, we manage to reduce this complexity to $\operatornam…
Mixture of neural fields for heterogeneous reconstruction in cryo-EM
针对冷冻电镜中混合构象与组成异质性样本建模不足的问题，提出Hydra方法，以K个神经场参数化结构，结合新的基于似然的损失函数，从头建模两类异质性，拓展了冷冻电镜的应用范围。
SegTSim: A Big Data Driven Segmented Temporal Simulation Framework for Heterogeneous Multivariate Systems
SegTSim是一个基于大数据的分段时间模拟框架，通过分段弹性建模、自适应门控及多因素融合，在美日汽车贸易数据上验证了其在扰动下的预测稳定性。
pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learning
针对异构联邦学习无法处理批次级数据异构的问题，提出pFedAFM方法，通过共享全局小特征提取器、交替训练策略和动态特征混合，实现批次级个性化并显著提升准确率。
A particular challenge for disease progression modeling is the heterogeneity of a disease and its manifestations in the patients. Existing approaches often assume the presence of a single disease progression characteristics which is unlikely for neurodegenerative disorders such as Parkinson's disease. In this paper, we propose a hierarchical time-series model that can discover multiple disease progression dynamics. The proposed model is an extension of an input-output hidden Markov model that takes into account the clinical assessments of patients' health status and prescribed medications. We …
Adaptive Graph Mixture of Residual Experts: Unsupervised Learning on Diverse Graphs with Heterogeneous Specialization
提出ADaMoRE框架，通过骨干-残差专家架构和结构感知门控网络，在无监督条件下实现异构图专家混合训练，提升数据效率与训练稳定性，在16个基准上达到最先进性能。
This paper studies the problem of estimating the means $\pmθ_{*}\in\mathbb{R}^{d}$ of a symmetric two-component Gaussian mixture $δ_{*}\cdot N(θ_{*},I)+(1-δ_{*})\cdot N(-θ_{*},I)$ where the weights $δ_{*}$ and $1-δ_{*}$ are unequal. Assuming that $δ_{*}$ is known, we show that the population version of the EM algorithm globally converges if the initial estimate has non-negative inner product with the mean of the larger weight component. This can be achieved by the trivial initialization $θ_{0}=0$. For the empirical iteration based on $n$ samples, we show that when initialized at $θ_{0}=0$, the…
Incorporating Expert Knowledge into Bayesian Causal Discovery of Mixtures of Directed Acyclic Graphs
本文提出基于贝叶斯实验设计原则和变分混合结构学习方法，用于处理异质领域的因果发现问题，通过专家反馈构建信息图先验，提升异质合成数据的结构学习性能，并在乳腺癌数据库中捕捉复杂分布。
MoHETS: Long-term Time Series Forecasting with Mixture-of-Heterogeneous-Experts
本文提出MoHETS，一种仅含编码器的Transformer模型，通过整合稀疏混合异质专家层，有效捕捉多尺度时间序列结构，提升非平稳动态的鲁棒性，并在多个基准测试中取得最佳性能，平均MSE降低12%。
HMoE: Heterogeneous Mixture of Experts for Language Modeling
本文提出异构专家混合模型HMoE，通过不同大小的专家处理不同复杂度的令牌，并设计训练目标促进小专家激活，以更少参数实现更低损失，超越传统同质MoE。
Graph-Driven Models for Gas Mixture Identification and Concentration Estimation on Heterogeneous Sensor Array Signals
针对异构传感器阵列气体分析泛化受限问题，提出集成时间图结构的GraphCapsNet和GraphANet模型，在分类和浓度估计任务上分别达到98.00%以上准确率和0.96以上R2得分，显著提升了工业应用的可扩展性。
Heterogeneous big data poses many challenges in machine learning. Its enormous scale, high dimensionality, and inherent uncertainty make almost every aspect of machine learning difficult, from providing enough processing power to maintaining model accuracy to protecting privacy. However, perhaps the most imposing problem is that big data is often interspersed with sensitive personal data. Hence, we propose a privacy-preserving hierarchical fuzzy neural network (PP-HFNN) to address these technical challenges while also alleviating privacy concerns. The network is trained with a two-stage optimi…
Mixture of Experts (MoE): A Big Data Perspective
本文从大数据视角综述了混合专家的基本原理、架构组件、关键技术挑战解决方案及典型应用，指出MoE将成为大数据时代AI的重要范式。
Graph-Regularized Learning of Gaussian Mixture Models
提出一种分布式环境下的 Gaussian Mixture Models 图正则化学习方法，利用相似性图指导节点间参数共享以避免传输原始数据，在异构和低样本情况下优于集中式与本地训练的模型。
pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning
针对模型异构个性化联邦学习中的隐私、性能与成本平衡问题，提出pFedMoE方法，通过为本地异构大模型分配共享小特征提取器与本地门控网络构建混合专家，实现数据级个性化并支持模型异构。
The combination and aggregation of knowledge from multiple neural networks can be commonly seen in the form of mixtures of experts. However, such combinations are usually done using networks trained on the same tasks, with little mention of the combination of heterogeneous pre-trained networks, especially in the data-free regime. This paper proposes multiple data-free methods for the combination of heterogeneous neural networks, ranging from the utilization of simple output logit statistics, to training specialized gating networks. The gating networks decide whether specific inputs belong to s…
Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees
本文提出一种无需重新训练的异构计算框架，通过数字计算处理噪声敏感的专家和密集激活模块，以提高大规模混合专家模型在模拟非理想特性下的鲁棒性和效率。
Decentralized EM Algorithm for Gaussian Mixtures under Data Heterogeneity and Partial Labeling
针对分布式联邦学习中数据异质性导致经典EM算法估计有偏的问题，提出动量网络EM（MNEM）算法和半监督MNEM（semi-MNEM）算法，实现渐近有效估计并加速收敛。
Mixture of Experts (MoE) are successful models for modeling heterogeneous data in many statistical learning problems including regression, clustering and classification. Generally fitted by maximum likelihood estimation via the well-known EM algorithm, their application to high-dimensional problems is still therefore challenging. We consider the problem of fitting and feature selection in MoE models, and propose a regularized maximum likelihood estimation approach that encourages sparse solutions for heterogeneous regression data models with potentially high-dimensional predictors. Unlike stat…
UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models
本文提出UB-SMoE，通过动态调节路由和通用伪梯度解决异构联邦学习中专家利用率失衡和Top-K路由非可微问题，实现低资源客户端的计算节省与性能提升。
FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models
FlexMoRE通过灵活的秩异质专家混合模型，在联邦训练的大型语言模型中实现更高效的性能，使用低秩适配器替代完整专家，提升任务表现并减少参数量。
Heterogeneous Variational Inference for Markov Degradation Hazard Models: Discretized Mixture with Interpretable Clusters
本文提出一种结合精细状态离散化、特征工程和可解释模型选择的框架，用于解决马尔可夫退化危险模型中的关键瓶颈问题，验证了变分推断在稳定性、收敛性和计算效率上的优势。
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
本文提出AutoMoE框架，利用NAS在FLOPs和延迟等约束下搜索异构稀疏MoE架构，实现token级自适应计算，并在NMT上以更低计算量达到接近SwitchTransformer的翻译性能。
Mixture of Heterogeneous Grouped Experts for Language Modeling
本文提出MoHGE，通过双层路由机制和分组辅助损失优化，实现资源感知的专家组合，减少参数量并平衡GPU负载，提升语言模型的效率与实用性。
FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts
针对联邦学习中的客户端异构性挑战，提出基于稀疏混合专家架构的个性化框架FedMoE，通过两阶段微调（启发式搜索构建子模型与模块化聚合）及全局专家推荐机制，有效解决数据异构问题并提升模型性能。
Anchor-MoE: A Mean-Anchored Mixture of Experts For Probabilistic Regression
本文提出Anchor-MoE模型，以任意点回归器的预测为锚定均值，通过可学习度量窗口核和软路由器将样本分派给混合密度网络专家进行异方差校正，在理论上达到极小极大最优L²风险速率，并在UCI回归基准上取得领先的概率回归性能。
Reconstructing Heterogeneous Biomolecules via Hierarchical Gaussian Mixtures and Part Discovery
针对冷冻电镜中分子构象柔性与部件缺失导致的异质性重建难题，提出基于分层高斯混合和部件分割的 CryoSPIRE 框架，并在 CryoBench 上达到最优。
Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search
提出自动化大规模搜索流水线，在LEMUR数据集生态中系统探索4专家异构MoE架构，发现搜索空间偏差并给出修正，识别出高性能与低效专家组合。
Federated learning (FL) is an important paradigm for training global models from decentralized data in a privacy-preserving way. Existing FL methods usually assume the global model can be trained on any participating client. However, in real applications, the devices of clients are usually heterogeneous, and have different computing power. Although big models like BERT have achieved huge success in AI, it is difficult to apply them to heterogeneous FL with weak clients. The straightforward solutions like removing the weak clients or using a small model to fit all clients will lead to some prob…
Mixture-of-Models: Unifying Heterogeneous Agents via N-Way Self-Evaluating Deliberation
通过N方式自我评估 deliberation 协议，统一异质代理，实现小模型集合在性能上超越大模型，建立新的硬件效率前沿。
Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments
针对联邦学习中模型与数据异构性问题，提出Mosaic框架，通过本地生成模型合成隐私保护数据，并利用混合专家模型蒸馏全局模型，在图像和多模态基准上超越现有方法。
HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs
本文提出HeterMoE系统，通过解耦注意力与专家计算、采用斑马并行重叠计算及非对称专家分配策略，在异构GPU上高效训练MoE模型，实现最高2.3倍加速并高效利用旧GPU。
Drift-aware Collaborative Assistance Mixture of Experts for Heterogeneous Multistream Learning
针对异构多流学习中的概念漂移问题，提出CAMEL框架，通过辅助专家和自主专家调谐器实现动态协作与容量自适应，提升泛化性和抗漂移能力。
Koopman operators model nonlinear dynamics as a linear dynamic system acting on a nonlinear function as the state. This nonstandard state is often called a Koopman observable and is usually approximated numerically by a superposition of functions drawn from a \textit{dictionary}. A widely used algorithm, is \textit{Extended Dynamic Mode Decomposition}, where the dictionary functions are drawn from a fixed, homogeneous class of functions. Recently, deep learning combined with EDMD has been used to learn novel dictionary functions in an algorithm called deep dynamic mode decomposition (deepDMD).…
Clustering algorithms start with a fixed divergence, which captures the possibly asymmetric distance between a sample and a centroid. In the mixture model setting, the sample distribution plays the same role. When all attributes have the same topology and dispersion, the data are said to be homogeneous. If the prior knowledge of the distribution is inaccurate or the set of plausible distributions is large, an adaptive approach is essential. The motivation is more compelling for heterogeneous data, where the dispersion or the topology differs among attributes. We propose an adaptive approach to…
Learning predictive models for combinations of heterogeneous proteomic data sources
本文研究了两种蛋白质混合物表达数据源的个体和联合效用，提出了一种模型融合方法以最大化异质数据集的结合效益。
We study the fundamental tradeoffs between statistical accuracy and computational tractability in the analysis of high dimensional heterogeneous data. As examples, we study sparse Gaussian mixture model, mixture of sparse linear regressions, and sparse phase retrieval model. For these models, we exploit an oracle-based computational model to establish conjecture-free computationally feasible minimax lower bounds, which quantify the minimum signal strength required for the existence of any algorithm that is both computationally tractable and statistically accurate. Our analysis shows that there…
Prognoses of Traumatic Brain Injury (TBI) outcomes are neither easily nor accurately determined from clinical indicators. This is due in part to the heterogeneity of damage inflicted to the brain, ultimately resulting in diverse and complex outcomes. Using a data-driven approach on many distinct data elements may be necessary to describe this large set of outcomes and thereby robustly depict the nuanced differences among TBI patients' recovery. In this work, we develop a method for modeling large heterogeneous data types relevant to TBI. Our approach is geared toward the probabilistic represen…
HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts
HFedMoE通过定制专家子集和稀疏性感知聚合策略，实现高效异构联邦学习微调。
Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
针对Transformer模型因果语言建模中KV缓存扩展性差的问题，提出mixSGA专家混合方法，通过逐Token路由、权重共享和辅助损失实现动态KV优化，在多模型上的效率和效果均优于静态基线。
Robust Federated Personalised Mean Estimation for the Gaussian Mixture Model
针对联邦学习中异构数据与客户端损坏并存的问题，提出高斯混合模型下个性化均值估计算法，误差与损坏比例近似线性，并给出同阶下界。
Although more layers and more parameters generally improve the accuracy of the models, such big models generally have high computational complexity and require big memory, which exceed the capacity of small devices for inference and incurs long training time. In addition, it is difficult to afford long training time and inference time of big models even in high performance servers, as well. As an efficient approach to compress a large deep model (a teacher model) to a compact model (a student model), knowledge distillation emerges as a promising approach to deal with the big models. Existing k…
Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills
提出Skill-MoE框架，通过推断查询所需技能进行实例级专家选择，并采用批推理策略降低开销，在单GPU上集成16个专家模型，在多个推理基准上平均提升8.15%。
Matrix decomposition is one of the fundamental tools to discover knowledge from big data generated by modern applications. However, it is still inefficient or infeasible to process very big data using such a method in a single machine. Moreover, big data are often distributedly collected and stored on different machines. Thus, such data generally bear strong heterogeneous noise. It is essential and useful to develop distributed matrix decomposition for big data analytics. Such a method should scale up well, model the heterogeneous noise, and address the communication issue in a distributed sys…
HAR-DoReMi: Optimizing Data Mixture for Self-Supervised Human Activity Recognition Across Heterogeneous IMU Datasets
本文提出 HAR-DoReMi 框架，通过 MSE 掩码重建与 Mahony 朝向对齐优化自监督 HAR 预训练数据混合，在跨异构 IMU 数据集迁移中以更少数据显著提升准确率。
UniFed: A Universal Federation of a Mixture of Highly Heterogeneous Medical Image Classification Tasks
针对联邦学习中医学数据与任务高度异构、通信成本高及收敛不一致的问题，提出通用联邦学习范式UniFed，通过动态模型调整、序列迁移等机制，在多类疾病诊断任务上实现了准确率、通信成本和收敛速度的提升。
Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths
MoA通过异质滑动窗口长度优化LLM推理效率，提升上下文长度和检索准确率，减少内存消耗并提高吞吐量。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。