Distributed networks and real-time systems are becoming the most important components for the new computer age, the Internet of Things (IoT), with huge data streams or data sets generated from sensors and data generated from existing legacy systems. The data generated offers the ability to measure, infer and understand environmental indicators, from delicate ecologies and natural resources to urban environments. This can be achieved through the analysis of the heterogeneous data sources (structured and unstructured). In this paper, we propose a distributed framework Event STream Processing Engine for Environmental Monitoring Domain (ESTemd) for the application of stream processing on heterogeneous environmental data. Our work in this area demonstrates the useful role big data techniques can play in an environmental decision support system, early warning and forecasting systems. The proposed framework addresses the challenges of data heterogeneity from heterogeneous systems and real time processing of huge environmental datasets through a publish/subscribe method via a unified data pipeline with the application of Apache Kafka for real time analytics.
Applying Large-Scale Distributed Computing to Structural Bioinformatics -- Bridging Legacy HPC Clusters With Big Data Technologies Using kafka-slurm-agent
Warsaw School of Economics（华沙经济大学）；University of Virginia（弗吉尼亚大学）
Kafka-ML: connecting the data stream with ML/AI frameworks
University of Málaga（马拉加大学）；IHP - Leibniz-Institut für Innovative Mikroelektronik（IHP - 莱布尼茨创新微电子研究所）；Brandenburg University of Technology Cottbus-Senftenberg（勃兰登堡科特布斯-森夫滕贝格工业大学）
SProBench: Stream Processing Benchmark for High Performance Computing Infrastructure
Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Center for Interdisciplinary Digital Sciences (CIDS) TUD Dresden University of Technology（德累斯顿工业大学可扩展数据分析与人工智能中心（ScaDS.AI）跨学科数字科学中心（CIDS））
BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training
Lionrock AI Lab, China Merchants Group, Hong Kong, China（狮岩人工智能实验室，中国商人集团，香港，中国）；Wuhan University, Wuhan, China（武汉大学，武汉，中国）；The Chinese University of Hong Kong, Shenzhen, China（香港中文大学（深圳））
SnapMode: An Intelligent and Distributed Large-Scale Fashion Image Retrieval Platform Based On Big Data and Deep Generative Adversarial Network Technologies
