Internet-native audio-visual services are witnessing rapid development. Among these services, object-based audio-visual services are gaining importance. In 2014, we established the Software Defined Media (SDM) consortium to target new research areas and markets involving object-based digital media and Internet-by-design audio-visual environments. In this paper, we introduce the SDM architecture that virtualizes networked audio-visual services along with the development of smart buildings and smart cities using Internet of Things (IoT) devices and smart building facilities. Moreover, we design the SDM architecture as a layered architecture to promote the development of innovative applications on the basis of rapid advancements in software-defined networking (SDN). Then, we implement a prototype system based on the architecture, present the system at an exhibition, and provide it as an SDM API to application developers at hackathons. Various types of applications are developed using the API at these events. An evaluation of SDM API access shows that the prototype SDM platform effectively provides 3D audio reproducibility and interactiveness for SDM applications.
Media Query Processing For The Internet-of-Things: Coupling Of Device Energy Consumption And Cloud Infrastructure Billing
音频/视觉识别和检索应用最近在面向物联网(IoT)的服务中获得了极大的关注,因为即使在低端嵌入式系统中,视频摄像机和音频处理芯片组也无处不在。在这种服务的最典型场景中,每个设备提取音频/视觉特征并将其压缩成特征描述符,特征描述符包括媒体查询。这些查询被上传到远程云计算服务,该服务执行用于分类或检索应用的内容匹配。这些服务的两个最关键的方面是:(i)在使用服务时控制设备能耗;(ii)减少云基础设施提供商产生的计费成本。在本文中,我们推导出设备能耗和所产生的云基础设施计费之间的最佳耦合的分析条件。我们的框架包含:产生和传输音频/视频查询的能耗、云基础设施的计费率、同时连接到同一云服务器的设备的数量、{每个设备集群的查询量约束,}以及每个设备的查询数据产生量的统计。我们的分析结果通过以下部署进行了验证:(i)设备端,包括在Beaglebone Linux嵌入式平台上计算的紧凑图像描述符(查询),并传输到Amazon Web Services(AWS)Simple Storage Service;(ii)云侧通过AWS弹性计算云(EC2)实例进行图像相似性检测,AWS Auto Scaling用于根据需求控制实例数量。
CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization
Hujing Digital Media and Entertainment Group(华景数字媒体与娱乐集团);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS部门);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院)