斯隆数字巡天(SDSS)科学数据库描述了超过1.4亿个物体,大小超过1.5 TB。SDSS目录存档服务器(CAS)通过SkyServer网站提供了对SDSS数据的几个级别的查询接口。大多数查询在几秒或几分钟内执行。但是,有些查询可能需要数小时或数天的时间,因为它们需要对最大的表进行非索引扫描,或者因为它们请求非常大的结果集,或者因为它们表示非常复杂的数据聚合。这些“怪物查询”不仅花费很长时间,还影响其他所有人的响应时间-其中一个或多个查询可能会阻塞整个系统。为了改善这一问题,我们开发了一个多服务器多队列批量作业提交和跟踪系统的CAS称为CasJobs。通过网络传输查询的非常大的结果集是另一个严重的问题。统计数据表明,这种数据传输大部分是不必要的;用户更愿意将结果存储在本地,以便允许进一步的连接和过滤。为了允许本地分析,开发了一个系统,在服务器端为用户提供他们自己的个人数据库(MyDB)。用户可以将数据传输到他们的MyDB,然后在将其提取到自己的机器之前进行进一步分析。MyDB表还提供了一种与协作者共享查询结果的方便方法,而无需下载它们。CasJobs是使用SOAP XML Web服务构建的,自2004年5月以来一直在运行。
英文摘要:
The Sloan Digital Sky Survey (SDSS) science database describes over 140 million objects and is over 1.5 TB in size. The SDSS Catalog Archive Server (CAS) provides several levels of query interface to the SDSS data via the SkyServer website. Most queries execute in seconds or minutes. However, some queries can take hours or days, either because they require non-index scans of the largest tables, or because they request very large result sets, or because they represent very complex aggregations of the data. These "monster queries" not only take a long time, they also affect response times for everyone else - one or more of them can clog the entire system. To ameliorate this problem, we developed a multi-server multi-queue batch job submission and tracking system for the CAS called CasJobs. The transfer of very large result sets from queries over the network is another serious problem. Statistics suggested that much of this data transfer is unnecessary; users would prefer to store results locally in order to allow further joins and filtering. To allow local analysis, a system was developed that gives users their own personal databases (MyDB) at the server side. Users may transfer data to their MyDB, and then perform further analysis before extracting it to their own machine. MyDB tables also provide a convenient way to share results of queries with collaborators without downloading them. CasJobs is built using SOAP XML Web services and has been in operation since May 2004.