Today, the prominence of data science within organizations has given rise to teams of data science workers collaborating on extracting insights from data, as opposed to individual data scientists working alone. However, we still lack a deep understanding of how data science workers collaborate in practice. In this work, we conducted an online survey with 183 participants who work in various aspects of data science. We focused on their reported interactions with each other (e.g., managers with engineers) and with different tools (e.g., Jupyter Notebook). We found that data science teams are extremely collaborative and work with a variety of stakeholders and tools during the six common steps of a data science workflow (e.g., clean data and train model). We also found that the collaborative practices workers employ, such as documentation, vary according to the kinds of tools they use. Based on these findings, we discuss design implications for supporting data science team collaborations and future research directions.
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Amazon AI（亚马逊人工智能）
We propose a novel three-stage FIND-RESOLVE-LABEL workflow for crowdsourced annotation to reduce ambiguity in task instructions and thus improve annotation quality. Stage 1 (FIND) asks the crowd to find examples whose correct label seems ambiguous given task instructions. Workers are also asked to provide a short tag which describes the ambiguous concept embodied by the specific instance found. We compare collaborative vs. non-collaborative designs for this stage. In Stage 2 (RESOLVE), the requester selects one or more of these ambiguous examples to label (resolving ambiguity). The new label(s…
Data science workers increasingly collaborate on large-scale projects before communicating insights to a broader audience in the form of visualization. While prior work has modeled how data science teams, oftentimes with distinct roles and work processes, communicate knowledge to outside stakeholders, we have little knowledge of how data science workers communicate intermediately before delivering the final products. In this work, we contribute a nuanced description of the intermediate communication process within data science teams. By analyzing interview data with 8 self-identified data scie…
FairFare: A Tool for Crowdsourcing Rideshare Data to Empower Labor Organizers
Penn State University（宾夕法尼亚州立大学）；Center for Information Technology Policy, Princeton University（普林斯顿大学信息技术政策中心）；University of Colorado Boulder（科罗拉多大学丹佛分校）；Princeton University（普林斯顿大学）；Cornflower Labs（Cornflower实验室）
Creating Interactive Behaviors in Early Sketch by Recording and Remixing Crowd Demonstrations
在设计图形用户界面（GUI）的早期阶段，外观（外观）可以通过草图轻松呈现，但感觉（交互行为）不能，并且通常需要对其工作原理进行描述（Myers et al. 2008）。我们建议使用众包来增强早期的草图，其中包含由集体“绿野仙踪”生成、使用和重用的交互行为，而不是像以前的工作那样由单个向导生成（Davis等人，2007）。这个演示展示了Apparition（Lasecki et al. 2015）的扩展，Apparition是一个群体驱动的原型设计工具，允许最终用户使用语音和草图创建功能GUI。在Apparition中，群组工作人员在共享画布上实时协作，以交互方式并在最终用户的帮助下优化用户请求的草图。我们的演示扩展了此功能，让群组工作人员“演示”行为所需的画布更改，并改进其演示以提高交互式行为的保真度。然后，该系统让工作人员“重新混合”这些行为，使创建未来的行为更有效率。
This volume contains the position papers presented at CSCW 2021 Workshop - Investigating and Mitigating Biases in Crowdsourced Data, held online on 23rd October 2021, at the 24th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2021). The workshop explored how specific crowdsourcing workflows, worker attributes, and work practices contribute to biases in data. The workshop also included discussions on research directions to mitigate labelling biases, particularly in a crowdsourced context, and the implications of such methods for the workers.
Evaluating Digital Work Instructions with Augmented Reality versus Paper-based Documents for Manual, Object-Specific Repair Tasks in a Case Study with Experienced Workers
Data scientists often collaborate with clients to analyze data to meet a client's needs. What does the end-to-end workflow of a data scientist's collaboration with clients look like throughout the lifetime of a project? To investigate this question, we interviewed ten data scientists (5 female, 4 male, 1 non-binary) in diverse roles across industry and academia. We discovered that they work with clients in a six-stage outer-loop workflow, which involves 1) laying groundwork by building trust before a project begins, 2) orienting to the constraints of the client's environment, 3) collaborativel…
The opacity of machine learning data is a significant threat to ethical data work and intelligible systems. Previous research has addressed this issue by proposing standardized checklists to document datasets. This paper expands that field of inquiry by proposing a shift of perspective: from documenting datasets toward documenting data production. We draw on participatory design and collaborate with data workers at two companies located in Bulgaria and Argentina, where the collection and annotation of data for machine learning are outsourced. Our investigation comprises 2.5 years of research, …
Data science and machine learning (DS/ML) are at the heart of the recent advancements of many Artificial Intelligence (AI) applications. There is an active research thread in AI, \autoai, that aims to develop systems for automating end-to-end the DS/ML Lifecycle. However, do DS and ML workers really want to automate their DS/ML workflow? To answer this question, we first synthesize a human-centered AutoML framework with 6 User Role/Personas, 10 Stages and 43 Sub-Tasks, 5 Levels of Automation, and 5 Types of Explanation, through reviewing research literature and marketing reports. Secondly, we …
The crowd sourced security industry, particularly bug bounty programs, has grown dramatically over the past years and has become the main source of software security reviews for many companies. However, the academic literature has largely omitted security teams, particularly in crowd work contexts. As such, we know very little about how distributed security teams organize, collaborate, and what technology needs they have. We fill this gap by conducting focus groups with the top five teams (out of 18,201 participating teams) of a computer security Capture-the-Flag (CTF) competition. We find tha…
In the era of digital transformation, new technological foundations and possibilities for collaboration, production as well as organization open up many opportunities to work differently in the future. The digitization of workflows results in new forms of working which is denoted by the term Work 4.0. In the context of Work 4.0, digital assistance systems play an important role as they give users additional situation-specific information about a workflow or a product via displays, mobile devices such as tablets and smartphones, or data glasses. Furthermore, such digital assistance systems can …
FareShare: A Tool for Labor Organizers to Estimate Lost Wages and Contest Arbitrary AI and Algorithmic Deactivations
Center for Information Technology Policy, Princeton University（信息科技政策中心，普林斯顿大学）；University of Colorado Boulder（科罗拉多大学博尔德分校）；Cornflower Labs（Cornflower实验室）；Penn State University（宾夕法尼亚州立大学）
