Crowdsourcing, human-in-the-loop big data lifecycles, human-AI-collaboration in work, human-AI-interface in human-in-the-loop work, AI for supporting humans in work, security and privacy in human-AI-collaboration, human factors, ELSI (ethical, legal and social is-sues) of human-in-the-loop systems.

The main purpose of this NII Shonan meeting is to bring together researchers from the multi-disciplinary fields of human-in-the-loop Big Data and AI as well as practitioners, to connect the latest research findings to real-world questions and examine the impact of crowdsourcing research in practice. Our goal is to increase impact of crowdsourcing and study how it contributes to the future of work.

Human-in-the-loop approaches are used extensively in collecting, processing, analyzing, integrating, utilizing, archiving and disposing data, augmenting knowledge, and deriving actionable insights. Human-in-the-loop became necessary in many real-world application domains, such as production design, citizen science, online job markets, and natural disaster response.

However, there is still work to be done to adapt methodologies and transfer research results across disciplines and to real-world applications. One of the reasons is that researchers tend to work in silo and focus on a specific aspect of the human-in-the-loop system. The universally accepted human-in-the-loop system architecture is yet to be agreed upon. It usually contains languages, data management systems, data processing systems, worker-task interfaces, and worker-resource managers. It often requires data-acquiring systems for real-world workers, and real-world recognition methods for realizing AI-workers. We stress that applications are a major component of the human-in-the-loop system architecture, since otherwise the system would not match real-world applications thereby hindering usability. We also note that the system must model human factors for addressing ELSI (Ethical, Legal, Social Issues), such as fairness and transparency. The human-in-the-loop system is unique in that the system itself has to deal with ELSI to a certain extent, while ELSI are usually dealt by its users but not by those who are not a part of it. Therefore, we aim to develop a coherent research agenda of crowdsourcing to mobilize momentum on its convergence.

We focus on not only the requester-centric view but also worker-centric one. An important outstanding question is how AI will influence future workforces – a common fear is that the machines will fully replace the need of human workers. Instead, we believe that a hybrid work-force will be the future of work comprising of a diverse group of machines and humans working together and achieving superior results compared to what either group could accomplish working alone.

Given the current state of progress of human-in-the-loop research, we believe that the time is ripe to embark on a fundamental approach to connect research results to real-world problems in human-in-the-loop Big Data and AI from both viewpoints. The proposal is to assemble researchers and practitioners to share experiences and review the latest research results on different components from different perspectives, identify the current mismatches and missing links between methodologies, research results, and real-world problems, review lessons learned, and identify future guidelines in human-in-the-loop system research. We will invite experts from different countries with different needs and legislations to study impact beyond frontiers. In particular, we will invite practitioners who have experience developing crowdsourcing platforms for online job markers (https://aworker.io/, https://www.qapa.fr/, https://www.mistertemp.com/), collaborative coding platforms (https://www.topcoder.com/), and citizen science (https://www.inaturalist.org/projects). We will invite researchers in the social sciences, psychology, laws, data management and mining, algorithms and systems.

It will build on a previous smaller-scale NSF workshop organized recently by Senjuti Basu Roy (http://convergence2018.info/). The main questions we will address are: ELSI, impact in practice, and the future of work. The concrete questions to be discussed at the meeting will be: a. What are gaps between the research results and problems in crowdsourcing in practice? b. What are ELSI in human-in-the-loop bigdata and AI and what can systems do for ELSI? c. On which aspects in practice human-in-the-loop services/systems will have an impact if the gaps are resolved? d. What is the roadmap? We also discuss how to foster collaboration between researchers and practitioners over the world, and whether we can develop any platforms such as standards and benchmarks to advance this area. Possible outputs of the seminar are a list of roadmap of best practices and research questions that arise from bridging the gap between research results and practices in different aspects and vertical scenarios for the evaluation of future research where collaboration among different components is important.