Cost and Quality Assurance in Crowdsourcing Workflows - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Preprints, Working Papers, ... Year : 2020

Cost and Quality Assurance in Crowdsourcing Workflows


Crowdsourcing platforms provide tools to replicate and distribute micro tasks (simple, independent work units) to the crowd and assemble results. However, real-life problems are often complex: they require to collect, organize, process or transform data, guarantee the quality of results, and meet budget constraints. Further, specifications and realization of intricate jobs on crowdsourcing platforms are still in their infancy. In this work, we combine workflows with crowdsourcing to enable efficient execution of complex tasks. Workflows provide ways to organize a complex task in phases and guide the overall realization. The challenge is to interface workflows and crowd-sourcing systems efficiently to achieve good accuracy of results at a reasonable cost. Standard "static" allocation of work in crowd-sourcing affects a fixed number of workers per micro-task to realize and aggregates the results. We propose synchronous and asynchronous dynamic worker allocation techniques on top of workflows, where decisions to replicate tasks, worker allocation and progress of the workflow execution depend on inferred tasks difficulty, worker expertise, confidence in answers and remaining budget. We evaluate the performance of this framework on a benchmark and show that the proposed approaches outperform static allocation in terms of cost and accuracy.
Fichier principal
Vignette du fichier
edbt-workflow.pdf (2.22 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-02964736 , version 1 (12-10-2020)


  • HAL Id : hal-02964736 , version 1


Loïc Hélouët, Zoltan Miklos, Rituraj Singh. Cost and Quality Assurance in Crowdsourcing Workflows. 2020. ⟨hal-02964736⟩
138 View
166 Download


Gmail Facebook X LinkedIn More