2013/10/21 by Muhammad Imran, Ioanna Lykourentzou, Imran, Muhammad +5
Computer Science · #Artificial Intelligence (cs.AI) #D.2.2 #Databases (cs.DB) #FOS: Computer and information sciences #H.4 #Software Engineering (cs.SE) #cs.AI #cs.DB #cs.SE
paper · pdf · doi:10.48550/arxiv.1310.5463
arxiv created 2014/08/04 · arxiv updated 2014/08/05
A crowdsourced stream processing system (CSP) is a system that incorporates crowdsourced tasks in the processing of a data stream. This can be seen as enabling crowdsourcing work to be applied on a sample of large-scale data at high speed, or equivalently, enabling stream processing to employ human intelligence. It also leads to a substantial expansion of the capabilities of data processing systems. Engineering a CSP system requires the combination of human and machine computation elements. From a general systems theory perspective, this means taking into account inherited as well as emerging properties from both these elements. In this paper, we position CSP systems within a broader taxonomy, outline a series of design principles and evaluation metrics, present an extensible framework for their design, and describe several design patterns. We showcase the capabilities of CSP systems by performing a case study that applies our proposed framework to the design and analysis of a real system (AIDR) that classifies social media messages during time-critical crisis events. Results show that compared to a pure stream processing system, AIDR can achieve a higher data classification accuracy, while compared to a pure crowdsourcing solution, the system makes better use of human workers by requiring much less manual work effort.