BargCrEx: A System for Bargaining Based Aggregation of Crowd and Expert Opinions in Crowdsourcing
Апстракт
Crowdsourcing and crowd voting systems are being increasingly used in societal, industry, and academic problems (labeling, recommendations, social choice, etc.) due to their possibility to exploit "wisdom of crowd" and obtain good quality solutions, and/or voter satisfaction, with high cost-efficiency. However, the decisions based on crowd vote aggregation do not guarantee high-quality results due to crowd voter data quality. Additionally, such decisions often do not satisfy the majority of voters due to data heterogeneity (multimodal or uniform vote distributions) and/or outliers, which cause traditional aggregation procedures (e.g., central tendency measures) to propose decisions with low voter satisfaction. In this research, we propose a system for the integration of crowd and expert knowledge in a crowdsourcing setting with limited resources. The system addresses the problem of sparse voting data by using machine learning models (matrix factorization and regression) for the estimat...ion of crowd and expert votes/grades. The problem of vote aggregation under multimodal or uniform vote distributions is addressed by the inclusion of expert votes and aggregation of crowd and expert votes based on optimization and bargaining models (Kalai-Smorodinsky and Nash) usually used in game theory. Experimental evaluation on real world and artificial problems showed that the bargaining-based aggregation outperforms the traditional methods in terms of cumulative satisfaction of experts and crowd. Additionally, the machine learning models showed satisfactory predictive performance and enabled cost reduction in the process of vote collection.
Кључне речи:
Matrix-factorisation / Machine learning / Expert knowledge / Crowd-voting / Bargaining modelsИзвор:
Group Decision and Negotiation, 2022, 31, 4, 789-818Издавач:
- Springer, Dordrecht
Финансирање / пројекти:
- Office for Naval Research, the United States [ONR - N62909-19-1-2008]
DOI: 10.1007/s10726-022-09783-0
ISSN: 0926-2644
PubMed: 35615756
WoS: 000801210900001
Scopus: 2-s2.0-85130217651
Институција/група
Fakultet organizacionih naukaTY - JOUR AU - Vukicević, Ana AU - Vukićević, Milan AU - Radovanović, Sandro AU - Delibašić, Boris PY - 2022 UR - https://rfos.fon.bg.ac.rs/handle/123456789/2292 AB - Crowdsourcing and crowd voting systems are being increasingly used in societal, industry, and academic problems (labeling, recommendations, social choice, etc.) due to their possibility to exploit "wisdom of crowd" and obtain good quality solutions, and/or voter satisfaction, with high cost-efficiency. However, the decisions based on crowd vote aggregation do not guarantee high-quality results due to crowd voter data quality. Additionally, such decisions often do not satisfy the majority of voters due to data heterogeneity (multimodal or uniform vote distributions) and/or outliers, which cause traditional aggregation procedures (e.g., central tendency measures) to propose decisions with low voter satisfaction. In this research, we propose a system for the integration of crowd and expert knowledge in a crowdsourcing setting with limited resources. The system addresses the problem of sparse voting data by using machine learning models (matrix factorization and regression) for the estimation of crowd and expert votes/grades. The problem of vote aggregation under multimodal or uniform vote distributions is addressed by the inclusion of expert votes and aggregation of crowd and expert votes based on optimization and bargaining models (Kalai-Smorodinsky and Nash) usually used in game theory. Experimental evaluation on real world and artificial problems showed that the bargaining-based aggregation outperforms the traditional methods in terms of cumulative satisfaction of experts and crowd. Additionally, the machine learning models showed satisfactory predictive performance and enabled cost reduction in the process of vote collection. PB - Springer, Dordrecht T2 - Group Decision and Negotiation T1 - BargCrEx: A System for Bargaining Based Aggregation of Crowd and Expert Opinions in Crowdsourcing EP - 818 IS - 4 SP - 789 VL - 31 DO - 10.1007/s10726-022-09783-0 UR - conv_2685 ER -
@article{ author = "Vukicević, Ana and Vukićević, Milan and Radovanović, Sandro and Delibašić, Boris", year = "2022", abstract = "Crowdsourcing and crowd voting systems are being increasingly used in societal, industry, and academic problems (labeling, recommendations, social choice, etc.) due to their possibility to exploit "wisdom of crowd" and obtain good quality solutions, and/or voter satisfaction, with high cost-efficiency. However, the decisions based on crowd vote aggregation do not guarantee high-quality results due to crowd voter data quality. Additionally, such decisions often do not satisfy the majority of voters due to data heterogeneity (multimodal or uniform vote distributions) and/or outliers, which cause traditional aggregation procedures (e.g., central tendency measures) to propose decisions with low voter satisfaction. In this research, we propose a system for the integration of crowd and expert knowledge in a crowdsourcing setting with limited resources. The system addresses the problem of sparse voting data by using machine learning models (matrix factorization and regression) for the estimation of crowd and expert votes/grades. The problem of vote aggregation under multimodal or uniform vote distributions is addressed by the inclusion of expert votes and aggregation of crowd and expert votes based on optimization and bargaining models (Kalai-Smorodinsky and Nash) usually used in game theory. Experimental evaluation on real world and artificial problems showed that the bargaining-based aggregation outperforms the traditional methods in terms of cumulative satisfaction of experts and crowd. Additionally, the machine learning models showed satisfactory predictive performance and enabled cost reduction in the process of vote collection.", publisher = "Springer, Dordrecht", journal = "Group Decision and Negotiation", title = "BargCrEx: A System for Bargaining Based Aggregation of Crowd and Expert Opinions in Crowdsourcing", pages = "818-789", number = "4", volume = "31", doi = "10.1007/s10726-022-09783-0", url = "conv_2685" }
Vukicević, A., Vukićević, M., Radovanović, S.,& Delibašić, B.. (2022). BargCrEx: A System for Bargaining Based Aggregation of Crowd and Expert Opinions in Crowdsourcing. in Group Decision and Negotiation Springer, Dordrecht., 31(4), 789-818. https://doi.org/10.1007/s10726-022-09783-0 conv_2685
Vukicević A, Vukićević M, Radovanović S, Delibašić B. BargCrEx: A System for Bargaining Based Aggregation of Crowd and Expert Opinions in Crowdsourcing. in Group Decision and Negotiation. 2022;31(4):789-818. doi:10.1007/s10726-022-09783-0 conv_2685 .
Vukicević, Ana, Vukićević, Milan, Radovanović, Sandro, Delibašić, Boris, "BargCrEx: A System for Bargaining Based Aggregation of Crowd and Expert Opinions in Crowdsourcing" in Group Decision and Negotiation, 31, no. 4 (2022):789-818, https://doi.org/10.1007/s10726-022-09783-0 ., conv_2685 .