Abstract
BACKGROUND: Machine learning assisted systematic reviewing may help to reduce the work burden in systematic reviews. The aim of this study is therefore to determine by a non-developer the performance of machine learning assisted systematic reviewing on previously published orthopaedic reviews in retrieving relevant papers. METHODS: Active learning for Systematic Reviews (ASReview) was tested against the results from three previously published systematic reviews in the field of orthopaedics with 20 iterations for each review. The reviews covered easy, intermediate and advanced scenarios. The outcomes of interest were the percentage work saved at 95% recall (WSS@95), the percentage work saved at 100% recall (WSS@100) and the percentage of relevant references identified after having screened the first 10% of the records (RRF@10). Means and corresponding [95% confidence intervals] were calculated. RESULTS: The WSS@95 was respectively 72 [71-74], 72 [72-73] and 50 [50-51] for the easy, intermediate and advanced scenarios. The WSS@100 was respectively 72 [71-73], 62 [61-63] and 37 [36-38] for the easy, intermediate and advanced scenarios. The RRF@10 was respectively 79 [78-81], 70 [69-71] and 58 [56-60] for the easy, intermediate and advanced scenarios. CONCLUSIONS: Machine learning assisted systematic reviewing was efficient in retrieving relevant papers for systematic review in orthopaedics. The majority of relevant papers were identified after screening only 10% of the papers. All relevant papers were identified after screening 30%-40% of the total papers meaning that 60%-70% of the work can potentially be saved.