Home > Publications database > In-depth evaluation of machine learning methods for semi-automating article screening in a systematic review of mechanistic literature. > print

001     180603
005     20241220120851.0
024 7 _ |a 10.1002/jrsm.1589
|2 doi
024 7 _ |a pmid:35798691
|2 pmid
024 7 _ |a 1759-2879
|2 ISSN
024 7 _ |a 1759-2887
|2 ISSN
024 7 _ |a altmetric:130843578
|2 altmetric
037 _ _ |a DKFZ-2022-01433
041 _ _ |a English
082 _ _ |a 900
100 1 _ |a Kebede, Mihiretu
|0 P:(DE-He78)547386e1dd3330f9f40321e89ec05354
|b 0
|e First author
|u dkfz
245 _ _ |a In-depth evaluation of machine learning methods for semi-automating article screening in a systematic review of mechanistic literature.
260 _ _ |a Sao Paulo
|c 2023
|b Programa de Estudos Pós-Graduados em História
336 7 _ |a article
|2 DRIVER
336 7 _ |a Output Types/Journal article
|2 DataCite
336 7 _ |a Journal Article
|b journal
|m journal
|0 PUB:(DE-HGF)16
|s 1678876342_29492
|2 PUB:(DE-HGF)
336 7 _ |a ARTICLE
|2 BibTeX
336 7 _ |a JOURNAL_ARTICLE
|2 ORCID
336 7 _ |a Journal Article
|0 0
|2 EndNote
500 _ _ |a #EA:C020#LA:C020# / 2023 Mar;14(2):156-172
520 _ _ |a We aimed to evaluate the performance of supervised machine learning algorithms in predicting articles relevant for full-text review in a systematic review. Overall, 16,430 manually screened titles/abstracts, including 861 references identified relevant for full-text review were used for the analysis. Of these, 40% (n=6573) were sub-divided for training (70%) and testing (30%) the algorithms. The remaining 60% (n=9857) were used as a validation set. We evaluated down- and up-sampling methods and compared unigram, bigram, and singular value decomposition (SVD) approaches. For each approach, Naïve Bayes, Support Vector Machines (SVM), regularized logistic regressions, Neural Networks, random forest, Logit boost, and XGBoost were implemented using simple term frequency or Tf-Idf feature representations. Performance was evaluated using sensitivity, specificity, precision and area under the Curve. We combined predictions of best-performing algorithms (Youden Index ≥0.3 with sensitivity/specificity≥70/60%). In down sample unigram approach, Naïve Bayes, SVM/quanteda text models with Tf-Idf, and linear SVM e1071 package with Tf-Idf achieved >90% sensitivity at specificity >65%. Combining the predictions of the 10 best-performing algorithms improved the performance to reach 95% sensitivity and 64% specificity in the validation set. Crude screening burden was reduced by 61% (5979) (adjusted: 80.3%) with 5% (27) false negativity rate. All the other approaches yielded relatively poorer performances. The down sampling unigram approach achieved good performance in our data. Combining the predictions of algorithms improved sensitivity while screening burden was reduced by almost two-third. Implementing machine learning approaches in title/abstract screening should be investigated further toward refining these tools and automating their implementation. This article is protected by copyright. All rights reserved.
536 _ _ |a 313 - Krebsrisikofaktoren und Prävention (POF4-313)
|0 G:(DE-HGF)POF4-313
|c POF4-313
|f POF IV
|x 0
588 _ _ |a Dataset connected to CrossRef, PubMed, , Journals: inrepo02.dkfz.de
650 _ 7 |a Automated screening
|2 Other
650 _ 7 |a Citation Screening
|2 Other
650 _ 7 |a Machine Learning
|2 Other
650 _ 7 |a NLP
|2 Other
650 _ 7 |a Natural Language Processing
|2 Other
650 _ 7 |a Systematic review
|2 Other
650 _ 7 |a Text mining
|2 Other
700 1 _ |a Le Cornet, Charlotte
|0 P:(DE-He78)2d4cac0f4bf270325b98fe07036dc6c6
|b 1
|u dkfz
700 1 _ |a Turzanski-Fortner, Renée
|0 P:(DE-He78)74a6af8347ec5cbd4b77e562e10ca1f2
|b 2
|e Last author
|u dkfz
773 _ _ |a 10.1002/jrsm.1589
|g p. jrsm.1589
|0 PERI:(DE-600)2548499-0
|n 2
|p 156-172
|t Cordis
|v 14
|y 2023
|x 1759-2879
909 C O |o oai:inrepo02.dkfz.de:180603
|p VDB
|p OpenAPC
|p openCost
910 1 _ |a Deutsches Krebsforschungszentrum
|0 I:(DE-588b)2036810-0
|k DKFZ
|b 0
|6 P:(DE-He78)547386e1dd3330f9f40321e89ec05354
910 1 _ |a Deutsches Krebsforschungszentrum
|0 I:(DE-588b)2036810-0
|k DKFZ
|b 1
|6 P:(DE-He78)2d4cac0f4bf270325b98fe07036dc6c6
910 1 _ |a Deutsches Krebsforschungszentrum
|0 I:(DE-588b)2036810-0
|k DKFZ
|b 2
|6 P:(DE-He78)74a6af8347ec5cbd4b77e562e10ca1f2
913 1 _ |a DE-HGF
|b Gesundheit
|l Krebsforschung
|1 G:(DE-HGF)POF4-310
|0 G:(DE-HGF)POF4-313
|3 G:(DE-HGF)POF4
|2 G:(DE-HGF)POF4-300
|4 G:(DE-HGF)POF
|v Krebsrisikofaktoren und Prävention
|x 0
914 1 _ |y 2022
915 _ _ |a DEAL Wiley
|0 StatID:(DE-HGF)3001
|2 StatID
|d 2021-02-03
|w ger
915 _ _ |a DBCoverage
|0 StatID:(DE-HGF)0160
|2 StatID
|b Essential Science Indicators
|d 2021-02-03
915 _ _ |a WoS
|0 StatID:(DE-HGF)0113
|2 StatID
|b Science Citation Index Expanded
|d 2021-02-03
915 _ _ |a JCR
|0 StatID:(DE-HGF)0100
|2 StatID
|b RES SYNTH METHODS : 2022
|d 2023-10-26
915 _ _ |a DBCoverage
|0 StatID:(DE-HGF)0300
|2 StatID
|b Medline
|d 2023-10-26
915 _ _ |a DBCoverage
|0 StatID:(DE-HGF)0600
|2 StatID
|b Ebsco Academic Search
|d 2023-10-26
915 _ _ |a Peer Review
|0 StatID:(DE-HGF)0030
|2 StatID
|b ASC
|d 2023-10-26
915 _ _ |a DBCoverage
|0 StatID:(DE-HGF)0199
|2 StatID
|b Clarivate Analytics Master Journal List
|d 2023-10-26
915 _ _ |a DBCoverage
|0 StatID:(DE-HGF)0150
|2 StatID
|b Web of Science Core Collection
|d 2023-10-26
915 _ _ |a DBCoverage
|0 StatID:(DE-HGF)1110
|2 StatID
|b Current Contents - Clinical Medicine
|d 2023-10-26
915 _ _ |a DBCoverage
|0 StatID:(DE-HGF)0200
|2 StatID
|b SCOPUS
|d 2023-10-26
915 _ _ |a IF >= 5
|0 StatID:(DE-HGF)9905
|2 StatID
|b RES SYNTH METHODS : 2022
|d 2023-10-26
915 p c |a APC keys set
|2 APC
|0 PC:(DE-HGF)0000
915 p c |a Local Funding
|2 APC
|0 PC:(DE-HGF)0001
920 2 _ |0 I:(DE-He78)C020-20160331
|k C020
|l C020 Epidemiologie von Krebs
|x 0
920 1 _ |0 I:(DE-He78)C020-20160331
|k C020
|l C020 Epidemiologie von Krebs
|x 0
920 0 _ |0 I:(DE-He78)C020-20160331
|k C020
|l C020 Epidemiologie von Krebs
|x 0
980 _ _ |a journal
980 _ _ |a VDB
980 _ _ |a I:(DE-He78)C020-20160331
980 _ _ |a UNRESTRICTED
980 _ _ |a APC

LibraryCollectionCLSMajorCLSMinorLanguageAuthor
Marc 21