Purpose　Many　science,　technology　and　innovation　(STI)　resources　are　attached　with　several　different　labels.　To　assign　automatically　the　resulting　labels　to　an　interested　instance,　many　approaches　with　good　performance　on　the　benchmark　datasets　have　been　proposed　for　multilabel　classification　task　in　the　literature.　Furthermore,　several　open-source　tools　implementing　these　approaches　have　also　been　developed.　However,　the　characteristics　of　real-world　multilabel　patent　and　publication　datasets　are　not　completely　in　line　with　those　of　benchmark　ones.　Therefore,　the　main　purpose　of　this　paper　is　to　evaluate　comprehensively　seven　multi-label　classification　methods　on　real-world　datasets.Design/methodology/approach　Three　real-world　datasets　(Biological-Sciences,　Health-Sciences,　and　USPTO)　from　SciGraph　and　USPTO　database　are　constructed.　Seven　multilabel　classification　methods　with　tuned　parameters　(dependency-LDA,　MLkNN,　LabelPowerset,　RAkEL,　TextCNN,　TexRNN,　and　TextRCNN)　are　comprehensively　compared　on　these　three　real-world　datasets.　To　evaluate　the　performance,　the　study　adopts　three　classification-based　metrics:　Macro-F1,　Micro-F1,　and　Hamming　Loss.Findings　The　TextCNN　and　TextRCNN　models　show　obvious　superiority　on　small-scale　datasets　with　more　complex　hierarchical　structure　of　labels　and　more　balanced　documentlabel　distribution　in　terms　of　macro-F1,　micro-F1　and　Hamming　Loss.　The　MLkNN　method　works　better　on　the　larger-scale　dataset　with　more　unbalanced　document-label　distribution.Research　limitations　Three　real-world　datasets　differ　in　the　following　aspects:　statement,　data　quality,　and　purposes.　Additionally,　open-source　tools　designed　for　multi-label　classification　also　have　intrinsic　differences　in　their　approaches　for　data　processing　and　feature　selection,　which　in　turn　impacts　the　performance　of　a　multi-label　classification　approach.　In　the　near　future,　we　will　enhance　experimental　precision　and　reinforce　the　validity　of　conclusions　by　employing　more　rigorous　control　over　variables　through　introducing　expanded　parameter　settings.Practical　implications　The　observed　Macro　F1　and　Micro　F1　scores　on　real-world　datasets　typically　fall　short　of　those　achieved　on　benchmark　datasets,　underscoring　the　complexity　of　real-world　multi-label　classification　tasks.　Approaches　leveraging　deep　learning　techniques　offer　promising　solutions　by　accommodating　the　hierarchical　relationships　and　interdependencies　among　labels.　With　ongoing　enhancements　in　deep　learning　algorithms　and　large-scale　models,　it　is　expected　that　the　efficacy　of　multi-label　classification　tasks　will　be　significantly　improved,　reaching　a　level　of　practical　utility　in　the　foreseeable　future.Originality/value　(1)　Seven　multi-label　classification　methods　are　comprehensively　compared　on　three　real-world　datasets.　(2)　The　TextCNN　and　TextRCNN　models　perform　better　on　small-scale　datasets　with　more　complex　hierarchical　structure　of　labels　and　more　balanced　document-label　distribution.　(3)　The　MLkNN　method　works　better　on　the　larger-scale　dataset　with　more　unbalanced　document-label　distribution.