Primeiras impressões na ciência: prevendo o impacto de artigos a partir de títulos e metadados básicos

Yonara Costa Magalhães; João Pedro Cavalcanti Azevedo; Antônio de Abreu Batista Junior; Jesús Pascual Mena-Chalco

doi:10.1590/1808-5245.32.149912

Authors

Yonara Costa Magalhães Universidade Ceuma https://orcid.org/0000-0001-5502-9634
João Pedro Cavalcanti Azevedo Universidade Federal do Maranhão https://orcid.org/0009-0003-2035-6872
Antônio de Abreu Batista Junior Universidade Federal do Maranhão https://orcid.org/0000-0002-6013-9704
Jesús Pascual Mena-Chalco Universidade Federal do ABC https://orcid.org/0000-0001-7509-5532

DOI:

https://doi.org/10.1590/1808-5245.32.149912

Keywords:

Citation Prediction, Scientific Impact, BERT, Article Titles, Scientometrics

Abstract

The assessment of scientific impact relies largely on citation-based indicators, which only consolidate over long periods, limiting their usefulness for early editorial decisions, research evaluation, and information management. This study investigates to what extent article titles, used alone or enriched with simple bibliographic metadata available prior to publication, can anticipate future citation impact. The problem is formulated as a binary classification task, distinguishing high-impact and low-impact articles. Experiments are conducted on 78,707 articles from Physical Review A, published by the American Physical Society, with impact defined using citation quartile grouping. A model based on contextual linguistic representations is employed and evaluated using stratified 5-fold cross-validation, comparing the exclusive use of titles with their combination with simple metadata, namely publication year, number of authors, and punctuation. Results are assessed using the Matthews Correlation Coefficient, a normalized metric whose maximum value is 1 and values close to 0 indicate random performance. The findings show limited performance for titles alone (MCC ≈ 0.283) and a substantial improvement when metadata are included (MCC ≈ 0.509), which is statistically significant (Wilcoxon, p < 0.05). These results highlight the potential of predictive approaches as support tools for research evaluation, editorial policies, and scholarly communication.

Downloads

Download data is not yet available.

Author Biographies

Yonara Costa Magalhães, Universidade Ceuma

Yonara Costa Magalhães é Mestre em Engenharia de Eletricidade pela Universidade Federal do Maranhão (UFMA) e Professora Titular na Universidade CEUMA, com atuação também como professora na UEMA e na UFMA. Sua pesquisa na área de Ciência da Computação concentra-se em análise e modelagem de sistemas, desenvolvimento de sistemas e tecnologia educacional. É pesquisadora do Núcleo de Pesquisa em Sistemas e Tecnologia da Informação (NuSTI/CNPq) e membro da Sociedade Brasileira de Computação (SBC).

João Pedro Cavalcanti Azevedo, Universidade Federal do Maranhão

João Pedro Cavalcanti Azevedo é Mestre em Ciência da Computação pela Universidade Federal do Maranhão (UFMA), com especializações em Análise de Dados, Matemática Aplicada e Biotecnologia. Sua atuação profissional e de pesquisa é focada em Inteligência Artificial, Ciência de Dados e Modelos de Linguagem (LLMs), com ênfase no desenvolvimento e aplicação de soluções de Inteligência Artificial Generativa (GenAI).

Antônio de Abreu Batista Junior, Universidade Federal do Maranhão

Antônio de Abreu Batista Júnior is an Adjunct Professor in the undergraduate program in Computer Science at the Federal University of Maranhão (UFMA) and holds a Ph.D. in Computer Science from the Federal University of ABC (UFABC). His research focuses on the field of Scientometrics, with publications in national and international conferences, books, and journals. He serves as a reviewer for high-impact journals in the field, such as Scientometrics, and is a collaborating professor in the Graduate Program in Computer Science at UFMA.

Jesús Pascual Mena-Chalco, Universidade Federal do ABC

Jesús P. Mena-Chalco é Professor Doutor na Universidade Federal do ABC (UFABC), com doutorado e pós-doutorado em Ciência da Computação pela Universidade de São Paulo (USP). Sua pesquisa se concentra na descoberta de conhecimento em grandes volumes de dados acadêmicos, com ênfase em Bibliometria, Cientometria e mineração de grafos. Seu trabalho de mestrado e doutorado recebeu prêmios de destaque em eventos científicos de prestígio. Atualmente, é pesquisador associado da Rede Nacional de Ciência para a Educação (CpE).

References

ALGABA, Andres et al. Large language models reflect human citation patterns with a heightened citation bias. In: FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 2025, Albuquerque. Proceedings [...]. Stroudsburg: Association for Computational Linguistics, 2025. p. 6829-6864.

BABA, Takahiro; BABA, Kensuke; IKEDA, Daisuke. Citation count prediction using abstracts. Journal of Web Engineering, Gistrup, v. 18, n. 1-3, p. 207-228, 2019. Disponível em: https://doi.org/10.13052/jwe1540-9589.18136. Acesso: 20 mar. 2026.

BROWN, Tom B. et al. Language models are few-shot learners. In: NEURAL INFORMATION PROCESSING SYSTEMS, 33., 2020, San Diego. Proceedings [...]. New York: Curran Associates, 2020. p. 1877-1901.

CHICCO, David; JURMAN, Giuseppe. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, New York, v. 21, n. 6, p. 1-13, 2020. Disponível em: https://doi.org/10.1186/s12864-019-6413-7. Acesso: 20 mar. 2026.

DEMŠAR, Janez. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, Massachusetts, v. 7, p. 1-30, 2006.

DEVLIN, Jacob et al. BERT: pre-training of deep bidirectional transformers for language understanding. In: CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 2019, Minneapolis. Proceedings [...]. Stroudsburg: Association for Computational Linguistics, 2019. p. 4171-4186.

GALKE, Lukas et al. Using titles vs. full-text as source for automated semantic document annotation. In: KNOWLEDGE CAPTURE CONFERENCE, 9, 2017, Austin. Proceedings [...]. New York: Association for Computing Machinery, 2017.

GARFIELD, Eugene. Citation indexes for science: a new dimension in documentation through association of ideas. Science, Washington, v. 122, n. 3159, p. 108-111, 1955. Disponível em: https://doi.org/10.1126/science.122.3159.108. Acesso: 20 mar. 2026.

HIRAKO, Jun; SASANO, Ryohei; TAKEDA, Koichi. CiMaTe: citation count prediction effectively leveraging the main text. arXiv, Ithaca, 6 Oct., 2024. Disponível em: https://doi.org/10.48550/arXiv.2410.04404. Acesso: 20 mar. 2026.

JEONG, Chanwoo et al. A context-aware citation recommendation model with BERT and graph convolutional networks. Scientometrics, New York, v. 124, n. 3, p. 1907-1922, 2020. Disponível em: https://doi.org/10.1007/s11192-020-03561-y. Acesso: 20 mar. 2026.

JI, Taoran et al. Citation forecasting with multi-context attention-aided dependency modeling. ACM Transactions on Knowledge Discovery from Data, New York, v. 18, n. 6, p. 1-23, 2024. Disponível em: https://doi.org/10.1145/3649140. Acesso: 20 mar. 2026.

KOHAVI, Ron. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, 14., 1995, Montreal. Proceedings [...]. San Francisco: Morgan Kaufmann, 1995. p. 1137-1143.

KOYEJO, Oluwasanmi et al. Consistent binary classification with generalized performance metrics. In: ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS, 27., 2014, Montreal. Proceedings [...]. New York: Curran Associates, 2014.

MA, Anqi; LIU, Yu; XU, Xiujuan; DONG, Tao. A deep-learning based citation count prediction model with paper metadata semantic features. Scientometrics, New York, v. 126, n. 8, p. 6803-6823, 2021. Disponível em: https://doi.org/10.1007/s11192-021-04033-7. Acesso: 20 mar. 2026.

NØRGAARD, Birgitte; LIE, Karen E.; LUND, Hans. Predictors of citation rates and the problem of citation bias: a scoping review. Journal of Clinical Epidemiology, Amsterdam, v. 190, p. 1-13, 2026. Disponível em: https://doi.org/10.1016/j.jclinepi.2025.112057. Acesso: 20 mar. 2026.

NEWMAN, Mark E. J. Prediction of highly cited papers. Europhysics Letters, Mulhouse, v. 105, n. 2, p. 28002, 2014. Disponível em: https://doi.org/10.1209/0295-5075/105/28002. Acesso: 20 mar. 2026.

SILVA, Maurício Coelho; WENDT, Lucas George; MOURA, Ana Maria Mielniczuk; ARAUJO, Ronaldo Ferreira. O sistema de recompensa científico na ótica da Ciência Aberta: dimensões de avaliação, características e desafios. Transinformação, Campinas, v. 36, p. 1-18, 2024. Disponível em: https://doi.org/10.1590/2318-0889202436e2410680. Acesso em: 20 de março de 2026.

STEGEHUIS, Clara; LITVAK, Nelly; WALTMAN, Ludo. Predicting the long-term citation impact of recent publications. Journal of informetrics, Amsterdam, v. 9, n. 3, p. 642-657, 2015. Disponível em: https://doi.org/10.1016/j.joi.2015.06.005. Acesso: 20 mar. 2026.

TEH, Phoey Lee; UWASOMBA, Chukwudi Festus. Impact of large language models on scholarly publication titles and abstracts: a comparative analysis. Journal of Social Computing, Beijing, v. 5, n. 2, p. 105-121, 2024. Disponível em: https://doi.org/10.23919/JSC.2024.0011. Acesso: 20 mar. 2026.

VASWANI, Ashish et al. Attention is all you need. In: NEURAL INFORMATION PROCESSING SYSTEMS, 30., 2017, Long Beach. Proceedings [...]. New York: Curran Associates, 2017. p. 5998-6008.

VERGOULIS, Thanasis. et al. Simplifying impact prediction for scientific articles. In: EDBT/ICDT 2021 Joint Conference Workshops, 2021, Nicosia. Proceedings [...]. Aachen: CEUR-WS.org, 2021.

VITAL JR., Adilson et al. Predicting citation impact of research papers using GPT and other text embeddings. Physica A: Statistical Mechanics and its Applications, Amsterdam, v. 674, p. 130789, 2025. Disponível em: https://doi.org/10.1016/j.physa.2025.130789. Acesso: 20 mar. 2026.

WANG, Dashun; SONG, Chaoming; BARABÁSI, Albert-László. Quantifying long-term scientific impact. Science, Washington, v. 342, n. 6154, p. 127-132, 2013. Disponível em: https://doi.org/10.1126/science.1237825. Acesso: 20 mar. 2026.

ZHANG, Fang; WU, Shengli. Predicting citation impact of academic papers across research areas using multiple models and early citations. Scientometrics, New York, v. 129, p. 4137-4166, 2024. Disponível em: https://doi.org/10.1007/s11192-024-05086-0. Acesso: 20 mar. 2026.