Human Gesture Recognition using CNNs for Human-Robot Interaction

Werikson Frederiko de Oliveira Alves; Michel Melo da Silva; Alexandre Santos Brandão

doi:10.22456/2175-2745.150950

Authors

Werikson Frederiko de Oliveira Alves Universidade Federal de Viçosa (UFV) https://orcid.org/0000-0002-3444-2849
Michel Melo da Silva Universidade Federal de Viçosa (UFV) https://orcid.org/0000-0002-2499-9619
Alexandre Santos Brandão Universidade Federal de Viçosa (UFV) https://orcid.org/0000-0001-5990-2218

DOI:

https://doi.org/10.22456/2175-2745.150950

Keywords:

gesture recognition, convolutional neural networks, transfer learning, human-robot interaction

Abstract

This paper proposes a deep learning approach for human gesture recognition to support real-time teleoperation of mobile robots in outdoor environments. A lightweight ResNet18 architecture is adapted via transfer learning, combining a personalized dataset captured with a RealSense camera and a reorganized subset of HMDB. A modular pipeline was developed and evaluated under nine experimental configurations, considering different optimizers, datasets, and backbone freezing levels. Results demonstrate that models trained with combined data and adaptive fine-tuning strategies achieve high accuracy and strong generalization under varied lighting and background conditions. The best configuration reached 97.9% test accuracy, reinforcing the potential of CNNs for robust gesture-based human–robot interaction.

Downloads

Download data is not yet available.

References

[1] MOYSIADIS, V. et al. An integrated real-time hand gesture recognition framework for human–robot interaction in agriculture. Applied Sciences, MDPI, v. 12, n. 16, p. 8160, 2022.

[2] LIU, C.; SZIRÁNYI, T. Gesture recognition for uav-based rescue operation based on deep learning. In: Improve. [S.l.: s.n.], 2021. p. 180–187.

[3] GUNASUNDARI, S. et al. Gesture controlled drone swarm system for violence detection using machine learning for women safety. In: SPRINGER. Int. Conf. on Intelligent Sustainable Systems. [S.l.], 2023. p. 221–236.

[4] PATRONA, F.; MADEMLIS, I.; PITAS, I. An overview of hand gesture languages for autonomous uav handling. 2021 Aerial Robotic Systems Physically Interacting with the Environment, IEEE, p. 1–7, 2021.

[5] SZELISKI, R. Computer vision: algorithms and applications. [S.l.]: Springer Nature, 2022.

[6] BASILIO, V. T.; CARVALHO, K.; BRANDÃO, A. Reconhecimento de ações por RNA em aplicações de robótica social. XIV Simpósio Brasileiro de Automação Inteligente, Ouro Preto/MG, 2019.

[7] RAY, P.; REDDY, S. S.; BANERJEE, T. Various dimension reduction techniques for high dimensional data analysis: a review. Artificial Intelligence Review, Springer, v. 54, n. 5, p. 3473–3515, 2021.

[8] GOODFELLOW, I. et al. Deep learning. [S.l.]: MIT press Cambridge, 2016. v. 1.

[9] HE, K. et al. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [S.l.: s.n.], 2016.

[10] ZHANG, A. et al. Dive into deep learning. [S.l.]: Cambridge University Press, 2023.

[11] POURBAHRAMI, S. et al. A survey of neighborhood construction algorithms for clustering and classifying data points. Computer Science Review, v. 38, p. 100315, 2020. ISSN 1574-0137.

[12] CARVALHO, K. B. de; BASÍLIO, V. T.; BRANDÃO, A. S. Action recognition for educational proposals applying concepts of social assistive robotics. Cognitive Systems Research, Elsevier, v. 71, p. 1–8, 2022.

[13] JEERU, S. et al. Depth camera based dataset of hand gestures. Data in Brief, v. 45, p. 108659, 2022. ISSN 2352-3409.

[14] ALVES, W. F. d. O.; MOREIRA, K. S.; BRANDÃO, A. S. Interação com o ambiente mediante classificação por distância euclidiana. In: Congresso Brasileiro de Automática. [S.l.: s.n.], 2022. v. 3, n. 1.

[15] SHEENA, C. V.; NARAYANAN, N. K. Static gesture classification and recognition using hog feature parameters and k-nn and svm-based machine learning algorithms. In: BAJPAI, M. K.; SINGH, K. K.; GIAKOS, G. (Ed.). Machine Vision and Augmented Intelligence—Theory and Applications. Singapore: Springer Singapore, 2021. p. 157–166. ISBN 978-981-16-5078-9.

[16] CARVALHO, K. B. de et al. Gestures-teleoperation of a heterogeneous multi-robot system. The Int. Journal of Advanced Manufacturing Technology, Springer, v. 118, n. 5, p. 1999–2015, 2022.

[17] REDDY, G. T. et al. Analysis of dimensionality reduction techniques on big data. IEEE Access, v. 8, p. 54776–54788, 2020.

[18] AYESHA, S.; HANIF, M. K.; TALIB, R. Overview and comparative study of dimensionality reduction techniques for high dimensional data. Information Fusion, v. 59, p. 44–58, 2020. ISSN 1566-2535.

[19] YOSINSKI, J. et al. How transferable are features in deep neural networks? Advances in neural information processing systems, v. 27, 2014.

[20] HUDSON, T. M. et al. Gesture and emotion recognition system for autonomous vehicle control. In: 2025 Brazilian Conference on Robotics (CROS). [S.l.: s.n.], 2025. v. 1, p. 1–5.

[21] ALVES, W. F. d. O.; BARCELOS, C. O.; BRANDÃO, A. S. A gesture recognition system for robot guidance applications. In: SOCIEDADE BRASILEIRA DE AUTOMÁTICA (SBA). Congresso Brasileiro de Automática (CBA). 2024. p. 1–6. Disponível em: https://www.sba.org.br/cba2024/papers/paper3971.pdf.

[22] KUEHNE, H. et al. HMDB: a large video database for human motion recognition. In: Proceedings of the International Conference on Computer Vision (ICCV). [S.l.: s.n.], 2011.