A Luminous Pen for Air-Writing: Hybrid Approach with Computer Vision and Vision–Language Models

Abstract

Air writing is a form of human–computer interaction that enables users to write in the air through different technological approaches, such as radio waves, wearable sensors, dedicated devices, and computer vision. In this work, we present a hybrid solution that combines the use of a luminous-tip pen with computer vision techniques to support pen detection and the air-writing process. The pen was designed and fabricated with a 3D printer and integrates an ESP32 microcontroller board, which provides Bluetooth connectivity for real-time communication with the computer. In addition to tracking movements for writing, the device can transmit specific commands to the system, such as starting or stopping the writing process and changing colors. This approach bridges the gap between wearable devices and vision-based methods, offering a practical and accessible alternative for air-writing applications. The system integrates vision–language models capable of recognizing both words and images. To determine which models were best suited for our system, we evaluated the performance of several candidates. Five participants contributed to the creation of the dataset. For drawing recognition, each participant produced images in the following classes: Tree, Moon, Cat, Heart, and Pen. For word recognition, the dataset included the following eight words: Purple, Window, Jungle, Pillow, Team, Doctor, Words, and Science. The results showed that the best-performing models achieved 76.00% accuracy with Kosmos-2 for image detection and 97.39% accuracy with Gemini 2.5 Flash for word detection, demonstrating satisfactory outcomes for the system implementation. The proposed system demonstrates the potential of combining electronic devices and advanced machine learning techniques for air-writing, with applications in education and assistive technologies.