Virtual Keyboard-Mouse in Real-Time using Hand Gesture and Voice Assistant

Document Type : Original Article

Authors

1 Higher Institute of Computers and Information Technology, Computer Depart., El. Shorouk Academy, Cairo, Egypt

2 Department of Mathematics, Faculty of science, Al-Azhar University, Cairo, Egypt

Abstract

The advancement of computer applications has increasingly facilitated everyday tasks, with recent innovations focusing on voice assistants and virtual input devices. This technology is particularly beneficial for individuals with mobility challenges or in situations where direct manual computer interaction is limited. Leveraging computer vision and artificial intelligence, these applications can interpret visual data, such as human movements, and make decisions to execute corresponding commands. Our desktop application incorporates a voice assistant, virtual mouse, and virtual keyboard to enhance accessibility and usability, especially for individuals with physical disabilities or those who prefer alternative input methods. Using Python, MediaPipe, and OpenCV, the application effectively processes and interprets user gestures, providing a responsive and efficient computing experience. MediaPipe’s features specifically contribute to the model’s precision, optimizing hand tracking and gesture recognition for AI-driven tasks. Users can control the computer cursor through various hand gestures, type on a virtual keyboard using colored caps or tapes, and perform essential actions like left-clicking and dragging items. This integrated solution aims to improve productivity, making computers more accessible and enhancing the overall digital experience for users. The fusion of AI and computer vision in such applications continues to drive forward innovative and inclusive computing solutions, promising a future of greater accessibility and convenience in human-computer interaction.

Keywords

Main Subjects

Files

Share

How to cite

Statistics