Efficient Text-Entry in Mixed Reality: Tap, Gaze & Pinch, SwEYEpe (CHI 2025 Late-Breaking Work)

Duration: Sep 2023 – Present
Role: Project Lead, Lead Software Engineer, Experimenter
Advisor: Dr. Steven K. Feiner
Status: Preparing for CHI 2025 Late-Breaking Work

Overview:
Sweyepe reimagines text-entry in mixed reality (MR) environments by utilizing gaze paths instead of traditional finger swipes on a keyboard. The system combines various input modalities, including tapping, gaze, pinching, and swiping, to create an intuitive and seamless text-entry experience. Users can enter text by simply looking at letters in a specific sequence, mimicking the “swiping” motion commonly used on mobile keyboards, but using eye movements.

The system employs a fixation detection algorithm to filter the user’s gaze path, ensuring that only intentional movements are considered. It then matches the user’s gaze path with an ideal one using a language model to further refine and predict the most likely word candidates. This approach leverages natural eye movement patterns to allow efficient and hands-free text input in MR, significantly enhancing usability and interaction.

Key Features:

• Multi-Modal Interaction: Combines various input modalities to offer natural text-entry experiences in MR.
• User-Centric Design: Focuses on usability and efficiency, ensuring that text-entry methods are intuitive for users.
• Evaluation: Conducted user studies to assess the effectiveness of the proposed methods in improving MR interaction.

My Contributions:

• Foundation Development: Built the project’s foundation, including both the front-end and back-end systems.
• Gesture and Eye Tracking System: Invented the gesture and eye-tracking based input system, enabling intuitive text entry through gaze path tracking.
• System Design: Implemented the fixation detection algorithm to filter gaze paths and developed the mechanism for matching ideal gaze paths with user gaze paths.

Significance:
Sweyepe aims to enhance user interaction in mixed reality by introducing more intuitive text-entry solutions. By combining various input modalities and refining user interactions through language modeling, the project seeks to create a more user-friendly and efficient MR environment.

Additional Notes:
Detailed insights into this project can be found in my Master’s thesis. The paper is currently being prepared for submission as a CHI2025 late-breaking work.