As a Research Prototyper at Meta Reality Labs, I collaborated with Adwait Sharma, Frances Lai, Tovi Grossman, and Stephanie Sentosa to develop the Grasp UI design space. Our GraspUI Research Paper was then accepted and was presented at DIS ’24, the ACM SIGCHI Conference on Designing Interactive Systems. Be sure to check out the video below for an overview of the work!
The everyday objects we hold in our hands afford several opportunities for microtasks. For example, a person might activate a dial-like control while holding a cup of coffee by using a thumb rotation gesture to adjust the volume of a music player, or tap the side with their index finger to toggle playback. Existing research however, primarily focused on situations where an object is already being firmly grasped.
Although existing object-centric gestures tend to focus on the user holding an object, visiting Reality Labs Research Scientist Adwait Sharma identified an opportunity to explore microtasks across the seven different phases of grasping:
- Reach: The initial phase of the grasp where the hand begins moving towards the object before any contact occurs with the object.
- Load: Occurs when the digits of the hand contact the object and ends when the object begins to lift off the surface it is resting on.
- Lift: Occurs as the object is being raised and ends when the object reaches an intended goal height.
- Hold: Occurs while the object is firmly grasped.
- Replace: Occurs when the object is being lowered and ends when the object is almost touching the surface it will rest on. This is the opposite of the Lift phase.
- Unload: Occurs when the object is supported by the surface and the task involving the object concludes. The digits of the hand begin to break contact with the object. This is the opposite of the Load phase.
- Depart: The termination phase of the grasp when the hand moves away from the object and ceases to have any contact with the object. This is the opposite of the Reach phase.
To explore the potential of these phases for microtasks, our team conducted eight ideation sessions with two groups of expert mixed reality researchers and practitioners. After identifying common applications and objects, these activities envisioned in the sessions were used as exemplars to generate ideas for interaction. In collaboration with the design team, Adwait and I developed a set of storyboards that build upon the concepts in the sessions, highlighting exemplar interactions for each phase. This involved sketching various concepts, as we ideated back and forth, before landing on the storyboards below.
I continued to support the project by helping debug, test, run, and record footage of an interactive GraspUI prototype built for the Hololens 2 with Unity and Python. The prototype (below) utilized the Hololens’ built-in hand tracker to precisely track hand positions in Unity and 3 classification models: an Object tracker, a Static hand pose tracker, and a Dynamic hand pose tracker implemented with MediaPipe.
Using the GraspUI storyboards and prototype, an empirical evaluation was conducted with end users to assess the user experience of the GraspUI gestures and to measure the overhead cost of performing the associated microtasks. Overall, participants reacted positively to the proposed gestures, with findings suggesting that performing microtasks wasn’t very demanding and that participants liked the idea of performing these microtasks. Performing microtasks during the grasp phases also resulted in a minimal duration overhead of 0.76 seconds, when compared to the baseline condition of performing these activities without any microtask. For the paper, I transformed the data into this data visualization below to communicate the results with the error bars depicting 95% confidence intervals.
For more details about our results and the implications they have for gesture design, be sure to check out our full GraspUI paper publication.
