UCL MI3 Facial Navigation

 

Overview on MI3

The Facial Navigation app website and help documentation can be found here on https://www.facenav.org

UCL MotionInput 3 ("MI3") is a suite of several touchless computing apps. The apps enables users to control the mouse and keyboard of a Windows 10 and 11 computer or laptop via a webcam and microphone in various ways.

MI3 Facial Navigation is one of those apps, intended for accessibility use as an Assistive Technology by tracking a users nose and combining speech recognition with facial gestures to simulate mouse clicks and keyboard presses.

For a full description on the other apps, see our parent website at www.touchlesscomputing.org

 

On some computers, we have found that the Microsoft Visual C++ Redistributable is not installed. If no launch window appears, please try installing this version for X64 Windows computer from here: https://docs.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170 

 

About the Facial Navigation App

Moving your face, with your nose as a centred axis, is similar to how a joystick would operate. Holding your face in a direction away from a centre position, moves the mouse in that direction.  Facial gestures and movements, such as a smile, opening and closing of the mouth, and raising one’s eyebrows, are used to trigger computer mouse actions. Users can alternatively use speech commands for the same actions instead. The user can also dictate in English straight into any text editing software, such as Microsoft Word documents, Outlook emails, browser pages and PowerPoint. 

This creates a joystick around the user’s nose to control the mouse, with the centering of the nose as the rest position. Users can move the mouse up, down, left or right, by moving their head with their neck and body. The blue box that the user will see is called the area of interest (AOI). Once the user’s nose is inside that box, it will evaluate the nose from the start-up position of where the user is. 

In speech modes, resetting the mouse’s starting point with an anchor to the user’s nose at their current position can be achieved by clearly saying out loud the word "BUTTERFLY". This is the most important keyword! It centres the joystick around the user’s nose to control the mouse, with the centre of the nose as the resting position. It also stops the mouse from moving if it is. This can be done whenever the user moves in their seat and they would like to reset the resting position of the mouse to their nose.

A mouse click on an icon or click in a game/app, can be done by saying "click", "double click" and "right click". Useful when playing games requiring arrow key presses, is saying the words “up”, “down”, “left”, and “right”.

Click and drag (left mouse button and holding it down) can be done by saying "hold left" and "release left". The same applies to the right mouse click, "hold right" and "release right".

There are four different modes that the application could be used with. These are:

1. Nose-Centred Mouse Navigation with Speech Commands for the mouse buttons

2. Nose-Centred Mouse Navigation with Facial Switches for the mouse buttons

3. Eye Gaze Directions for Mouse Navigation with Speech Commands for the mouse buttons (Experimental)

4. Eye Gaze Directions for Mouse Navigation with Facial Switches for the mouse buttons (Experimental)

 

Instructions on how to use the Facial Navigation app

Once it has been installed, the app can be found in the Start Menu under the folder “UCL MotionInput v3” with the icon labelled “UCL MI3 Facial Navigation”. Running this program will show a settings launch screen as follows. 

Here the mode of use is selected along with a number of other preferences. Once these are selected, the [Launch] button will close the window and start the application. The keyword “BUTTERFLY” is an important key word to remember in the speech modes. It will let you centre the nose joystick to wherever you have moved your position to. So don’t feel that you have to lean into a direction to move the mouse, just say “butterfly” to reset the centre of the joystick. Saying “butterfly” also stops the mouse from moving when you are not physically moving. Press the “ESC” key on the keyboard to stop the tracking at any time.

Whenever the other settings need to be readjusted the comma [,] key on the device’s keyboard can be pressed to reopen the settings window. Alternatively, using the speech command ‘settings’ will do the same, as will going back into the Start Menu and running the program again. 

There are six settings sections each of which contains a group of user options. These are General, Facial Mode Options, Eyes Mode Options, Nose Mode Options, Camera and Advanced. 

1. General 

This section is for the specification of the Tracking Mode and Method the user wishes to use the application. The Tracking Mode refers to the user preference of whether they would like the computer mouse cursor to track and respond to the user’s facial movement, focusing on the nose position and movement direction, or the direction of their eye gaze. Nose Mode requires some neck mobility as it is needed when pointing the nose up, down, left, and right. The Method option is for the selection of the triggers of mouse and keyboard actions. These could be either via Speech commands or via Facial Gestures. 

2. Facial Mode Options

This section is available during Nose Mode with Facial Gestures Method and Eyes Mode with Facial Gestures. It allows the user to select which mouse action they would like each facial gesture to correspond to. Currently, the available options are Smile, Open Mouth, Fish Face, Raise Eyebrows, Tilt Head Left, Tilt Head Left, Turn Head Left and Turn Head Right. 

Each of these could be mapped to a mouse Left Click, Right Click, Double Click, Left Hold/Release, and Right Hold/Release. During Eyes Mode, an additional option for On/Off Eye Gaze is available to start and stop the mouse cursor movement.

When the tick box for the Calibrate Tilt/Turn option is selected, the application will run a calibration at Launch. This allows the user to make the range of motion, needed to trigger an action, comfortable to their needs. 

1. Eyes Mode Options 

Currently, there is a single option specific to this section which dictates the mouse movement speed across the screen. The lower the number selected, the slower the mouse movement speed. 

You may find Eye Gaze tracking working better for users without wearing glasses; however, it works with clear glasses as well. 

2. Nose Mode Options

This section also includes a mouse speed setting, similar to the one in Eyes Mode Options. However, it also includes a Distance setting. The Distance option is only applied during Nose Mode with Speech Method. A lower number would set the movement response of the mouse cursor to be triggered much easier. A larger number would slow that response and to move the cursor would require a bigger movement. 

3. Camera

The first Camera option is for the selection of a default camera video source. This setting may need to be adjusted when the computer device has two or more webcams available. If the application is not able to automatically select the correct camera the user wishes to use, then a different one may be specified here. Most commonly, changing Camera 1 to Camera 2 is likely to be a solution. 

The Low Light indicator is set to ‘OFF’ by default. If the application is used in a place with low lighting, changing this to ‘ON’ will ensure improved recognition of visual control commands. 

4. Advanced

Currently, there is a single Advanced option which enables and disables the FPS (Frames Per Second) number. FPS is the rate of frames (pictures) produced every second. The higher the number is, the smoother and better the interaction with the device will be. By default, this number is displayed, at the bottom right corner of the camera window. Changing this option to ‘OFF’ will hide this number. 

Modes and Methods Options

1. Nose Navigation with Facial Gestures

This setup also allows two speech commands which are ‘settings’ to open the control panel window and ‘butterfly’ to reposition the joystick location. ‘Joystick’  refers to the blue lined square surrounding the position center of the nose tip. Below are the Action options for each Facial Gesture available. 

2. Nose Navigation with Speech Commands

There is a number of available speech commands that can be used. The table below lists the trigger words and phrases along with their corresponding computer actions. The speech commands available to use with Speech capabilities for Nose and Eyes Navigation are very similar. 

The General category includes the words ‘settings’ and ‘help’ which open the settings control panel and a user help file. The Joystick adjustments category includes speech commands which help to tailor the mouse movement to the user preference. Mouse actions are the available button clicks the user could use. Transcription, or also called Dictation, category describes the commands which activate/deactive live capturing and typing of speech. The last two categories, common keys/hotkeys and windows operations, cover useful combinations of keyboard keys. 

 

3. Eyes Navigation with Facial Gestures - this feature is experimental and is still under development for improvements and optimisations.

‘Settings’ speech command is available. 

 

Examples of using the speech key phrases

1. Say "click", "double click" and "right click" for those mouse buttons.
2. In any editable text field, including in Word, Outlook, Teams etc; say "transcribe". Speak with short English sentences and pause, and it should appear. To end transcribe mode, say "transcribe" again to turn it off.
3. In any browser or office app, "page up" and "page down".
4. In powerpoint you can say "next", "previous", "show fullscreen" and "escape".
5. In a keyboard driven PC game where you can set keyboard mappings, you could set up to say up, "down", "left", "right", "enter"
6. For users who are learning to program and code, the software can also be used with Visual Studio and has a number of shortcuts to make programming more accessible. A description of this can be found at: https://hrussellzfac023.github.io/VisualNav/Controls.html