Enhancing creativity and play through accessible projector-based interactive PC-control touch technology

Miriam E. N. Begnum

MediaLT, Jerikoveien 22, N-1067 Oslo, Norway

Kyrre M. Begnum

Faculty of Engineering, Oslo University College, P.O.Box4 St. Olavs plass, N-0130 Oslo, Norway

Johan Finstadsveen

MediaLT, Jerikoveien 22, N-1067 Oslo, Norway


Abstract: Standard computer peripherals are often challenging to use for mobility impaired, and in particular controlling computer mouse movements efficiently and without risking strain injuries. Projected touch screen technology show promise in this respect. This paper presents a Projected Interactive PC-control pilot (PIP) solution for computer interaction. The paper focus on testing and improving the usability of the PIP solution through iterative user tests with mobility impaired children. It describes prototype improvements aimed at fulfilling the requirements of users with reduced motor skills, and discusses challenges and key findings from the extensive usability tests. Our results demonstrates that interactive touch based solutions may enable heavily impaired children to independently partake in creative activities and play, and points to the value of creating a touch based computer interaction solution tailored to the needs of this user group.

1. Introduction

Computer technology is central to the modern societies, and continuous use of technology is becoming a prevalent part of everyday life. For disabled people, technology may be of great value - compensating and providing alternate ways of doing things. At the same time, standard technologies may be difficult to use, and system adjustments along with alternative peripherals are often needed.

Most standard peripherals today are designed to avoid strain injuries for non-disabled computer users, and not to lessen the impact of symptoms such as impaired body movement, muscle stiffness and weakness, spasticity and pain. Ergonomically designed equipment exists - typically computer mice and keyboards. Computer mice are often optimized towards the gaming industry, providing alternative sensitivities and weights but using standard designs. Computer interaction peripherals designed to accommodate specific users are available as assistive technologies. Examples are keyboards with fewer keys, larger keys or enhanced key label visibility, key guards, foot or head operated mice and software for enhancing mouse control (medGadget 2005; IBM 2007).

However, through the project Parkinson's IT challenges (PIKT), it became evident that few existing alternative peripherals show the innovative potential of projected touch screen technology for motor disabled users; a projected interactive PC-management (PIP) solution indicated touch technologies may provide a beneficial alternative interaction technique for people with mobility impairments (Begnum 2010).

Mobility impaired is a user group with many sub-populations, and thus various challenges and demands towards technical solutions. A number of disabilities fall within the category of mobility impairment, ranging from reduced stamina to complete paralysis. These disabilities may affect body movements and muscle coordination, give various degrees of paralysis, stiffness and/or spasticity, as well as loss of muscle strength. The focus of the PIKT project were on users with Parkinson's disease, with mild to moderate mobility disabilities. This paper examines the usefulness of the technology for children with different types of significant and severe motor disabilities.

Existing interactive touch technologies are not tailored for motor disabled users. Our aim was to investigate the usefulness, as well as further the development, of an affordable, easy-to-use PIP solution tailored to user group needs, offering an efficient and effective way to interact with a computer without unnecessary fatigue, pain and strain.

2. Background

Creativity and play is an important part of child development (Lemus 2008). They are fun and social activities in which friendships and social groups are built, even helping to reduce stress (Ginsburg 2007). Many researches argue creativity and play is important for brain growth, motor control and general child development socially, emotionally and cognitively. For example, there seems to be strong evidence that play can serve literacy by: (a) providing settings that promote literacy activity, skills, and strategies; (b) serving as a language experience that can build connections between oral and written modes of expression; and (c) providing opportunities to teach and learn literacy (Roskos 2001), and Frost (Frost 1998) discuss the role of play, recess, games and art in cognitive, language, social, emotional, and physical development.

Disabled children have the same desire to play and use their creativity as non-disabled, but not equal opportunities. Many activities, such as drawing a picture, can be inhibited due to poor motor control. Technological solutions could provide disabled children with the means to overcome some of these obstacles (Brænde 2003). For example, games can be played at the computer and art software exists which is specialized for children, like Tux-paint. However, when using a standard keyboard/mouse interface mobility impaired children face the same accessibility issues as for regular computer usage. The challenges are particularly great when it comes to tasks which require a high degree of accuracy motor coordination skill, such as for example drawing.

The PIP solution consist of a point-and-click device with an infrared LED, a projector, a Bluetooth enabled computer and Wii-mote controller. The projector, acting as the computer monitor, projects the screen on any chosen surface, typically downwards from a high position onto a table in front of the user. With a raised projector, a large screen surface is projected, reducing the need for detailed motor coordination skills as the surface area reaches dimensions such as 30" across. A Nintendo Wii-mote controller is mounted next to the projector, pointing its infrared camera towards the same surface as the projected screen.

The control and clicking device consists of a simple circuit and is attached to the index finger. A button at the bottom closes the circuit every time the user "touches" a surface. At the top, over the nail, sits the LED, emitting light when the circuit is closed. The Wii-mote controller above the table is connected via Bluetooth to the computer, registering every time the iFinger's button is pressed, i.e. when someone with the iFinger touches the screen on the projected surface. The position of the click is communicated to the computer, initiating a pointer movement or a click.

Utilizing the potential of this affordable technology was showcased by Johnny Chung Lee, as he was working at the Carnegie Mellon University. He posted videos of the low-cost touch screens functionality on YouTube which has been watched by millions of people and was elected best educational video in 2007 (Lee 2008).

The control and clicking device was improved since first tested by PC users with PD in the PIKT project. Instead of a glove solution, a finger attached pointing device was created (iFinger). Our iFinger material is stretchy to fit different finger sizes and the design allow for switching from finger to finger including the left handed. Low fidelity clicking device prototypes was tested on students from the Oslo University College (HiO), and general flaws and errors detected. The battery was changed to a smaller variant, a click-button was added to increase affordance and LED type, placement and position was optimized. These changes resulted in prototype "A". Prior to this study some challenges regarding inaccurate aiming/positioning of the infrared light from the diode was detected in prototype A, and as a result, prototype B was built.

3. Usability tests

Focus was on testing usability and benefit of the touch based interactive technology for different types and severities of motor disability, and to optimize the technical improvements of the iFinger unit accordingly. Tests were thus conducted iteratively, improving the technology based on detected user needs prior to each new test iteration, aiming to increase usability and benefit with each iteration.

The decision was made to use children as testers. They represent a realistic test audience for the here-and-now prototype evaluation - and not focused on future potential. In addition, children seem more open to new interaction ideas, adapting to new techniques more quickly than adults.

Further, the choice was made to test the device for play: drawing and gaming. Research on virtual reality and children with Cerebral Palsy shows creative games (such as drawing and music programs) in particular increase feeling of coping, contentment and motivation (Reid 2004; Lager 2009). The idea was that a play test scenario would make it easier to recruit testers, motivate the testers, make longer tests possible (less tiring when inspiring) as well as test the solution more thoroughly than for office use. Also, drawing requires fine motor skills, thus challenging and testing pointer control. When gaming, the pointing/control device is stressed with regards to the rapid clicking and movements made. The number of interaction events per minute when gaming is also far greater than for routine desktop procedures.

Tux Paint was selected as the drawing program. Tux Paint (New Breed Software 2010) is designed for children 3-12 years of age, has a user friendly and intuitive design and amusing sound effects.

The figure shows a screen dump of Tux Paint, with large buttons and user friendly design.

Figure 1. Tux Paint Screen shot

Minabc.no is an online gaming portal with entertaining colorful flash-based games designed for children of all ages. The user receives visual and/or auditive feedback for every action taken. Most games have the possibility for tailored degree of difficulty, including the size of clickable elements. Many games also have motivational factors included, such as an image becoming gradually more visible as tasks are completed. 5 different games from this portal were used in our tests: "Find pairs" (either identical animals or one animal and the related letter), "Count the fish" (fish swimming on screen), "Figure Memory", "Drag and drop puzzle" and "Connect-the-dots numerical drawing".

Screen shots top left and top middle: Find pairs, top right corner: Count the fish, bottom left corner: Figure Memory, bottom middle: Drag and drop puzzle, bottom right corner: Connect the dots.

Figure 2. Screen shots of the 5 games tested

Seven children with various severe mobility disabilities were selected by a Norwegian school for students with neurological disabilities and/or multiple disabilities as testers, ranging from 6 to 15 years of age. Table 1 presents the seven testers. Prior to tests, we visited the children at their school to get to know them and establish communication with testers and participating teachers, map out physical needs regarding game and iFinger adaptations and explain test set ups. All tests in the test series also took place at the school. Each tester had a teacher at his/her side during tests, aiding verbal and non-verbal communication with the facilitator. In addition there were two observers placed at a distance. Tests were non-structured, reflecting play. A test journal was used for each tester, suggesting scenarios to complete and logging each test. Observers were provided with copies in which resulting data was noted. In addition, hypothesis for following tests were registered, such as tester specific challenges and assumptions on prototype improvements.

In total, the PIP solution was iteratively tested through an iterative test series by 7 children with various severe mobility disabilities - a total of 17 different tests conducted in 6 iterations:

Table 1. Mobility disabled child testers
Test subject Description
"Barney" Skin disease inflicts pains, slow growth and maturing, learning disability, some involuntary movements and reduced fine motor coordination, reduced muscle strength, is able to walk.
"Phoebe" Significantly slowed psychological and physical development, weakened and slowed movements, significantly reduced fine motor coordination in both arms, large involuntary arm movements particularly in right arm, reduced concentration.
"Rose" Reduced motor control and coordination, weak musculature and significantly reduced fine motor coordination, communication difficulties.
"Sarah" Whole body affected by CP, severe motor disability, able to grip and release with right hand. Socially and cognitively very well versed, some communication difficulties (roll-talk/bliss user).
"Pam" Left handed, reduced wrist mobility, reduced extension of wrist and finger joints, involuntary hand contraction (closing), some communication difficulties.
"Joey" Acquired brain damage, hand contraction, frequent and large involuntary movements, reduced precision and motor control.
"Hillary" Degenerative muscle disease, weakened and diminishing muscle strength. Cognitively and socially very well versed.

In addition to the iterative tests, one of the child testers (with Cerebral Palsy) and her teacher were subsequently followed through a 7 weeks period of independent use at the school. at the school. The main goal of this was to observe how teacher and student would utilize the solution in the education, and if and how an external institution could use the PIP solution without expert knowledge.

4. Results

Prototype B drawing test: Barney, Rose and Phoebe

Barney managed to draw a circle following a circle shape, and demonstrated pointer control and click precision. He drew freely and enjoyed playing using the PIP solution. Rose familiarized herself with the PIP solution, but was very withdrawn. Phoebe did not have the strength to use the click button on prototype B, and tested with prototype A instead. She found drawing extremely fun, and drew painting upon painting, learning even advanced features of Tux Paint. She had some interaction difficulties linked to button insensitivity and prototype A inaccuracy.

The added click-button in prototype B was not sensitive enough for the testers, due to reduced muscle strength. Prototype C was modified using two buttons of smaller size, together forming a larger and more sensitive clickable area. The new button design was also more robust, placed in between two surfaces. More iFinger size diversity were needed.

Prototype C drawing test: Sarah, Pam and Joey

Physical test surroundings did not fit Sarah, who had challenges using the set-up table. In addition, her finger positioning was inexpedient, resulting in jagged drawing and low click responsiveness. Sarah had no difficulties understanding the software or the PIP solution, and thus overcame the interaction difficulties - for example repeating clicks until registered. She very much enjoyed using the geometrical figures, and especially liked drawing red hearts. Pam mastered both pointer precision and the click technique after some practice. Pam very much enjoyed painting, and so did Joey. Joey had some trouble understanding the software, but was an eager tester and artist.

Clicking techniques amongst the children made it expedient to further increase the clickable area. This was done in prototype D. Some calibration issues were also discovered due to a loosely fastened Wii control.

Prototype D drawing test: Barney, Rose and Phoebe

As button sensitivity was increased, Barney now had better click response. He also drew with a standard computer mouse for a part of the test to compare against the iFinger. The iFinger produced significantly improved precision. Barney now had no interaction difficulties aside from some random involuntary twitches affecting click registration. Rose had some cognitive challenges using Tux Paint, and needed assistance changing colors etc. She had some interaction difficulties related to precision, but found the drawing test fun. The iFinger was still not fitting Phoebe well, resulting in difficulties reaching the click area. Still, click responsiveness and sensitivity was notably improved, making drawing more exciting and fun. iFinger robustness was tested, with luck, as Phoebe experienced some large, speeded spasm where she banged the iFinger violently against the table.

Prototype D now used two larger sized buttons, reducing the resistance of pressure from 1.6 nm in previous prototypes to 1.3 nm, but ease of clicking still needed some improvement. The design had weak affordance, as several testers tried clicking with their thumb on the iFinger button. This was improved in prototype E. Steel edges on the buttons were painful, and were removed.

Prototype E drawing test: Sarah, Pam and Hillary

The final prototype was built in two versions, using two different battery technologies (SR44 in addition to the LR1). The iFinger surface facing towards the projected screen was enlarged in prototype E, increasing ease of clicking, but reducing click precision. Even so, Sarah's precision was improved compared to prototype C. Both she and Pam enjoyed drawing, although still having some interaction challenges. Hillary tried iFinger for the first time, instantly displaying good click and pointer precision, and even managed handwriting. She controlled the software with ease, changing colors and modes, opening new sheets etc. No interaction difficulties were identified.

Prototype E game test: Barney and Joey

Barney tested the games "Find pairs", "Drag and drop puzzle", "Count the fish" and "Figure Memory" with great success. Joey tested "Find pairs" and "Figure Memory". Joey was at times disturbed by violent involuntary movements, but with practice, both click technique and precision was managed.

Prototype E game test: Phoebe, Pam and Sarah

Phoebe tested the game "Find pairs", but even on the easiest level, this game was too complicated cognitively for her. She interacted seemingly without any purpose, and eventually the test were ended. Pam and Sarah also tested "Find pairs". A couple of games were to challenging cognitively for Pam, but she managed to complete the easy and medium versions of "Find pairs". Even though her precision was good, the position of her hand was not optimal, and she did not have the accuracy skills needed for "Count the fish". Sarah quickly selected matching figures and letters, herself increasing game difficulty to the maximum. Sarah had to practice interaction and precision skills for a while to master the games "Drag and drop puzzle" and "Count the fish". Her memory was exercised during "Figure Memory". For Sarah, the gaming test was a huge success.

Long-term institutional usage

Sarah was selected to try out the solution over a longer period of time after the tests series were completed. In collaboration with the school board and the art teacher, the solution was set up in the art education classroom. Even though mounted on lockable wheels, and thus movable, Sarah mainly used the solution during art lessons. The IT responsible at the school was given a quick introduction to the set up and use of the PIP-solution.

Feedback from the school after the long-term test was that Sarah had on several occasions used the PIP solution for the entire class (45 minutes) without fatigue and continued to enjoy using it. The school managed to independently utilize and maintain the technology, moving and calibrating it when needed.

5. Discussion

The child testers had various severe and complex motor disabilities, affecting both movements, fine motor skills and cognitive understanding. Still, all testers were able to master using the PIP-solution, where as not all master standard computer equipment. All testers spontaneous started drawing using the iFinger when placed at the test table, finding the testing fun and exciting, and even desired that we should come back and test some more. They often arrived smiling, remembering how to use the PIP solution and looking forward to the test. During briefings, they emphasized the positive feelings of being able to create something and "to draw on the table, like the other kids".

The PIKT project showed mouse clicking and pointer control were challenging users with reduced motor skills, in particular retaining mouse pointer control while initiating a click, and largely unsolved. The PIP solution however seemed to support both areas simultaneously. Our tests show the PIP interaction solution provide children not able to draw with a pencil on paper the possibility of mastering unassisted freehand drawing. 

Users could lean in on the projected surface to support their movements, reducing fatigue and strain, stabilizing the pointer while steering and steadying hand and fingers while clicking. However, since testers often had limited motion range, projecting a larger screen to enlarge interactive elements was not always helpful. Troubles during the tests were mainly related to technical prototype weaknesses and not using the iFinger in the correct manner. The PIP solution was iteratively and incrementally improved to better suit the needs of people with reduced motor skills. Enlarged clickable area, increased sensitivity and improved affordance in the final prototype solved most of the interaction difficulties. Further aspects needing attention are:

Teachers and school personnel continued giving positive feedback long after the study were completed and the long-term test scenario demonstrated how the PIP solution can be utilized by a school institution, independently managing the solution without needing extensive training or technical skills. This shows that the solution is accessible both financially as technically. Our tests show the iFinger device is treated very roughly when used. We envisage an institution/user having multiple iFingers, simply instantly replacing one if broken.

Creating a touch based computer interaction solution for users with reduced motor skills seem both beneficial and possible, although not trivial. Our tests emphasized the importance of a high level of personal adjustment possibilities for the user group. The PIP pilot offer choice of screen surface (floor, wall, table...) and size, projected height and direction, and elasticity and size of pointer unit. The possibilities for personal adjustment should however be furthered, especially with regards to the pointing device; possibilities for individual preference on arm/hand device mounting (on hand, one or several fingers, easy to take on/off...), strategy for pointing and clicking, click sensitivity and size of clickable area. In addition, specific additions should be considered, e.g. users with involuntary hand contractions probably need help keeping the hand open.

It would be interesting to test the PIP-solution in a more multi-modal interaction environment, e.g. alongside a keyboard, for general PC control and usage. If iFinger precision is further improved, the solution may be a highly interesting interaction system for more advanced gaming - possibly giving an aim advantage and a more realistic feel.

6. Conclusion

This paper demonstrates how a projected interactive touch screen technology is promising for users with significant mobility impairments, providing pointer and click control literary at your finger tips. Prototypes were iteratively and incrementally improved through a series of 17 tests on 7 different children of ages 6-15 years with various mobility impairments. Our tests illustrate how the touch based interactive technology have potential as a compensator and enabler for the challenges facing people with motor disabilities. It empowered motor disabled children, facilitating them to independently part-take in games and creative arts they could not exercise with standard computer peripherals and providing them with the means to creatively express themselves through drawing. The possibilities of individual adaptation proved a vital solution attribute in relation to the targeted use group, e.g. with regards to mouse clicking sensitivity.

7. Acknowledgements

The authors would like to thank all those who contributed to this study. In particular, we would like to acknowledge the contributions of Petter Skolla and Jonas Taftø Rødfoss through their bachelor project work.

8. References