Trials
The evaluation procedure for the project involved trials in Spain and Greece, taking place in different times and using different versions of the system. Forty users were involved in total; 19 in Spain and 21 in Greece. In Spain, users were recruited from the groups of older people that Ingema works with. In Greece, users were recruited from the elderly friendship clubs that AEDA has dirrect access to through the Municipality of Athens. The PeerAssist system was prepared and complemented with various options and features, most of these options being from deliverable D2.2. (PeerAssist use scenarios definition) and from deliverable D2.3. (Definition of global requirements, trials and demonstration), in which the user’s preferences and needs were collected.
In Spain, the methodology for the trials, was developed so as to take place in pairs. More specifically, there was a user and one evaluator in one room and another user with the another evaluator in a different room. Once users arrived to the lab where the trials were conducted, they were instructed to use the system by following a concrete route designed by the PeerAssist consortium. During the evaluations, the users were guided by an assisting person that was physically present to make them feel more comfortable during the trials. After the first trials in Spain, a second date was arranged for the next week in order to let them test the Peerassist system again and report any differences in their perception of the Peerassist system between the two trials.
The trials in Greece followed those in Spain, taking advantage of the expertise and overall evaluation obtained there, and using an improved user interface that was designed to address the users' comments from the Spanish trials. The evaluation methodology in Greece included both pairs of users that tried the system at AEDA premises, but also users that were communicating through PeerAssist from their houses, experiencing the same set of scenarios from their own comfortable environment, to allow more realistic experience. Some photographs of the trials are included below.
In both sites, the following features of PeerAssist system were evaluated:
| Feature | Description |
| Usability | How easy and pleasant are these features to use. This is a quality attribute that assesses how easy user interfaces are to use. The word "usability" also refers to methods for improving ease-of-use during the design process. It is also the functional evaluation of the capacities of the system: usefulness related to the designed capabilities of the system, the new alternative capabilities the user can be benefited from and the indirect synergistic relations between the user and the system interaction. |
| Acceptability | The response related to the acceptance, tolerance and how much a user likes the system he/she tries. It includes personal likeness, trustworthiness reliability etc. Two factors influence the decision: perceived usefulness and perceived ease-of-use. |
| Accessibility | This is the degree to which a product, device, service, or environment is available to as many people as possible. It describes its ease of reach, use and understanding. In terms of user experience design it can also be related to the overall comprehensibility of the information and features presented to a user. It contributes to shorten the learning curve attached with the system. |
| Time to complete the tasks | The time a user spends to complete specific tasks. Naturally, this influences the other features above. |
The study was carried out in 9 steps:
1. Introduction and informed consent
2. Sociodemographic data
3. Scenario 1 “User profile” – After-Scenario Questionnaire (ASQ)
4. Scenario 2 “Social profile” – After-Scenario Questionnaire (ASQ)
5. Scenario 3 “Search service” – After-Scenario Questionnaire (ASQ)
6. Scenario 4 “Caregivers” – After-Scenario Questionnaire (ASQ)
7. Scenario 5 “Notifications” – After-Scenario Questionnaire (ASQ)
8. Post-Study System Usability Questionnaire (PSSUQ)
9. Closing
Regarding the user profile, initially the inclusion criteria to participate in the evaluation was, older adults of 60+ years old, without cognitive impairment, in good physical and mental conditions, living independently and being active. On the other hand, taking into account the essence of the PeerAssist project, it was not necessary that the users have been familiarized with the new technologies.
| User profile | Older adult (+60 years old) Without cognitive impairment Good physically Good mental condictions Living independently Active life style |
The users carried out some tasks, which were developed based on the scenarios of deliverable D2.2. (PeerAssist use scenarios definition) and explained in detail in deliverable D6.1 (Evaluation design report). During the evaluation, each user was accompanied by one assistant. The main functions of that assistant was:
- Introduce details about the platform.
- Explain the consent form and possible doubts or questions.
- Help if they are not able to perform the tasks after trying to do it.
- Observe users behavior.
- Pass questionnaires.
Regarding the collecting data, there are different methodologies available to proceed. One of these classifications is depending on the degree of interaction between the users and the assistants. According to this classification, there is a division of three categories: observation only, interacting with the user and method supplements. It is very important to select the method that will best address the goals of our study, as well as resources available to conduct the evaluations.
In our case, due to the necessity of interaction between the user and the assistant, we chose the mixed option of observation and ad-hoc interview with validated questionnaires. One of the main advantages of this mixed methodology is the opportunity to see the user interacting with the system and in some concrete moments to ask a question regarding her/his execution.
All the users followed the same route in order to be able to interact with other participants. The route can be seen below:
CONCLUSIONS
As the users were people from 60 to 82 years old and not all of them very skillful using a computer, results have been lower than if we had been working with more experienced users. As an example, it has to be mentioned that many users did not know terms such as blog, chat, video call, etc. Because of that, some of them needed the help of the assistants to carry out the different tasks: complete the profile, choose the options, etc. However, some users carried out the different tasks without much help, including the users that took the Peerassist terminals at home and run the trials from there.
Many of the users reported that, with a previous training and more hands on experience, they could use the Peerassist system without any difficulty. Less experienced users said that they prefer the phone when they have to communicate with just one person, but if they had to communicate the same message to many people, they would choose the group chat or the video call. Some of these users said that they would rather prefer the video call compared to the chat, because they are not used to write with a keyboard or have difficulties to communicate by writing, and they don't want to expose this difficulty to others. All of the users highly appreciated the emergency notifications to the caregivers as a way to get immediate help in case of need, and also the availability to have access to external services.
The differences between the Greek and Spanish results were primarily due to the improved user interface and the relative better familiarity of the Greek users with the computers. Especially in Spain, less experienced users needed the help of the assistants in order to accomplish the scenarios.
Finally,despite the fact that the users liked the idea of the system, they would hardly pay for buy it at this stage. Based on this fact, exploitation plans are currently targeting other sources of revenue, such as advertisements, external services, public funding, etc.
To conclude, the evaluation outcome was very promising. For a trial version, the users appreciated its usefulness and would really wait to see it in real use. Despite the small time of training, they were able to explore all its capabilities and use it relatively easy. Nevertheless, for large-scale use, special attention should be paid to the training needs of the users, especially non-experienced ones. The weaknesses identified can all be easily covered with moderate effort from the consortium.