Common use of Evaluation Methodology Clause in Contracts

Evaluation Methodology. The evaluation methodology to be used in the project will consist of an expert-based evaluation, a user-based evaluation in a controlled environment, and a user-based evaluation at the elderly home. The methodology will further provide a combination of recognized qualitative and quantitative usability analysis methods to report the findings, covering the project’s pre-trials as well as the project’s final trial. Qualitative analysis components such as user personal comments and expert observations will be used. For the quantitative analysis of the system, questionnaires which will be filled in by the end users as well as their caregivers were constructed (see Appendix A and Appendix B). The pre-trials questionnaires are simpler, as certain features of the complete system will not be possible to asses due to their prototype nature. However, the trial questionnaires, along with automatically gathered measurements will provide a full picture for every indicator mentioned in section 4.2. Furthermore, a selection questionnaire (see Appendix C) will be used to ensure that the end users sample participating in the trials will be representative of the general target audience of the system. The constructed questionnaires incorporate elements of standardized and validated questionnaires adapted to our system. In detail, the trials and pre-trial questionnaires comprise of questions adapted from the System Usability Scale (SUS) and User Success Rate (USR) [1] [2], which are widely used to assess the usability of a system. Parts of Social Presence questionnaires were used in order to measure the realism and the engagement involving the avatar system [3] [4] as well as the Perception of the Personality of the avatar by the user [5]. Furthermore, questions aiming to gauge the Quality of Life of the end user were adapted and included from the Q-LES-Q-SF (Quality of Life Enjoyment and Satisfaction Questionnaire – Short Form) questionnaire. In order to access the indicators of objective 5, the ▇▇▇▇▇ ▇▇▇▇▇▇ Interview (ZBI) questionnaire, which aims in assessing the reduction of the burden of care of the caregivers, was adopted (mainly for the trial questionnaire).

Evaluation Methodology. The evaluation methodology to be used in the project will consist of an expert-based evaluationevaluation (during pre-trials), a user-based evaluation in a controlled environmentenvironment (during pre-trials), and a user-based evaluation at the elderly homehome (trial phase). The methodology will further provide a combination of recognized qualitative and quantitative usability analysis methods to report the findings, covering the project’s pre-trials as well as the project’s final trial. Qualitative analysis components such as user personal comments in the form of structured questionnaires and focus groups as well as expert observations observations, will be used. For the quantitative analysis of the system, questionnaires which will be filled in by the end users as well as their caregivers were constructed (see Appendix A and Appendix B). The pre-pre- trials questionnaires are simpler, as certain features of the complete system will not be possible to asses due to their prototype nature. However, the trial questionnaires, along with automatically gathered measurements will provide a full picture for every indicator mentioned in section 4.24.4. Furthermore, a selection questionnaire (see Appendix C) will be used to ensure that the end users sample participating in the trials will be representative of the general target audience of the system. The constructed questionnaires incorporate elements of standardized and validated questionnaires adapted to our system. In detail, the trials and pre-trial questionnaires comprise of questions adapted from the System Usability Scale (SUS) and User Success Rate (USR) [1] [2], which are widely used to assess the usability of a system. Parts of Social Presence questionnaires were used in order to measure the realism and the engagement involving the avatar system [3] [4] as well as the Perception of the Personality of the avatar by the user [5]. Furthermore, questions aiming WHOQL-BREF [17] questionnaire is used to gauge measure the Quality of Life of the end user were adapted and included from the Q-LES-Q-SF (Quality of Life Enjoyment and Satisfaction Questionnaire – Short Form) questionnaireelder. In order to access the indicators of objective 5, the ▇▇▇▇▇ ▇▇▇▇▇▇ Interview (ZBI) questionnaire, which aims in assessing the reduction of the burden of care of the caregivers, was adopted (mainly for adapted in order to build the questionnaire “PART G – Care Demand” in the trial questionnaire. Also the Groningen Activity Restriction Scale (GARS) was included in the trial questionnaire which is a non-disease-specific instrument to measure disability in activities of daily living (ADL) and instrumental activities of daily living (IADL). It was developed in studies of Dutch samples consisting of elderly or chronically ill people. The psychometric properties of the GARS demonstrated in these studies were highly satisfactory [15]. The Almere model [16] of technology acceptance which is specifically developed to test the acceptance of assistive social agents by elderly users and widely tested by ▇▇▇▇▇▇▇ et all was also included in the trial questionnaire. In order to assess the effectiveness of the avatar as a tool in terms of its appearance and the users’ perception in interacting with it, the Godspeed questionnaire was adopted [14]. The Godspeed questionnaire is a validated measurement of the perception of the user in interacting with a robotic or a virtual character. Godspeed consists of five parts that assess i) anthropomorphism, ii) animacy, iii) likeability, iv) perceived intelligence and v) perceived safety. In the context of Miraculous-Life this tool allows us to examine any problems with the behavior and/or the appearance of the avatar and how these affect the interaction of our users with it. 1) Slightly agree, (+2) Tend to agree and (+3) Strongly agree, are used. Part F of the Trials’ questionnaire uses a Likert-type scale from -2 to 2 ((-2) Very Poor, (-1) Poor, (0) Fair, (+1) Good, (+ 2) Very Good) while for Part G a Likert-type scale in the range of 0 to 4 is used ((0) Never, (1) Rarely, (2) Sometimes, (3) Quite Frequently, (4) Nearly Always). Finally, Part F of the Pre-Trials’ and Part H of the Trials’ questionnaire gauges the moral aspects of the system. In order to evaluate the filled in questionnaires, each question is associated with specific objectives and indicators, allowing the extraction of a numeric score for each one of them (see Section 4.4). The numeric score for each indicator will be provided by summing up the score of the associated questions and then normalizing these sums as a percentage. For indicators where there are automatic measurements, the normalized scores will be factored in along with the (also normalized) measurements in order to produce an overall score. The pre-trial scores will serve as a pointer to which aspects of the system need to be refined and reworked while the trial scores will be used to evaluate the whole system. Specifically the trial questionnaires will be filled in by the end users at the early stages of the trials and once more at the end of the trials allowing the comparison and gauging of the improvement on the system’s objectives. The overall evaluation approach of Miraculous-Life includes:  Project’s pre-trials (month 8 and month 16): 1) Expert-based evaluation phase (see section 4.1.1) 2) User-based evaluation phase in a controlled environment (see section 4.1.2)  Project’s trial (month 26 – month 32): 1) User-based evaluation phase at home (see section 4.1.3) which is associated to the final system evaluation.