Assessing the implementation of user-centred design standards on assistive technology for persons with visual impairments: a systematic review

Despite scientific and technological advances in the field of assistive technology (AT) for people with visual impairment (VI), technological designs are frequently based on a poor understanding of the physical and social context of use, resulting in devices that are less than optimal for their intended beneficiaries. To resolve this situation, user-centred approaches in the development process of AT have been widely adopted in recent years. However, there is a lack of systematization on the application of this approach. This systematic review registered in PROSPERO (CRD42022307466), assesses the application of the ISO 9241-210 human-centred design principles in allegedly “user-centred designed” AT developments for persons with VI (see Supplementary PROSPERO Protocol). The results point to a wide variation of the depth of understanding of user needs, a poor characterization of the application of the User Centred Design (UCD) approach in the initial design phases or in the early prototyping, and a vague description of user feedback and device iteration. Among the principles set out in ISO 9241-210, the application of 5.6: “the design team includes multidisciplinary skills and perspectives” is the one for which the least evidence is found. The results show there is not enough evidence to fully assess the impact of UCD in (1) promoting innovation regarding AT products and practices, and (2) Judging if AT produced following such standards is leading to better user access, wellbeing outcomes and satisfaction. To address this gap it is necessary to, first, generate better implementation of UCD in AT development and second, to strengthen evidence regarding the implementation and outcomes of using UCD for AT. To better engage with the realities of persons with VI, we propose capacity building across development teams regarding UCD, its principles and components; better planning for UCD implementation; and cross-fertilization across engineering disciplines and social and clinical science.

Systematic Review Registration: https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=307466 PROSPERO (CRD42022307466).

Introduction

User-Centred Design (UCD) has gained a stronger presence in Assistive Technology (AT) development over the last decade (1). This approach promotes the involvement of end users in all stages of the design process, elicitation and understanding of their needs, and characterization of social contexts as the basis for an iterative design process (2, 3). Therefore, UCD adoption is believed to lead to better products (4). However, there is limited evidence regarding the implementation of these approaches or if their results are having the intended impact across their target populations, particularly regarding AT (5, 6). This study aims to assess the application of ISO 9241-210 human-centred design principles in the allegedly “user-centred designed” assistive technology developments for persons with Visual Impairments (VI).

The Global Report on Assistive Technology (GREAT) states that children and adults with disabilities lack access to AT, particularly in low-and-middle-income Countries (LMICs) where access was reported to be as low as 3% (7). The current lack of access to AT reflects not only an economic gap but a severe malfunction of social provision and coverage schemes as well as in AT design and development (8). Nevertheless, UCD and international standards' adoption can help to alleviate these shortfalls by guiding the development of better and more efficient AT solutions responding to the users' priorities. Disability is very diverse and persons with different impairments, namely sensorial, physical or cognitive or multiples, benefit from different technological solutions; we need to learn more about similarities, as well as differences. Therefore, in this paper, the focus is on AT for persons with VI. Worldwide, there are approximately 39 million people with severe VI or blindness (9). Although not all-disabling loss of sight can be addressed by AT, for persons who are blind (visual acuity worse than 3/60) some tools such as walking canes, screen readers, or braille embossers, amongst others, are of great help. The Global report on disability calls for action and standard setting in a variety of AT related fields, particularly regarding access (7). Thus, investigating how internationally adopted standards are implemented for technology design is relevant to close the AT gap. The upcoming section explores relevant international standards for the production of AT.

AT and international standards

International standards play a key role in the development, production and distribution of technology (10). The existence of clear, accessible, and commonly accepted International Standards is vital for the manufacture of products that can be globally implemented and commercialized. Standardization enhances product quality, safety, and reliability, it also allows for higher interoperability and compatibility in different contexts and reduces maintenance complexity and costs (11). There are different studies on the positive effect of international standardization on trade, industry and management (10, 12), including evidence of the reduction of barriers to the export and acceptance of products between different global regions, including products exported from LMICs to high income Countries (13). Given the current AT access gap and lack of evidence on how available technology responds to the needs of persons with VI in LMIC it is relevant to look at how and whether the adoption of these standards can lead to better and more efficient AT. Furthermore, infrastructure for AT production and a well-defined value chain have an impact on AT access, nonetheless, this is outside the scope of this paper.

The standard of user-centred design

UCD is recognized by the International Organization for Standardization (ISO) in their standard ISO 9241-210, where it is described as an “approach to system design and development that aims to make interactive systems more usable by focusing on system use and applying human factors/ergonomics and usability knowledge and techniques” (3). The standard presents a framework giving examples of activities that can be developed when adopting the approach. Furthermore, it clarifies that UCD is complementary to existing design methodologies, for example regarding usability (14) and Measurement of quality in use ISO/IEC 25022, amongst others. UCD is guided by the following 6 principles: (I) the design is based upon an explicit understanding of users, tasks and environments; (II) users are involved throughout design and development; (III) the design is driven and refined by user-centred evaluation; (IV) the process is iterative; (V) the design addresses the whole user experience; and (VI) the design team includes multidisciplinary skills and perspectives.

There is very narrow empirical evidence on the impact of standards on innovation, particularly regarding AT (15). However, forthcoming empirical literature shows a positive influence of standards on the diffusion of technical knowledge and their contribution to macroeconomic growth. For example, a set of studies performed within different countries showed that the contribution of standards to the growth rate in each of the evaluated countries was equivalent to “0.9% in Germany, 0.8% in France and Australia, 0.3% in the UK and 0.2% in Canada” (12). Another set of studies, performed by the ISO in several companies from different sectors in ten countries, showed an overall increase between 0.5% to 4% in the companies’ annual sales revenues provided by the implementation of international standards (13, 16, 17).

The adoption of a user-centred design approach in the development process of AT has increased in recent years. This systematic review assesses the application of the ISO 9241-210 Human-centred design principles in the “user-centred designed” AT developments for persons with VI. The goal is to better understand how systematically the approach has been applied in the design and development of AT.

Method

The present review followed the Prisma guidelines for systematic reviews seeking to answer the next question (Supplementary PRISMA Checklist) (18).

Does the existing literature provide sufficient evidence on how developments of user-centred designed assistive technology for persons with visual impairments apply the human-centred design principles of the ISO 9241-210?

The multidisciplinary databases Science Direct, Scopus, PubMed Central and Web of Science, were defined as primary sources. The electronic searches were performed in January 2022 and updated on June 2022. The keywords visual impairments (blindness and low vision), user-centred design, and assistive technology were used as search terms. The search period was established between January 2012 and March 2022. Considering that standards take time to become known and applied, a gap of two years was left between the publication of ISO 9241-210 (2010) and the start date of the search (2012). In any case, the application of the previous standard (ISO 13407) was considered during the revision. The complete Web of Science search strategy, was adapted for the other databases:

Search string: [(“visual+ impair+” OR “visual+  disab+” OR blind OR “low vision”) AND (“user-centred design” OR “human-centred design” OR “ISO 9241-210” NOT “universal design”) AND (assistive technology)]

Eligibility criteria

Inclusion criteria

• Topic of study: papers are describing the design and/or development process of user centred designed assistive technology for visually impaired persons.

• Type of scientific material to analyse: Peer-reviewed journals: Quantitative, qualitative, and mixed methods empirical studies. Except for systematic reviews and meta analyses, letters and editorials.

• Studies available in English/Spanish/Portuguese/French.

• Full text available.

• Full conference papers.

Exclusion criteria

• Articles that are not exclusively addressed to persons with VI.

• Articles describing assistive technology design or developments addressed for persons with VI without any consideration to the UCD approach.

Study selection

All search results were imported into an EndNote database. Duplicates were removed. Abstracts and titles that were noticeably unrelated to the review topic were dismissed. Two researchers independently screened the titles and abstracts against the eligibility criteria and selected those that met the inclusion criteria. Full-text reports were retrieved and again assessed for final eligibility. Reasons for excluding full-text reports were documented. The selected studies were analysed with a standardised data extraction form. Inter-rater agreement was 87.12%, Disagreements during the selection process were discussed in a consensus meeting with a third reviewer who helped to solve the discrepancies.

Data extraction

To meet the aim of the study, all data related to the application of each of the six principles of ISO 9241-210 was extracted and grouped under each of the principles for further analysis.

The following data on the study characteristics were also extracted for contextual purposes:

• Data about the publication (authors, title of the article and the journal), aims, methods, design approaches (usability testing, workshops, interviews, focus groups, think-aloud, observation, including others.), frameworks, and studies design.

• Data about the designs or developments: areas covered (according to the ISO 9999:2016).

• Data about participants: sample size, socio-demographic characteristics, inclusion and exclusion criteria, type of impairments (low vision or blindness).

• Setting: country.

Data quality assessment

The methodological quality of the selected studies was assessed using the Mixed Methods Appraisal Tool (MMAT) (19). This tool was selected for its applicability in evaluating qualitative, quantitative, and mixed methods empirical studies. Studies were assessed over five components which vary according to the nature of the study. For the qualitative studies, the following criteria were assessed: The appropriateness of the qualitative approach (1), the data collection methods (2), and the data analysis (3) to answer the research question. In addition, that the resultś interpretation was supported by the data (4), and the coherence between all these parts of the study (5). Regarding the quantitative studies, the assessment addressed (1) the relevance of the sampling strategy according the research question. (2) The representativeness of the sample. (3) The validity, reliability and coherence of the measurements. (4) The minimization of the risk of nonresponse bias, and (5) the appropriateness of the statistical analysis. The items evaluated for mixed methods studies, were: (1) the pertinence for applying a mixed methods design to answer the research question. (2) The integration of the quantitative and qualitative components of the study. (3) The general interpretations resulting from consolidation of qualitative and quantitative results. (4) The presentation and explanation of dissonances between the components findings. (5) The quality criteria of each of the components of the study. The instrument encourages not to estimate an overall score to rate the quality of each study, but to analyse and discuss each criterion. It was implemented independently by two researchers to perform, and disagreements were resolved through discussion with a third researcher. Following the indications of the MMAT, studies were not excluded based on the methodological quality. Instead, the results of the evaluation are addressed in the discussion and conclusions. Two of the authors jointly (MRV and LMO) assessed the risk of bias and quality of the articles. Disagreements were resolved by consulting (MACh) to achieve consensus.

Data analysis and synthesis

All search results and their respective reasons for inclusion or exclusion were documented through The PRISMA flowchart. The qualitative evidence was analysed and synthesised using thematic synthesis approach guided by the Human-centred design approach described in the ISO 9241-210. The analysis and synthesis was carried out by two researchers. A third researcher assessed Inter rater reliability.

Protocol and registration

The protocol describing this systematic review methodology was previously registered in PROSPERO (CRD42022307466).

Results

The search in databases retrieved 348 references, After removing three papers that were duplicated, 310 papers were discarded through the abstract screening stage. The remaining 35 were subjected to a full-text review. Among them, four literature reviews and seven empirical studies were discarded for not meeting the inclusion criteria (see Figure 1).

FIGURE 1

Figure 1. Prisma flow diagram.

Overview of included studies

Themes covered in the literature and geo-representation

The results retrieved contributions from 16 countries. Regarding countries' distribution by income category, it should be noted that 87.5% (24) of the papers included are from high-income countries, four from middle-income, and none from low-income countries (Table 1). AT designs or developments covered the following areas (according to the ISO 9999:2016) (20) https://sciwheel.com/work/citation?ids=14235244&pre=&suf=&sa=0: activities and participation relating to personal mobility and transportation (ten papers); communication and information management (five papers); education and training in skills (six papers); work activities and participation in employment (one paper); and for assistive products for self-care activities and participation in self-care (two papers).

TABLE 1

Table 1. Number of studies per country.

Most of the papers (21/24) explicitly declare using/implementing the UCD framework of reference. However, the implemented framework, namely the ISO 9241-210 (or its previous version ISO 13407) was referenced in only five of the reviewed articles (21–25). Excluding the aforementioned studies, only three papers (22, 26, 27), cited a reference, other than ISO 9241-210, for UCD, specifically Cheverst et al. (28) and Nielsen (29).

Further characteristics of the studies included in the review are summarised in Table 2. These data contextualise the results of both the quality assessment of the studies (e.g., aims of the studies, data on population samples, data collection instruments, etc.) and the application of the principles of ISO 9241-210 (e.g., characteristics of participants and their involvement in the studies, among others), which are presented below.

TABLE 2

Table 2. Characteristics of the included studies.

Quality assessment

Table 3 sums up the quality appraisal of the included studies following the MMAT criteria. Three papers were classified as qualitative, 15 as quantitative, and six as mixed methods studies. Only four papers met all quality criteria; nine papers met between 3 and 4 criteria, and 11 papers met between 1 and 2 quality criteria. When comparing the papers according to their type or nature, the ones that showed the highest compliance with the quality criteria were the qualitative studies. As for the quantitative studies, the first two criteria related to sampling strategy and representativeness of the sample were the least fulfilled. In this type of studies, in which probability samples would be expected, it was common to find undelineated information on the target population, lack of information on the selection criteria, non-probability samples and small sample sizes. On the other hand, as far as data analysis is concerned, the description of instruments such as questionnaires, their methods of analysis and the full presentation of their results are missing. In the mixed methods studies, the description of the application of the UCD approach took precedence over the definition of the mixed nature of the study. The methods of data analysis were clearly presented in the quantitative component of all the mixed methods studies, in the qualitative component only in one study.

TABLE 3

Table 3. Quality assessment.

Adoption of UCD principles

The aim of this systematic review is to examine and describe the application (or absence of application) of the ISO 9241-210’s principles in AT developments for persons with VI, based on searches in multidisciplinary databases. Here, we present an in-depth analysis of findings pertaining to the development and adoption of the UCD principles.

The available evidence shows that papers documenting the development of AT for persons with VI tend to present a more detailed description of the state of the art in terms of the systems requirements than a proper characterization of the context of use or preferences and needs of the target user. Also, there is a strong emphasis on usability-oriented studies (64.29%). AT developments tend to engage users mostly at the end of the process to test if the product can be used. More than one third (35.71%) of the articles did exactly that. While 21.4% presented usability evaluation and results as part of the user-centred design process. Another 21.4% aimed to apply the user-centred design process in the early stages of the design or development process, looking at the feasibility of using the device and highlighting the compatibility and advantages of using participatory design methodologies with the UCD approach.

Table 4 succinctly outlines the results of the application of the principles of ISO 9241-210 in the papers included in this review. The following sections expand on the findings of the detailed peer review on the application of this standard, grouping them under each of the six principles.

TABLE 4

Table 4. Data extraction from reviewed papers on the application of the principles of ISO 9241-210 in assistive technology design or development processes.

The design is based upon an explicit understanding of users, tasks, and environments

In reviewing the application of this principle in the available evidence, we sought compliance with the following points: identification of user and stakeholder groups, understanding of users’ needs and description of the context of use: “specified users, having specified goals, performing specified tasks”.

All the reviewed articles reported the participation of VI users, even though six studies complemented their samples with non-VI participants. Regarding sample sizes, the number of participants varies from two to 128 (55 VI, 71 non-VI), being 12 participants the number that appears the most often. Only seven articles reported sample sizes with more than 12 users. There is no clear rationale for why and how these samples were designed and selected, especially when considering the quantitative methodologies. Only one paper, Nimmolrat et al. reported the sampling strategy and inclusion criteria for a sample of 60 participants (26).

Quantitative standards regarding sampling were not observed either. Mascetti et al, reported difficulties in recruiting test subjects with VI or blindness (30). Under that argument, the paper added non-VI participants to the study and reported on results that merge data from both non-VI individuals and VI individuals. Najjar et al., whose sample consisted of 10 non-VI and seven VI participants, noted their limitations without being specific or addressing bias on the data analysis (31). Connors et al. acknowledged that their sample size (7 blind adolescents) was “relatively small” and limited to carry out a correlation-based analysis (32).

When qualitative methods were applied, no standard sampling techniques nor quality assurance practices for qualitative sampling were reported, e.g., characterization of patterns, and variations among the participants, data saturation. Studies such as Aziz et al., argued that the sample size (8) was sufficient considering its qualitative nature, without providing any further rationale (33). Furthermore, Conradie et al. claimed that two focus groups (sample sizes 9 and 12) “served to reveal the experiences and knowledge of blind persons” to the researchers which make it possible to sketch broad user needs within the target group and specifying varying degrees of mobility needs and assistive device demands (21).

In terms of characterising users, participants' details were poorly described. The sex of the sample members was reported by 19 papers, the same number of articles stated the age of the participants, four of the studies were addressed to minors (8–17); two other papers reported participation of teenagers and young adults; the rest of the studies included adults only in different age ranges (18–78). Other types of data reported were: the participant's skills related to the use of the designed AT (14 papers); the education level of the participants (10 papers), their occupation (five papers), the number of years lived with visual disability or the year when the disability was acquired (five papers), and the use of aids (two papers). Environment of use was often not mentioned. Only one paper reported an analysis of the physical environment in which the product will be used, the user's social and organisational milieu and the technical environment and associated technical constraints (23).

As for stakeholder identification, 8 papers mentioned the involvement of stakeholders. Four papers declared the participation of academics (principals, teachers and O&M trainers), three studies included representatives from disabled people's organisations, and four studies included technology experts (22–25, 34–37). Bateman et al., included all the above (34).

Users are involved throughout the design and development

All of the reviewed articles reported user involvement, but rarely throughout all of the stages of the design process. Four of the studies present data from the design phases in which participants were actively involved (21, 24, 35, 37). Their involvement included “in-depth requirements analysis” of users and stakeholders' feedback through a series of UCD comprehensive methods. Mattheiss, et al. first centred on analysing the requirements in the areas of Orientation and Mobility (O&M) training and accessible video game play to later work on the first iterations of the design, implementation, and evaluation of the developed game editor (24). In this case, authors declared the involvement of children (end-users) as design partners.

User participation in the final stages, namely for evaluating the solutions, was stated by 5 studies: 4 in usability testing (33, 38–40) and 1 in field testing (41). Involvement, both in the design and testing phases, was reported by 14 studies (22, 25–27, 30–32, 34, 36, 42–46). Although, it is pertinent to point out that: on the one hand, some studies mention the involvement of the participants at the beginning and at the end of the process, but not in all the stages of the process. On the other hand, how users were involved tend to be unclear and reporting of such involvement tends to be rather superficial, for example, Najjar et al. mentioned the identification of potential users' requirements, but these are not presented in the paper (31), instead a previous study is referred to. Nimmolrat et al. provide a better description of users' participation during the design process (26). Ntakolia et al. detailed user's participation in the design and development phases, however, usability testing was done with blindfolded non-VI participants only (23).

The design is driven and refined by user-centred evaluation

The use of user-centred evaluation tends to be more explicit, explained and applied in studies that used qualitative methods, such as behavioural observation, think aloud techniques, in-depth interviews, and focus groups, among others. These kinds of evaluation methods allow the user's perspective to be addressed early (21–26, 33, 34, 37, 46). The analysis of the context of use could determine the user's needs against which the preliminary design solutions will be tested.

Usability evaluations reported on the evidence collected included both quantitative and qualitative methods. There is a stronger emphasis on quantitative scales to assess usability, such as the System Usability Scale. In addition, usability was assessed in terms of the performance of the technology and other quantitative variables related to efficiency (time) and effectiveness. Except for the studies by Najjar et al. and Giraud et.al, the studies which applied quantitative methods for usability testing, also reported userś feedback without specifying the methods used to gather that data (31, 40), e.g., Lopes et al., stated that subjects had the chance to use the device and were asked to give feedback, but did not describe the methods for data collection (41).

There is a lack of real-world scenarios when evaluating AT. Some studies claim that this was because they are focused on preliminary solutions, and in some others, because the study has pure research and non-commercial orientation (24, 27, 31, 40). Ntakolia et al., excluded users from its evaluation process of the prototype, reporting that future research would include VI users for the usability test (23).

The process is iterative

In reviewing compliance with iteration, which dictates the iterative repetition of a sequence of steps until the desired outcome is achieved, it is important to remember that not all the included articles report the complete UCD process, but some focus on design and several, as presented initially, are limited to the usability evaluation of prototypes. Thus, all the studies reported iterations, or the intention to make them, based on the feedback gathered from studies' participants. Iteration involves not only the prototype but also the descriptions and specifications, the refinement of information from the feedback obtained during the development process and in usability testing, is also considered. From this perspective, a noteworthy study on this subject is that of Bateman et al. where design and re-designs were submitted to preliminary tests with expert users (34). Finally, the usability test conducted with 12 students confirmed that the previously expressed needs regarding accessibility and effectiveness were met. The authors went beyond mentioning that an iterative UCD process was carried out, in fact, they went on to explain the information gathered and the stakeholder's characteristics through every round of interviews. The iterations made and the preliminary test results were also detailed. In other words, iterations were placed in the context of use.

Another interesting example is the study by Shi et al., where two studies were conducted to understand how to design effective, interactive 3D models for education purposes for blind students (36). In the first study, two design workshops were performed with teachers of VI students (TVIs) in which suggestions from conceptual designs were aggregated. Then, the second study was performed with three teachers of VI students, not only to design, but to deploy sample interactive 3D models over seven weeks. In-depth work with individual TVIs, and deployment of interactive 3D models in their classrooms were reported by the researchers, resulting in improvements to the prior system and mobile application development that supports the use of interactive 3D printed models in an educational setting. Additionally, the authors stated that based on the feedback from the second study, the mobile application could be further improved.

Although in less depth than the cases previously discussed, Conradie et al. and Mattheiss et al. highlighted the importance of rapid prototyping in the execution of iterations (21, 24). Adebiyi et al. and Feiz et al. emphasised the effectiveness of the “Wizard of Oz” technique in achieving development improvements (39, 46).This technique consists of a tactic used for low fidelity prototyping in which the participant receives instructions in order to perform tasks while testing a prototype, and a human simulates the behaviour of the completed AT. For example, for a navigation device, a person will simulate the task that the device will perform by providing vocal instructions to the users.

In turn, Doush and Pontelli reported “iterative modifications have been applied to the system based on empirical studies carried out with the participation of sighted and blind users”. However, they do not describe the iterations performed or how these studies were conducted, nor do they explain why non-visually impaired participants were involved (38). Likewise, Mascetti et. al, stated that “during the design of the auditory guiding modes several test subjects were asked to use the application and provide feedback” this was done via informal test (30).

The design addresses the whole user experience

ISO 9241 stresses that usability goes beyond “making products easy to use”, by considering perceptual and emotional aspects as keys to understanding the user's experience from their own perspective.

Still, several studies assessed usability mainly by considering parameters such as ease of use, accessibility, or satisfaction with the device (30, 31, 38–40, 42). These studies applied quantitative scales. To have information to improve the device, three papers reported to have included questionnaires or open-ended questions (not described in the papers) (30, 38, 42). In yet another case, in which only System Usability Scale (SUS) was applied, feedback from users was reported as results of “anecdotal comments” (39).

Mascetti et al., reported as a result of feedback from participants after evaluation of the prototype, that they did not desire to hold a mobile phone in one hand while holding a cane in the other (42). This type of information evidences that the characterisation of users' needs and preferences was not carried out at an early stage and therefore, users' previous experiences and perspective were not addressed.

Other feedback refers to the time the user needs to get familiar with the device, the need for more training time was expressed by the participants in the studies conducted by Doush and Pontelli and by Mascetti et al. (38, 42, 46). It was also stated by Najjar et al. (31). On the other hand, although Giraud et al., did not include feedback within the methods or results, they did express the future need of conducting semi-structured interviews “in order to collect the perceptions of users with blindness of such a tool (advantages, risks, opportunities)” (40).

Alternatively, preferences and expectations were mainly assessed in the studies of Mattheiss et al., and Aziz et al.: design for skills development in VI children through an interactive learning material and a videogame, respectively (24, 33).

Furthermore, Sánchez obtained feedback from users regarding their emotions (25). Colley et al., also considered affective state variables, namely “control over the situation” in the analysis (35). Nimmolrat et al., assessed satisfaction with the functionality of the application through interviews (26). Finally, eight studies (28.57%) based their development on the available literature and did not include collecting any empirical data.

The design team includes multidisciplinary skills and perspectives

The large majority of studies did not report multidisciplinary skills and perspectives. Only two studies described the research team. Facanha et al., stated that the design team included four undergraduate students in the computer sciences, an assistive technology researcher and an ophthalmologist (22). Shi et al., mentioned that one of the researchers of the team is an expert in education for students with visual impairments, and that they included an accessibility specialist (36). Nevertheless, some authors did report collaborations: Kammoun et al., declared the participation of different engineering research groups in human perception, human-computer interaction, audio and acoustic, and spatial cognition and perception. Further, the authors mentioned the project leader, is an interdisciplinary research group in Human Computer Interaction (43). Other reports of collaborations outside the engineering team are: Lopes et al., who mentioned a collaboration with the Department of Communication and Arts of Aveiro University (41) and Younis et al., who declared a research collaboration with the Department of Health Services Research in the UK (45).

Discussion

The literature reports a growing trend in the application of user-centred design in the development of assistive technology for the visually impaired persons (47). However, the results show that evidence on the effective implementation of UCD with VI on the design of AT is scarce. Publications show that the principles of the ISO 9241-210 (user-centred design) tended to be not fully applied, despite being called guiding principles and despite the increasing availability of models and frameworks that could facilitate their application (48).

The focus was on the system requirements, not its user

The information on the system architecture reported in the state of the art in the analysed studies, was prioritised over the participants' needs with respect to AT. For these papers, it was common not to find specifics on sample size calculation and participant's selection. Participation of potential users was low and was accompanied by a superficial description of their profiles. Users' involvement was reported mainly in usability assessment at the end of the process rather than in design phases, and when users’ involvement was reported in design phases, it was usually not thoroughly described. This contrasts with the extent in which technological aspects of the development were informed. Also, a stronger focus on the verification of the system, over its validation, was observed. According to quality management standards, such as the ISO 9001, independent validation and verification (V&V) processes need to be performed to determine if a developed system meets the defined requirements and specifications and fulfils its intended purpose (49, 50). Specifically, the verification process focuses on the system's requirements (“Did we developed the system right?”) while the validation process focuses on the system's worthiness, i.e., if it fulfils its intended purpose, user expectations, etc. (“Did we developed the right system?”) (51).

For usability assessment, most studies used surveys and quantitative methods to gather information. Though standard parameters of quality on those methods, such as rationale for power and limitation of the sample size calculation, were not met. Feedback from users, when present, took the form of “informal”, “casual” or “anecdotal” data. Moreover, in these studies iterations are often mentioned in the evaluation phase and not in the design phase. At this point, it is important to emphasise that according to the ISO 9241-210, iterations should be done throughout the process and not only at the evaluation stage.

Regarding usability, ISO 9241-210 states the need to go beyond the concepts of ease of use and effectiveness, and to incorporate userś experience. In this perspective, the standard recommends to consider the userś skills, habits and personal goals, as well as emotional aspects and experiences of previous solutions or alternatives. Notwithstanding, the studies under consideration fall short in assessing the whole users' experience, there was little or no information on the social and environmental context in which these devices were intended to be used. In this regard infrastructural constraints such as internet availability, road safety or social aspects like stigma are not accounted for.

Moving towards better understanding of the final users

Characterization of user needs was often unstructured, lacked robustness or tended to be underreported. This trend has been previously observed in the study of requirements elicitation techniques (48). This was also observed in the present review. Among the reasons given to justify such behaviour are limited resources, time and endeavour to conduct a thorough requirements assessment process (48). Similarly, some of the studies in this review reported major logistical challenges in recruiting participants.

There is a growing number of articles that seek to better engage with users of AT. This was generally achieved either because they took care to obtain larger samples under previously defined selection criteria or because they selected more appropriate methods (qualitative or mixed) with respect to the objective pursued, or due to both reasons (23–26, 43). It is also pertinent to highlight the importance of having included stakeholders in these studies (22–26, 36, 37, 43, 44).

Researches did not fully apply all ISO 9241-210's principles. However, it can be argued that a better compliance to the first principle (the design is based upon an explicit understanding of users, tasks, and environments) increased the probability of applying the subsequent four principles. The fact that some studies integrated participatory design approaches into the methodology boosted the involvement of participants in the whole process (22, 24, 26, 27, 36, 44). The participation of both potential users and stakeholders in the early design phases and throughout the process, as well as the type of instruments applied to collect information, allowed the design to be “driven and refined by user-centred evaluation”. Iterations were reported both in the information collected to guide the design, and in the prototypes.

Regarding usability, in addition to the application of validated surveys and the analysis of system performance parameters, qualitative methods were used to obtain feedback from users in a more systematic and deeper way, and to compare it with the initial information from the context of use. In some cases, the user experience was assessed in a more comprehensive way by considering the emotions evoked through interaction during the prototypes assessment (24, 33, 35).

Multidisciplinary skills and perspectives

So far, the application of five of the six principles of the ISO 9241-210 in the reviewed articles has been discussed. Regarding the application of the last principle “The design team includes multidisciplinary skills and perspectives” in the reviewed studies, this is where the least evidence was reported. Although the standard does not define the need for broad heterogeneity of the disciplines involved in the process, since it is designed to guide processes of different natures, it is understood that disciplines from diverse fields are needed to elicit and comprehend userś needs and to address the system's requirements. Multidisciplinary teams would not only allow dealing with the issues related to technology but also those that have to do with the users' functionality, and above all, it would facilitate the mixed methodological approach.

As for the literature, in addition to designers and engineers, it is proposed in specific cases to work with clinicians, health professionals or rehabilitation professionals and with Commercial specialists (47, 49).

From user centred to person centred

Literature addresses two streams of user-centred design, one in which the “user” is placed at the centre of the design process and another, which focuses on the “person” (52). The main difference lies in the fact that the first considers the interaction between the user and the product and “is concerned with ensuring that artifacts function as intended by the designers”. While the latter also accounts for context-determined interactions and focuses on “enabling many individual or cultural conceptions to unfold into uninterrupted interfaces with technology.” Giacomin et al., add that products acquire meaning when used by persons, and that it is the understanding of that meaning that should guide the design. In their words, “the natural focus of questions, insights and activities is on the people for whom the product, system or service is intended, rather than on the designer's personal creative process or on the material and technological substrates of the artefact” (52). We might say that most articles followed the first trend (user-centred), while there is less evidence of the second understanding (person-centred) when it comes to AT.

Limitations

Although this systematic review process was based on best practice in conducting systematic reviews (PRISMA), like any research, has its limitations. Some studies on the topic may have been missed, either because they were not found within the four selected databases or because there is a chance that some studies are not covered by the search string.

To reduce this risk, we selected multidisciplinary electronic databases relevant to the topics addressed. The key terms and search string were tested. In addition, as reported in the methodology, strategies to increase inter-rater reliability in the studies selection and data extraction were carried out.

Conclusion

This review explores how well the principles of ISO 9241-210 are applied in the case of AT. As for the implications, on the one hand, it highlights that the application of the UCD approach is not standardised in the field of AT design for the visually impaired. Although there is a standard that guides the implementation of the approach and has been thoroughly reviewed by ISO experts, it has not been embraced in this field. On the other hand, there is also a lack of methodological rigour in understanding the needs of users in their context, revealing that people are not at the centre of the process in a generalised manner.

These findings confirm the outcomes of the studies quality appraisal. The areas of concern in the quality of the studies include non-probabilistic and small samples in the quantitative studies. Further, lack of rigour in the analysis and description of some questionnaires and in the collection of feedback from users. Absence of description of the methodology of mixed methods studies and omission of the analysis of qualitative data and methods. These results serve as an input to understand the nature of the problem and to look for solutions that aim to improve the research processes and therefore the development of better products.

Furthermore, it is evident that the developments are carried out in a disarticulated manner, so that recommendations made by international authorities on the subject, such as those given by WHO in GREAT (7), are also disregarded.

Based on these findings, we emphasise the need to pay greater attention to the principle: “Users are involved throughout the design and development”, meaningfully engaging with users would lead to better identification of their needs and preferences. It shall also improve the possibility to have better recruitment procedures, representative samples and more representative and robust results.

Engaging with users will require a broad level of expertise and full implementation of ISO 9241-210 principle 5.7 “The design team includes multidisciplinary skills and perspectives”. Transdisciplinarity could have reduced methodological flaws observed in the literature today. Transdisciplinarity shall open the possibility of cross-fertilization between the different fields of knowledge and in conjunction with people with visual impairments as potential direct users and with their stakeholders. The application of this principle shall enable design teams to include not only diverse classes of engineers, but also designers, health and rehabilitation professionals, social scientists including disability scholars, anthropologists, and economists, among others. Such teams shall be better equipped to develop and apply a range of methodologies that understand the social, physiological, cultural and technological needs of the target users and develop AT that responds to them. The strengthening of this last principle of the standard would lead the work towards the consolidation of adequate methodologies to gain a better understanding of how AT could enable visually impaired people to live the lives they would like to live.

Design of AT should be focusing on enhancing the user's agency, bodily integrity, and capabilities, and not trying to “fix disabled bodies”. Evidence collected suggests that assistive technology has focused on functional deficiency solely, namely impairment rather than in enhancing wellbeing for its users (6, 53). The latter seems to be the prevalent approach today as users are for the most part not meaningfully included in the design and development and only call to test a final product that aims to provide a “solution”. “Nothing About Us Without Us” should resonate with the design, development and implementation of any technological development that concerns persons with disabilities.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author contributions

LMO-E: prepared conceptualization, methodology data curation, formal analysis, validation, visualization, writing original draft, and writing, reviewing, and editing. MAC: contributed in conceptualization, data curation, writing. Original draft, project administration, validation, writing, reviewing, editing. KS: project administration, fund raising, validation, writing, reviewing, editing. SH: project administration, fund raising, validation, writing, reviewing, editing. MAS: reviewing, editing. AM: reviewing, editing. MR-V: prepared conceptualization, methodology data curation, formal analysis, validation, writing original draft, and writing, reviewing, and editing, project administration, fund raising and supervision. All authors contributed to the article and approved the submitted version.

Funding

MR-V was supported by the Ambizione grant (PZOOP1_186035), funded by the Swiss National Science Foundation. LMO-E and MAC were supported by the Fondation Gelbert (0751-2020).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fresc.2023.1238158/full#supplementary-material

References

1. Blind K, Von Laer M. Paving the path: drivers of standardization participation at ISO. J Technol Transf. (2022) 47:1115–34. doi: 10.1007/s10961-021-09871-4

CrossRef Full Text | Google Scholar

2. Thomas V, Remy C, Bates O. The limits of HCD: reimagining the anthropocentricity of ISO 9241-210. LIMITS ‘17 workshop on computing within limits (2017). p. 85–92.

3. International Organization for Standardization. ISO 9241-210:2019. Part 210: human-centred design for interactive systems. Standarization IOf. ISO 9241: ergonomics of human-system interaction (2019).

4. Chammas A, Quaresma M, Mont’Alvão C. A closer look on the user centred design. Procedia Manuf. (2015) 3:5397–404. doi: 10.1016/j.promfg.2015.07.656

CrossRef Full Text | Google Scholar

5. Magnier C, Thomann G, Villeneuve F, Zwolinsk P. Investigation of methods for the design of assistive device: UCD and medical tools. IDMME_P30; Bordeaux, France (2010). p. 5397–404.

6. ChavarriaVaron MA, Mugeere A, Schönenberger K, Hurst S, Rivas-Velarde M, Havarria MA, et al. Design approaches for creating person centered, context sensitive, and sustainable assistive technology with the global south. In: Stein MA, Lazar J, editors. Accessible technology and the developing world. Oxford: Oxford University Press (2021). p. 248–68.

7. World Health Organization (WHO) and the United Nations Children’s Fund (UNICEF). Global report on assistive technology. Geneva: World Health Organization and the United Nations Children’s Fund (UNICEF) (2022).

8. Stein M, Lazar J. In: Stein M, Lazar J, editors. Accessible technology and the developing world. Oxford: Oxford University Press (2021). p. 18.

9. World Health Organization (WHO). World report on vision. Geneva: World Health Organization, Department of Noncommunicable Diseases (2019. Report No.: WHO/NMH/NVI/19.12.

10. Banerjee T. Assessment of the impact of ISO standards – a study of Chinese firms. J Emerg Technol Innov Res. (2020) 7(9):1040–5. https://api.semanticscholar.org/CorpusID:235374739

Google Scholar

11. Martincic CJ. A brief history of ISO. Pittsburgh: University of Pittsburgh. (1997). Available at: http://www.sis.pitt.edu/mbsclass/standards/martincic/isohistr.htm (Cited 07, 2022).

12. Ferguson W. Impact of the ISO 9000 series standards on industrial marketing. Ind Mark Manag. (1996) 25:305–10. doi: 10.1016/0019-8501(96)00013-2

CrossRef Full Text | Google Scholar

13. Martincus CV, Castresana S, Castagnino T. ISO standards: a certificate to expand exports? Firm-level evidence from Argentina. Rev Int Econ. (2010) 18(5):896–912. doi: 10.1111/j.1467-9396.2010.00915.x

CrossRef Full Text | Google Scholar

14. International Organization for Standardization. ISO 9241-11:2018. Part 11: usability: definitions and concepts. In: Standardization IOf, editors. ISO 9241: Ergonomics of human-system interaction. Geneva: ISO (2018).

15. Blind K. The impact of standardisation and standards on innovation. In: Publishing EE , editors. Handbook of innovation policy impact. London: Edward Elgar Publishing Limited (2016). p. 423–49.

16. International Organization for Standardization. Economic benefits of standards: international case studies. 1st ed. Geneva: ISO (2011).

17. International Organization for Standardization. Economic benefits of standards: international case studies. 2nd ed. Geneva: ISO (2012).

18. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffman TC, Murlow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Rev. (2021) 10(89):2–10. doi: 10.1186/s13643-021-01626-4

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Hong QNN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. Mixed methods appraisal tool (MMAT). Version 2018. Registration of Copyright (#1148552). Industry Canada: Canadian Intellectual Property Office (2018).

20. International Organization for Standardization. ISO 9999:2016. Assistive products for persons with disability—classification and terminology. Geneva: International Organization for Standardization (2016).

21. Conradie PD, De Marez L, Saldien J. Participation is blind: involving low vision lead users in product development. Procedia Comput Sci. (2015) 67:48–56. doi: 10.1016/j.procs.2015.09.248

CrossRef Full Text | Google Scholar

22. Façanha AR, Araújo MC, Viana W, Sánchez J. Design and evaluation of mobile sensing technologies for identifying medicines by people with visual disabilities. Int J Pervasive Comput Commun. (2019) 15(3/4):121–43. doi: 10.1108/IJPCC-02-2019-0006

CrossRef Full Text | Google Scholar

23. Ntakolia C, Dimas G, Lakovidis DK. User-centered system design for assisted navigation of visually impaired individuals in outdoor cultural environments. Univers Access Inf Soc. (2020) 21:249–74. doi: 10.1007/s10209-020-00764-1

CrossRef Full Text | Google Scholar

24. Mattheiss E, Regal G, Sellitsch D, Tscheligi M. User-centred design with visually impaired pupils: a case study of a game editor for orientation and mobility training. Int J Child Comput Interact (2017) 11:12–8. doi: 10.1016/j.ijcci.2016.11.001

CrossRef Full Text | Google Scholar

25. Sánchez J. Development of navigation skills through audio haptic videogaming in learners who are blind. Procedia Comput Sci. (2012) 14:102–10. doi: 10.1016/j.procs.2012.10.012

CrossRef Full Text | Google Scholar

26. Nimmolrat A, Khuwuthyakorn P, Wientong P, Thinnukool O. Pharmaceutical mobile application for visually-impaired people in Thailand: development and implementation. BMC Med Inform Decis Mak. (2021) 21(1):1–19. doi: 10.1186/s12911-020-01362-0

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Yeh FH, Yang CC. Assisting the visually impaired to deal with telephone interview jobs using information and commutation technology. Res Dev Disabil. (2014) 35(12):3462–8. doi: 10.1016/j.ridd.2014.08.009

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Cheverst K, Clarke K, Dewsbury G, Hemmings T, Kember S, Rodden T, et al. Designing assistive technologies for medication regimes in care settings. Univers Access Inf Soc. (2003) 2(3):235–42. doi: 10.1007/s10209-003-0055-9

CrossRef Full Text | Google Scholar

29. Nielsen J. Heuristic evaluation. In: Nielsen J, Mack RL, editors. Usability inspection methods. New York: Wiley (1994).

30. Mascetti S, Picinali L, Gerino A, Ahmetovic D, Bernareggi C. Sonification of guidance data during road crossing for people with visual impairments or blindness. Int J Hum Comput Stud. (2016) 85:16–26. doi: 10.1016/j.ijhcs.2015.08.003

CrossRef Full Text | Google Scholar

31. Najjar AB, Al-Issa AR, Hosny M. Dynamic indoor path planning for the visually impaired. J King Saud Univ Comput Inf Sci. (2022) 34:7014–24. doi: 10.1016/j.jksuci.2022.03.004

CrossRef Full Text | Google Scholar

32. Connors E, Chrastil E, Sánchez J, Merabet L. Action video game play and transfer of navigation and spatial cognition skills in adolescents who are blind. Front Hum Neurosci (2014) 8(133):1–7. doi: 10.3389/fnhum.2014.00133

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Aziz N, Mutalib AA, Sarif SM. User experience of interactive assistive courseware for low vision learners (AC4LV): initial round. TEM J (2017) 6(3):488–96: doi: 10.18421/TEM63-09

CrossRef Full Text | Google Scholar

34. Bateman A, Zhao OK, Bajcsy AV, Jennings MC, Toth BN, Cohen AJ, et al. A user-centered design and analysis of an electrostatic haptic touchscreen system for students with visual impairments. Int J Hum Comput Stud. (2018) 109:102–11. doi: 10.1016/j.ijhcs.2017.09.004

CrossRef Full Text | Google Scholar

35. Colley M, Walch M, Gugenheimer J, Askari A, Ruzkio E. Towards inclusive external communication of autonomous vehicles for pedestrians with vision impairments. Proc SIGCHI conf hum factor comput syst. (2020). p. 1–14.

36. Shi L, Lawson H, Zhang Z, Azenkot S. Designing interactive 3D printed models with teachers of the visually impaired. (2019). p. 1–14.

37. Fidyka A, Matamala A. Audio description in 360° videos: results from focus groups in Barcelona and kraków. Transl Spaces (Neth.). (2018) 7(2):285–303. doi: 10.1075/ts.18018.fid

CrossRef Full Text | Google Scholar

38. Doush IA, Pontelli E. Non-visual navigation of spreadsheets. Total Qual Manag Bus Excell. (2013) 12(2):143–59. doi: 10.1007/s10209-012-0272-1

CrossRef Full Text | Google Scholar

39. Adebiyi A, Sorrentino P, Bohlool S, Zhang C, Arditti M, Goodrich G, et al. Assessment of feedback modalities for wearable visual aids in blind mobility. PLoS One. (2017) 12(2):e0170531. doi: 10.1371/journal.pone.0170531

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Giraud S, Thérouanne P, Steiner DD. Web accessibility: filtering redundant and irrelevant information improves website usability for blind users. Int J Hum Comput Stud. (2018) 111:23–35. doi: 10.1016/j.ijhcs.2017.10.011

CrossRef Full Text | Google Scholar

41. Lopes SI, Vieira JM, Lopes OF, Rosa PR, Dias NA. Mobifree: a set of electronic mobility aids for the blind. Procedia Comput Sci. (2012) 14:10–9. doi: 10.1016/j.procs.2012.10.002

CrossRef Full Text | Google Scholar

42. Mascetti S, Ahmetovic D, Gerino A, Bernareggi C, Busso M, Rizzi A. Robust traffic lights detection on mobile devices for pedestrians with visual impairment. Comput Vis Image Underst. (2016) 148:123–35. doi: 10.1016/j.cviu.2015.11.017

CrossRef Full Text | Google Scholar

43. Kammoun S, Parseihian G, Gutierrez O, Brilhault A, Serpa A, Raynal M, et al. Navigation and space perception assistance for the visually impaired: the NAVIG project. IRBM. (2012) 33(2):182–9. doi: 10.1016/j.irbm.2012.01.009

CrossRef Full Text | Google Scholar

44. Lee K, Hong J, Jarjue E, Mensah EE, Kacorri H. From the lab to people’s home: lessons from accessing blind participants’ interactions via smart glasses in remote studies. Proceedings of the 19th international web for all conference (2022). p. 1–11.

45. Younis O, Al-Nuaimy W, Rowe F, Alomari M. A smart context-aware hazard attention system to help people with peripheral vision loss. Sensors. (2019) 19(7):1630. doi: 10.3390/s19071630

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Feiz S, Borodin A, Bi X, Ramakrishnan IV. Towards enabling blind people to fill out paper forms with a wearable smartphone assistant. Proc Graph Interface. (2021) 2021:156–65. doi: 10.20380/GI2021.18

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Real S, Araujo A. Navigation systems for the blind and visually impaired: past work, challenges, and open problems. J Sens. (2019) 19(15):1–16. doi: 10.3390/s19153404

CrossRef Full Text | Google Scholar

48. Blackburn SJ, Cudd PA. A discussion of systematic user requirements gathering from a population who require assistive technology. Technol Disabil. (2012) 24(3):193–204. doi: 10.3233/TAD-2012-0352

CrossRef Full Text | Google Scholar

49. International Organization for Standardization. ISO 9001:2015 quality management systems – requirements. 5th ed. Geneva: ISO (2015).

50. GHTF Study Group 3. Quality management systems – process validation guidance. The Global Harmonization Task Force (GHTF) (2004).

51. Project Management Institute (PMI) Standard. A guide to the project management body of knowledge (PMBOK guide). 4th ed. PMI (2011).

52. Giacomin J. What is human centred design? Des J. (2014) 17(4):608–19. doi: 10.2752/175630614X14056185480186

CrossRef Full Text | Google Scholar

53. Verlager A. Decloaking disability: images of disability and technology in science fiction media [M.Sc. Thesis]. Massachusetts Institute of Technology (2006).