Validity, Reliability, and Usability of the Automated Method for Testing Auditory Sensitivity With Malay Instructional Video Among Malaysian Population
Article information
Abstract
Background and Objectives
The Automated Method for Testing Auditory Sensitivity (AMTAS) is a promising automated hearing assessment tool currently available as an alternative for conducting hearing assessments. The aim of this study was to assess the validity, reliability, and usability of the AMTAS in a Malaysian population.
Subjects and Methods
This study included 100 participants who participated in validity testing, 30 in reliability testing, and 232 participants who answered the Malay version of the mHealth Application Usability Questionnaire (M-MAUQ). The test was conducted at a general hospital and district hospital. Hearing thresholds were compared between conventional pure-tone audiometry and AMTAS for validation. The hearing threshold of the AMTAS was measured twice to investigate the test-retest reliability. The usability of the AMTAS was analyzed based on M-MAUQ scores for ease of use, interface and satisfaction, and usefulness after the participants used the AMTAS.
Results
The mean difference in hearing thresholds between these two methods ranged from 1.25 dB to 3.40 dB at the general hospital and 0.30 dB to 3.35 dB at the district hospital, with over 95% of participants exhibiting a mean difference within ±10 dB. The intraclass correlation coefficient for the test-retest reliability of the AMTAS ranged from 0.64 to 0.99 across tested frequencies. The overall mean scores of the M-MAUQ for each item ranged from 6.31 to 6.80 out of a full score of 7, indicating excellent usability of the AMTAS.
Conclusions
This study revealed that the AMTAS is a valid and reliable approach for measuring hearing status in the Malaysian population, demonstrating high user satisfaction.
Introduction
Hearing loss is a global health issue, with the World Health Organization (WHO) reporting 1.5 billion individuals affected worldwide [1]. This rising prevalence of affected individuals has outnumbered the availability of audiologists who can provide hearing healthcare services [2]. With a population of 34 million, Malaysia exemplifies this disparity, with one audiologist for every 42,500 Malaysians [3]. As of 2023, although nearly 800 certified local audiologists serve in various public and private employment sectors across the country, this figure falls short of the projected need for audiologists at a ratio of 1:500 [4].
In response, the Ministry of Health Malaysia has introduced and implemented initiatives such as teleaudiology, funding hearing health programs, community-based rehabilitation programs, and training more audiologists to improve access to hearing healthcare services, particularly in rural and underserved areas [5-9]. Innovative solutions such as automated hearing assessment tools have the potential to address the mentioned issues and reduce the gap between the need and the capacity of audiologists.
The Automated Method for Testing Auditory Sensitivity (AMTAS) is a promising automated hearing assessment tool currently available as an alternative for conducting hearing assessments. AMTAS is a self-administered prototype computer-based audiometer capable of providing hearing thresholds without the direct involvement of an audiologist. AMTAS has been validated for over 15 years through multiple published studies and has demonstrated its accuracy and reliability, showing comparable results to manual pure-tone audiometry (PTA) in controlled settings [10-13]. However, despite these advancements, AMTAS has primarily been validated in Western countries, leaving a gap in verifying its applicability in a non-native English-speaking country like Malaysia. Given Malaysia’s multilingual society, the translation and validation of automated healthcare tools like AMTAS must ensure their applicability and efficiency across diverse populations.
According to Margolis, et al. [11], AMTAS uses user feedback and a single-interval, YES-NO psychophysical approach. The interval during which the tonal signal is presented is visually marked on the touchscreen in front of the listener. After this interval, the person taps YES or NO. Randomly interspersed “catch trials” play no tone and display a feedback message if the participant selected YES incorrectly. The system adaptively adjusts the signal intensity to determine the hearing threshold. In addition, masking noise is continuously applied to the non-test ear at levels chosen to prevent audibility in the non-test ear without over-masking [11].
To implement the AMTAS in Malaysia, user acceptance is a critical determinant of the success of any healthcare technology. The Technology Acceptance Model (TAM) highlights perceived ease of use and perceived usefulness as key factors influencing the adoption of new systems [14]. As a result, many researchers have developed questionnaires such as the Post-Study System Usability Questionnaire (PSSUQ), Telehealth Usability Questionnaire (TUQ), and mHealth Application Usability Questionnaire (MAUQ), which provide robust methodologies for evaluating these parameters [15-17]. Notably, the Malay version of the MAUQ (M-MAUQ) has demonstrated validity and reliability for evaluating mobile health tools, reinforcing the importance of localized usability assessments [18].
This initial investigation involved the selection of normal-hearing individuals to evaluate the baseline performance and cultural applicability of the AMTAS system. The main objective was to assess the impact of cultural and linguistic differences on the accuracy and usability of the system among individuals without hearing impairment. Demonstrating effectiveness among normal-hearing listeners is an essential initial step; if the system fails to perform consistently within this demographic, it is improbable that valid results will be obtained when utilized with individuals experiencing hearing loss. This method establishes a systematic basis for assessing the system’s applicability in wider clinical contexts.
As stated earlier, many studies have focused on the validation and reliability of AMTAS, and limited studies have focused on the feedback from participants who have used AMTAS. A study in Singapore found that most participants reported that AMTAS was easy to use and that testing was fast [19]. A trial conducted in Veterans Administration outpatient clinics in the United States found good acceptability by patients and clinicians [20]. This study provides a comprehensive and concise analysis of the validity and reliability of AMTAS among Malaysians, particularly delving into the usability aspect of AMTAS based on M-MAUQ scores rated by the participants.
Subjects and Methods
This study was approved by the Human Research Ethics Committee Universiti Sains Malaysia (JEPeM-USM) with study protocol code USM/JEPeM/KK/23080602. This study was conducted in two phases. Phase I focused on two key task components: translating the AMTAS instructional video into Malay and evaluating face validation of the M-MAUQ. The translation process involved transcribing and translating the English script of the instructional video into Bahasa Malaysia. The translated script was reviewed by the School of Languages, Literacies, and Translation (SoLLaT), and the final translated version was recorded in a sound studio to produce the Malay version of the AMTAS instructional video. This step was crucial as the video was unavailable in Malay, and maintaining its accuracy was vital.
The second task was the face validation of the M-MAUQ, following Yusoff ’s (2019) guidelines [21]. This is to assess the clarity and comprehensibility of the questionnaire items using a Likert scale. Thirty participants from various backgrounds took part in this assessment. The face validity index (FVI) was then calculated to assess the clarity and relevance of the questionnaire items for the local population.
Phase II focused on evaluating the validity, reliability, and usability of the AMTAS with a Malay instructional video. For this, a stratified sampling method was used to select participants based on five age groups with different ethnicities (Table 1). This study employed the GSI AMTAS Flex system, comprising the GSI AMTAS Flex software (Version 1.5.1, Build 40; Grason-Stadler Inc., 2023), utilized on a Microsoft Surface Go 3 Tablet (Device name: TABLET-M2ERRMMV, S/N: 0B33H6H214933F) and connected to RadioEar DD450 headphones (P/N: 8509294, S/N: WTB04242).
Demographic characteristics of participants in AMTAS validity, reliability, and usability testing
AMTAS is a computer-controlled threshold-measurement method that utilizes a single-interval, YES-NO, forced-choice paradigm with feedback. Stimuli are presented with a bracketing technique similar to the Hughson-Westlake method [22]. Stimulus levels descend in 10-dB steps until a “NO” response occurs and then bracket in 5-dB steps. Threshold is defined as the level at which two YES responses, each following NO responses, occur within a 5-dB range. If the two NO responses occur at different levels, the higher of the two levels is defined as threshold. The higher of the two levels was chosen because it produced better agreement with manual audiometry. The interval during which the tonal signal is presented is visually marked on the touchscreen in front of the listener. After this interval, the person taps YES or NO, indicating if a tone was heard. Randomly interspersed “catch trials” in which no tone was presented are used to assess the listener’s accuracy in responding. When the listener responds YES (false alarm) when there was no tone, a feedback message is presented informing the listener that a YES response occurred when there was no tone. After each threshold determination, a quality check is executed in which a tone 5 dB above threshold is presented. If the listener responds NO, a quality check fail is recorded. Quality indicators include number of false alarms, number of quality check fails, test-retest differences at 1,000 Hz, and the average response time of the listener. The quality check measures are entered into a validated formula that predicts the overall accuracy of the test.
Participants are provided with written instructions and respond using a touchscreen interface. The signal level is adjusted adaptively according to the participant’s responses to estimate pure-tone thresholds. The system incorporates ambient noise monitoring and is structured for self-administration, necessitating approximately 7–10 minutes for completion. The calibrations for the microphone and headphones were valid during the data collection period. Validity was determined by comparing thresholds obtained via AMTAS with those obtained via conventional PTA. PTA was conducted by an experienced audiologist following the Ministry of Health’s standard operating procedures [23]. Air conduction thresholds were tested atcross six frequencies (250, 500, 1,000, 2,000, 4,000, and 8,000 Hz) with AMTAS and manual testing, with three-frequency pure- tone averages (500, 1,000, and 2,000 Hz) were calculated. On the other hand, AMTAS testing was overseen by trained personnel who ensured proper headphone placement before leaving the participant to complete the test independently. The testing order between PTA and AMTAS was alternated to avoid any sequence bias.
To assess the test-retest reliability of AMTAS, the procedure was administered twice to each participant under the same test conditions. The hearing thresholds obtained from both sessions were then compared to determine the consistency of the results. Statistical analysis was conducted using the intra-class correlation coefficient (ICC), which is a robust measure commonly used to evaluate the reliability of repeated measurements. IBM SPSS Statistics, version 28 (IBM Corp.) was used for data analyses. A significance level of 0.05 was set for all inferential statistical analyses, and descriptive analyses were applied as appropriate to summarize and describe the data.
Following the completion of the validation and reliability measurements, all participants were asked to complete the M-MAUQ to evaluate the usability of the AMTAS system. Additionally, to further assess usability, a separate group of participants who were not involved in the validation and reliability phases were recruited. These individuals used the AMTAS for the first time and subsequently completed the M-MAUQ based on their initial experience. For the reliability and validation testing, assessments were conducted in two different clinical environments: a sound-treated room at a general hospital (GH) and a quiet room at a district hospital (DH). This approach was chosen to represent varying clinical settings. Participants were pre-selected from outpatients, their companions, and random visitors at both GH and DH. Eligibility criteria included age between 20 and 69 years old, being literate in Malay, no prior experience with hearing tests, and possessing normal or corrected vision sufficient to view a computer screen. Individuals with any visible or active ear infections were excluded. Before testing, participants underwent a brief background check, otoscopic examination, and tympanometry to ensure ear health. A total of 100 participants completed the validity testing, 30 participated in the reliability testing, and 232 individuals completed the M-MAUQ to evaluate the usability of AMTAS.
RESULTS
Face validation of M-MAUQ
The FVI was calculated based on Yusoff ’s [21] guidelines, with an FVI threshold typically set between 0.80 and 0.83. For this objective, two types of FVI were calculated: the item-level FVI (I-FVI) and the scale-level FVI (S-FVI/Ave), the latter being the average of individual I-FVI scores or raters’ clarity and comprehension ratings. In this study, I-FVI and S-FVI/Ave were excellent, each at 0.98, indicating high clarity, relevance, and understanding of the M-MAUQ items. These findings align with those of Mustafa, et al. [18], supporting the evidence that M-MAUQ can be used to assess usability by the target population.
Validation of AMTAS
At GH, AMTAS thresholds ranged from 5.60 dB to 23.95 dB, while PTA thresholds ranged from 7.35 dB to 25.20 dB across all tested frequencies. The average hearing threshold differences between AMTAS and conventional testing ranged from 1.25 dB to 3.40 dB, remaining well within the acceptable ±10 dB range. Moreover, more than 95% of participants demonstrated threshold differences within these clinically acceptable values. Similar results were observed at DH, with slightly higher thresholds due to increased background noise. At DH, AMTAS thresholds ranged from 14.60 dB to 25.10 dB and PTA from 17.75 dB to 27.15 dB across frequencies. The mean differences remained within the acceptable range.
The regression analysis of six frequencies (250, 500, 1,000, 2,000, 4,000, and 8,000 Hz) revealed that AMTAS thresholds can strongly predict PTA thresholds. The R-squared values ranged from 0.88 to 0.96, indicating that AMTAS explains a substantial portion of the variance in PTA thresholds, with the lowest explained variance being 88.1% at 500 Hz and the highest being 95.7% at 4,000 Hz. Moreover, the regression analyses were all statistically significant (p< 0.001), reinforcing the validity of AMTAS.
Reliability of AMTAS
Test-retest reliability was analyzed to assess the reliability of AMTAS. This involved performing AMTAS twice and comparing the thresholds obtained from both measurements. ICC values at GH ranged from 0.64 (250 Hz), indicating moderate reliability, to 0.99 (4,000 Hz to 8,000 Hz), indicating excellent reliability across frequencies. At DH, ICC values ranged from 0.71 to 0.92, indicating good to excellent reliability. ICC scores between 0.5 and 0.75 indicate moderate reliability, 0.75 to 0.9 suggest good reliability, and above 0.9 signify excellent reliability [24]. Results also showed that the testing order did not affect the outcomes, indicating that whether PTA or AMTAS was performed first produced equivalent results.
Usability of AMTAS
The usability of AMTAS was assessed across M-MAUQ’s three domains: ease of use, interface and satisfaction, and usefulness. Overall mean scores ranged from 6.31 to 6.80 out of a full score of 7 (Fig. 1). This range indicates positive feedback from its users, with scores close to the maximum reflecting excellent usability. Reliability was assessed using Cronbach’s alpha, calculated at 0.96 for the M-MAUQ. A Cronbach’s alpha value above 0.90 typically signifies that the items are highly correlated, indicating that the scale is reliable. Further analysis was conducted to investigate the relationship between usability scores and participant’s sociodemographic data. No significant differences in usability scores were found based on sex, age, race, education, or language proficiency.
Usability of AMTAS analyzed based on M-MAUQ scores for the three domains. AMTAS, Automated Method for Testing Auditory Sensitivity; M-MAUQ, Malay version of the mHealth App Usability Questionnaire; M, mean; SD, standard deviation.
Discussion
Translating the AMTAS instructional video into a nation’s national language and mother tongue is crucial to ensuring its message is delivered effectively to the local population. The reason is that instructions are an integral part of a hearing testing procedure, making users more comfortable and confident when performing tests. Next, face validation of M-MAUQ was essential to ensure the usability questionnaire was appropriate and understandable for the Malaysian population. Face validity index scores above 0.79 indicate that the questionnaire items are relevant to the domain and understandable for the target users. The high scores for I-FVI and S-FVI of 0.98 confirm that the translated items were clear and comprehensible to the participants. Therefore, it can be concluded that the M-MAUQ has excellent validity and reliability and can be used to assess usability among adults who are native Malay speakers [18].
The validation of AMTAS with Malay instructional videos showed that the thresholds obtained through the automated method were highly agreeable to those obtained via conventional PTA. The differences between AMTAS and PTA thresholds were within the acceptable ±10 dB range across all tested frequencies, with the highest agreement observed at 250 Hz (99%) and the lowest at 8,000 Hz (95%) at the GH. These results align with prior studies that demonstrated the accuracy of AMTAS in providing reliable threshold measurements comparable to manual audiometry [11,13,19]. Even in a setting with higher ambient noise levels, such as at a DH, AMTAS still strongly correlate with PTA, further supporting its validity in diverse clinical environments. This consistency across different settings highlights its potential to address Malaysia’s increasing demand for hearing assessments.
The regression analysis demonstrated that AMTAS is a strong predictor of manual PTA thresholds across frequencies, with high R-squared values (ranging from 0.88 to 0.96), further supporting the accuracy and reliability of AMTAS in estimating hearing thresholds. These findings align with previous studies that have validated the accuracy of automated audiometry, showing that AMTAS can provide results comparable to manual PTA [11,13]. It further supports the idea that AMTAS is a reliable tool for clinical audiological assessments, especially when manual testing is unavailable.
The reliability of AMTAS was thoroughly assessed through test-retest analysis, which revealed strong ICCs across various frequencies. At GH, the reliability ranged from moderate (ICC=0.64) at 250 Hz to excellent (ICC=0.99) at higher frequencies such as 4,000 Hz and 8,000 Hz. Similar trends were observed at DH, where ICC values ranged from 0.71 to 0.92, indicating high-frequency reliability. These findings are consistent with previous research, which found that AMTAS provides reliable and consistent threshold measurements [10-12,20]. The minimal differences between test-retest measurements and the low significant differences further affirm the reliability of AMTAS, suggesting that it can be used confidently in clinical settings.
The usability of AMTAS, as evaluated through the M-MAUQ, received positive feedback from participants, with mean scores indicating high usability across all domains. The overall mean scores for ease of use, interface, satisfaction, and usefulness ranged from 6.31 to 6.80 out of 7, reflecting excellent user satisfaction. The high Cronbach’s alpha value of 0.96 indicates strong internal consistency within the usability questionnaire, supporting the reliability of the participant responses. These analyses align with previous findings highlighting the importance of user-friendly interfaces in successfully adopting health technologies [17].
The study’s strengths include adapting AMTAS for Malay speakers and testing it with diverse participants. Beyond translating the instructional video, one significant step for future research would be to translate the entire AMTAS interface into Bahasa Malaysia. Noise differences at testing sites and absence of bone conduction testing are limitations in this study. Future studies should aim to minimise noise variability, incorporate bone conduction testing, and include hearing-impaired subjects to provide a more comprehensive assessment.
This study is important in several ways. First, it supports efforts to improve accessibility to hearing healthcare, especially in underserved communities, by utilizing an automated, self-administered testing system. Second, the integration of AMTAS has the potential to reduce the workload and burnout experienced by audiologists by enabling task delegation in routine hearing screening. Third, this research adds to the body of local evidence by evaluating the applicability of AMTAS in both clinical and community-based settings within Malaysia. From a clinical standpoint, it presents a practical tool that can be deployed in diverse settings, including remote and resource-limited environments.
In terms of novelty, this is the first study in Malaysia to validate the AMTAS system using a Malay-translated instructional video, thereby addressing the linguistic and cultural needs of the local population. It is also the first to assess AMTAS across both clinical and community settings in Malaysia using the M-MAUQ to evaluate its usability. This localized approach ensures relevance and acceptability among Malaysian users. Furthermore, AMTAS demonstrated effective usability across a broad age range, suggesting its potential for widespread implementation regardless of age demographics.
In conclusion, this study shows AMTAS’s validity, reliability, and usability, displaying thresholds comparable to manual audiometry and demonstrating high user satisfaction. AMTAS is well-suited for integration into Malaysian clinical practice, addressing the demand for accessible hearing assessments. Future research should include bone conduction testing and standardizing testing environments across diverse clinical and community settings to expand its applicability.
Notes
Conflicts of Interest
The authors have no financial conflicts of interest.
Author Contributions
Conceptualization: Mohd Fadzil Nor Rashid, Nor Hidayah Mohammed Hatta. Data curation: Mohd Fadzil Nor Rashid, Nor Hidayah Mohammed Hatta, Wan Najibah Wan Mohamad. Formal analysis: Mohd Fadzil Nor Rashid, Nor Hidayah Mohammed Hatta, Mohd Normani Zakaria. Funding acquisition: Mohd Fadzil Nor Rashid. Investigation: Nor Hidayah Mohammed Hatta. Methodology: Mohd Fadzil Nor Rashid, Nor Hidayah Mohammed Hatta. Project administration: Mohd Fadzil Nor Rashid, Nor Hidayah Mohammed Hatta. Supervision: Mohd Fadzil Nor Rashid, Wan Najibah Wan Mohamad, Mohd Normani Zakaria. Validation: Mohd Fadzil Nor Rashid. Writing—original draft: Mohd Fadzil Nor Rashid, Nor Hidayah Mohammed Hatta. Writing—review & editing: all authors. Approval of final manuscript: all authors.
Funding Statement
This research was funded by a grant from Universiti Sains Malaysia, Short-Term Grant with Project No: 304/PPSK/6315805.
Acknowledgments
The authors would like to express their sincere appreciation to Professor Robert H. Margolis, PhD, of Audiology Incorporated, for his valuable time and expert insights that greatly contributed to the preparation of this article. The authors also gratefully acknowledge MKS Medic Sdn. Bhd. for their continuous support and for providing access to AMTAS during the data collection phase.