Test and Verification Methodology

An essential part of most auditory research is based on sensitive, reliable and valid test methods. Progress in hearing aid technology and outcome measures require constant revision of existing test methods and development of new methods. In some of the studies at ORCA Europe, focus has been on test methods per se. In other studies, method development, modification, and validation has been a secondary but necessary task. Hearing aid performance assessment can be conducted in field trials or in the laboratory, and the relationship between these two types of evaluations is also of special interest.
Analysis of paired comparison data


In a number of projects at ORCA Europe, Paired Comparison methods have been applied to pairs of hearing-aid settings to find preference or to evaluate other attributes, such as loudness or listening comfort. The analysis of such data can simplest be done using a “win count” concept. An analysis method developed by Dahlquist and Leijon (2003), using a Linear Gaussian model, has been the starting point for developing a Bayesian method for the analysis of Paired Comparison data, which was the aim of this project.

Main Outcomes

A Paired Comparisons software package has been developed. The software takes “preferred alternative” and “magnitude rating” as input parameters, and provides results in terms of credibility for one object being selected more often than the other. The analysis could be applied on both individual and group comparison data.
Conformity of paired comparison results obtained in laboratory and field trials


There has been a certain focus in the hearing aid research community on the design and application of more realistic laboratory tests. To some extent, the degree of realism in such tests is evaluated by comparing their outcome with data collected by test subjects in the field. Thus, an important question is how we can collect reliable data in the field. The aim of the current study was to investigate ways to collect detailed and valid information in the field and compare the findings to results from commonly used laboratory tests.

Main Outcomes

Data were collected by paired comparisons of two hearing aid settings in the field (diary, interview and questionnaire) and laboratory (Paired Comparison judgments, HINT and ORCA Speech test) using a double-blind design. Relationships between laboratory and field measurements were performed using Spearman’s rank correlation coefficient between measures. Assuming the reported overall preference after the field trial is the “true preference”, a statistically significant correlation was found for 6 out of 7 diary categories and for 6 out of 10 questionnaire questions. In contrast, the Paired Comparison data from the lab showed poor agreement with the “true preference”. 

Publications and Presentations

SPIN (Speech in Noise Workshop), Copenhagen (Denmark), 2015

TeMA Hörsel (National conference for Audiologists, Engineers and ENT specialists), Malmö (Sweden), 2015 
Analysis of Confusion Matrix Data


The aim was to develop and validate a test method for analyzing outcome from confusion matrices, resulting from speech phoneme perception tests, typically of the type VCV and CVCVC. 

Main Outcomes

The Bayesian method for analysis of confusion matrices has been described and discussed theoretically in an article (see below). The method has been implemented in a software package, and it has been found to be useful for analysis of data from VCV and the ORCA Speech Test.

Publications and Presentations

Leijon A, Henter GE, Dahlquist M. (2016) Bayesian analysis of phoneme confusion matrices. IEEE/ACM Transactions on Audio, Speech and Language Processing 24: 469-482.
Discrepancy between Real Ear Measured and NOAH module predicted gain


The aim of this project was to investigate and analyze the differences between gain targets provided by a number of hearing aid manufacturers, and actual individual gain measured with Real Ear Measurement (REM) in a number of ears.

Main Outcomes

The individually measured gain was compared to the manufacturers “target gain” in the fitting software. We concluded that, for the estimated real ear targets from the manufacturer, the accuracy was high for ears with acoustical properties similar to those of an ear simulator and when a "typical" REUG was measured. We also saw that some devices were more appropriate for specific ears than others. To know which device to use and how much gain really is provided, it is essential to verify the hearing aid gain in the ear when fitting hearing aids.

Publications and Presentations

TeMA Hörsel (National conference for Audiologists, Engineers and ENT specialists), Linköping (Sweden), 2012. 
ORCA speech test in Swedish


In 2008, ORCA Europe’s sister organization ORCA USA developed a syllable recognition test using a nonsense word material for the American language, named ORCA Speech Test. The overall aim of the current project was to develop a Swedish version of the ORCA Speech Test, and to develop a test that could be used to detect and analyze the effects a given signal processing algorithm could have on different phonemes and/or phoneme groups.

Main Outcomes

Recordings, analyses, and selection of items for a Swedish speech material were made. A first version of the ORCA Speech Test for Swedish was tested on 22 participants with normal hearing. In the initial testing, randomized lists containing 108 nonsense CVCVC words were presented at different signal-to-noise ratios to the participants. Participants were asked to repeat the words and their answers were scored manually by the test leader. Based on the results, the number of items was reduced to 41 nonsense words which formed a shorter and more applicable test list.

Comparison of predictive measures of speech recognition after noise reduction processing


In hearing aid research there is a constant need for the development of verification measures that could reliably predict and thus, at least partially, replace often time-consuming listening tests performed by test subjects. Hence, the aim of this study was to evaluate a number of objective measures concerning their qualification for predicting perceived sound quality and experimentally measured speech-recognition benefit of noise reduction algorithms.

Main outcomes

Speech test material was presented to 20 hearing impaired and 10 normal hearing listeners, both in a reference condition and processed by three NR algorithms, respectively. An adaptive procedure was used to obtain individual, NR algorithm related signal-to-noise ratios which correspond to equal speech-recognition performance. This allowed for a direct comparison between the measured and predicted NR algorithm performance. In addition, the results of sound quality ratings were compared to predicted sound quality.   Only one out of nine tested predictive measures could predict the speech test results reliably for both listener groups. In general, prediction performance was better for the group of hearing impaired listeners and the majority of correctly predicting measures performed short-time analyses and based their calculations on the correlation between clean speech and the processed noisy speech. The results from the sound quality related measures were difficult to interpret, but most of the predictive measures showed the trend to rank order the three noise reduction algorithms correctly.

Publications and presentations

Smeds K, Leijon A, Wolters F, Hammarstedt A, Båsjö S, Hertzman S. (2014) Comparison of predictive measures of speech recognition after noise reduction processing. Journal of Acoustical Society of America 136: 1363-1374.

Nilsson, A. (2010). Evaluation of theoretical measures to assess their ability to predict the performance of noise reduction algorithms. (MSc), KTH, Stockholm.   

Wolters, F. (2010). Predictive ability of objective methods in terms of speech intelligibility and sound quality for noise reduction systems in hearing aids. (Bachelor of Engineering), University of Applied Sciences, Oldenburg, Stockholm.

ISAAR (International Symposium on Auditory and Audiological Research) in Nyborg (Denmark), August 2011

STAF (National Swedish audiology conference for engineers and technicians), Uppsala (Sweden), 2011. 

AES (International conference of the Audio Engineering Society), Piteå (Sweden), 2010. 

Audionomdagarna (National Swedish conference for Audiologists), Västerås (Sweden), 2010.

STAF (National Swedish conference for engineers and technicians), Eskilstuna (Sweden), 2010.

IHCON (International Hearing Aid Research Conference), Lake Tahoe, California (USA), 2010.


Björns Trädgårdsgränd 1

Tel: +46 8 455 67 60

E-mail: orca@orca-eu.info


We are located at Medborgarplatsen at Södermalm in Stockholm, a central location with good communications.

Google Map of Maria Bangata 4 Location on Google Maps