Objectives

The main objective for evaluating acoustic sets is to determine the quality of a product by specified criteria on an absolute scale and in comparison with other products. Success of acoustic sets, as good products, depends on many factors. We can single out the following most important factors for users: exterior, ergonomics, functionality, and audio quality. A product price is also an important factor when you buy an acoustic set. But the most important parameter is the price/quality ratio compared to closest competitors.

Overview

In order to compare speakers with each other as well as to get an insight into each product, we should examine and compare products by all key parameters that determine the integral quality.

It's very useful to compare products directly and use a reference section for an absolute model.

Equipment

Our laboratory is equipped with a rack-mount studio mastering interface Lynx Studio Aurora 8 ($3000) and high-quality active studio monitors ADAM S2.5A ($5000). We also have a consumer section from a DVD-Audio/SACD player Pioneer 757Ai (higher category) and a multi-channel amplifier Pioneer VSX-AX5i, connected via non jitter interface i-Link. Acoustic sets B&W 6xS. The total cost of the consumer section is about $6000.

We use reference DVD-Audio / SACD discs as well as a selection of the best CD-DA records.

Objective Tests

Objective parameters are used in the course of analysis for higher precision – the main parameters that can be obtained in tests. They are frequency response, axial and extra-axial parameters, harmonic distortions in a service band, analysis of the second and the third harmonics separately, spectrum screenshots at pilot frequencies.

Frequency response characterizes a service band of a device and timbre nonuniformity. Extra axial frequency response characterizes HF directivity. Frequency response at extreme timbre control positions demonstrates capacities of active speakers. Harmonic level shows the audible component of harmonic distortions. This useful parameter is not registered by most testlabs. We use our own method of growing aliquant logarithmic sinusoids of fixed frequency with a music interval to measure harmonic factor (2+3 harmonics).

We take measurements using a special firmware complex. We use a lengthy logarithmically growing sinusoid signal with self-tuning phase to measure frequency response. It allows to get more real readings, because this test signal is closer to music in energy terms. It also eliminates artificial addition of theoretically obtained LF range by approximated modeling results.

Subjective Tests

It's very difficult to put auditory sensations into words. There are several approaches to describing instrument sounds. The most popular approach is to use emotional epithets that do not belong to music terms as well as to technical terms used to describe physical processes. For example, you can come across such terms as "tasty bass", "grownup sound", "intimate immersion degree". Another approach consists in rating such parameters as "stage width", "image stability", and subsequent manipulations with these figures. Such word combinations have no strict definitions or standard scientifically grounded basis. It's up to a user to interpret them. It has to do with esoteric anti-scientific subjective emotional (audiophile) point system, individual for each listener. Value of such ratings is questionable at best, even if you try to detect some logic in them. It's impossible to draw useful conclusions on price/quality ratio or how good the speaker design is.

The point system used in our test procedure is based on the following:

1. Scientific basis to understand physical processes (spectral analysis, temporal analysis, etc). Acoustic system (speaker) is a complex non-linear electroacoustic device with a complex of properties. That's why evaluations may include physical terms from the Acoustics section.

2. Musical approach. A signal reproduced by speakers is music, so it can be analyzed by musical criteria. So we can evaluate audio quality without inventing new terms.

A complex approach is an advantage. For example, a phrase "a good attack of the acoustic system allows more accurate rendition of accents" combines scientific, technical, and musical terms. But a phrase "intimately mystical sound of speakers is agitating and emotionally overwhelming" does not bear any useful information, except about hyperthymia of the listener.

This approach also has drawbacks. For example, a reader is assumed to have a certain minimal qualification level. On the other hand, test results are always intended for interested people, who are ready to learn new things, if they can really help them choose a device or expand their horizon.

AES20

AES (Audio Engineering Society) is a professional society that specializes in audio technologies only. AES draws its membership from amongst leading engineers, scientists, and other famous persons. This society was established to share know-how, work out specifications and recommendations for engineers, publish research works in the audio field to stimulate its development.

AES20-1996/2002 "Subjective evaluation of loudspeakers" was created in 1996, updated and confirmed in 2002. This standard is a set of requirements and recommendations for subjective evaluation of speakers.

For the six years of comparing acoustic sets, we have gradually come to similar comparison methods independently. But we take AES20-1996/2002 critically. In our opinion, many points cannot be used. We'll refer only to those paragraphs of this standard, with which we fully agree.

Requirements to Experts

Listeners should meet high requirements. Evaluation precision and credibility depends much on expertise of listeners.

Only experienced specialists can act as judges in tournaments, they should know all ins and outs of the sport, not fans. Just as well, dilettantes with inadequate fabricated scale of emotional ratings cannot evaluate acoustic sets. They don't understand the musical and technical sides of the audio field.

Only trained experienced listeners should be allowed to evaluate speakers. (AES20-1996/2002 7.7.1)

Untrained adequate listeners can take part in additional tests, which results should be analyzed separately. (AES20-1996/2002 7.7.2)

Listeners with subjective preferences of brands or certain speaker models cannot take part in tests. Any privies should also be excluded.

Listeners shouldn't note acoustic peculiarities in a lengthy test. The test shouldn't be long to avoid listening fatigue. (AES20-1996/2002 7.8.1.)

Testing

AES20 requires double blind testing (DBT). (AES20-1996/2002 7.10.1.)

DBT has a drawback - a listener is under increased mental stress (owing to the risk to make a mistake). Insufficiently trained listeners may provide random unsystematic DBT results. To carry out DBT, loudspeakers shall be hidden from view with a screen. The screen shall be visually opaque and acoustically transparent. For example, the screen can be made of polyester double-knit fabric. (AES20-1996/2002 7.10.3.)

Only digital records can be used as media. (AES20-1996/2002 6.1) We use the same PCM-WAV files in 16 bit 44.1 kHz and 24 bit 96 kHz formats, digitally copied from CD-DA and DVD-Audio tracks. The files are stored on a hard drive. Speakers must be compared using the same records.

The core selections shall include acoustically recorded music selections of recognizable instruments, the audio shouldn't be processed much. (AES20-1996/2002 6.2.2)

Test volume depends on a speaker class. Speakers should play in their working conditions, so it makes no sense to specify a certain volume level. Speaker volume should be the same for direct comparisons. It's a must.

Duration of test fragments should be about 10 seconds, switch interval shouldn't exceed 5 seconds. A listener should switch to the original and back. The longer is the switch interval, the less adequate results are, because human audio memory is not long. If a pause is long, a listener preserves only emotional memories and loses sound details.

To determine objective characteristics, we use measurements instead of optional listening to technical signals (AES20-1996/2002 6.3)

Test Report

Sound peculiarities of each model are written down in a report.
The total score consists of the following components:

1. audio quality in the absolute category
2. audio quality in its class
3. price/audio quality ratio
4. evaluation of applications (music, home theatre, professional audio equipment)

Write a comment below. No registration needed!