Listening Tests

MIAMI, FL., DEC. 20 — SDMI’s statistical analysis of its listening tests indicate that Blue Spike’s Giovanni is the only statistically inaudible system of the technologies remaining under consideration in SDMI’s watermarking trials, confirming results of tests in the UK, Japan and US in the past two years that commented on Blue Spike’s peerless transparency and fidelity.

The statistical analysis, completed by Eugene Ericksen, Ph.D., a consultant from Philadelphia specializing in statistical and survey analyses, showed Blue Spike’s Giovanni – alone among the three SDMI proponents’ technologies – had listening test results that indicated it is statistically inaudible.

SDMI conducted test sessions with expert listeners in New York, Japan, Nashville, London and Los Angeles in September and October. Systems from Blue Spike and the two other remaining SDMI proponents were used to encode watermarks into the samples. Judges at these test sessions heard the same piece of music 8 times and were asked to report whether or not a watermark was present. Each technology was heard more than 100 times by different listeners, using 10 different music samples.

To evaluate the test results for each watermarking technology, Dr. Ericksen drew up a model of results based on the assumption that the listeners were guessing and compared that with the results of the actual tests. Comparing the actual results of a survey or a series of experiments against this kind of random model is an exercise statisticians conduct to assess the authenticity of their results.

Assuming that the chance of guessing correctly on each sample was 50 percent, Dr. Ericksen formulated probabilities that the listeners had correctly identified the presence or absence of a watermark by random chance. By that calculation, Giovanni far higher probability of being identified by random chance than the remaining proponents’ technologies.

“[T]he [Blue Spike] results are not incompatible with the random assumption,” Dr. Ericksen wrote. Of the two other technologies surveyed in the listening tests, Dr. Ericksen wrote that “their results are incompatible with the assumption of randomness” meaning perceptions were more truly instrumental in informing the listeners’ identifications. In sum, by the results of the tests and by statistical conventions, there was only one technology that could be deemed statistically inaudible: Blue Spike’s Giovanni.

For listeners who successfully identified the presence or absence of watermarks in the music they heard in 8 of 8 samples, for example, the probability that Blue Spike’s technology was identified by random chance was 6.54%; in 7 of 8 samples, 16.98%; in 6 of 8 samples, 0.19% and in 5 of 8 samples, 5.45%. For the two remaining SDMI proponents, the probability that chance played a role in informing the listeners’ perceptions was consistently less than 1%. (Statisticians usually consider results above 5% in this kind of null hypothesis test to be significant and of higher confidence than those below 5%.)

What this means is that for Blue Spike’s results there was a much greater probability that, statistically speaking, listeners’ identifications were being made by accident. In the context of a listening test, the greater the probability that identifications of a given technology’s watermarks were made by chance, the greater the indication that the technology’s watermarks were statistically inaudible.

In concluding his discussion of the listening results compared with the random model, Dr. Ericksen wrote, “Testing the reliability of our results, I conclude that we are about 75 percent certain that [Blue Spike] hides the watermarks more effectively than either CRL or [Verance].”

SDMI’s tests and statistical analysis arrive after some months of rising contention about the audibility of Verance’s watermarking system which was selected by the 4C Entity – comprised of DVD-Audio co-developers IBM, Intel, Matsushita and Toshiba – for DVD-Audio copy-protection. (The Verance system was also selected by the SDMI for its Phase I specification.)

Audio engineers at listening sessions, in online forums and conference panels have criticized the watermarking system chosen for DVD-Audio as easily audible. They have questioned the thoroughness of testing for the watermarking system itself and have speculated that the higher sampling rates of new audio formats would accentuate audible artifacts that watermarking systems may introduce.

It is already apparent that the watermarking audibility issue is forcing labels that issue high-fidelity recordings to question the selection of the copy-control watermarking technology. Telarc announced last month that the company would not watermark its DVD-Audio releases.

CEO Scott Moskowitz said, “Blue Spike’s audibility results clearly distinguish Giovanni and our Trusted Transaction architecture as the only viable alternative to the current 4C-content protection scheme. We compete with the 4C proposal and have held DVD-audio licensing separate from our SDMI-proposed terms and conditions.”

Although Blue Spike has achieved superior fidelity with its automated embedder, as required in the SDMI Phase II Call for Proposals, the company has long advocated key-based watermarking systems that rely on human judgments – as opposed to automated coding systems. A key-based watermarking system, in the hands of an experienced audio engineer, would enable the most transparent embedding of watermarks.

To date however, SDMI has not supported key-based interactive embedders and would not accept such a system as a response to the Call for Proposals, which specifically required automated embedding. Blue Spike, drawing on extensive testing and years of development, has determined that digital watermarking regimes can only achieve their highest fidelity with key-based systems.