Group 6: Compiling (VC++)

The previous compile test was weak at multithreading. That was its main problem. As a result, computers with more processor cores did not get a proper advantage over computers with fewer cores. As it turned out, this problem can be solved quite easily: there is no need to change our compiler or even a project. What is needed is to tweak the project and compile options. So this benchmark is both new and old at the same time. Old because we haven't changed the compiler (Microsoft Visual Studio 2008) or the project (Ogre 3D -- Open Source 3D Graphics Engine). New because the benchmark behaves differently now, and it's much more responsive to computers with many processors cores.

Technical details

Software: Microsoft Visual Studio 2008 (9.0.21022.8 RTM), Microsoft DirectX SDK (March 2009). Source code for compilation: modified Ogre3D, Visual C++.Net 2008 (9.0) Precompiled Dependencies. This test measures the time it takes to compile the whole project. The test is performed five times, results obtained are averaged out.

Group 7: Java

From our point of view, it's quite logical to add this new group to our test method. Even if it includes only one benchmark that tests execution speed of programs written in a certain language. Later on we may expand this group of tests (as well as the compile group...) However, it's quite representative even now: from the point of view of execution on computers of end users, Java is indeed the most popular of programming languages, and its Windows implementation from Sun Microsystems is the most popular implementation of JRE (Java Runtime Environment). What concerns the test itself, we use a benchmark from SPEC again -- SPECjvm2008. It's quite detailed, and it includes lots of subtests from various fields: Compiler (compilation with OpenJDK), Compress (LZW compression), Crypto (AES/DES, RSA encryption, MD5withRSA, SHA1withRSA, SHA1withDSA, and SHA256withRSA verification), Derby (this benchmark uses an open source database, written completely in Java), MPEGAudio (mp3 decoding based on the LGPL library JLayer), Scimark (popular scientific floating-point benchmark; by the way, we use its MAPLE version in the corresponding test), Serial (parallel-series conversion), Sunflow (visualization with a multithreaded render engine supporting global illumination), XML (applying stylesheets to XML documents and validation of XML documents (xsd). This benchmark features partial optimizations for multithreading (not available in all tests).

Technical details

Software: SPECjvm2008, Sun JRE 6.13 for Windows x64. Startup instructions come with the benchmark. Additional startup options: "-i 5" (5 iterations). Results are averaged out by the benchmark itself.

Group 8: Audio encoding

This time we solved the problem of practically complete absence of multithreaded audio codecs with the help of the dBpoweramp shell, which can start several encoding processes simultaneously. Thus, it makes encoding a multithreaded process even if we use single-threaded codecs. Another advantage of dBpoweramp is its integrated benchmark that shows the ratio between audio encoding and its playback (for example, 20 points mean that a 20-minute audio stream was encoded for one minute). We conducted an express examination of this benchmark (we compared results of several processors in the benchmark and then measured how much time it took to encode selected WAV files with dBpoweramp Music Converter) to make sure that the integrated benchmark is adequate, stable, and predictable. Unfortunately, we had to abandon two codecs (Musepack and Windows Media Audio), because the dBpoweramp benchmark does not work correctly with them. However, the list of codecs is still quite imposing: MP3 (LAME), Apple Lossless, FLAC, Nero AAC, Monkeys Audio, and OGG Vorbis.

Technical details

Software: dBpoweramp R13.1, dBpoweramp Bench Mark Tester. Each test is performed five times for each codec. Test results of each codec are averages out, and the total score in this group of tests is calculated as a geomean of results demonstrated by all codecs.

Group 9: Video encoding

It's a traditional part for our test methods, which evolves from version to version. In the new test method we added another format, which hasn't happened for a long time already. So, in this part of the test method we evaluate the speed of transcoding a video file from one format into another. From what to what? And with what codec?

1. The AVI file (DV) with fragments of home video is to be converted into MPEG2 (DVD) with Canopus ProCoder. It's a classical task for many users, who prefer to convert their video to DVD.
2. The MPEG2 file (1920x1080, 5-min fragment of the 'Iron Man' movie) is transcoded into the most popular formats used to rip movies: DivX, XviD, x264, VC-1.

Thus, our tests will be interesting to people, who process their home video archives, and to those users, who want to rip DVDs in their video libraries on their own.

Technical details

1. Software: Grass Valley ProCoder 3.0.5. Source: DV-format file (637 MB). Options and settings: ProCoder project (file paths can be adjusted in any text editor, even in Notepad). This test measures encoding time. The test is run five times, results obtained are averaged out.
2. Software: DivX Pro 7.0, VirtualDub x86 1.8.8, AVISynth 2.5.8, AC3Filter 1.51a. Source: 5-min fragment from the 'Iron Man' movie (from 00:02:00 to 00: 07:00), MPEG2, 1920x1080, average bitrate 6000 (to reduce the file size)*. Options and settings: VirtualDub project. This test measures encoding time. The test is run five times, results obtained are averaged out.
3. Software: XviD 1.2.1 Final Release, VirtualDub x86 1.8.8, AVISynth 2.5.8, AC3Filter 1.51a. Source: see the previous test. Options and settings: VirtualDub project. This test measures encoding time. The test is run five times, results obtained are averaged out.
4. Software: x264 x86 r1163, AVISynth 2.5.8, AC3Filter 1.51a. Source: see the previous test. Options and settings (startup command): x264.exe --crf 24.0 --ref 4 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --weightb --direct auto --subme 6 --trellis 1 --partitions all --8x8dct --threads auto --thread-input --no-dct-decimate --progress -o temp.avi file.avs. This test measures encoding time. The test is run five times, results obtained are averaged out.
5. Software: Mainconcept Reference 1.6. Source: see the previous test. Options and settings: Mainconcept Reference project (you can edit file paths manually in Notepad). This test measures encoding time. The test is run five times, results obtained are averaged out.

* We cannot upload this fragment, as it will be infringement of law, so we give you an accurate description, so that you could make the file on your own.

The total score in this group is calculated as a geomean of results demonstrated by all tests.

Write a comment below. No registration needed!