Version 1.0

Time goes on, there appear new notebook models, new applications, new benchmarks. Capacities of our test lab grow as well. Perhaps it's time to lay down our notebook test procedure. This article contains a structure of a typical review with our comments.

So, that's what we start with -

Specifications

Frankly speaking, there exist just a few platforms. So if you have a look at specifications and don't notice (for example) a new graphics controller, you may already predict the performance results of a given notebook.

Then -

Bundle

A standard software bundle for watching DVD or recoding optical discs (if a notebook is equipped with a proper drive) is a must. Various trifles, like a wireless mouse, may be a pro as well as a con for different users (they don't come shipped for nothing).

Subjective part

Overview

Evaluating the exterior and convenience of the port layout. Real weight. Photos of the front panel, rear panel (with the lid open), flanks, and bottom of a notebook.

Ergonomics and usability

Evaluating usability of a given notebook. Keyboard, touchpad, application buttons, media players, etc. As for me, I find it very difficult to lay down some objective rating criteria. That's why this part contains mostly descriptive material, conclusions coming to author's convenient/not convenient. If the key size differs from the standard 19 mm, new dimensions are to be mentioned. This part also describes interesting proprietary software that comes shipped with a notebook.

That's the end of the subjective part. There follow objective results of instrumental tests.

Objective test results

Audio tests

We test the line-out signal in RightMark Audio Analyzer 5.5 using the Terratec DMX 6fire sound card. Total results are published in the main review. Program-generated report is published in a separate document.

Matrix tests

The tests will be carried out according to the LCD test procedure. You can read the details in this article. The only difference has to do with the limited number of settings in notebook matrices (in fact there is only one hardware setting - brightness), we carry out the tests with maximum brightness. The main review shows a response time, the average value and deviations of black and white spot luminance and contrast, as well as angles of view, measured by contrast for CR 10:1 and 5:1. Full results and graphs are published in a separate document.

Performance and battery life tests

We use BAPCO tests (the most adequate tests in our opinion) to determine performance and battery life of a notebook. Their only drawback is the horrible (I mean it) technical support. Thus, you are not guaranteed that the test will run on a given notebook correctly.

Testing a notebook, operating on a battery. We use BAPCO MobileMark 2005 Patch 1 here. Portable/Laptop power scheme. All additional power management utilities are disabled. If necessary, proprietary power management utilities are reviewed separately. Display is never turned off. According to the original test procedure, when a notebook works on a battery, its display brightness is set to minimum, exceeding 60 cd/m². Brightness is measured with the SpyderPRO (PANTONE) colorimeter using OptiCAL software. The package consists of four components.

1. Office Productivity 2002SE. The set of applications is no different from MobileMark 2002.
   • Adobe Photoshop 6.0.1
   • Netscape Communicator 6.01
   • Macromedia Flash 5.0
   • Microsoft Word Excel PowerPoint Outlook 2002
   • Network Associates McAfee VirusScan 5.13
   • WinZip Computing WinZip 8.0

   The audio volume is set to minimum during tests. This test shows Performance Rating (notebook performance versus a reference platform) and Battery Life Rating (battery life time in minutes during the test)

2. Reader 2002SE. This test emulates reading War and Peace in html format. We use Netscape Communicator 6.01. The result is Battery Life Rating
3. DVD playback 2005. During this test we cycle a BMW commercial (1 hour 55 minutes) using InterVideo WinDVD 6.0. Audio output is through headphones. According to the test procedure, audio volume must be comfortable. But this test automatically sets the volume to maximum at startup. So we carry out the test using this volume level.
4. Wireless browsing 2005. Determining the effect of wireless networking on the battery life. The result is Battery Life Rating. This test emulates wireless Internet operations. The test will be included into the test procedure as soon as we configure a necessary server.

Besides, we may publish test-generated reports in a separate page.

You can read the details about this test, scoring procedures, and other aspects of MobileMark 2005 in this document (pdf format, English, ~1MB).

Testing a notebook, operating on mains power supply. The main test is BAPCO SysMark 2004SE. Portable/Laptop power scheme. Test tasks are divided into two subtests - Internet Content Creation and Office Productivity.

1. Internet Content Creation. This group consists of the following subtests.
   • 3D Creation. A 3D model is rendered into a bitmap image in Descreet 3ds max 5.1 simultaneously with creating a web page in Macromedia Dreamweaver MX. 3D animation is then converted into a vector format.
   • 2D Creation. Editing audio and video data in Adobe Premiere 6.5 and exporting a movie. While the movie is being exported, we edit the bmp image (from the previous test) in Adobe Photoshop 7.01 and write a resulting image. After the export is completed, we use Adobe After Effects 5.5 to add special effects to the movie
   • Web Publications. We unpack a Zip archive in WinZip Computing WinZip 8.1 simultaneously with editing and optimizing the file with vector animation, which we got at Stage I, in Macromedia Dreamweaver MX. After that, the movie we got at Stage II is encoded in Windows Media Encoder 9 into a format, suitable for broadband Internet broadcasting. In conclusion we do a final edit of the web site in Macromedia Dreamweaver MX and scan the system for viruses using McAfee VirusScan 7.0.

2. Office Productivity. This group consists of the following subtests.
   • Communication. It emulates receiving emails in Microsoft Outlook 2002. While an email with zip attachments is being received, McAfee VirusScan 7.0 is scanning the system for viruses. In conclusion we use Microsoft Internet Explorer to browse fragments of the web site, created during Internet Content Creation test.
   • Document Creation. Editing a document in Microsoft Word 2002. Dictating a document using ScanSoft Dragon NaturallySpeaking 6. Creating a presentation in Microsoft PowerPoint 2002.
   • Data Analysis. Working with a database in Microsoft Access 2002. Archiving documents with WinZip Computing WinZip 8.1. Importing query results into Microsoft Excel 2002 and creating diagrams.

The result is Performance Rating - notebook performance versus a reference platform. We calculate the rating for each group of applications, the rating of subtests and the total system rating. Each potential user may consult the results he/she is interested in. This version of the test package is approved by AMD, who previously made a complaint about the performance rating procedure in BAPCO tests. You can read the details about this test, scoring procedures, and other aspects of SysMark 2004SE in this document (pdf format, English, ~1.5MB). Unfortunately, the current version of the test script for Adobe Premiere 6.5 is unstable with HD Audio cards in notebooks based on Intel 915M chipset. That's why the Internet Content Creation test is not always correct. We hope that this problem will be fixed.

We are going to use BAPCO SysMark 2004SE together with stable SysMark 2002 for some time. Unfortunately, this test is not subdivided into groups. Besides, program versions are already old, though the general ideology is the same.

3D performance tests

The main test is FutureMark 3DMark03. We use Patch 3.30. Test settings are by default.

Portable/Laptop power scheme. The results - 3DMarks for all resolutions supported. In some cases a review of a new controller may also include fps values for games from the test package, but with their parameters set not by default (mainly with different settings). You can read about the test in this document (pdf format, English, ~1.1MB). Reviews of new graphics controllers may also use notebook results in real games. In our opinion, it makes sense to test integrated video controllers only once - when a new representative appears on the market.

Memory bandwidth tests

This test confirms (when the results are close to the expected) that the notebook assemblage is of high quality. If the results obtained are much lower than the expected, we can establish a fact that the notebook has problems. We plan on using RightMark Memory Analyzer. Always On power scheme. We plan on publishing Quick Test results - average and peak memory throughput for reading and writing.

In our opinion, the current test procedure offers an adequate evaluation of notebook capacities. The set of tests may be expanded in future.

Write a comment below. No registration needed!