The OnePlus 3T and the Meizu Pro 6, smartphones from two of the most promising up-and-coming manufacturers, have apparently been cheating on their benchmark tests, making their phones appear to be faster on paper than they actually are.
Benchmark cheating seems to be back
XDA-Developers reports that the practice of cheating on benchmark tests has reared its ugly head again after being beaten down years ago following the discovery that almost every smartphone manufacturer except Google/ Motorola was cheating. The point of benchmark tests is to show just how fast smartphones are, and manufacturers were taking steps to make their phones be faster, but not in ways that would actually impact real-world usage.
In other words, they boosted their smartphones enough to perform better in the tests but not in ways that would benefit real users. Some of the practices that were being used included forcing clock speeds to their highest settings to create special states of higher power or special clock speeds that only became available during benchmark tests. And all of these efforts were performed only to gain “a couple percentage point increases in benchmark,” explains XDA-Developers.
Continued from part one... Q1 hedge fund letters, conference, scoops etc Abrams and his team want to understand the fundamental economics of every opportunity because, "It is easy to tell what has been, and it is easy to tell what is today, but the biggest deal for the investor is to . . . SORRY! Read More
OnePlus accused of cheating on benchmark tests
The group states that all the developers that were caught red-handed cheating on the benchmark tests quit due to the widespread public outcry. Primate Labs made its Geekbench 4 test much longer than Geekbench 3 to curb OEMs’ ability to cheat on the benchmarks, but apparently, some companies have figured out how to beat the new tests and haven’t been swayed by the backlash following the previous bout of cheating.
This time, XDA discovered “something strange” when testing the OnePlus 3T that couldn’t be reproduced on the Xiaomi Mi Note 2, the Google Pixel XL or other devices with Qualcomm’s Snapdragon 821 chip. Editor-in-Chief Mario Serrafero was looking at the way Qualcomm boosts the CPU clock speed when opening apps, and he saw that some apps on the OnePlus 3T didn’t fall back to their “normal idling speeds” after they actually opened.
A special build of Geekbench
After further testing, the developers concluded that the OnePlus 3T was targeting specific apps and benchmarks by name and then “entering an alternate CPU scaling mode to pump up their benchmark scores.” They said that although the OnePlus 3 wasn’t cheating when they reviewed it, a software update added it, which demonstrates that it’s not enough to just run tests when a phone is first released.
They contacted OnePlus regarding the results of their tests with a special secret Geekbench build, and the manufacturer told them that it would stop targeting benchmarking apps, although it will still do so for games.
A whole new extreme of benchmark cheating?
XDA-Developers then began testing phones by other manufacturers and learned that HTC, Xiaomi, Google, Sony, Huawei and a number of others appear to pass the test using a special Geekbench build. However, they believe that the Meizu Pro 6 Plus running on the Exynos 8890 is also cheating. In fact, they said the phone “took the benchmark cheating to another extreme.” The phone offers a “Balance Mode” and “Performance Mode,” but they call this toggle “nothing more than a parlor trick,” as the “Performance Mode” performs no better on the special Geekbench build than the “Balance Mode.”
It seems as if the phone targets benchmarking apps because upon opening one, the phone recommends that the user switch to Performance Mode. This does boost the phone’s performance in the benchmarking apps by targeting them by name, but when using the secret Geekbench build, apparently “Performance Mode” does no better than “Balance Mode.”