List of Android Benchmarks, Optimizations Results & Benchmarks Analysis

Linaro Connect Europe occurred in Copenhagen on Oct. 29 – Nov. 2 2012 (LCE12) and included 3 mini-summits about Android, big.LITTLE and ARMv8. Linaro has recently posted the presentation slides and I’ll have a look at a few of those slides in details and try to post information that I feel can be interesting.

Today. I’ll go over the “Benchmarking and Optimization Opportunities” slides, where we can learn which Android Benchmarks Linaro use, how they’ve decreased benchmark results variance, which parts of the system are actually tested by benchmarks (profiling), and what they plan to do to further optimize Android on ARM.

When I do some reviews, I usually simply use Antutu and Quadrant benchmarks to assess the performance of the devices, but Linaro uses many more benchmarks which I’ll list below. I’ll provide 2 links for each benchmark. The first link is to the official website and the other link(s) to Google Play.

• AndEBench – This is an Android benchmark by EEMCB (The Embedded Microprocessor Consortium Benchmark) that focuses on CPU and Dalvik interpreter performance, and gives you two numbers of iteration: one for native mode, one for Java mode. It will show graphs with comparison to a limited number (5) of other Android devices. It is available free of charge on Google Play.
• Antutu – Antutu must be the most widely used benchmark for Android. It gives you a total score as well as separate score for RAM, CPU integer, CPU float, GPU 2D, GPU 3D, database I/O and SD card read/write speed. You can download it on Google Play. (Note: Version 3.0 will be release on the 26th of November with several improvements, and scores won’t be comparable to 2.x versions)
• Linpack – The LINPACK Benchmarks are normally used to measure the floating point computing power of a given processor, but Greene Computing, the company behind this benchmark, explains that in Android, “this test is more a reflection of the state of the Android Dalvik Virtual Machine than of the floating point performance of the underlying processor”. There are 2 versions in Google play: Linpack for Android (free) and Linpack Pro for Android ($1.99) . The only difference between the two is the Pro version does not show ads.
• CaffeineMark – This benchmark is probably the reference when it comes to test Java performance on different platforms/operating systems. The Android app shows the results in text format for 6 different tests and an overall score,  There’s no database to compare with other devices.
• GLBenchmark – GLBenchmark measures and compares the performance of OpenGL ES implementation of different mobile devices. The tests focus on graphics and computation capabilities. This is one of the most demanding benchmark to test the GPU 3D capabilities. The app is available free of charge on Google Play.
• GeekBench 2 – Geekbench measures the CPU (float and integer) and memory performance of your processor. It is available for Android, iOS, Linux, Mac and Windows, and features an online browser. The Android version costs $0.99 on Google Play.
• Quadrant – Another widely benchmark for mobile devices, capable of measuring CPU, memory, I/O and 3D graphics performance. There are three versions: Standard (free), Advanced ($2.99) and Professional ($24.99). All versions perform the same benchmarks, but the standard version displays ads, and does not allow to submit your device score. The Advanced and Professional version appear identical but commercial use is not allowed by the Advanced version.
• Vellamo – Vellamo is designed by Qualcomm and measures HTML5 & CPU performance, as well as some extra tests such as touchscreen responsiveness, advanced video and octane (Google Javascript benchmark). Vellamo benchmark can be downloaded for free on Google Play.
• NBench – Android NBench is a native port of BYTE’s BYTEmark CPU benchmark. The benchmark measures CPU, FPU, memory and C compiler performance for ARMv5, ARM v7 and Mips32 CPUs. It is a free download on Google Play.
• Methanol – Methanol is JavaScript based browser page load benchmark. This is Javascript and HTML code, so there’s no app to install. You need to copy the files in your device, start a web browser and pass the parameters in the URL. For example: http://localhost/fire.html?iter=6&skipped=1&synchStart=0&reportToUrl=1

Those are all the benchmarks integrated into LAVA at Linaro, so that they can test the performance of the software developed by the Linaro team. There are also other benchmarks such as RgbenchMM & RgBandwidth, but the ones listed should be the main ones.

You may have seen the demo that shows Linaro Android is 100% faster than Stock Android on TI Pandaboard. Finally, the massive performance improvement was due to another benchmark software (oxBench) that use vsync to render images, where a small increase in actual performance may lead to a doubling of the fps. Still the optimization work at Linaro does pay off, and they provided some other results with AOSP + Linaro Patches build on Panda 4460 measured
with 3.2 TILT stable kernel vs AOSP build with 3.2 kernel:

• Antutu RAM 21 %
• CaffeineMark String 24 %
• Quadrant Mem 81 %
• Antutu Float 10 %
• 0xBench Sunspider 10 %
• 0xBench DrawArc,DrawCircle,DrawRect >40 %
• 0xBench Garbage Collection 11 %
• Vellamo Sunspider 14 %
• Vellamo V8 Benchmark 8 %

They’ve also profiled the benchmarks and provided a large table in the presentation slides that shows what part of Android (Compiler, libc, JIT, libdvm, libskia, libm, kernel, webkit or V8 Engine) each benchmark is actually mostly testing. Most benchmarks use the compiler and/or JIT a lot, except BBench (a web-page rendering benchmark) that mainly gives an idea of webkit performance.

The final part of the presentation is about optimization opportunities, and they expect to further leverage optimization for libc & libm, work on optimizing Bionic, Dalvik JIT and V8, as well as optimize code than runs on ARM Cortex A15 processors.

Linaro also posted videos but in a format I don’t really like as it is not per presentation, but per mini-summit. This means each video lasts 2 hours and those are Google Hangout recording which sometimes show just the audience and when the slides are shown they are not always readable.  I tried to watch during the summit, but I quickly gave up.

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress