And now for something completely different.
My past as a Commodore 64 demo programmer means that I find benchmarks quite irresistible. What is the fastest way to accomplish a given task?
Following Steve Jobs’ denunciation of Flash as slow, Sean Christmann recently posted his Guimark2 test suite for comparing vector, bitmap, and text rendering across platforms. Sean built Flash and HTML versions of the tests. One of the clear results was that HTML5 just isn’t that fast yet.
But being mostly a Java programmer, I couldn’t help building Java versions (perhaps I wanted to counter the perception that Java is slow). You can try the Java tests here, Vector, Bitmap, Text and compare them to the tests on the Guimark2 page.
Here are the results from the two machines I tested on:
| Test/OS | XP, HTML | XP, Flash | XP, Java | OS X, HTML | OS X, Flash | OS X, Java |
|---|---|---|---|---|---|---|
| Vector | 11.9 | 21.9 | 20.7 | 4.5 | 19.8 | 31.9 |
| Bitmap | 4.3 | 12.7 | 208.2 | 13.0 | 12.9 | 42.0 |
| Text | 19.2 | 1.5 | 11.4 | 29.9 | 16.6 | 14.1 |
Test results in Frames Per Second (Higher is better)
As you can see, on my two test machines, the Vector test is fastest in Java on OS X, the Bitmap test is fastest in Java on XP (5 times faster than any other platform/language), and the Text test is fastest in HTML on OS X.
Is this always the case? No. As can also be witnessed on the Guimark2 page, these results are highly variable between different OS configurations, browsers. As far as I can make out, really high Java bitmap scores require an ATI or NVidia graphics card, and may not always be present on Windows Vista or 7. On OS X, Flash runs faster in 32-bit Safari, while Java runs fastest in 64-bit Safari (tested here). And so on.
Benchmarking is a lot more complicated than it used to be, but I just thought I’d share these results. The tests also link to the source code if you are interested. Feel free to post your results here or on the Guimark2 page.
PS. Test machines:
- 2007 Lenovo ThinkPad T60p running Windows XP, Firefox 3.6, Java 1.6, Flash 10.
- 2010 15″ MacBook Pro running Snow Leopard, Safari (64-bit), Java 1.6, Flash 10.
Very interesting benchmarks, but I’m not really surprised by the results. Java was always fastest that Flash.
But, speaking of browser-based technology, I think speed is not the only factor that developers must care about. Broadcasting and accessibility are almost as important.
IMHO, this is the main reason that led Flash to “beat” Java as a browser-game devolepment platform. HTML/JS is great, but slow and very dependant on the browser used by players.
The use of a plugin is a good solution to this issue, as long as your players do not have to install the plugin. Too bad for Java, Unity, Shockwave, Vitalize, Silverlight and so on. They are/were more robust technologies, but with too few plugin penetration.
@DDams Yes, I agree completely. The installed base beats everything else. You may have the most amazing technology – it doesn’t matter.
Here are some stats about the installed base of different plugins. http://www.statowl.com/plugin_overview.php
This puts Flash is at 96%, Java 80%, Silverlight 45%. I couldn’t find any numbers for Unity.
Is it a typo, or Bitmap in Java is actually 5 times faster on XP than on Safari?
(sorry for the double post …)
For some reason I did not see this line “the Bitmap test is fastest in Java on XP (5 times faster than any other platform/language)” … So my actual question is: why so much difference? Is it a JVM issue with OS X?
@gentimouton The bitmap test really is that much faster on XP. (Someone reported getting 600 FPS on their Windows 7 machine too.)
The reason seems to be that Java 1.6_10 introduced a Direct3D pipeline for rendering graphics on Windows.
When this is active, graphics should be drawn directly using your graphics card, giving a major speedup. The bitmap test probably gets the largest speedup because GPUs excel at drawing multiple copies of the same bitmap.
There exists a similar OpenGL rendering system that you’d imagine could work on OS X, but it does not seem to be available on that platform.
i believe your java bitmap you linked here is the wrong version
use the one you posted on post 47 on sean tread
i used it and men i went fro 39 fps to 560 fps huge leap
now lets find ways to apply this to our daily use because clearly nobody tweak with those in mind there are probably lot of other tweak avail we dont yet lol since ecmascript 5 is such a langage
that its hard to wrap our mind around the concept they use!
good work by the way.
by the way flash and java both use ecmascript.
@carol I am linking to the same site from here and from the GuiMark2 page, so it should give the same results!
ty men in your link to sun i found what my problem was
this work only if you have window classic view in w7 (me its 64) or vista
thats what it worked before and wasnt working now if you find other with slow problem ,this is probably the reason.men i love this tweak 516 fps on java!hell ya now if some programer can find a use for this tweak. i hear there is an opengl version that make java work the same but its way slower then this lol
@ddams:ecmascript is java and flash.both use ecmascript today.
the main issue with ecmascript was it was deemed to simple.
not versatile enough even thos flash use the same base lol.
bitmap is graphic right?and ill play a major role in the coming years .if the difference between the op way and regular method
was just 2 time faster i wouldnt bother to post again ,but me on w7 64 bit with an average machine(today spec)going from 39 fps regular bitmap to 516 fps poster way.this is a huge leep .
bit enough to warrent investigating.i checked with flash same bitmap test i got 18 fps .im sorry but the difference is just to great everything is visual now a day.ecmascript has been used by java and flash for years .now with the new ecmascript5 and the rest the futur couldnt be brighter
your idea is great but it cant be applied because there is a major bug at microsoft end!
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6670586 thats why we havent seen this pickup.sun isnt moving till microsoft fix their problem since we know ms isnt very fast on adapting to latest.this might take a while .shame tho because as we saw here tthe idea has tremendous potential for the cloud and the web!
@carol It’s a little unclear to me if the same problem will always occur on Windows 7.
Update: I also get 200FPS on the same machine now that I have installed Windows 7 on it.
vista and w7 only.since the bug is related to aero.
as for the link i posted that prevent hardware acceleration its because of the gdi-d3d relashionship.ms didnt fix it in the past.one thing is sure its very easy to get 500 fps on bitmap (with your tip)with a computer less then a year old on w7 64 bit!my last try did 605 fps(set to 600)(channel interleave set to 12 bit instead of xor,bios)
been toying with ubuntu.java acceleration in there is inconsistant.but the idea you provided as potential be it trough opengl or ms but it will need to be polished or a walktrough from java to make sure we get consistant result
but i get 580 fps on java bitmap with an amd 550 ,ati 5770.
easy way to garanty 2d acceleration?
leave the ressource monitor running
and run you version of window in classic view
(always work for me whatever im doing,only way i found to garanty acceleration
sadly this week,mozilla made a change in minefield(firefox)the result!your trick no longer work.they probably doont know thias tweak or they would have enabled it has much as they can in the browser.sad week for the browser users!
this trick doesnt work anymore because of a ms change.i do hope ms play correct this and play nicer with java cause skype use java.and going from 600 fps to 37 fps because of ms idea is plain wrong
the new update ms made avail this week brought back some speed we were at 37 fps now we re at 150 fps still far from the 600 we had but better then what we had.
with latest java build we re back to 600 +fps in java bitmap test (cheer)
i am happy to report we re in the 1000 to 1100 fps on this test.this tweak work great for java user playing say oh runescape!i had some 1300 run but i think i changed some tweak that affected these fps!but i am very happy with these speed!this will be my last post on this subject!just thot i would let people know the evolution!