Browser Benchmarks November 5 2010

It's been a few months since I last benchmarked web browsers, so I figured it was time for an update. There's just something innately fun about running benchmarks. These are first and foremost for my curiosity and amusement, not because I think it matters if Browser A is 0.3% faster at Sunspider than Browser B. However, at least last time there were very measurable gaps in performance and results of each browser - more than I would have thought.

This time I'm going to separate the results of official stable browser releases and beta releases.

Browser releases continue to be frequent, and I don't think it is going to slow down. Chrome in particular appears to now plan on major releases every 6 weeks. Last benchmark was Chrome 5 - this time it's Chrome 7. Opera has some modest improvements with 10.63, Firefox 4 is on beta 6 now, up from beta 1. IE 9 Platform Preview has also gone through some more iterations (though it is hard to find)

System specs are the same as last time, except for new video card drivers, which may affect 3d accelerated HTML5 tests.

Environment:

• Intel Core i5 750 @ 3.36GHz (Turbo Disabled)
• 4GB RAM
• 80GB Intel SSD (G1)
• ATI Radeon HD 4850 (Catalyst 10.10 drivers)
• Windows 7 64-bit Home Premium

Fresh boot. No other apps or system tray programs running aside from Microsoft Security Essentials.
Each browser was run by itself, with only one tab for the benchmark itself.

I'll start with Sunspider again. Only this time with Sunspider 0.9.1 (I didn't realize that was available last time):

• Chrome 7.0.517.41 - 208.3ms +/- 0.5% [Full Result]
• Opera 10.63 - 224.4ms +/-0.5% [Full Result]
• Safari 5.02 - 291.3ms +/- 0.6% [Full Result]
• Firefox 3.6.12 - 638.1ms +/- 2.9% [Full Result]
• IE8 - 3614.1ms +/- 0.2% [Full Result]

 Separately, beta browsers:

• IE9 Platform Preview 6 - 249.1ms +/- 1.0% [Full Result]
• Firefox 4 Beta 6 - 364.4ms +/- 3.5 % [Full Result]

No real surprises - Chrome continues to lead, but IE9 and Firefox 4 betas are catching up to the 3 leaders. There is much less spread than last time.

I also decided to run The Kraken, a new benchmark the Mozilla team released in September.This will be the only benchmark Firefox 4 beta will overtake the others in. I am curious how accurate of a representation this benchmark will be of the future.

• Opera 10.63 - 10515.7ms +/- 0.5% [Full Result]
• Chrome 7.0.517.41 -12046.8ms +/- 0.4% [Full Result]
• Firefox 3.6.12 - 14467.1ms +/- 0.1% [Full Result]
• Safari 5.02 - 14589.5ms +/- 0.1% [Full Result]
• IE8 - I did not wait for this to finish.

Separately, beta browsers:

• Firefox 4 Beta 6 - 9768.9ms +/- 0.1% [Full Result] 
• IE9 Platform Preview 6 - 21015.6ms +/- 0.2% [Full Result]

Firefox 4 beta 6 leads Opera 10.63 by a slight margin. Chrome trails by a bit more, followed by Safari and Firefox 3.

Peacekeeper:

Firefox 3 appears twice because I didn't realize 3.6.12 was out at first. All browsers except Opera see gains from last time. This could be due to my updated graphics driver.

ACID3:

• Chrome 7.0.517.41 - 100/100
• Safari 5.0.2 - 100/100
• Opera 10.63 - 100/100
• Firefox 3.6.12 - 94/100
• IE8 - 20/100

Beta Browsers:

• Firefox 4 Beta 6 - 97/100
• IE9 Platform Preview 6 - 95/100 (noticeable improvement from preview 3)

CSS3 Selectors Test:

• Chrome 7.0.517.41 - Passes 574/574 tests
• Safari 5.0.2 - Passes 574/574 tests
• Opera 10.63 - Passes 574/574 tests
• Firefox 3.6.12 - Passes 574/574 tests
• IE8 - Passes 345/574 tests

Beta Browsers:

• Firefox 4 Beta 6 - Passes 574/574 tests
• IE9 Platform Preview 6 - Passes 574/574 tests

I must say, the IE9 team seems to be taking web standards pretty seriously. I recommend taking a look at their test center. This will be good for everyone. I'm not a web developer currently, so I can't say if they are truly implementing everything that's important, or just a higher amount of obscure features, but in the end it is a net win. One of the reasons I've always liked Opera is that they strive to adopt web standards.

I noticed something weird in Firefox 4 and Firefox 3.6.12 (it must be a recent thing). By default, when I type into the search box after a fresh install - nothing happens. The browser appears to now require search providers, a la IE, except it doesn't come with any by default.

UPDATE: This is why beta browsers are listed separately. I installed Firefox 3.6.12 on my work PC and it does not have this issue - neither does Firefox 4 beta 6, once it is installed The above must be a fluke. My guess is that one of the Firefox 4 betas somehow corrupted or cleared out the search engine extensions, and this also affected Firefox 3.6.12 on the same machine.

Overall the browser situation is getting better and better. When IE9 comes out, IE8 can finally start going away - it is really the only browser that I can't recommend. Most of the rest come down to personal choice. 

Do you want the absolute fastest browser, but with a few less features? Chrome.
Do you want the ultimate in customization, if you want to download some extensions? Firefox.
Do you want more features built in, and almost as fast as Chrome? Opera.
I'm not sure if there is a compelling reason to use Safari 5 on Windows, but it also isn't a bad choice.

Until next time.

Browser Benchmarks - July 8, 2010 - Ready. Set. Fight!

Update: More recent benchmark

Yesterday my Opera browser at work auto-updated to 10.60. Opera's auto update is finally coming in line with Firefox...still not quite as nice, but it's getting there. More interesting was the dev blog about 10.60, and I saw elsewhere that Firefox 4.0 beta 1 was out, and also checked up on IE9 to see how things were progressing. Somewhere along the way, I also found the Peacekeeper benchmark and decided I wanted to benchmark some browsers to see what the current state-of-play is. I will be using that for overall (HTML5 / DOM / Javascript) performance, and Sunspider for pure Javascript performance. I'm also throwing ACID3 in there for a sense of where each browser is standards-wise. 

Environment:

System: Intel Core i5 750 @ 3.36GHz, 4GB RAM, 80GB Intel SSD (G1), ATI Radeon HD 4850

OS: Windows 7 64-bit Home Premium

Fresh boot. No other apps or system tray programs running aside from Microsoft Security Essentials.

Each browser was run by itself, with only one tab for the benchmark itself.

First off, I'll start with Sunspider. These are the final numbers, but I have linked the full results for each. I don't have any fancy graphs, so I will order them fastest to slowest.

1. Chrome 5.0.375.99 - 224.0ms +/- 2.0% [Full Result]
2. Opera 10.60 final - 231.4ms +/- 1.5% [Full Result]
3. Safari 5.33.16.0 - 273.4ms +/- 2.3% [Full Result]
4. IE 9 Preview 3 (1.9.7874.6000) -  293.6ms +/- 0.7% [Full Result]
5. Firefox 4.0 Beta 1 - 406.8ms +/- 1.9% [Full Result]
6. Firefox 3.6.6 - 575.6ms +/- 1.1% [Full Result]
7. IE 8 - 3555.8ms +/- 0.6% [Full Result]

Chrome has the lead on pure Javascript performance with its V8 Javascript engine, but Opera is not far behind. It's only trailing by 3.3%, barely more than the margin of error. Next up is Safari, taking 22% longer. IE9 Preview is showing great promise, at 31% behind Chrome. 

Firefox 4 beta is not in the same ballpark currently, at just more than half the speed of Chrome. It is a good increase over Firefox 3.6.6, which would've been almost a 3x difference, but it is a ways off. The big takeaway is that the IE9 team appears to now be ahead of the Firefox 4 team on Javascript performance...times are certainly changing.

IE8 is included as a baseline, and because a lot of users are still on IE.

Next up is Peacekeeper, for this I have screenshots and the full results here.

Opera 10.60 leads here overall. Chrome is close, at 16% lower score. It's worth noting just how far ahead Chrome and Opera are over every other browser in the overall category. Safari 5 is next, at less than half the score. Firefox 4b1 is on the heels of Safari, and then there are the rest.

What's interesting to note here is that even though Safari & Chrome are both webkit based browsers, clearly the Chrome team is going the extra mile on performance.

I want to drill into what makes up these scores, as this benchmark is new to me, but I think that will be in a follow-up post as this post is already almost a week late if you look at the date..

Finally, lets look at ACID 3.

• Chrome 5.0.375.99 - 100/100
• Opera 10.60 - 100/100
• Safari 5  - 100/100
• Firefox 4 Beta 1 -  97/100
• Firefox 3.6.6 - 94/100
• IE 9 Preview 3 -  83/100
• IE8 20/100 (FAIL)

Chrome, Opera, and Safari all receive full marks. I did not compare pixel to pixel, but they have had good track records with ACID tests. Firefox 4 Beta 1 is getting close to passing, slightly better than Firefox 3.6.6. IE9 has come a long way from IE8 but still has a ways to go in order to pass ACID 3.

Regardless of your preferred browser, there is some intense competition now in Javascript engines and general rendering performance, resulting in the experience improving for everyone. It is an exciting time, and I don't think you can go wrong with Chrome, Firefox, Opera, or Safari. Rendering and performance are more than good enough in all of them, so it comes down to other usability / features. Once IE9 is closer to final, I may even be able to recommend that.

Update: More Recent Benchmark