The Magento Enteprise 1.13 performance benchmark

When Baruch Toledano (Magento) told us that he wanted a meaningful and realistic benchmark of Magento Enterprise 1.13 with a real customer traffic, we were delighted. He came to us since we are doing benchmark & optimization with Magento since August 2008.

We told him it would not be a complacency benchmark or a kinder garden walk and that we’ll give hell to the software to really test it to the ground. We then called our friends at AdFab, with whom we’re working with for years on the GEMO customer. This duet (NBS & AdFab) is behind the customer site for years and we already made a lot of optimization, tests, tweaking for them altogether.

So, as you understood, this is a co-production between :

Test protocol

We decided to use a pretty harsh testing protocol, to be as realistic as possible. We captured the traffic of the first day of sales in France (which took place on 5th of January 2013), multiplexed and randomized it (see below) and reinjected it to an exact similar copy of the codebase + infrastructure.

The Multiplexing purpose was to stay realistic in the injected traffic (GET & POST, front & backend work) and be able to amplify the traffic. To put it short, we started to replay the logs at different offsets and randomized the queries to avoid the cache performance measurement to be impaired by the repetitions of the same session.

Test environment

  • Infrastructure : 2 front servers (16 cores i5540 / 16 Go of RAM) and 1 database server (20 cores, 40 Go of RAM, also hosting the Redis / Memcached cache systems)
  • Disks & backbone : Stored on a NetAPP NAS, 10 Gb/s backbone
  • Linux & virtualization : Debian v6.0.1 & Xen, kernel 2.6.32
  • LAMP stack version : PHP 5.3.3.7, Zend engine 2.3.0, Mysql 5.1.49
  • Verrsion of the Virtual machine image (internal reference of NBS, V18 Magento optimized env)
  • Customer : Gemo / Eram
  • Catalog size & structure : 25000 simples products, 7000 configurable, 300 bundled
  • Normal daily traffic: 80 000
  • Peak traffic : 150 000
  • Various : the infrastructure provides Nginx Reverse proxy in front (in all setup), no local varnish booster was installed

IMPORTANT Setup precision:

The 1.9 and 1.12 versions were using Memcached for fast cache backend and sessions storage and database for the slow cache backend.

The 1.13 was configured with the Redis cache backend in fast & slow, sessions were still stored in the memcached. A very meaningful part of the performance boost seen below comes from the usage of Redis which is now supported natively by the Magento 1.13 version.

Magento & customer codebase version

The customer codebase was the exact same version across all tests, whatever version of Magento EE was used, except of course the local.xml file.

The Magento versions used & tested where the following :

  • Enterprise 1.9.1.1
  • Enterprise 1.12.0.2
  • Enterprise 1.13-alpha-2

Results

We were able to run two types of tests, unit tests made to check a precise performance enhancement on a specific field of functions and Real traffic injection, which consists in replaying a Log (Get & Post) of a day of work (see above).

Unit test (Scenarii)

Scenarii structures

Unit Load tests
  • Homepage scenario : we only call the homepage.
  • Catalog scenario
    1. We call the homepage
    2. then a random category page
    3. then a random product page

(The randomization technic is based on a xquery path giving dynamicaly the list of available categories / products. A link is then picked randomly.)

  • Add To Cart scenario
    1. The homepage
    2. a random category page
    3. a random product page
    4. add to cart

(The add to cart is based on a xquery path giving super attributes among which a simple product is chosen.)

  • Checkout scenario
    1. The homepage
    2. a random category page
    3. a random product page
    4. add to cart
    5. cart page
    6. register account
    7. saveMethod
    8. save Billing
    9. save ShippingMethod
    10. payment (check)
    11. save order
    12. success page

(The register account is randomly generated.)

Raw results

Homepage Catalog
UV/D RT UV/D RT
1.9 EE 353 664 2,22 285 768 5,15
1.12 EE 431 928 0,18 457 561 3,44
1.13 EE 432 072 0,08 775 872 0,62

(UV/D means Unique Visitor per Day, RT = Average Response Time)

Add To cart Checkout
UV RT Adds/min UV RT Checkouts/min
1.9 EE 106 488 9,25 194 16 416 0,83 10
1.12 EE 144 576 8,75 251 31 104 1,21 18
1.13 EE 183 168 6,25 320 68 256 1,87 40

Analyzes

The 1.13 version we tested was bringing a meaningful boost in performances. The unit test of homepage loading made a gap between 1.12 and 1.13 of 56% improvement, meaning the core load time, was on average 56% lower than with the 1.12. On the catalog browsing, the gap was even deeper with a tremendous boost of 82% when comparing 1.12 and 1.13 only.

The add to cart test included homepage landing, an access to a random page of the catalog, a random product page and finally an add to cart action. (the products are configurable products). The average time for the scenario playing of the 1.9 was 9,25 seconds, which lowered to 8,75 in 1.12 and a significant 6,25 second in 1.13.

More interesting or readable, the number of “add to cart” per minute ramped up from 194 in 1.9 to a blazing 320 in 1.13. If we include the full checkout we jump from 10 per minutes to 40, which is more than doubled between the 1.12 and 1.13 and 4 times more than with a 1.9.

Also noticeable, we were able to make 775 872 “virtual” people browse the catalog over a day with a 1.13 where only 457 561 could be hosted in a 1.12 (and 285 768 in a 1.9).

Real traffic

Method

AdFab has developed a LogAnalyzer software using Talend ETL : It recompose sessions in an Apache log file and create a format ready to be played by our loadUI scenario. They also developed a Groovy LoadUI component able to parse randomly or sequentially the file. This way, we could replay a real life e-commerce experience. We could also “multiplex it (playing it quicker or from multi-injectors) or randomize it.

For the test, we’ve taken the log of a normal sales day on gemo.fr : 35 000 unique visitors and 474 checkouts.

We have to precise that for the POST data, we’ve proceeded this way :

  • add to cart : color and size of the product chosen by the customer
  • register : random creation
  • Login : a randomly picked testing login
  • payment : we switched from a bank payment to a check payment

Results

Realistic
UV RT
1.9 EE 137 928 4,6
1.12 EE 158 231 3,2
1.13 EE 166 584 1,87

Analyzes

The 1.13 EE version ramped up to traffic easily with a load time per page (all pages, from the homepage to account creation and so on) stayed below 1,87 seconds, which is nearly two times faster than with the 1.12 and 3 times than with the 1.9.

From our complementary finding, we could gear the traffic up to a million unique visitor per day with the exact same setup (2 fronts, 1 DB) where we usually struggle to get past ~250 000 unique visitor per day with 1.9 and 400 000 with a 1.12. That would represent a time four compared to the 1.9 and a 2.5x , with the exact same infrastructure.

Conclusion

Redis changes the deal a lot and makes a huge difference. The codebase itself benefited from a nice lifting and is already itself bringing more performances. We would have loved to test it with the same memcached environment also but had no time for it. Still the performance increase is very meaningful and the add to cart and checkout benefit a lot from this new release, allowing merchant to scale better, without soaring their hosting costs.

Credits

This test was a race against time. Even if realistic, we could have done better and more accurate, more scenarii, with variations (memcached, redis, etc) and many other points. Still, it was a great achivement and making a realist traffic load test was a performance we are proud of.

  • NBS System provided the infrastructure, tweaking and support to both AdFab & eBay / Magento team on top of the usual super optimized Magento environment.
  • AdFab constructed the scenarii and more important the tool to manage “real traffic” injection and runned the benchmarks
  • Magento Inc (eBay group) provided support and bug correction along with the 1.13 alpha 2 core private release
  • A great & massive thanks to all involved eBay members that helped a lot
  • Gemo provided the authorization to use the logs and codebase along with its name to showcase these results

Feel free to contact us for realistic load test & traffic simulation with your own logs or to get either Magento optimized managed services (NBS System) or Development (AdFab).

  • NBS System team members involved: Philippe Humeau, Adrien, Florent, Jean Baptiste et Vincent.
  • AdFab team members involved: Gregory Besson, Matthieu, Hervé, Emmanuel, Arnaud.
  • eBay / Magento team members involved: Baruch Toledano, Joe, Keren, Krupa, Muna, Maksym, Vivek, Nicholay, Andrey and Max.
  • Also a massive thank to Gemo’s team member Christophe Danion & Laurence Desterke-Aznar.

[no_toc]

Philippe Humeau
Philippe Humeau
Philippe co-founded NBS System in 1999. After a focus on cybersecurity, which he never gave up, he discovered a passion for e-commerce from 2008 on. Pentester, CTO, CCO then CEO, Philippe’s multifaceted profile drove him to becoming OT Group’s Marketing and Strategy Director.