eValid™ -- Automated Web Quality Solution Browser-Based, Client-Side, Functional Testing & Validation, Load & Performance Tuning, Page Timing, Website Analysis, and Rich Internet Application Monitoring. © Copyright 2000-2012 by Software Research, Inc.

Summary
Using eValid LoadTest capability we ran a series of experiments that involved driving a mobile web application in ramped loading scenarios, up to 1,000 BUs in parallel. The results we generated strongly suggest that a performance limitation is an easily recognizable bottleneck in the infrastructure of the mobile application web server stack.

Chart of Derived Internal Response Times (Click to See Full Image)

Conclusions About Capacity
The chart graphs the basic page download time of the retrieval of train departure details into the mobile device, as a function of the total number of simultaneous users, which grows linearly over the 30-minute scenario duration. This chart suggests that the server-stack I/O capacity has been reached. You can see on the curve that the delays begin to build at a level of about 150 BUs.

Working backward and assuming linearity in the driven load, we estimate that the actual I/O channel capacity of the application server is 10 Mbps.

Technical Details
Here are the technical details about how this experiment was done.

• UserAgentString Setting
  The script sets evalid to the iPhone UserAgentString (UAS) as follows:

  "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+
  	(KHTML, like Gecko) Version/3.0 Mobile/1C25 Safari/419.3"

• LoadTest Ramp Schedule
  The ramp scenario is 1[1]1,000 BUs at the rate of one new BU per second, so the ramp up required just over 1,000 seconds = 0.27 hours = 16.67 minutes. To do this we used 10 separate user desktops on one machine, allocated at 100 BUs per desktop.

• Driver Machine Details
  For this experiment we used a machine image on the Amazon Compute Cloud. The driver machine was an Amazon EC2 "m2.4xlarge" instance, which has 26 ECUs (Equivalent Computing Units), 8 cores, 68.4 GBytes RAM, and I/O bandwidth = 1 Gbps (1,000 Mbps).

  During the experiment we observed the percentage of CPU utilization when all 1,000 BUs were running was ~45%. At the same time, the percentage of RAM used by all 1,000 BUs was ~20%.

• Test Download Volume
  We measured the total amount of material downloaded by a singleton playback as 443 KBytes. This measurement was done by running the script one time, while noting the total Bytes Downloaded in the test summary page.

• Steady State Driver I/O Channel Usage
  In steady state, with 1,000 BUs running in parallel, each BU implies 443 KB/cycle downloaded (singleton data volume), completing at ~325 secs/cycle.

  The channel on the driver machine is 1,000 Mbps, so this I/O load is only ~10% of the driver machine channel capacity.

• Driver Machine Desktop Screenshot
  Here is a screen view of the driver computer with 10 users' desktops operating in parallel. If you look carefully you will see that each of the user desktops shows ≥ 100 BUs running per desktop, for a total of ≥ 1,000 BUs overall.

  
  Click For Full-Size Image

  
  

Playback Script
The playback script is shown here. The actual URL and some of the script details have been redacted.

# # Recording by eValid V9 # Copyright © 2011 by Software Research, Inc. # Recording made on: Microsoft Windows Vista # ProjectID "LoadTests" GroupID "Demos" TestID "Mobile-1000" LogID "AUTO" ScreenSize 1280 1024 FontSize 0 DeleteCache DeleteCookies InitLink "about:blank" Wait 3000 # Set up for Apple iPhone 4 emulation... SetUserAgent "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+" \ " (KHTML, like Gecko) Version/3.0 Mobile/1C25 Safari/419.3" Wait 2236 IndexSet 0 IndexFindElement 0 DOWN "innerText" "Mobile XXXXXXXXXX" "" ValueSet "_window_$USER_$NUMB" ValuePutElement 0 "target" "" IndexFollowLink 0 "" # The subwindow now will report being the specified device... SubFrameSet 1 "Mobile Web APPLICATION" Wait 5192 ResetTimer FollowLink 1 24 "Real Time Departures" "http://XXXXXXXXXX/schedules/eta/" \ "index.aspx" "" SyncOnText 1 "Departures as of" "" SyncOnText 1 "min" "" SyncOnText 1 " car)" "" # Click internal stopwatch and save data for chart... ElapsedTime "Train Schedule" SaveRecord DownloadTime.txt "%T \tInstance: $USER_$NUMB_$_RunCount\tTime: %E" Wait 1983