IBM Systems & Technology Group SPECweb2005™ in the Real World: Using

IBM Systems & Technology Group SPECweb2005 in the Real World: Using Internet Information Server (IIS) and PHP 01/27/2008 Sam Warner Steve Worley 2008 SPEC Benchmark Workshop IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Abstract Analysis of various systems Using IIS as web server Using PHP to server dynamic content SPECweb2005/PHP more representative to “real world” Results comparison PHP to published JSP results Dual-socket results and Multi-socket results IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Introduction SPECweb benchmark has brought ability to show the relative performance of various systems manufactured in the Web Serving market SPECweb can be used for Research efforts at universities Within computer equipment manufactures to evaluate alternative technologies In the purchase process by Information Technology (IT) professionals SPECweb has two alternatives to enable performance evaluation Java Server Pages (JSP ) All current published results Top results are unrealistic at approximately 40000 users Hypertext Preprocessor (PHP) This paper provides results comparison not found in published results Help insure the benchmark remains useful in IT customer purchase process IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Overview of the SPECweb2005 Benchmark Standard Performance Evaluation Corporation benchmark for evaluating the performance of World Wide Web Servers Supersedes SPECweb99 and SPECweb99 SSL Three workloads representing market segment that are part of the benchmark SPECweb2005 Banking SPECweb2005 Ecommerce SPECweb2005 Support Each workload variant enables measuring the maximum number of simultaneous user sessions Still achieving specific QOS metrics Still achieving error rate requirements IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Overview of the SPECweb2005 Benchmark Benchmark score Individual sub-metric scores indicate the total number of simultaneous user sessions the server can support Overall score is the geometric mean of the three sub-metrics normalized to a reference platform score A score of 100 represents the same overall performance of the reference platform A score of 20,000 represents a score 200 times that of the reference platform The benchmark does not require a specific choice of web server software Only requirement is that the web server support HTTP 1.1 and SSL (HTTPS) Implementation of the web server will lead to differences in observed performance on the same system under test Current performance using the JSP kit achieves performance which is roughly 370x the reference platform IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Top 15 SPECweb2005 /JSP Scores Rank Web Server Chips Cores Processor 1 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 2 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 3 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 4 v1.2.0 4(x86 64) Intel Xeon X7350 Processor 4 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 4 v1.2.0 4(x86 64) Intel Xeon E7330 Processor 5 Rock Web Server v1.4.6 (x86 64), Rock JSP/Servlet Container 2 v1.3.1 4(x86 64) Intel Xeon Processor X5365 6 Rock Web Server v1.4.3 (x86 64), Rock JSP/Servlet Container 2 v1.3.0 4(x86 64) Intel XEON 5365 processor 7 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 4 v1.2.0 2(x86 64) AMD Opteron 8222SE 8 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 4 v1.2.0 2(x86 64) AMD Opteron 8220 9 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 2 v1.2.0 4(x86 64) Intel Xeon Processor X5355 10 Rock Web Server v1.4.0 (x86 64), Rock JSP/Servlet Container 4 v1.2.0 2(x86 64) AMD Opteron 8220 11 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 2 v1.2.0 4(x86 64) Intel Xeon X5355 Processor 12 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 2 v1.2.0 4(x86 64) Intel Xeon Processor X5355 13 Rock Web Server v1.4.0 (x86 64), Rock JSP/Servlet Container 4 v1.2.0 2(x86 64) Intel Xeon 7140M Processor 14 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 2 v1.2.0 4(x86 64) Intel Xeon X5365 Processor 15 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 2 v1.2.0 4(x86 64) Intel Xeon X5355 Processor IBM, INTEL, 2008 Result 37,001 37,001 30,261 26,119 26,077 22,332 22,254 21,470 20,387 20,235 19,931 19,661 18,981 18,931 18,917

Intel Corporation and IBM Systems & Technology Group Overview of Top Fifteen Results All fifteen submitted with one of two web severs Sun Java System Web Server 7.0 Accoria Rock Web Server V1.4 13 of the top 15 results Rock Web Server is undeniably fast No measurable market share from market analysis firms All results available at SPEC website http://www.spec.org IBM, INTEL, 2008 2008 SPEC Benchmark Workshop

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop SPECweb2005 and the Real World IT customers use of performance benchmarks Seen as way to evaluate the complex combination of Number of users supported Content development language Web server software Web server hardware Don’t have much time to understand the subtleties of a benchmark Forces OEMs to be constrained by competitive pressures to produce leading results Causes the publication of SPECweb2005 to diverge from real world IT performance IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop SPECweb2005 and the Real World Trend of web server connections supported Range of 20-2000 sustained customer connections As processing capacities have improved, so has Concerns and processing for security Complexity in formulating the responses for web server requests SPECweb99 Results ranged from 400 to 25000 for the five years of publication Meanwhile, web server connection rates remained in the 1000s SPECweb2005 /JSP Two and four socket systems have begun to achieve levels approaching 50,000 simultaneous users for a duration of ½ hour. In contrast to research papers where published connection loads tend to range from 2000 to 3000 users SPECweb2005 /PHP By comparing to SPECweb2005 /JSP attempt is to bridge gap between real world loads and published results IT industry may be better served by comparisons to connection loads seen in their environments IBM, INTEL, 2008

2008 SPEC Benchmark Workshop Intel Corporation and IBM Systems & Technology Group SPECweb2005 /JSP Scores since Inception SPECweb2005/JSP Scores since inception 50x change in Connections sustained in two years 70 40000 60 SPECweb2005/JSP Scores 35000 50 30000 40 25000 20000 30 15000 20 10000 10 5000 0 Sep-05 0 Mar-06 Aug-06 SPECweb2005/JSP Score Jun-07 # sockets May-07 Memory (GB) IBM, INTEL, 2008 Jun-07 # of Nets Memory(GB), Number Netwokrs, Number of Sockets 45000

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Analysis of Currently Published SPECweb2005 Results Request for Information and Request for Proposal processes have led many OEMs to develop publications for SPECweb2005 . Forces OEMs to use software combinations that enable leading performance results Competition Great way to insure that the final purchased product meets the needs of the customer If benchmark closely matches the end-user environment, benchmark produces an informed purchase Top 15 results show connection rates that are 10-20x higher than connection loads that are normally seen by web servers. IBM, INTEL, 2008

2008 SPEC Benchmark Workshop Intel Corporation and IBM Systems & Technology Group Comparison of Web Servers Of top fifteen SPECweb2005 results, thirteen use the Accoria Rock web Server v1.4 Proprietary Non-open source No measurable market share Web Servers discussed in the paper Web Server Creator Windows Yes Linux No Solaris No CGI Yes FastCGI Yes Servlet No ASP.NET Yes IIS Microsoft Rock Web Server Accoria Networks Yes Yes Yes Yes Yes Yes No Sun Java System Sun Microsystems Yes Yes Yes Yes Yes Yes No IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Web Servers Currently Used in Submissions Choice of Web Server up to OEM Results prepared as market collateral Market conditions dictate the decision of web server Best possible combination of hardware and software If less than ideal component used then consumer education required Results response to RFI Choice of highly relevant results to customer or industry leading results Lower cost to educate customer by simply achieving highest possible result Net result is that neither Apache or IIS, both recognized in their widespread use, have been used in any submissions in the last two years IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Comparison of JSP to PHP performance Rank Web Server Chips Cores Processor 1 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 2 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 3 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 4 v1.2.04(x86 64) Intel Xeon X7350 Processor 4 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 4 v1.2.04(x86 64) Intel Xeon E7330 Processor 5 Rock Web Server v1.4.6 (x86 64), Rock JSP/Servlet Container 2 v1.3.14(x86 64) Intel Xeon Processor X5365 6 Rock Web Server v1.4.3 (x86 64), Rock JSP/Servlet Container 2 v1.3.04(x86 64) Intel XEON 5365 processor 7 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 4 v1.2.02(x86 64) AMD Opteron 8222SE 8 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 4 v1.2.02(x86 64) AMD Opteron 8220 9 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 2 v1.2.04(x86 64) Intel Xeon Processor X5355 10 Rock Web Server v1.4.0 (x86 64), Rock JSP/Servlet Container 4 v1.2.02(x86 64) AMD Opteron 8220 11 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 2 v1.2.04(x86 64) Intel Xeon X5355 Processor 12 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 2 v1.2.04(x86 64) Intel Xeon Processor X5355 13 Rock Web Server v1.4.0 (x86 64), Rock JSP/Servlet Container 4 v1.2.02(x86 64) Intel Xeon 7140M Processor 14 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 2 v1.2.04(x86 64) Intel Xeon X5365 Processor 15 Rock Web Server v1.4.1 (x86 64), Rock JSP/Servlet Container 2 v1.2.04(x86 64) Intel Xeon X5355 Processor IBM, INTEL, 2008 JSP Results PHP Results na na na na na na na na 20387 1083 na 19931 1083 19661 1083 18981 1143 na 18917 1083

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop JSP use in submissions Presently all of the results submitted have used the JSP kit While JSP is widely used, it is not the primary content development language used To achieve leading industry results an OEM needs to use a software stack that has the lowest overhead to generate a response. The current implementations of SPECweb2005 in JSP has lower overhead than PHP IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Analysis of Real World Workloads Industry results on SPECweb2005 /JSP have improved by 50x since inception Publications from universities show that true connection loads trend at levels 1/10th of those observed in publications. SPECweb99 this same disconnect Using the SPECweb2005 /PHP kit on equivalent systems we find that the change in connection loads observed is not as high. Intel Xeon X5355 JSP shows 18,000-20,000 connections PHP shows 1000 connections IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Web Server Statistics Aversa and Bestavros Paper TCP connection load of between 496 and 663 requests served Response time between 0.92s and 0.26s Grottke paper Connection rates of 390 connections/second (c/s) Cao paper Connection rates of between 0.18 c/s and 34 c/s Connection rates of between 2.41 c/s and 230 c/s Seems reasonable to look at a scenario where the top connection rates are in the 2000-3000 connections/second range IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Discussion of use of IIS According to various web server market analysis companies approximately 35% of web servers on the internet use IIS Tends to be used in websites with higher traffic volumes No web server market analysis companies are able to measure the market share represented by Rock Web Server One is left to wonder how the IT industry correlates the performance reported from SPECweb2005 submissions to their own environment IBM, INTEL, 2008

2008 SPEC Benchmark Workshop Intel Corporation and IBM Systems & Technology Group Web Server Market Share % Internet Market Share 100 80 60 40 20 0 Alexa 1000 Media Metrix 500 100 Hot NetCraft Non-active NetCraft Active Market Research Firm Apache Developer Apache Microsoft Google lighttpd Sun IIS Netscape Zues 7-Oct Percent 7-Nov Percent 68,155,320 47.73% 76,028,287 50.76% 53,017,735 37.13% 53,679,916 35.84% 7,763,516 5.44% 7,910,879 5.28% 1,541,779 1.08% 1,505,122 1.00% 2,262,019 1.58% 619,262 0.41% IBM, INTEL, 2008 Change 3.03 - 1.29 - 0.15 - 0.07 Yes

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Discussion of use of PHP Variety of market analysis companies have analyzed the percentage of the market share for the various content development languages. These research firms show levels from 35% to 75% for PHP Depends on whether the analysis includes all known websites or newer websites being actively developed The trend is that PHP development has increased from 25% to 50% of content development from 2001 to 2007 IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop SPECweb2005 /PHP Results and Analysis The key value in a benchmark is its ability to accurately predict for the market segment intended the value of the new hardware or software component. With the SPECweb2005 /PHP kit there appears to be more similarity Hardware requirements to achieve peak results are decreased dramatically. With the SPECweb2005 /PHP results are closer to the default OEM configurations IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Dual-Socket Results System chosen was Intel Xeon 5355 processor dual socket platform Support had the highest connection load Banking produced the lowest connection load Processor utilization 95% processor utilizations for Support and Ecommerce 80% processor utilization for Banking Banking workload using PHP has software serialization due in part to the session state having higher processing and IO requirements that limit the processor utilization achievable X5350 X5355 SPECweb2005 Score Support Bank Ecommerce 964 1900 750 1400 1082 2200 800 1600 IBM, INTEL, 2008 Simultaneous Connec 1259 1412

2008 SPEC Benchmark Workshop Intel Corporation and IBM Systems & Technology Group Dual-Socket Response Times Response times for Ecommerce where nearly equivalent to SPECweb2005 /JSP results Response times for Support were approximately 0.83x that of JSP Response times for Banking were lower at 0.47x that of JSP Response times are nearly the same across the three workloads. PHP JSP SPECweb2005 Avg Resp Avg Resp Support Ecommerce Banking Time (sec) 5.133 1.432 0.652 Time (sec) 6.334 1.511 1.366 IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Multi-Socket Results System chosen was Intel Xeon 7140M processor Equivalent characteristics observed to two-socket system Connection rates improved slightly for Ecommerce and Banking Slight degradation on the Support workload Processor Utilization Support and Ecommerce higher than 90% Banking slightly higher than 70% (Due to high software serialization) 7140M SPECweb2005 Score Support Bank Ecommerce 1141 2100 900 1750 IBM, INTEL, 2008 Simultaneous C 1490

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Comparison of Configurations to Published Results Major determining factor for choice of JSP kit is the path length for JSP to server the dynamic content is much shorter than for PHP to perform the same task Significant contrast in hardware requirements required for a top performing JSP result and a top performing PHP result Network JSP requires 5-18 subnets of 1Gb PHP drops to two 1Gb subnets Disk JSP requires 10 times the storage 4Socket 2Socket kit variant PHP JSP PHP JSP Memory 32 64 Network 3 17 physical disks 4 58 SPECweb2005 Score 1083 18981 16 32 2 9 4 58 1083 20387 IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC Benchmark Workshop Conclusions By using the SPECweb2005 /PHP kit a new system can be evaluated at a level of performance with respect to connection loads and response times that more closely matches a real world scenario. The amount of hardware required to analyze a new system with the SPECweb2005 /PHP kit is much smaller than using the SPECweb2005 /JSP kit IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group www.cs.ualberta.ca/ amaral/SPEC08 IBM, INTEL, 2008 2008 SPEC Benchmark Workshop