Infrastructure to Support Extreme Self-Service Session #20653

Infrastructure to Support Extreme Self-Service Session #20653 March 15, 2006 Alliance 2006 Conference Nashville, Tennessee

Presenter Carol Jordan I have been working on PeopleSoft applications since 1997, and have been at Stanford since November 2000. I am the Infrastructure Manager for the team supporting Student and HR Systems. In addition to the PeopleSoft products, our group also supports Kronos, Resource 25, OnBase Imaging, Resumix and a custom Workflow application. 2

Overview Before PeopleSoft: - Limited functionality for students - using a web-enabled mainframe application - allowing only 35 concurrent student sessions With PeopleSoft HR/SA and Portal: - Many functions for students - and for faculty, advisors and staff - and no portal sessions are turned away 3

Agenda/Contents In this presentation: Extreme self-service defined Our rough start Where we are now, how we got here What we learned 4

Overview: Stanford University Stanford University is a private university located 30 miles south of San Francisco and just north of Silicon Valley. There are currently 6,700 undergraduate and 8,000 graduate students enrolled, with 1,775 faculty. Founded in 1891, the university was established by Jane and Leland Stanford in memory of their son, Leland Stanford, Jr., who died of typhoid fever at the age of 15. 5

Overview: Application Administration The Application Administration team is part of Administrative Systems, supporting many of the applications used campus-wide. There are three AppAdmin teams, supporting PeopleSoft, Oracle Financials, and Reporting. The team supporting PeopleSoft is made up of 5 System Analysts, 3 Oracle DBAs and 3 Windows System Administrators. Developers and other support staff are part of a separate team within Administrative Systems. 6

Overview: Application Administration PeopleSoft products currently installed: Campus Solutions 8 SP1 PeopleTools 8.22.05 Enterprise Portal 8.8 SP1 PeopleTools 8.44.03 Enterprise Learning Management 8.8 SP1 PeopleTools 8.45.12 All on Oracle 9.2.0.6 7

Extreme Self-Service, defined Lots of functionality for students, faculty, advisors and staff and Lots of users – close to 35,000 logins on our busiest days 8

Extreme Self-Service, defined Lots of functionality for students File or adjust study list and elect grading options Request an official transcript to be mailed Print history of courses and grades Print Enrollment Certification Declare major and minor Apply to graduate Update addresses, emergency contacts Maintain FERPA elections View and accept financial aid View student bill and print statement View advisors View degree progress, milestones, program summary Coming in March enter or read course evaluations Coming in April pay student bill, add money to card plan Links to other applications: Student Housing, IT Services bill (Pinnacle), Office of Development 9

Extreme Self-Service, defined Lots of functionality for Faculty Review class lists Send e-mail to students in a class Submit and change grades Assign a grade proxy Review previously-submitted grades Coming in March read students’ course evaluations for Advisors View lists of current and past advisees View study lists, grades and unofficial transcripts for current advisees Place and release holds of current advisees for Staff View on-line pay statement Make W-4 elections Enter direct-deposit elections Register for training, take on-line tests through PeopleSoft ELM Link to applications: Kronos, on-line W-2 10

Extreme Self-Service, defined Lots of users: 1050 30,000 900 25,000 750 20,000 600 15,000 450 10,000 300 5,000 150 Total logins by day Jan Dec Nov Oct Sep Aug Jul Jun May Apr Mar 0 Feb 0 Max Logins per Quarter-hour 35,000 Jan Total Logins Axess Usage Summary: Logins - 2005 Max Logins Per Quarter-hour 11

Our Rough Start Upgrade to Version 8 went OK First couple of months went OK but the system could not support the sustained demand of our busiest days – the first two days of the new academic year 12

Our Rough Start Stanford Daily articles with headlines like, “Axess Problems Plague First Week” What students saw: log-in attempts that failed ‘connection refused’ errors between HR/SA and Portal pages that didn’t load 13

Our Rough Start CIO magazine article: ‘Big Mess on Campus’ “Stanford's IT was still struggling with integrating the enterprise systems when the newly launched PeopleSoft Web portal (called Axess) crashed last fall Axess couldn't handle the load of all the returning students trying to log in ” 14

Where we are now We are able to handle high load without issues Business offices are planning to roll out new functionality using the Axess portal and we’ve managed to stay out of the papers for a while (fingers crossed) 15

Obligatory Hardware Diagram 16

How We Got Here Upgraded hardware, added hardware Server hosting custom single sign-on was the biggest, earliest bottleneck Original web servers were old and slow Got the OK to order new hardware after capacity issues hit Lessons: Go with your gut if you’re worrying about it, fix it; if you can’t fix it, have a contingency plan Fight the requests to minimize costs by reusing old hardware 17

How We Got Here Web-server changes for HR/SA and Portal Upgraded JRE Upgraded Jolt and WebLogic to the latest rolling patch Reduce network disconnect timeout Used WebLogic console to fine-tune JVM heapsize (768M) and thread-count (100) on Windows web servers Set Windows PIA service to restart automatically if it goes down Installed two logical web-servers per physical server to improve memory utilization under Windows Separated administrative and self-service users, to set different time-outs, breadcrumbs, and other configuration parameters for different types of users 18

How We Got Here Application-server changes for HR/SA and Portal: Upgraded JRE Upgraded Tuxedo to latest rolling patch Reduced client-cleanup timeout For Portal and ELM: implemented shared application-server cache For Portal and ELM: re-configured Tuxedo domains to eliminate spawning – always start the maximum number of PSAPPSRVs 35 per server for each of two Portal application servers 8 per server for each of two ELM application servers 15-25 per server for each of two HR/SA application servers 19

What we learned We can’t predict load with certainty High load is loosely tied to the beginning of the term, when fees are due and students need to know where their classes are so we implemented defensive monitoring to send an e-mail when at 400 logins per quarter-hour Sample e-mail text: The Axess login count as of 00:16, 03/13/2006 is 550 in a 15-minute interval. 20

What we learned Because we count logins, we can see patterns in user activity: 03/12/2006 03/12/2006 03/12/2006 03/12/2006 03/12/2006 03/12/2006 03/13/2006 03/13/2006 03/13/2006 03/13/2006 03/13/2006 03/13/2006 03/13/2006 03/13/2006 22:30 22:45 23:00 23:15 23:30 23:45 00:00 00:15 00:30 00:45 01:00 01:15 01:30 01:45 57 66 94 80 105 240 550 197 181 128 100 90 Spring enrollment opened at 12:01am on Monday 3/13 76 57 21

What we learned Monitor at many levels At the web-server layer: We monitor the load-balanced site and individual (logical) web servers and the HTTP port used by the load-balancer probe At the application-server layer: We monitor for Tuxedo processes UNIX SysAdmins monitor memory usage, add swap space as needed We monitor for CPU utilization (sample e-mail follows) 22

CPU monitor – sample e-mail Info from sar Time: 00:40:10 usr%: 77 sys%: 10 wio%: 0 idle%: 14 Top output load averages: 4.76, 9.29, 15.18 00:40:10 106 processes: 101 sleeping, 1 zombie, 4 on cpu Memory: 8192M real, 1143M free, 6577M swap in use, 18G swap free PID 26073 9800 10991 10819 USERNAME THR PRI NICE a2k prd 4 0 0 a2k prd 4 58 0 a2k prd 4 50 0 a2k prd 4 58 0 SIZE 203M 256M 245M 267M RES 187M 238M 227M 251M STATE cpu0 sleep sleep sleep TIME CPU COMMAND 4:40 10.43% PSAPPSRV 13:49 8.83% PSAPPSRV 11:18 7.61% PSAPPSRV 13:33 6.82% PSAPPSRV 23

What we learned Use your PeopleSoft tools! PeopleSoft Performance Monitor for Portal and ELM WebLogic console for web servers 24

What we learned Keep Tools up-to-date – watch for BEA updates Move AppMessaging to a dedicated host – keep back-end processing out of the transactional infrastructure Nobody hits ‘logoff’ – configure your web servers to close idle connections quickly Performance testing is mandatory for new projects Configuration review, top-to-bottom, with PeopleSoft Consulting was helpful 25

Questions? 26

Contacts Carol Jordan Infrastructure Manager, Student and HR Systems Administrative Systems Stanford University E-mail: [email protected] Jeff Butler Lead System Analyst, Student & HR Systems Stanford University E-mail: [email protected] Ross Shimabukuro System Analyst, Student & HR Systems Stanford University E-mail: [email protected] 27

This presentation and all Alliance 2006 presentations are available for download from the Conference Site Presentations from previous meetings are also available