Launching and Supporting Killzone Shadow Fall Jorrit

Launching and Supporting Killzone Shadow Fall Jorrit Rouwé [email protected] Lead Game Tech, Guerrilla Games 1

Introduction Killzone Shadow Fall is a First Person Shooter 24 player team based gameplay Users can create game modes PS4 launch title 2

Before Launch 3

Road to Launch Previous Title: Killzone 3 – Closed beta ( 100 players within Sony) – Semi public beta ( 15K players invited) – Public beta ( 25K players concurrent) PS4 launch – Limited hardware – Max 100 player test 4

Development Focus Telemetry Load testing Monitoring Kill Switches 5

Telemetry – Collecting Game Metrics Storage Game Server Visualization Processing Telemetry Event Collector Redshift Tableau 6

Frame Rate Heat Map – The Terminal 55 FPS: Problem area! 7

SP - Average Deaths Per Player Per Level 8

MP - Death Balance – The Forest VSA failing to spawn camp Objective A 100 % HGH HGH Spawn 100% VSA HGH failing to spawn camp HGH being spawn camped Objective C Objective B VSA Spawn 9

Development Focus Telemetry Load testing Monitoring Kill Switches 10

Load Testing – What to Test Functionality – – – – Match making (e.g. day night cycle, featuring game mode) Score posting Leaderboards Failures – – – – Loss of server Temporary loss of connectivity PlayStation Network outage / slow response Server update 11

Load Testing – Framework PlayStation Network Game Server Sim 1.N Sim N 1.2N 12

Load Testing – Framework Mock PlayStation Network Game Server Sim 1.N Sim N 1.2N 13

Load Testing – Framework Mock PlayStation Network Game Server Share state & validation Sim 1.N Sim N 1.2N 14

Load Testing – Framework Mock PlayStation Network Game Server Telemetry & Monitoring Behavioral Verification Share state & validation Sim 1.N Sim N 1.2N 15

Development Focus Telemetry Load testing Monitoring Kill Switches 16

Monitoring What to monitor – – – – – – Player count REST API timings / errors Database throughput / errors Connectivity between servers (game servers / PSN) Low level OS (disc / CPU / memory / network) Backups 24/7 on call support for failures 17

Development Focus Telemetry Load testing Monitoring Kill Switches 18

Kill Switches Block Block Block Block apps (killzone.com, public API, live tiles) individual REST APIs (e.g. post score) hard to test things (e.g. game merging) misc. components (e.g. telemetry) 19

After Launch 20

Match Making - What Players Expect Low ping Full games Team balanced – Player count – Skill level A choice of game modes No dropped connections 21

Average Ping Between Regions 22

Average Ping Between Regions - Conclusion Matchmaking tries to avoid cross region! – Smaller regions / game modes don’t have enough players – Friends may want to play cross region – 98% success in large, 80% in smallest region 23

Ping – Cable vs Wireless Play Time Wireless Dropped connections Wired Wireless 15 ms slower! Educate players! 24

The Hunt for Server Locations (Sao Paulo) Sao Paulo 25

The Hunt for Server Locations (Dallas) 26

The Hunt for Server Locations (Istanbul) 27

The Hunt For Server Locations – Conclusion Geographically close does not mean low ping – Possible packet: US East US West US East – Very hard for players to understand – We don’t offer region selection Too many regions too few players per region 28

Match Making - What Players Expect Low ping Full games Team balanced – Player count – Skill level A choice of game modes No dropped connections 29

Joining Full Games Join alone full game Parties of more than 2 very rare! 8 player party 8 vs 1 game. 30

Match Making - What Players Expect Low ping Full games Team balanced – Player count – Skill level A choice of game modes No dropped connections 31

Team Balance - ELO Glicko Divide by time played Discard not played long enough 32

Team Balance - ELO Glicko Win 33

Team Balance - ELO Glicko Loss 34

Team Balance Skill based separation too many groups Player count balance most important Use ELO skill rating to – Determine team when sizes equal – Shuffle teams at end of round 35

Match Making - What Players Expect Low ping Full games Team balanced – Player count – Skill level A choice of game modes No dropped connections 36

A Choice of Game Modes 37

Match Making - What Players Expect Low ping Full games Team balanced – Player count – Skill level A choice of game modes No dropped connections 38

Errors Per Hour Over Time PlayStation Plus check PSN authentication Buffer increase fix (A/B test) 39

Errors Per Hour Per Router Technicolor TG587: UDP socket cannot connect to multiple IP. 40

Conclusion Had very little downtime Kill switches used on few occasions Match making is a hard problem Telemetry helped a lot 41

Questions? Also see: ‘The Server Architecture Behind Killzone Shadow Fall’ http://jrouwe.nl/serverarchkzsf/ 42

Bonus Slides 43

Ping – Educate Players! Max score not possible on wireless 44

The Hunt for Server Locations (Milan) 45

The Hunt for Server Locations (Frankfurt) 46