Exercise Keen Edge 22 Lessons Learned From A Perfect Storm Ellen

Exercise Keen Edge 22 Lessons Learned From A Perfect Storm Ellen F Roland ROLANDS & ASSOCIATES LLC ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 1

Exercise Lessons Learned Why Hold Computer Aided Exercises (CAX) Opportunity to assess performance and capabilities of forces under realistic conditions. Ability to test alternative Courses of Action (COA) After Action Review (AAR) Process Is Important Identify Recurring Problems Develop Corrective Actions Ensure Others Avoid Repeating Past Mistakes AAR Is A Mandated Exercise Audience / Staff Requirement Collect Lessons Learned Analyze Information Looking For Recurring Problems Disseminate All Information To Potential Users Implement Corrective Action And Validate Results ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 2

Tech Support Lessons Learned Not Just For The Exercise Audience / Staff Same Process Should Be Followed For Exercise Construct Exercise Computer Simulation Driver Purpose Of This Presentation Is To Provide AAR Results From A Stressful CAX Keen Edge 22 – Held January 2022 Why Partly Mea Culpa (My Mistake) Partly To Set Record Straight Mostly For The Benefit Of Lessons Learned Conference Paper Is The AAR Report ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 3

Exercise Keen Edge 22 Keen Edge 22 Joint Command Post Exercise United States US Indo-Pacific Command (US INDO-PACOM) US Forces Japan Japan Japanese Self Defense Force (JSDF) Goal / Purpose Increase both nation’s readiness to respond to a variety of crises and contingencies in the Indo-Pacific region Remain a combat credible deterrence in an increasingly complex environment. ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 4

Joint Theater Level Simulation Global Operations (JTLS-GO) JTLS-GO Basic Characteristics Interactive, Web-Enabled Environment No Embedded Data Or Doctrine (Data Driven) Stochastic World-Wide Representation Forces Represented At The Operational Level Dynamically Adapted to User Requirements Since 1983 From Concept To First Exercise Use – 6 Months Step-Wise Improvement Prioritize Improvement Agile Implementation Immediate Feedback From Users Repeat ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 5

Perfect Storm What Factors Lead To Model Unreliability New Code – First Use JTLS 6.1.0.0 New Hardware – New Servers With Serious Heat Issue New Users – COVID-19 Significant Expertise Loss Project Management – At US INDO-PACOM 3 Years – 1 st Exercise Computer System Management – New Team Interface Controllers / Players / SMEs – 80% New Team Model Support – 50% Expert, 30% Limited, 20% New New Data – Bilateral Data Development For First Time Keen Edge 22 Each Factor Played A Role In The Perfect Storm ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 6

Lessons Learned Process Collect Data / Information All JTLS-GO Exercises Maintain A Manual Exercise Log All User Questions Explanation Of All Crashes Notes Concerning Other Issues Encountered Summary Of Exercise Logs Is In Conference Paper Lesson Learned 1: R&A Personnel were at two sites, Interface Controllers at four sites. Impossible to keep everyone informed and keep log accurate. Rumors and misinformation were rampant. JTLS-GO is a web-enabled system. The log should also be web-enabled so everyone can contribute and understand what is going on. ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 7

Lessons Learned Process Analyze Collected Data Encountered Issues Were Categorized User Question – Requires Training or Documentation Update Minor Issue – No Impact On Exercise Execution Code Fix Issue – Exercise Impact - Fixed During Exercise Period Work Around Issue – Required Significant Time To Solve Engineering Change Proposal – Requested New Improvement Hardware Issue ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 8

Lessons Learned Process Type Of Log Entries (128 Total) Hardware ECP User Work Around Code Fix Minor User Work Around ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company Minor Hardware Code Fix ECP 9

Model Unavailable Time Definition And Types CEP – Code Error Personnel – Code Error Unavailable analysis Based On Poor Decisions CEP Unsynchronized Hardware – Hardware Unavailable Unsynchronized - Model Hardware More Than One Hour Behind Real Time Personnel ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 10

Lies, Dammed Lies, and Statistics Mark Twain: “There are three kinds of lies. Lies, Dammed Lies, and Statistics” Summary Exercise time CEP Hardware Personnel Unsynchronized Available ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 11

Lessons Learned Process Disseminate Lessons Learned Official Lessons Learned Paper This Briefing Implement Corrective Action Continuing Process ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 12

Perfect Storm – New Code New Code Keen Edge 22 Used JTLS 6.1.0 First Use Case Developed During COVID Lockdown Tested Using Remote Test Site Attempted 24-Hour Test Support Communication Difficulty Between Test Shifts Regression Testing Was Not Stressed Model Non-Availability Due To Code Errors Nine Model Crash Situations Over 18 Days 6 Crash Situations - Existed In Previous Version 1 Crash Happened Twice – Existed In Previous Version 1 Crash Attributed Directly To New JTLS 6.1 Code Average Downtime – 30 Minutes ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 13

Perfect Storm – New Code Lesson Learned 2: One of the more interesting items of note, is that only one model problem was identified during testing and training. Given the number of model problems encountered during the exercise implies that the pre-exercise functional tests are not testing normal exercise functional requirements. This could be attributed to one or both of the following reasons: During the test period, the JTLS-GO operators were still learning the model basics which are well tested. The testing checklist did not cover any of the new features developed over the previous two years. It is worthwhile for JTLS-GO operators or JTLS-GO training staff to create their own test checklists from the information provided in these logs and expand their test procedures for future exercises. ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 14

Perfect Storm – New Hardware New Hardware Server Network Consisted Of Nine Machines Three New Super Servers – 128 G-Bytes Memory And Fast One Used As File Server / Model Machine Two Used To Run System Processes Six Older Servers – 32 G-Bytes Memory And Adequate Speed Main Issue Super Servers Had Heat Issue During Period Of High Computation CPU Over Heated CPU Went Into Automatic Reduced Speed Mode Until Cool Model Fell Behind Desired Clock Model Attempted To Run At Faster Game Speeds To Catch Up Systems Were Constantly In High Computation Mode Never Ending Cycle ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 15

Perfect Storm – New Hardware New Hardware (Con’t) Heat Issue Solutions - Extraordinary / Harmful Steps To Cool Down Machines To Reduce Model Computations Lesson Learned 3: There were indications of hardware issues during the testing period, but they were attributed to various non-serious possibilities. For example, it was normal operating procedure to unload a new version of the database the last thing each night of the test period and run the model as fast as possible to ensure there were no database induced problems. Several times the systems would be down the next morning. System management personnel hypothesized there was a power failure at night, or the air conditioning was disabled at night. Hypotheticals are not proof. If these problems had been addressed during the test period, many of the issues could have been addressed under less stressful conditions. ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 16

Perfect Storm – New Hardware Lesson Learned 4: Bigger is not always better. Lesson Learned 5: Cost saving measures don’t always cost less. The first hardware switch was made without technical support. The hardware swap went fine but then it was learned that the ports for the new IP address were not open. It took 90 minutes to call the proper people in to open the needed ports. Lesson Learned 6: Blaming hardware takes pressure off model developers. More time should have been allocated to determining how to reduce model computations vice how to solve hardware issues. There were two major model computations that could have been easily solved if time had been allocated to analyze the situation . Proper use of JTLS-GO Operations Areas Proper use of JTLS-GO Directed Search Areas ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 17

Perfect Storm – New Users New Users COVID-19 Resulted In Two Year Exercise Hiatus Almost All Personnel Were Considered New Users Technical Control 50% Experience 30% Limited Experience 20% Little Experience Computer System / Network Management 100% Little Experience In Exercise / Computer Environment Interface Controllers Four Remote Locations – Approximately 80% First Time Users Exercise Coordinator Limited JTLS-GO Experience ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 18

Perfect Storm – New Users Lesson Learned 7: Although COVID-19 hiatus was a unique situation, extra time should have been allocated to training. There were players that had no idea how to operate the user interface tools. Each remote site was responsible for their own training. A more centralized training and testing plan was needed. Lesson Learned 8: Information provided to one site would have been useful to other sites. A web-enabled logging system is needed to disseminate information. One model crash happened twice because the word to not use the Magic Air Ops order under a unique situation was never passed to all sites. A logging system that all users should read at the start of their shift could have solved numerous problems and questions. ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 19

Perfect Storm – New Users Lesson Learned 9: “Aye, aye sir” is not always the correct answer. New personnel were hesitant to firmly but politely voice issues with selected decisions: The questionable computer cooling technique was known by many to be a poor idea, but no one spoke up. I heard one person after the exercise say: “When I heard what they were doing, I walked out of the room”. More importantly, once implemented, no one checked on the status of the situation and an irreparable hardware failure occurred. Weapon supplies were not properly built in the database and missions were having trouble meeting launch requirements. Ninety minutes before launch time, missions reported their problem to the Exercise Audience. Air Staff objected to the Exercise Audience notification and ordered the JTLS-GO operator to stop generating the reports. From post-exercise interviews, the operator knew the consequences, but did not voice a strong objection. The Exercise Audience no longer was notified but the operator was also not notified. The end result was that missions were canceled before the operator could fix the situation. ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 20

Perfect Storm – New Data New Data Database Was Huge – Target List Started Out With Over 120,000 Targets Testing Showed This Was A Problem No One Would Take Responsibility For Cutting Down Target List Finally, Database Team Settled On 60,000 Targets All Indications Were 90% Of These Did Not Add To Training Undetermined If This Has Been Revisited / Confirmed Bilateral Database Development Database Development Is Web- Enabled Each Coalition Partner Developed Their Own Data Misperception Of Data Meaning Led To One Model Crash ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 21

Perfect Storm – New Data Lesson Learned 10: Exercise Audience representatives must be more integrated into the Database Development process. Understandably, this is easier said than done, due to real-world job requirements Lesson Learned 11: Maintaining consistent and regular oversight of database collaborative efforts with coalition partners is paramount to having a successful JTLS-GO event ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 22

What Went Right Lessons Learned / AAR Process Concentrates On What Went Wrong And Why Remember To Capture What Went Right When Establishing Corrective Actions Don’t Undo What Went Right ROLANDS & ASSOCIATES A Valkyrie Enterprises Owned Company 23

Questio ns 24 ROLANDS & ASSOCIATES Corporation A Wholly Owned Subsidiary Of Valkyrie Enterprises LLC