E132: DISCUSSION-BASED EXERCISE DESIGN AND EVALUATION COURSE Unit 4:

E132: DISCUSSION-BASED EXERCISE DESIGN AND EVALUATION COURSE Unit 4: Discussion-Based Exercises Overview FEMA News Photo Visual 4.1

Unit Objectives By the end of this unit you will be able to: ‒ ‒ ‒ ‒ Describe the types of discussion-based exercises Describe the eight exercise design steps Develop SMART objectives Design, develop, and conduct a seminar for the Liberty County Policy Group Visual 4.2

Types of Discussion-Based Exercises There are four types of discussion-based exercises: 1. 2. 3. 4. Seminars Workshops Tabletop exercises Games Visual 4.3

Progressive Approach Discussion-based exercises are the first step in the progressive approach to exercises Visual 4.4

Eight Exercise Design Steps 1. 2. 3. 4. 5. 6. 7. 8. Assess needs Define the scope Write a statement of purpose Define objectives Compose a narrative Write major and detailed events List expected actions Prepare messages Modified for discussionbased exercises Visual 4.5

Eight Exercise Design Steps Why use the Eight Exercise Design Steps? ‒ The process is critical to having a systematic exercise design ‒ They make the exercise process much simpler and smoother ‒ They are one way of implementing HSEEP ‒ They are a FEMA and exercise industry best practice ‒ They improve exercises ‒ This is where the “mastery” comes in Visual 4.6

Step 1: Assess Needs The best way to determine whether you need an exercise—and what kind of exercise is needed— is to conduct an exercise needs assessment An exercise needs assessment: ‒ Defines problems ‒ Establishes reasons to do an exercise ‒ Identifies capabilities to be exercised Visual 4.7

Begin With Your Plan An exercise needs assessment should begin with a review of the emergency plan and should address: ‒ ‒ ‒ ‒ ‒ Hazards Secondary hazards Hazard priority Area Plans and procedures ‒ ‒ ‒ ‒ Capabilities Players Program areas Past exercises Visual 4.8

Lessons Learned Learn from past exercises: ‒ ‒ ‒ ‒ Who participated? To what extent were objectives achieved? What lessons were learned? What problems were revealed, and what is needed to resolve them? ‒ What improvements have been made since, and have they been tested? Visual 4.9

Needs Assessment Results Primary and secondary hazards Problems, weak capabilities Skills requiring practice Improvements requiring testing Untested facilities, personnel, equipment Weaknesses in emergency plan or SOPs Need for role clarification Need for certain types of exercises Visual 4.10

Needs Assessment Results (cont’d.) Did you cover the topics listed on the prior slide when assessing the needs for your Multi-year TEP? Discussion Visual 4.11

Step 2: Define the Scope Defining the scope Setting realistic limits Factors that help define scope: ‒ ‒ ‒ ‒ ‒ ‒ Expense Availability of personnel and other resources Seriousness of the problem Capacity of the exercise to address the problem Planning team’s skills and experience Exercise length Visual 4.12

Six Key Elements Scope includes the: ‒ ‒ ‒ ‒ ‒ ‒ Type of emergency/hazard Location Capabilities Participants Exercise type Duration FEMA News Photo Visual 4.13

Six Key Elements (cont’d.) When defining the scope of your exercise, what are some factors you would consider in choosing the type of emergency/hazard? Discussion Visual 4.14

Six Key Elements (cont’d.) Type of Emergency/Hazard ‒ In choosing the type of emergency, ask yourself: What emergencies will generate the types of actions that need to be practiced? What are the high-priority hazards the organization faces? What hazards have not been recently exercised? Are there problems that have recently developed? Are there policy concerns surrounding the hazard/scenario? What plans are involved with this hazard/scenario? What secondary events might develop as the scenario develops? Visual 4.15

Six Key Elements (cont’d.) Location ‒ Identify the location where the simulated event will occur ‒ Consider the logistics of the chosen site Visual 4.16

Six Key Elements (cont’d.) Capabilities ‒ List the capabilities that the players will exercise (should be supported by the TEP) ‒ List the operations or functions that the participants will discuss ‒ Be sure that the procedures within a certain function are clear and narrowly defined Visual 4.17

Six Key Elements (cont’d.) Participants ‒ Narrow the list to those organizations and individuals who are required to hold a good discussion or carry out the actions ‒ More is not always better Visual 4.18

Six Key Elements (cont’d.) Exercise Type ‒ What exercises are most needed? ‒ What experience have personnel had with various types of exercises? ‒ What stress level do we want? ‒ What types of exercises are mandated by regulatory requirements? Visual 4.19

Step 3: Write a Statement of Purpose Purpose statement: A broad statement of the exercise goal ‒ Governs objectives, which determine subsequent steps ‒ Clarifies reasons for the exercise ‒ Is useful for communicating with media and community Visual 4.20

Developing the Purpose Statement Incorporate the scope decisions (type of emergency, location, capabilities, organizations, and exercise type) into a single sentence Include a date Sample purpose statements in Participant Guide Visual 4.21

Step 4: Define Objectives A description of the capabilities and expected actions that should occur ‒ What the exercise is intended to accomplish FEMA News Photo Visual 4.22

Why Define Objectives? Objectives are essential for: ‒ ‒ ‒ ‒ Design process Exercise conduct Evaluation Improvement planning Visual 4.23

How are Objectives Determined? Many objectives become evident at the time of the exercise needs assessment Objectives are also arrived at by breaking down a purpose statement into its logical components Visual 4.24

How Many Objectives? Typically: ‒ Tabletop exercises between 3 to 5 ‒ Functional exercises between 4 to 7 ‒ Full-scale exercises 8 The number of objectives selected for an exercise should be limited Visual 4.25

What Makes a “Good” Objective? Clear, concise, focused on performance Should contain: ‒ ‒ ‒ ‒ Action, stated in observable terms The entity (agency/jurisdiction) performing that action Conditions under which the action will be performed Standards (or level) of performance Should state who should do what under what conditions according to what standards Visual 4.26

Example of a “Good” Objective Specific action Who Discuss the plans for Central City’s incident command activation of essential HAZMAT personnel during a chemical incident in Central City in accordance with existing standard operating procedures (SOPs). Condition Standard Visual 4.27

Example of a “Good” Objective (cont’d.) Standard Who Condition Within 15 minutes after the evacuation notice is given, members of the EOC will complete notification procedures to school administrators. Specific action Visual 4.28

SMART Objectives Specific Measurable Achievable Relevant Time-bound Visual 4.29

Writing SMART Objectives How do you write SMART objectives for a TTX? Discussion Visual 4.30

Writing SMART Objectives (cont’d.) Objectives for discussion-based exercises typically focus on strategic, policy-oriented issues Objectives for operations-based exercises typically focus on the integration of multiple entities and systems-level and tactical-level issues that are physically demonstrated either in an EOC or in the field Visual 4.31

Writing SMART Objectives (cont’d.) Discussion-based exercise example: Restoration of Lifelines Evaluate Central City’s recovery plan to manage clearing and restoration activities, including the restoration of essential gas, electric, oil, communications, water, sewer, transportation/transportation infrastructure, and other utilities in the aftermath of a catastrophic incident Visual 4.32

Writing SMART Objectives (cont’d.) Share an example of an objective from an exercise you have designed. Discussion Visual 4.33

Word Choice Use concrete words Pay attention to the verb that describes participant performance ‒ Use action verbs ‒ Avoid vague verbs Visual 4.34

Word Choice (cont’d.) Useful action verbs include: ‒ ‒ ‒ ‒ ‒ ‒ ‒ Assess Clarify Define Determine Demonstrate Establish Evaluate ‒ ‒ ‒ ‒ ‒ ‒ ‒ Examine Explain Identify Inspect List Notify Operate ‒ ‒ ‒ ‒ ‒ Prepare Record Report Show Test Additional verbs provided in Participant Guide Visual 4.35

Is This a Useful Objective? To demonstrate an understanding of the procedures necessary in protecting responder health and safety Visual 4.36

Activity #2: Good or Poor Objectives? Visual 4.37

Activity #2: Good or Poor Objectives? (cont’d.) 1. Refer to Activity #2: Example: Good or Poor Objectives? Worksheet in your Participant Guide 2. Review the objectives and determine if they are good or poor 3. Record your answers on the worksheet 4. Work individually 5. You have 10 minutes to complete the activity Visual 4.38

Activity #3: TTX Objectives Visual 4.39

Activity #3: TTX Objectives (cont’d.) 1. We will run through an example first, then it will be your turn Visual 4.40

Activity #3: TTX Objectives Example ‒ ‒ ‒ ‒ Who: Central City Policy Group What: Long-term housing Conditions: Post-catastrophic dam failure Standards: National Long Term Housing Strategy Visual 4.41

Activity #3: TTX Objectives (cont’d.) SMART Objective ‒ The Central City Policy Group will discuss long-term housing strategies and options in response to a catastrophic dam failure in accordance with the National Long Term Housing Strategy. Visual 4.42

Activity #3: TTX Objectives (cont’d.) 1. Refer to Activity #3: TTX Objectives Worksheet in your Participant Guide (2 pages) Part One: Create SMART objectives for a discussion-based exercise using the information provided Part Two: Create SMART objectives for a discussionbased exercise using your group’s disaster category and priority 2. Record your answers on the worksheet Visual 4.43

Activity #3: TTX Objectives (cont’d.) PART ONE Objective #1: ‒ ‒ ‒ ‒ Who: Central City Policy Group What: Citizen warning Condition: Tornado Standard: Central City/Liberty County Basic Emergency Plan Visual 4.44

Activity #3: TTX Objectives (cont’d.) PART ONE (cont’d.) Objective #2: ‒ ‒ ‒ ‒ Who: Central City Policy Group What: Economic Recovery Condition: Category 3 Hurricane Standard: FEMA’s National Disaster Recovery Framework and Recovery Support Functions Visual 4.45

Activity #3: TTX Objectives (cont’d.) PART ONE (cont’d.) Objective #3: ‒ ‒ ‒ ‒ Who: Central City Policy Group What: Mandatory Evacuation Condition: Category 3 Hurricane Standard: Central City/Liberty County Basic Emergency Plan Visual 4.46

Activity #3: TTX Objectives (cont’d.) PART TWO 1. Create two SMART objectives for a discussion-based exercise applicable to your group’s disaster category and priority 2. Record your answers on the worksheet 3. Work in your groups 4. You have 15 minutes to complete PART TWO Visual 4.47

Step 5: Compose a Narrative Narrative: A brief description of the scenario events that have occurred up to the beginning of the exercise ‒ Sets participant interest ‒ Provides information that sets the stage for later action Visual 4.48

Narrative Characteristics A good narrative: ‒ ‒ ‒ ‒ ‒ ‒ Is located in the SitMan Is usually divided up into modules that represent time Is very specific Is phrased in present tense Is written in short sentences May develop the situation chronologically (event with warning time) ‒ May emphasize the emergency environment Visual 4.49

Example: Airplane Crash Narrative Read the Airplane Crash Narrative in your PG, and then answer the following questions: 1. What is the event? 2. How fast, strong, deep, or dangerous is it? 3. How did you find out? 4. What response has been made? 5. What damage has been reported? 6. What is the sequence of events? 7. 8. 9. 10. How much time has elapsed? Was there advance warning? Where does it take place? What are the relevant weather conditions? 11. What other factors would influence emergency procedures? 12. What is predicted for the future? Visual 4.50

SitMan Core documentation that provides the background for a facilitated exercise Supports the scenario narrative Serves as the primary reference material for ALL participants during conduct Expected actions do not go in the SitMan Visual 4.51

SitMan (cont’d.) Generally include the following information: ‒ ‒ ‒ ‒ ‒ Exercise scope, objectives, and core capabilities Exercise assumptions and artificialities Instructions for exercise participants Exercise structure (e.g., the order of the modules) Exercise scenario background (including scenario location information) ‒ Discussion questions and key issues ‒ Schedule of events Visual 4.52

SitMan (cont’d.) Appendices may include: ‒ Relevant documents (plans, SOPs, etc.) ‒ Specific threat information for jurisdictions or organizations ‒ Material safety data sheet or fact sheet, where appropriate ‒ List of reference terms Visual 4.53

SitMan (cont’d.) Introduction section ‒ Provides an overview of the exercise as well as an exercise agenda Scenario Narrative section ‒ Divided into distinct, chronologically sequenced modules ‒ Each module: Represents a specific time segment of the overall scenario Is followed up with discussion questions Visual 4.54

SitMan (cont’d.) Share a few examples of good discussion questions you have used in the past, or if you have a “go-to” set of discussion questions. Discussion Visual 4.55

Step 6: Write Major and Detailed Events Major and detailed events are occurrences that take place after, and as a result of, the emergency described in the narrative The goal is to provide a structure that will: ‒ Link the simulated event to the actions you want people to take ‒ Provide unity to the exercise Visual 4.56

Step 7: List Expected Actions The responses, actions, or decisions that you want participants to have or carry out to demonstrate competence It’s necessary to identify expected actions to develop discussion questions and guide facilitation Knowing who should know what allows you to develop discussion questions that target those participants and responsibilities as well as provide a guide for your facilitators to follow Visual 4.57

Step 8: Prepare Messages Messages (injects) are used to communicate detailed events to exercise participants Messages serve one purpose: to evoke a response FEMA/Heather Beebe Visual 4.58

Participant Feedback Forms After the exercise is conducted, you’ll need to capture the participants’ feedback on the exercise design Participant feedback forms ‒ Ask for input regarding observed strengths and areas for improvement ‒ Provide players the opportunity to give constructive criticism about the design, control, or logistics of the exercise All exercises should use participant feedback forms Visual 4.59

Participant Feedback Forms (cont’d.) At a minimum, questions on the Participant Feedback Form should solicit: ‒ Impressions about exercise conduct and logistics ‒ Strengths and areas for improvement pertaining to the implementation of participating agencies and organizations’ policies, plans, and SOPs; ‒ Improvements to future exercises ‒ Level of satisfaction Visual 4.60

Participant Feedback Forms (cont’d.) It is important to customize the Participant Feedback Form to capture the exact information you need for your specific exercise There are any number of ways you can customize a participant feedback form Visual 4.61

Planning Meetings Concepts & Objectives (C&O) Meeting* Initial Planning Meeting (IPM)** Mid-Term Planning Meeting (MPM)* Master Scenario Events List (MSEL) Meeting* Final Planning Meeting (FPM) * Not necessary or typically not conducted for TTXs or operations-based exercises ** C&O Meeting topics are covered at the IPM when a C&O Meeting is not conducted Visual 4.62

Planning Meetings (cont’d.) Share how you conduct your planning meetings. ‒ What topics do you discuss? ‒ What are your outcomes? Discussion Visual 4.63

TTX Demonstration Video FEMA/Eilis Maynard Visual 4.64

Unit Summary In this unit you learned: ‒ About the four types of discussion-based exercises ‒ The eight exercise design steps ‒ The application of the eight exercise design steps to a discussion-based exercise Visual 4.65

Questions? Visual 4.66

Proficiency Demonstration #2: Seminar Visual 4.67

Proficiency Demonstration #2: Seminar Purpose: ‒ Design, develop, and conduct a seminar about the Multi-year TEPs you developed in Proficiency Demonstration #1 Total time: 2 hours 40 minutes ‒ 1 hour to prepare the seminar ‒ 1 hour 10 minutes (two 35-minute rounds, including questions) for the presentations ‒ 20 minutes for peer/instructor feedback ‒ 10 minutes for internal group feedback Visual 4.68

Proficiency Demonstration #2: Seminar (cont’d.) Deliverables: 1. 30-minute seminar on your group’s Multi-year TEP (PowerPoint) 2. Participant feedback forms (HSEEP template) Visual 4.69

Proficiency Demonstration #2: Seminar (cont’d.) Scenario: ‒ The Liberty County Emergency Management Policy Group has requested a seminar on the newly developed Liberty County/State of Columbia Multi-year TEP ‒ Base your seminar on the Multi-year TEPs you developed in Proficiency Demonstration #1 Visual 4.70

Proficiency Demonstration #2: Seminar (cont’d.) Scenario (cont’d): ‒ You will present your 30-minute seminar to another group who will be acting as the Liberty County Policy Group: Group 1 presents to Group 3 Group 2 presents to Group 4 Group 3 presents to Group 1 Group 4 presents to Group 2 ‒ The goal is to sell your Multi-year TEP to the Policy Group and get the group’s approval for the necessary funding Visual 4.71

Proficiency Demonstration #3: Tabletop Exercise (cont’d.) Scenario (cont’d.): ‒ Determine which members of the Liberty County Policy Group will be audience members for your seminar A list of Policy Group members is provided in the ESSD Get name tents from instructor ‒ Each member of the group that will be acting as the seminar audience must be assigned a Policy Group role Refer to the Policy Group Job Descriptions in your Participant Guide Visual 4.72

Proficiency Demonstration #2: Seminar (cont’d.) Group roles: Each group must select the following roles: ‒ Group Leader (at least one) ‒ Scribe (at least one) ‒ Presenter/Facilitator (no more than two) Visual 4.73