PROVIDERS AS CO-REVIEWERS OF ACT TEAM FIDELITY ASSESSMENTS:

PROVIDERS AS CO-REVIEWERS OF ACT TEAM FIDELITY ASSESSMENTS: ESTABLISHING FEASIBILITY & ACCEPTABILITY Maria Monroe-DeVita, PhD Lorna Moser, PhD Sarah Kopelovich, PhD MacKenzie Hughes, BA Bryan Stiles, BA Roselyn Peterson, BA Stacy Smith, MEd Society for Implementation Research Collaboration Conference September 8, 2017

THE FIDELITY DILEMMA Fidelity is an important implementati on outcome Shown to predict bett er clinical outcomes across EBPs/EBTs Increasingly used for quality improvement But can be costly and burdensome Several methods have been used to reduce costs and/or improve feasibility We present a pragmati c approach uti lizing providers as co-reviewers within the context of ACT fi delity assessments Primary goal: establish feasibility and acceptability

ACT: A BRIEF OVERVIEW An EBP for adults with SMI Multi disciplinary team shares caseload; no brokering

Typical ACT Team Staffing Position Team Leader Full Team Half Team (serves 80-100) (serves 42-50) 1 FTE 1 FTE Psychiatric Care Provider/Prescriber 16 hours per 50 consumers 16 hours per 50 consumers Registered Nurses 3 FTE 1.5 - 2 FTE Peer Specialist 1 FTE 1 FTE Masters’ level Clinicians* 4 FTE 2 FTE BA-level CMs* 1 – 3 FTE 1.5 – 2.5 FTE *Substance Abuse Specialist 1 FTE 1 FTE *Vocational Specialist 1 FTE 1 FTE 1 – 1.5 FTE 1 FTE Program Assistant (non-clinical) 4

ACT: A BRIEF OVERVIEW An EBP for adults with SMI Multi disciplinary team shares caseload; no brokering Services primarily provided in vivo Multi ple contacts & intensive services 24/7 Integrates other evidence-based practi ces; not just case management

A Snapshot Integratedof ACT Dual Services Case Pharmacological Disorders Treatment Treatment Psychiatric Motivational Rehabilitation Interviewing ACT ACT management Wellness Wellness Management Management Services Cognitive EBTs for Behavioral Crisis Services Psychiatric MH/COD Therapy Rehabilitation Supported Supported Employment Employment

ACT: A BRIEF OVERVIEW An EBP for adults with SMI Multi disciplinary team shares caseload; no brokering Services primarily provided in vivo Multi ple contacts & intensive services 24/7 Integrates other evidence-based practi ces; not just case management Strengths-based, person-centered while balancing asserti ve engagement

TOOL FOR MEASUREMENT OF ACT (TMACT) 47 items; 5-point anchored scales 6 subscales: 1. 2. 3. 4. 5. 6. Operations & Structure (OS): 12 items Core Team (CT): 7 items Specialist Team (ST): 8 items Core Practices (CP): 8 items Evidence-Based Practices (EP): 8 items Person-Centered Planning Practices (PP): 4 items Monroe-DeVita, Moser, & Teague, 2013

EXAMPLE TMACT ITEMS & RATING SCALE

ACT FIDELITY REVIEW PROCESS Two independent reviewers Team completes survey & spreadsheet beforehand Onsite review 1¾ days Review randomly selected charts ( 20%) Observe daily team meeting, treatment planning, community-based services Conduct interviews with team members & clients Reviewers independently rate/consensus rate Write feedback report, focused on performance improvement recommendati ons

OVERVIEW OF APPROACH Piloti ng a co-reviewer process to fi delity reviews of 91 ACT teams in Washington and North Carolina NC began in 2013, WA in 2014

16 ACT TEAMS IN WASHINGTON Compass Skagit DESC Kitsap Mental Health Services Lake Whatcom Center Compass – Snohomish WHATCOM SAN JUAN North Sound BHO OKANOGAN CLALLAM SNOHOMISH Pioneer Human Service KING King County s Salish BHO JEFFERSON GRAYS HARBOR Thurston Mason BHO KITSAP PEND OREILLE Optum Pierce BHO LINCOLN SPOKANE North Central BHO ADAMS Navos Southeast LEWIS COWLITZ CLARK SKAMANIA Southwest Washington RSA WHITMAN FRANKLIN Great Rivers BHO Community Services Northwest DOUGLAS Frontier Behavior al Health GRANT THURSTON WAHKIAKUM Spokane County Regional BHO Catholic Family & Child Services KITTITAS MASON PACIFIC CHELAN BHO PIERCE New Great Rivers Team STEVENS SKAGIT ISLAND Behavioral Health Resources FERRY GARFIELD YAKIMA COLUMBIA Greater Columbia BHO BENTON WALLA WALLA Full Team KLICKITAT Multicare ASOTIN Lourde s Comprehensiv e Half Team Between Full &

75 ACT TEAMS IN NORTH CAROLINA

OVERVIEW OF APPROACH Piloti ng a provider co-reviewer process to fi delity reviews of 91 ACT teams in Washington and North Carolina Lead fi delity reviewer based at a university (UNC, UW) or state mental health authority (NC only) Co-reviewers: Program managers/clinical supervisors Team leaders Psychiatrists, nurse practitioners Other ACT team members (WA only, starting in 2015) Co-reviewers unpaid; travel reimbursement provided

Fidelity Review Training Training Washington North Carolina Yes (group training) Yes (group training) Trainee group joins TMACT Trainer to review ACT team. All participate in data collection, interviewing, ratings, and consensus calls Single or small group trainees join pair of seasoned TMACT evaluators. Trainees collect data, make ratings, participate in consensus. Training Review of ACT Team 2 n/a Single trainee joins pair of seasoned TMACT evaluators. Trainees collect data, assist with interviews, make ratings, participate in consensus. Booster Trainings n/a Annual – TMACTer Summit TMACT WalkThrough (1 day) Training Review of ACT Team 1

TRAINED CO-REVIEWERS BY TEAM ROLE (N 53) Washington Co-Reviewers (n 33) Case Manager Vocational 3% Specialist ARNP 3% 3% North Carolina Co-Reviewers (n 20) Psychiatrist 5% Program Manager 6% Assistant Team Leader 6% Team Leader 40% CD Specialist 6% Nurse 15% MHP 18% Team Leader 45% Program Manager/ Supervisor 50%

CURRENT CO-REVIEWERS (N 32) Washington Co-Reviewers (n 18) North Carolina Co-Reviewers (n 14) ARNP 6%, n 1 CD Specialist 6%, n 1 Psychiatrist 7%, n 1 Team Leader 33%, n 6 Nurse 17%, n 3 Team Leader 36%, n 5 Case Manager 5%, n 1 MHP 28%, n 5 Assistant Team Leader 5%, n 1 Program Manager/ Supervisor 57%, n 8

SURVEY METHODS Distributed a 14-item REDCap survey to all current WA and NC ACT fi delity co-reviewers (N 32) Parti cipants were assigned a unique identi fi er to maintain confi denti ality Response Rates 30/32 responded (94% response rate) Washington: 88% (15/17) North Carolina: 100% (15/15)

ESTIMATED FIDELITY REVIEW BURDEN Expectations (When Signing Up) 1 review/year in addition to the training review (2 first year, 1 subsequent years) May be asked to complete up to 4 reviews/year Number of Fidelity Assessments Conducted Per Reviewer Washington (n 15) Mean 2.33 / Median 3 / Range 1-4 North Carolina (n 15) Mean 7.8 / Median 9 / Range 1-16 Estimated Hours Spent Per Review Mean 31.2 Median 30 Range 20 – 50 Mean 37.2 Median 33 Range 20 – 72

Are you interested in doing more or fewer than the expected number of fidelity reviews each year? Reviews WA (n 15) NC (n 15) More 73% (n 11) 33% (n 5) Fewer 7% (n 1) 0% (n 0) Same 20% (n 3) 67% (n 10)

WHY DID YOU DECIDE TO BECOME A CO-REVIEWER? Improving withinteam outcomes: “gain new ideas for my team, so I can improve my team’s fi delity and overall outcomes for our clients.” Personal interest/ investment 4%, n 2 Agency directed 4%, n 2 Facilitate content mastery of ACT 10%, n 5 Learning about/ networking with other teams 25%, n 13 Quality Improvement 10%, n 5 Increase knowledge of ACT fidelity 21%, n 11 Improve personal leadership/augment job performance 11%, n 5 Improving within-team outcomes 15%, n 8

BENEFITS

ATTITUDES TOWARDS CONDUCTING FIDELITY REVIEWS (N 25) To what extent has serving as a coreviewer provided an opportunity to better learn about ACT? Clarifies existing ACT standards Provide opportunities 4% to collaborate 8% 5 Other 12% 4.7 4 Experiencing novel ways of working within ACT & incorporating into team 42% 3 2 1 0 Familiarization with TMACT 19% Provide holistic view of teams 15%

ATTITUDES TOWARDS CONDUCTING FIDELITY REVIEWS (N 24) To what extent has serving as a fidelity coreviewer provided an opportunity to better learn about ACT fidelity or the ACT fidelity tool (TMACT)? Learning more about reviewer role 8% Exposure to high fidelity teams 12% 5 4.67 4 Understanding fidelity vis-à-vis TMACT 48% Other 12% 3 2 1 0 peer consultation 20%

ATTITUDES TOWARDS CONDUCTING FIDELITY REVIEWS (N 18) To what extent have you enjoyed your experience as a fidelity co-reviewer? Having an impact on reviewer’s own team 5% 5 4.53 Continued learning 19% 4 3 Networking with other teams/ relationship building 48% 2 Other 28% 1 0

DO YOU FEEL YOUR ACT TEAM HAS BEEN POSITIVELY AFFECTED? 100% Yes Examples: Learning tricks & tools from other teams to bring back & implement within own team Seeing how well other teams function has been a motivator to achieve same Allows reviewer to have a more objective view on items that his/her own team needs to improve on Good to know what reviewers are looking for during the fidelity review process

RELATIONSHIP BETWEEN ACT TEAMS WITH A FIDELITY CO-REVIEWER AND FIDELITY RATINGS (NC TEAMS ONLY) ACT Team Overall Average Fidelity Rating* Whether Reviewer is an ACT Team Member or Supervisor Not a Team Member or Supervisor Team Member or Supervisor 90% (n 18) 10% (n 2) (Rating 3.7 or above) 56.6% (n 30) 43.4% (n 23) Total 65.8% (n 48) 34.2% (n 25) Provisional Certification (Rating 3.0 - 3.69) Met Certification Cutoff *TMACT fidelity ratings range from 1 to 5 χ2 7.192, p .006

ACT TEAMS WITH HIGHER TMACT RATINGS Further examined teams that met certi fi cati on cutoff s (rati ng of 3.7 and higher) TMACT fi delity scores were higher among those ACT teams that had at least one team member or supervisor conducti ng fi delity reviews of other teams (r s .314, p .007)

CHALLENGES

CO-REVIEWER TURNOVER Washington Turnover: 45% (15/33) trained coreviewers are no longer conducti ng fi delity reviews 67% (10/15) co-reviewers left their teams 33% (5/15) co-reviewers were promoted North Carolina Turnover: 15% (3/20) trained coreviewers are no longer conducti ng fi delity reviews 66% (2/3) co-reviewers left their teams 33% (1/3) co-reviewers were promoted In NC, 15% (3/20) of reviewers remain as reviewers, but work for the state or at UNC now

DO YOU FEEL YOUR ACT TEAM HAS BEEN NEGATIVELY AFFECTED? 93% (n 28) No 7% (n 2) Yes Negati ve Eff ects: Scheduling diffi culti es to ensure coverage while co-reviewer was out of offi ce Team was short staffed at the ti me of review. High workload when co-reviewer returned to offi ce

DID YOU ENCOUNTER BARRIERS/OBSTACLES AS PART OF THE FIDELITY REVIEW PROCESS? Equal distributi on between states on experiencing barriers related to the process 47% No n 14 53% Yes n 16

DID YOU ENCOUNTER BARRIERS/OBSTACLES AS PART OF THE FIDELITY REVIEW PROCESS? Work burden Burden associated placed upon with review other 5%, n 1 members in Travel reviewer’s reimburseme home agency nt process 10%, n 2 40%, n 8 Travel logistics 10%, n 2 Review logistics 10%, n 2 Work burden returning to agency after a review 25%, n 5

FUTURE DIRECTIONS

PRACTICE & POLICY IMPLICATIONS Fidelity reviews of ACT teams can be resource- and ti meintensive, but are needed to ensure adherence to the model and ultimately, better clinical outcomes. Our data suggest that this potenti ally less resourceintensive fi delity review model is feasible, acceptable to program stakeholders, and benefi cial to their home teams. The signifi cant relati onship between having a team member conduct fi delity reviews and fi delity scores is encouraging suggests that reviewers are learning about best practices by conducting these reviews. Off ers a potenti ally less costly opti on for reinforcing best practi ces (and improving outcomes) within ACT & other EBPs

FUTURE DIRECTIONS Are ACT team members’ fi delity assessments as valid and reliable? What are the costs of diff erent methods of fi delity assessment? How much cheaper is this approach? What factors should be considered in implementi ng this method for conducti ng fi delity reviews in other states? Especially when fidelity is tied to certification or funding Need to further assess other teams’ percepti ons of a co-reviewer who is also an ACT team member

THANK YOU! [email protected]