J U LY 9 , 2 0 1 5 Proposed Changes to the Demand Response Baseline

J U LY 9 , 2 0 1 5 Proposed Changes to the Demand Response Baseline Method Simplifying administration and maintaining baseline accuracy Laurie Corcoran S E N I O R A N A LY S T D E M A N D R E S O U R C E S T R AT E G Y

BACKGROUND Why modify the baseline method?

Background As mentioned at the May 2015 Markets Committee meeting, the current 90/10 baseline methodology is administratively complex The ISO is proposing to replace the 90/10 baseline methodology with a method which performs comparably or better in terms of accuracy, bias, variability, and reduces the administrative burden Additionally, the new methodology must calculate baselines for two additional day types at the time of full integration: Saturdays, and Sunday/Holidays 3

Current Baseline Methodology The ISO’s current 90/10 baseline methodology: – Estimates the expected energy consumption of an asset by simulating a 10-day rolling average of meter data from previous days on which the asset was not dispatched – Under this approach, the ISO takes 90 percent of the previously calculated baseline for each 5-minute interval of an Operating Day, and then adds to each interval 10 percent of the interval meter data from the most recent day on which the asset was not dispatched Comprehensive analysis of baseline methodologies conducted by DNV GL/KEMA (“DNV”) for PJM in 2011 demonstrated that ISO’s 90/10 baseline methodology was more accurate, and had lower bias and variability compared to other known methodologies 4

Administrative Issues The meter data used to compute the 90/10 baseline remains embedded in the calculated baseline forever – i.e., the 90/10 baseline methodology is said to have an “infinite tail” – The ISO and participants must store and manage many months of data to potentially recalculate baselines if any meter data is found to be incorrect even though the impact of any set of data becomes de minimus over time Additionally, the 90/10 baseline methodology involves a recursive algorithm in which today’s calculated baseline is a function of previously calculated baselines, so that: – Every time a data correction is made, every subsequent baseline, starting with the baseline for the first day that was computed using the corrected data, needs to be recalculated; – Any data corrections during a resettlement period impact all future settlements; and – When data issues are identified following the end of a data resubmission period, unwinding the impacts of such data is complex. 5

PROPOSED BASELINE MODIFICATIONS

Baseline Analysis In 2014, the ISO analyzed alternative baseline methodologies in comparison to the ISO’s 90/10 baseline methodology to determine which one(s) would: – Achieve comparable or better accuracy, bias, and variability – Address the administrative issues with the 90/10 methodology The analysis found that a “Mean 10 of 10” baseline using a 6 week look back period: – Performs comparably to or better than the current 90/10 methodology in terms of accuracy, bias, and variability* – Reduces the administrative burden by: Eliminating the infinite tail Addressing the recursive nature of the 90/10 methodology – Simplifies the calculation of Saturday and Sunday/Holiday baselines at the time of full integration * Details on the analysis, including the metrics used to determine accuracy, bias, and variability are available in the Appendix section of this presentation. 7

The ISO Proposes to Replace the Current Baseline with the Mean 10 of 10 Baseline Methodology The Mean 10 of 10 baseline is a 10-day rolling average of meter data from previous days on which the asset was not dispatched – Baseline accuracy, bias, variability are preserved – To address the infinite tail issue, the historical look-back period is limited to 6 weeks* – To address the recursive nature of the ISO 90/10, the baseline calculation uses only meter data from ten historical days – i.e., prior day baselines are not used to compute future baselines – Baselines for Saturdays and Sunday/Holidays are 6-day rolling averages of meter data* *The decisions to limit the historical look back period to 6 weeks, and calculate Saturday and Sunday/holiday baselines as 6-day rolling averages, are explained in the Appendix section of this presentation. 8

Tariff Changes There are two places where the Tariff must be changed to replace the 90/10 with the Mean 10 of 10 baseline methodology: – Section III.8A of the Tariff The Mean 10 of 10 methodology will be used to calculate baselines for single day type (weekdays) – Section III.8B of the Tariff For full integration (now proposed to be implemented on June 1, 2018), the Mean 10 of 10 methodology will be used to calculate baselines for three day types (weekdays, Saturdays, and Sunday/Holidays) 9

Proposed Stakeholder Schedule Timing July 2015 August 2015 Action Introduce a proposal to modify the ISO 90/10 baseline methodology. Proposed market rule changes to implement a Mean 10 of 10 baseline methodology are introduced for stakeholder review and consideration. September 2015 Markets Committee vote on market rule changes. October 2015 Participants Committee vote on market rule changes. Filing of rule changes with the Commission. 10

11

APPENDIX Alternative Baseline Analysis

ISO Analysis of Baseline Administration Issues Possible options to address the 90/10 baseline administration issues – To address the infinite tail issue, use a finite look back period – To address the recursion issue, calculate baselines using only meter data – i.e., do not include prior day baselines in the baseline calculation Because the ISO 90/10 baseline methodology is recursive by nature, other methodologies needed to be identified and analyzed 13

Solutions to Baseline Administration Issues Consider a finite look back period of approximately 6 weeks – Multiple baseline analyses in the last 10 years have considered the length of the look back period, and periods of 30 to 60 days have been studied – Initially, the ISO considered a look back period of 30 calendar days, however, in a 30 day period there may be only 4 Saturdays and 4 Sunday/holidays – In order to ensure more historical data for Saturdays and Sunday/holidays is available for the baseline calculation, while respecting a finite look back period, the ISO extended the look back to 30 non-holiday weekdays (approximately 6 weeks) For non-holiday weekdays, limit the look back to the most recent 30 days of that day type For Saturdays and Sunday/holidays, limit the look back to the most recent 6 days of that day type Consider calculating baselines using only meter data, and not using previously computed baselines – Avoids issue with ISO 90/10 where any changes to previous day baselines impact future baselines 14

Analysis of Alternatives to the Current ISO 90/10 Baseline Methodology Given the recursive nature of the ISO 90/10 methodology, analysis was conducted to identify a new approach that: – Achieves comparable or better accuracy, bias, and variability – Addresses the administrative issues with the 90/10 methodology With analytical support provided by DNV, the ISO examined: – Six alternative baseline methodologies in comparison to ISO 90/10 – The impact of limiting the look back period to approximately 6 weeks, and calculating baselines using only meter data – Baselines for 3 day types: Non-holiday weekdays, Saturdays, and Sunday/holidays – Adjusting baselines using the Real-Time Forecast Adjustment Method* (“dynamic baseline adjustment”) accepted by the FERC on 1/9/2015 (See ISO New England Inc. and New England Power Pool, 150 FERC ¶ 61,007) * http://www.iso-ne.com/static-assets/documents/committees/comm wkgrps/mrkts comm/mrkts/mtrls/2014/may672014/a03 iso presentation 05 06 14.pptx 15

Alternative Baseline Methodologies Median 10 of 10 baseline – Median of the ten most recent non-event, non-holiday, weekdays Mean 10 of 10 baseline – Simple average of the ten most recent non-event, non-holiday, weekdays Weighted 10 of 10 baseline – Weighted average of the ten most recent non-event, nonholiday, weekdays. Maximum weight is given to the 1st day or closest day (15%) and the weight assigned to each next day is gradually decreased giving the least weight to the 10th day or the furthest day (6%) Median 90-10 baseline – This baseline is a combination of the Median 10 of 10 and the Weighted 10 of 10 baselines Top 5 of 10 baseline – Simple average of top 5 of the ten most recent non-event, non-holiday, weekdays, where the top days are based on the daily average hourly load. If there are fewer than ten non-event days in the last 30 days, then fill in as needed with most recent event days Middle 2 of 10 baseline – Simple average of middle 2 (i.e., median) of the ten most recent non-event, non-holiday, weekdays. The two middle days of the 10 are determined based on the daily average hourly load 16

Analysis of Alternative Baseline Methodologies For purposes of the analysis, the current ISO 90/10 baseline methodology, incorporated the following: – The dynamic baseline adjustment – Saturday and Sunday/holiday baselines were calculated from 10 days of the same day type For all alternate baseline methodologies, the 10 most recent non-event, non-holiday weekdays were chosen from the look back period as inputs to the weekday baseline calculation – Where less than 10 non-event, non-holiday, weekdays were available, the most recent interruption weekdays were included as needed – For Saturday and Sunday/holiday baselines, the most recent 6 days of the same day type (whether non-event or event days) were chosen as inputs 17

Comparison Metrics Alternate baseline methodologies were compared on aggregate metrics for accuracy, bias, and variability – Accuracy is a measure of how closely the baseline predicts the asset’s actual load – Bias is the systematic tendency of a baseline method to over-predict or underpredict actual load – Variability is a measure of how well the baseline predicts actual load under many different conditions (e.g., time of day, day of week) and across many different customers Baseline performance was measured during simulated hourly events – Each baseline is adjusted as if the asset were asked to reduce for an hour, on the hour, for all hours included in the analysis – The difference between the adjusted baseline and actual load (error) in each hour serves as basis of comparison Errors are calculated for each asset, for each 5-minute interval and then integrated to hourly values 18

Comparison Metrics (cont.) Basis of comparison – Actual Error: adjusted baseline minus actual load – Relative Error: actual error divided by average annual hourly load Metrics (calculated across all assets) – Mean actual error Measures bias: reflects the actual magnitude of asset load; may be affected by outliers – Median Relative Error Measures bias: gives assets the same weight regardless of their actual size – Median Relative Root Mean Squared Error (“RRMSE”) or Accuracy Combined measure of bias and variability: weights large errors more heavily than small errors – 95th percentile of relative standard deviation Measures variability: gives an estimate of variability (on a relative basis) at higher level 19

Comparison Metrics (cont.) Additionally, metrics were analyzed based on various classifications – Day type (weekday, Saturdays, Sunday/Holiday) – Season (Summer, Winter) 20

Comparison of Alternative Baselines * Mean 10 of 10 ISO 90/10 Top 5 of 10 Median 10 of 10 Median 90-10 Middle 2 of 10 Weighted 10 of 10 *Calculated across all assets and seasons between October 1, 2012-September 30, 2013 21

Comparison of Alternative Baselines – Excluding Lowest Performers * Mean 10 of 10 ISO 90/10 Median 10 of 10 Median 90-10 Weighted 10 of 10 *Calculated across all assets and seasons between October 1, 2012-September 30, 2013 22

Outcome of Comprehensive Analysis A group of three methodologies (Mean 10 of 10, Weighted 10 of 10, Median 90-10) consistently performed better than Median 10 of 10, Top 5 of 10, and Middle 2 of 10, and performed very similarly to each other and to ISO 90/10 Out of the top performing alternatives, the ISO recommends the Mean 10 of 10 methodology to replace ISO 90/10 due to its simplicity 23