In 2015, as part of sweeping new goals designed to deliver a doubling of energy efficiency and renewable energy, Governor Brown signed the California Clean Energy and Pollution Reduction Act of 2015 (De León. SB 350), and Energy Efficiency legislation (Williams. AB 802). Together, these bills were meant to accelerate the deployment of energy efficiency by authorizing utilities to deliver pay-for-performance efficiency programs, and establishing a new data-driven, streamlined approach to efficiency called normalized metered energy consumption (NMEC).

The effectiveness of NMEC as a solution depends on a portfolio-level approach to energy efficiency. However,  with its March 23rd release of the draft Rulebook for Custom Program and Projects Based on Normalized Metered Energy Consumption (NMEC), the CPUC continues to focus on regulations at the building level, and is classifying NMEC as part of the same custom review process that has hindered market innovation and held back the scaling of energy efficiency.

These comments lay out an alternative, data-driven policy path for California that is based on tests of NMEC viability conducted on roughly 50 million California investor owned utility meters. This alternative path is rooted in rigorous statistical analysis and supports the goals of AB 802 and SB 350 by relying on the proven stability of portfolio-level savings for both residential and most types of commercial buildings.

With this approach, the CPUC could focus on ensuring that statewide goals are met by regulating the outcomes of utility portfolios rather than continuing with the current system of onerous project level custom review and measure-level accounting. Utilities could leverage pay-for-performance to create markets for efficiency by procuring time- and location-based savings to support electrification, load-balancing of renewables, and non-wires alternatives--and perhaps most importantly migrate away from current deemed savings estimates to instead track and reward actual results for the grid. Private industry, investors, and insurers could provide innovative solutions and capital to fuel market growth. Building owners could be saving money on their bills. All while creating hundreds of thousands of jobs and saving the environment.

High Opportunity Programs and Pilots (HOPPs) required by AB 802 to test this approach to NMEC are underway, but these new regulations are being put in place before results are in. One example is PG&E’s residential pay-for-performance program based on CalTRACK NMEC methods run by the open-source OpenEEmeter. This pilot incorporates the use of investment-grade metered performance insurance as well as payment contracts that include bonuses for efficiency delivered during the times of day that reduce the burden of the duck curve. Similar approaches to measuring energy efficiency and pay-for-performance are also getting underway in Oregon, New York, and Massachusetts.

In contrast to the approach of the pay-for-performance pilots now being conducted in California and under development in multiple states, the CPUC’s draft NMEC Rulebook requires all site-level NMEC to undergo the same custom review process that the CPUC itself has acknowledged needs review, and has already been flagged as problematic by the California Efficiency + Demand Management Council filing on SB 350 goals, where they cautioned that “savings estimates for these sectors will be challenging until there are significant improvements to the current custom project ex ante review by the CPUC staff and its consultants.” This issue is also the the subject of the Council’s legislation SB 1131.

At the heart of the CPUC’s treatment of custom projects is the reliance on traditional site-level M&V methods for the definition of “normalization”. As written, the new CPUC rule would require detailed and costly engineering and adjustments to achieve ASHRAE 14 level “accuracy” for individual buildings and customers.

Classifying NMEC the in the same category as traditional custom projects will add thousands of dollars in new transaction costs for building owners and efficiency entrepreneurs, and require an army of consultants that are tasked with providing regulatory oversight.

If the CPUC sticks with this classification, custom engineering to adjust baselines and to account for non-routine events in the performance period may be required on up to 80 percent of all buildings in the state. This finding varied only slightly even when using machine learning and other custom models.

Adhering to this site-level criteria would create substantial barriers to scaling efficiency in the market. Fortunately, however, we now have the data to show that there is a much more efficient approach to achieving confidence in NMEC outcomes through policies based on portfolios rather than individual buildings.

‍

Figure 1. Distribution of site-level CVRMSE from Caltrack 1.0 models in a sample of 80,000 office buildings across California. Half of all sites would require engineering adjustments to the baseline to achieve the CPUC / ASHRAE 14 standard.

‍

‍

Achieving Confidence in NMEC Through Portfolios

ASHRAE Guideline 14, from which the CPUC Guidelines are derived, specifies that the maximum acceptable level of savings uncertainty at the building-level is 50 percent (at the 68 percent confidence level). A better strategy is through aggregation of individual projects, which can, in most cases, deliver less than 25 percent portfolio-level savings uncertainty (at the 90 percent confidence level) with very reasonably sized portfolios.

As part of the CalTRACK 2.0 methods update process, the CEC contracted with OpenEE to test the CalTRACK methods  on non-participants across nearly 50 million meters using the open-source OpenEEmeter. The test covered all four California IOUs, building sectors, and climate zones.

CalTRACK empirical tests found that NMEC models can achieve the ASHRAE 14 uncertainty thresholds at a portfolio level for both residential portfolios and most types of commercial buildings, without the need for extensive site-level non-routine adjustments on most buildings.

These findings mean that by focusing on portfolio-level uncertainty, we can maintain a high level of accountability while avoiding the overwhelming transaction costs associated with the CPUC custom review process, as well as the customer incurred costs of detailed ongoing engineering adjustments for NMEC projects.

The graphic below shows the results of millions of CalTRACK NMEC models that were run on actual commercial building data to identify those sectors where, (A) NMEC at a portfolio-level can apply, (B) requires submetering or adjustments, (C) needs additional variables (i.e. schedules for schools).

‍

It just so happens that the building types that would be eligible for this type of approach are also the ones that have generated the majority of savings in the 2013-2015 set of programs.  Most of the existing program interventions can transition to a portfolio approach.    

‍

The power of portfolio aggregation is illustrated in the following chart. The fractional savings uncertainty at the portfolio level is plotted against portfolio size for random samples of office buildings in California, with the site-level filter set at a very lax level of 100 percent. For example, with average site-level savings as low as 10 percent, a portfolio of a little over 200 buildings can easily achieve an uncertainty of lower than 25 percent.

‍

Using this data we can now demonstrate that NMEC can achieve regulatory requirements for confidence through portfolios across most building sectors. Rather than tight control of each project, permissive building-level thresholds should be set in order to include a greater number of acceptable buildings, and instead focus on achieving acceptable portfolio-level confidence.

The utility and ratepayer will almost always have sufficient portfolio assets to ensure that mutual benefits are being delivered with enough confidence to assess portfolio level impacts, without requiring that adjustments on each building be tracked, counted, and regulated. As the CPUC has acknowledged, there is a need for “a regulatory framework to enable scaled implementation”.  

Based on empirical research, a portfolio pathway for utilities to continue using NMEC as a means to enable markets to scale energy efficiency, is the regulatory framework needed to “reduce time and cost of ex-post evaluation and to facilitate deployment of whole building approaches for commercial buildings.”

Taking a portfolio approach allows the CPUC to regulate the confidence of energy savings claims made by utilities based on the performance of their portfolio to ensure impact was achieved, without having to micromanage the details of each project.

In order to enable the scale required to achieve the State’s SB 350 efficiency goals, it is necessary that the CPUC NMEC Rulebook includes a pathway that enables a portfolio based approach to achieving confidence in savings. The following three simple recommendations would ensure savings are delivered that meet the level of confidence specified by the CPUC and be counted towards the State’s goals:

• For use cases where confidence in portfolio-level performance is required (e.g. aggregator-driven pay-for-performance, non-wires alternatives (NWA) procurements, electrification), we recommend using a permissive building-level CVRMSE threshold (100 percent is recommended as a default in CalTRACK), but requiring that a portfolio-level fractional savings uncertainty threshold be respected.
• Establishing a portfolio-level uncertainty threshold will depend on the use case and should be set by the procurer. For example, an NWA procurement may require less than 15 percent uncertainty, while a regular pay-for-performance program may require 25 percent. An alternative approach could use a discount rate based on the uncertainty of a portfolio.
• For use cases where high confidence in individual building results is required (e.g. customer-facing performance-based incentives), ASHRAE Guideline 14 thresholds should be used.

This exercise has also illustrated that transparent methods and open source tools are critical to bringing accountability to ratepayer funded programs. Replicable standards build trust in the system and allow proper alignment of incentives up and down the chain of market actors, including regulators.

We hope that the CPUC will consider this newly available information on the effective application of portfolios to increase confidence in NMEC results, and develop policies that can ensure ratepayers see real results at the meter as many believe AB 802 and SB 350 intended.

Regulators and policy makers need to embrace SB 350, AB 802 and the NMEC framework as an opportunity to leverage smart meters and data to enable markets that will deliver the business and technical innovation and private investment required to achieve the goal of doubling energy efficiency by 2030.

The good news is that the current NMEC Rulebook is only a draft, and may only apply to a small group of projects. There remains an opportunity for utilities, industry, advocates, and like minded regulators to provide the CPUC a path forward that can achieve confidence in NMEC and enable the scale required to deliver on California’s aggressive state efficiency goals.

_____________

The empirical analysis provided in this paper is based on statewide out of sample testing of the OpenEEmeter running CalTRACK Methods v.1. Results will vary by building type, location, climate, data granularity, and potentially other factors, but can be calculated given the data for a specific sample of buildings from the target population. While aggregation can dramatically reduce portfolio-level savings uncertainty, it does not eliminate inherent systemic biases due to model choice, implementation variance, imbalanced application of non-routine adjustments, unaccounted for independent variables, or population trends. Developing robust, transparent, and standardized methods that account for the major sources of bias is the foundational to portfolio aggregation.