In the last few years, a growing number of utilities have embraced performance-based markets for energy efficiency as an alternative to traditional deemed savings programs. In these new programs, aggregators in the market compete for customers and to optimize a cash flow that is based on metered savings (NMEC in California speak) valued and paid based on what is delivered over time.

Pay-for-performance markets for efficiency and other behind-the-meter flexible resources hold the promise of unlocking innovation, allowing utilities to integrate time- and location-sensitive demand flexibility into an ever-cleaner grid, and drawing in private financing at the scale needed for these resources to reach their decarbonization potential.

They could also very easily devolve into a mess. 

Historically, flexibility and energy efficiency savings have been calculated using a diverse set of methods with few clear standards for the hundreds of little choices made by engineers. However, continuing this approach in a pay-for-performance world simply won’t work.

If there aren't pre-defined and easily recognizable weights and measures, we are going to spend all our time arguing about who's got the right answer. The problem is, when it comes to efficiency, there is no single “right answer.” 

The only viable solution to this problem is to use transparent and reproducible methods and open-source software so that all those little engineering knobs have a default position that can easily be specified in a contract, and everyone in the transaction can replicate and verify the numbers.

Let's look at a real-world example to understand the sensitivity of these calculations and why standardized methods are required. 

We start with an actual whole-building retrofit that was completed on a home in Tracy, California. To determine how much the retrofit saved, we used the latest version of the OpenEEmeter, which can be set to run CalTRACK Methods, or configured based on custom settings. In this example, we ran the OpenEEmeter twice, making just a simple tweak to how the software is configured.

In our first calculation, we deviated from the CalTRACK methods in one simple way: we picked the weather station for normalization based on the nearest location--a perfectly reasonable and typical method. The results showed that the building saved 15 percent of its baseline gas consumption since doing the retrofit. 

We then re-ran the OpenEEmeter using the full CalTRACK methods, which, for weather station assignment, include both distance and climate zone. As a result, the customer’s metered gas savings decreased dramatically, to 12 percent of the baseline-- a 20 percent drop. 

Which answer is right, 12 percent or 15 percent?‍

In this example, the Stockton Airport weather station is 21.8 miles from this particular home, while the Livermore Airport station is 20.5 miles away. 

Using Livermore, the closest station, seems logical. However, while the Stockton station is slightly further from the home, it is in the same climate zone, while Livermore is not. 

If you specify CalTRACK, the correct weather station is Stockton and the correct savings are 12 percent. 

If you didn’t specify, there is no "right answer.” 

Parties are left to fight over a 20 percent difference in savings, and payments for performance, based on a weedy technical detail that has historically never had a clear, correct answer. However, this one simple choice might be the difference between an aggregator making or losing money, or a utility overpaying for savings.

CalTRACK Methods and OpenEEmeter were developed to solve this problem.

The CalTRACK Methods address hundreds of potential discrepancies through an open process that tests and specifies M&V methods that cover the full calculation of savings, including describing how raw data should be cleaned, assigning weather stations, regression modeling, generating uncertainties, aggregating results, and hundreds of other small but critical method choices and edge cases that occur within each of these steps.

For the particular issue above, the CalTRACK Method for weather station matching was developed by testing potential options against a unit test designed to compare different approaches. CalTRACK Section 2.4 specifies the best performing method, taking climate zone, data sufficiency checks, and proximity into account -- a method that was proven to provide a better match and more accurate results.

When the OpenEEmeter is run using CalTRACK Methods, everyone gets the same answer based on defensible methods and data science that can be specified in a contract and verified to have been implemented correctly. The checklist on the right lists the minimum steps that must be verifiable for each asset to declare a calculation CalTRACK compliant. 

It’s clearly not enough to just review and approve NMEC software, and while open source is critical so every party can run the calculations and be confident in implementation, it isn't by itself sufficient. To be sure of a result, one must be sure of the methods being used, how they are implemented, and be able to verify the software was run correctly at every step, each time a savings calculation is generated. 

Because money is changing hands based on the results of these calculations, it is critical that there is an agreement in advance as to exactly how they are run, and that any deviation from what is agreed to is approved by both parties. NMEC calculation is part of a contract for payment and needs to be explicit and reproducible by everyone in an NMEC transaction. If you deviate from the agreement, those are custom changes and need review. There can be no black boxes.

NMEC calculations are the keys to the cash register. Transparency and the ability to verify is required -- weights and measures must be standardized and available to all parties for a market to function.

If NMEC is not transparent and standardized, we are back to square one -- efficiency as usual. A system in which every project is “custom” and every party has to look over each other’s shoulders and argue about whose black-box results are better. This has already proven it is not a recipe for scale. 

It’s a good thing that California regulators, utilities, and stakeholders spent the last seven years developing just such a transparent and standardized open-source system to support metered efficiency, in the CalTRACK Methods and the OpenEEmeter.

Everyone can participate in the CalTRACK Methods process and has the same access to the OpenEEmeter code (hosted by Linux Foundation Energy)  for free, and without restriction.

It’s time to stop arguing about proprietary interests and black box math and instead collaborate on transparent, open-source measurement so we can focus on what matters: deploying and achieving scale in the market.

‍

‍