‍

On August 14th, 2020, high temperatures stressed the California grid to a breaking point, resulting in statewide blackouts. Real-time marginal pricing skyrocketed to over $1000/MWh, and the California Independent System Operator (CAISO) called on behind-the-meter demand response resources to help reduce demand and stabilize the grid. 

Companies such as OhmConnect alerted their customers to a significant event, incentivizing them to reduce demand during the peak hours in the early evening. However, OhmConnect and other third-party demand response providers could not deliver their full potential, nor were they compensated for the megawatts they did deliver. A lack of available data, inconsistent methods for accurate measurement, complex market rules, arbitrary market caps, and heavily discounted value for delivered load reductions contributed to these problems.

Because the day-ahead locational marginal pricing (LMPs) did not reach the price cap during the rolling blackouts, many third party demand response providers did not dispatch their resources at all. OhmConnect, which did dispatch to help alleviate the crisis, ended up losing hundreds of thousands of dollars because it dispatched during the August 14th grid crisis.

To help OhmConnect, utilities, and regulators understand how and why this happened -- and more importantly, what can be done to prevent it from happening again -- Recurve carried out a load impact analysis of a fraction of OhmConnect’s participants during the hours of the August 14th demand response event.

This analysis conclusively demonstrated the efficacy of demand response and showed how it is possible to measure it as a grid resource accurately. However, for OhmConnect and other Demand Response providers to achieve their full potential, regulators must create fair market rules that leverage smart-meter data and advanced, settlement-quality revenue-grade metering to allow demand flexibility to compete on an equal footing as other resources.

Current Bespoke M&V for Demand Response Is Holding the Market Back

OhmConnect is a Residential Demand Response Virtual Power Plant (VPP) developer based in California with over 150,000 customers in the state. OhmConnect pays people to save energy in response to grid events and is in turn paid by utilities and CAISO for demand response and resource adequacy. During periods of peak usage, OhmConnect engages users through a combination of communication, gamification, economic rewards, and direct control of grid-edge devices to reduce demand. 

Under such conditions as encountered on August 14, any amount of energy saved or shifted off-peak is valuable. However, gauging the impacts of a demand-side intervention to deploy the most effective interventions and fairly compensate DR companies for the savings they deliver requires establishing a robust counterfactual that estimates customer energy consumption in the program’s absence.

Lack of consistent measurement is a significant impediment to companies throughout this space. In an article in Microgrid Knowledge covering their recently announced Alphabet (Google) funded $100M virtual power plant, OhmConnect CEO Cisco DeVries commented that, 

“Depending on which agency we’re dealing with, there are five different ways that they count how our energy reduction is measured. And they change those all the time...In order to create a long-term investment in flexible load, it’s important to know how to count the megawatts. This affects what the consumers are paid as well as the types of devices that are controlled.”

Current tactics for quantifying the impacts of behind the meter resources such as those used to assess demand response and resource adequacy in California rely on methods designed before the ability to access and process smart meter data became widely available. In addition, the lack of an adequate privacy framework has restricted access to the data sets needed to identify appropriate comparison groups and thus enable accurate measurement. As a result, different methods for calculating savings produce different results. 

The California Load Impact Protocols illustrate this problem. With 149 pages of flowcharts and narratives describing general methods for a range of use cases to measure demand response resources, the Protocols give multiple answers to the same question from different agencies and evaluators, even when looking at the same event. Each of these methods is “correct”  within the context of its implementation code and hundreds of embedded engineering choices. 

However, all of these Protocols are limited by simplistic and easily biased methods, along with a lack of data.  For example, the 10-in-10 calculations measure the DR baseline by taking an average of the customer’s last ten days of usage at the DR event time. This approach fails to account that the event day is usually much hotter than other days by its very nature. 

In fact, California set multiple heat records during the August heat storm. Lacking access to population data for a comparison group, a “same day adjustment” is made to scale the event to that day's energy use. There are multiple methods to calculate this adjustment, each producing different results with a range of known biases. In addition, the adjustment is often capped, though the caps can vary depending on baselining methodologies (80-120% or 71-140%).

The problem is compounded by the fact that a customer’s baseline is affected explicitly by other DR events. This means that responding to an event one day can reduce the value of future event periods and create a dynamic where long-term and consistent energy efficiency or seasonal load shaping to fight the duck curve undermines the value of event-based DR. 

In its 2018 Load Impact Evaluation for OhmConnect’s DR Resource report, Convergence Data Analytics summed up the need for a comparison group and more advanced methods:

These issues are not specific to OhmConnect. Jason Michaels, Chief Commercial Officer at Leap, a company whose distributed energy exchange platform integrates residential smart thermostats, commercial EV charging stations, municipal water treatment facility pumps, and other technologies, recently explained the problem:

“Almost by definition, current baseline methods understate performance on the days when the grid has the greatest need for demand response, resulting in reduced incentive to support the grid in future events. More accurate methods for measurement and verification will help companies like Leap bring more flexible demand from local distributed energy resources to help balance the grid."

Utilizing Advanced Open-Source M&V to Provide Revenue-Grade Settlement Quality Measurement

To demonstrate how it is now possible to overcome these barriers, Recurve was able to work with OhmConnect customer data and privacy-protected population data from our research partner MCE to provide revenue-grade analysis of the net impact to load shape from the OhmConnect virtual power plant during the August 14th grid blackout event.

To conduct its analysis, Recurve combined the open-source OpenEEmeter and GRIDmeter™ methods and code with the Energy Differential Privacy methods that have been developed with the partnership of DOE, NREL, and MCE. When applied consistently, this approach overcomes current measurement barriers and improves confidence in the delivered DR resource. 

In this way, DR resources can be correctly valued and fully deployed to stabilize and decarbonize the grid. To demonstrate this approach, Recurve carried out a load impact analysis of a fraction of OhmConnect’s participants during the hours of the August 14th demand response event.

Recurve’s measurement methods involve a two-step calculation. First, a meter-level hourly baseline is created for every treated and comparison pool customer using the hourly CalTRACK methods and the OpenEEmeter. The CalTRACK calculations yield hourly counterfactuals for each meter, based on a model of past consumption adjusted for weather and occupancy.

Next, the GRIDmeter™ methods and code draw a comparison group sample from the comparison pool. At the heart of these methods, multidimensional stratified sampling on key usage characteristics is utilized to produce a comparison group that accurately reproduces the load shapes across the entire distribution of treatment customers. 

Using the CalTRACK model outputs, the difference between the observed hourly consumption and the counterfactual is measured for both treatment and comparison group customers. Taking the difference (known as the “difference of differences”) between these two sets of measurements yields the comparison-group adjusted hourly load impacts.  

This calculation is considered revenue-grade because it is based on open-source methods and code and is transparent and auditable by all parties. 

These results are also considered settlement quality, as they derive the load-shape impact that is directly attributable to OhmConnect, net of exogenous factors. These results represent the net impact of OhmConnect’s VPP to the grid.

Revenue-Grade Demand Flexibility

The figure below shows observed consumption (open circles) and the CalTRACK hourly counterfactual (dashed curve) results for the treatment group of OhmConnect customers on the date of the demand response event (Aug. 14, 2020).

The next figure shows analogous results for the comparison group selected by the open-source GRIDmeter™. The difference between the curves in the figure below represents the exogenous change in the comparable population. For example, perhaps these event hours were unusually hot (likely), or customers were hearing on the radio about a flex alert and taking action even without OhmConnect. This comparison group enables us to net those effects out.

The fuzzy line around the comparison group is noise that is introduced via Energy Differential Privacy. This noise enables the use of population data by protecting the privacy of individual customer records.

‍Settlement-Quality Net Impact to Loadshape

Finally, the next plot combines the measurements of the first two figures to show the final load impact measurement for treated customers. We call this value settlement-quality because it is net of exogenous effects (such as more than one flex alert being called during the same period or the effects of extreme weather) and represents an accurate analysis of the impact OhmConnect had on the load shape of its customers.

With an Energy Differential Privacy protected comparison group, Recurve is providing a transparent revenue-grade calculator of the settlement quality net hourly impact to demand delivered by OhmConnect’s customers in the MCE service territory during this event window.

This analysis conclusively demonstrates both the efficacy of demand response and the ability to measure it accurately as a grid resource.

Given the vast potential of event-driven demand response and long term demand flexibility, we believe it is time to treat them as resources on par with more traditional supply-side solutions. In addition to providing reliable and low-cost solutions for grid challenges, demand response and flexibility save customers money on their energy bills and reduce the cost of capital investments necessary to improve their buildings by making them grid-interactive.

It is time for regulators to facilitate the transition to a more nimble, decarbonized grid by leveraging smart meter data and advanced settlement-quality, revenue-grade metering to provide fair market rules that allow demand flexibility to compete on an equal footing with other resources.

Learn more about this research and findings by reading Recurve’s full report for NREL and DOE, which utilizes customer data provided by OhmConnect and population smart meter data provided by MCE.