Energy consumption patterns at the building level are generally difficult to model, due to the impact of human behavior. While statistical models can control for observable factors such as weather and occupancy patterns, human behavior is more stochastic and difficult to measure. However, energy consumption over portfolios of large numbers of buildings tend to be much more stable and manageable. Portfolios are generally composed of buildings that share common characteristics (e.g. building type, location, installed measures etc.)

Fractional Savings Uncertainty

The uncertainty in model predictions can be expressed through the savings uncertainty at a specified confidence level. When this is expressed as a percentage of the average metered  savings, it is referred to as Fractional Savings Uncertainty (FSU). Smaller FSUs indicate higher confidence in the savings estimates. For example, a building or group of buildings that have saved 1000 kWh on average with a 10% FSU, will have a confidence interval between 900 and 1100 kWh, while a different group of buildings with the same average savings and 50% FSU will have a confidence interval between 500 and 1500 kWh. ASHRAE Guideline 14 specifies that the maximum acceptable level of fractional savings uncertainty at the building-level is 50 percent (at the 68 percent confidence level). CalTRACK tests have shown that aggregation of individual projects, can, in most cases, deliver less than 25 percent portfolio-level savings uncertainty (at the 90 percent confidence level) with very reasonably sized portfolios. It must be noted that, by its nature, FSU decreases with larger portfolios and deeper savings.

The Case for an Aggregated Approach

CalTRACK empirical tests found that NMEC models can achieve promising uncertainty thresholds at a portfolio level for both residential portfolios and most types of small and medium commercial buildings, without the need for extensive site-level non-routine adjustments on most buildings. The utility and ratepayer will almost always have sufficient portfolio assets to ensure that mutual benefits are being delivered with enough confidence to assess portfolio level impacts, without requiring that adjustments on each building be tracked, counted, and regulated and without having to micromanage the details of each project. Moreover, site-level thresholds on model fit metrics (e.g. CVRMSE) tend to exclude a large proportion of potential candidates for energy efficiency retrofits, while offering little to no increased confidence in portfolio-level savings.

Use Case Recommendations

While the aggregated approach offers a pathway for scaling energy efficiency in certain types of buildings, it is not a panacea and is not suitable in sectors with fewer buildings or where the buildings have a relatively high rate of energy consumption. CalTRACK includes the following set of recommendations for applying program rules using an aggregated approach:

• For use cases where confidence in portfolio-level performance is required (e.g. aggregator-driven pay-for-performance, non-wires alternatives (NWA) procurements, electrification), we recommend using a permissive building-level CVRMSE threshold (100 percent is recommended as a default in CalTRACK), but requiring that a portfolio-level fractional savings uncertainty threshold be respected.
• Establishing a portfolio-level uncertainty threshold will depend on the use case and should be set by the procurer. For example, an NWA procurement may require less than 15 percent uncertainty, while a regular pay-for-performance program may require 25 percent. An alternative approach could use a discount rate based on the uncertainty of a portfolio.
• For use cases where high confidence in individual building results is required (e.g. customer-facing performance-based incentives, large commercial and industrial sectors), ASHRAE Guideline 14 thresholds should be used.