As energy efficiency programs shed their legacy programs and ritualistic black box evaluations, innovation is emerging in the form of new market platforms, consumer-oriented business models, and new technologies that drive greater efficiency results.

Even as the larger direction in the industry is evident, massive uncertainty hovers around program evaluation. Our current system spends upwards of $250M a year on a morass of evaluation, measurement, and verification (EM&V) studies, where results can vary by double digits depending on which evaluator is running the calculations, or what software is in use -- even when vendors are using very similar “methods.” This uncertainty has proven to be a deal killer for investment.

A consistent, replicable, and transparent system for measuring the results of energy efficiency is the cornerstone of future markets. As we move towards competitive markets in which contractors are paid on performance, rather than expected savings, it is essential that we have a standard and replicable weights and measures in place in order to create manageable risk that markets can invest in.

Rather than compete to convince regulators and utilities that one efficiency “yardstick” is better than anyone else’s (and then, if successful, keep the exact length of that yardstick a secret), we need to come together and agree what a yard is, and then make measuring it ubiquitous, transparent, and available to all parties.

Ultimately, no party can win by having a proprietary “better” unit of measure. The key to efficient trade and markets is confidence that all parties measure and value outcomes based on the same metrics.

If we can develop and improve measurement with the goal of convincing regulators and procurement at utilities that efficiency is a reliable resource, we all win.

For efficiency to succeed, we need to remove the barriers to measurement and encourage innovation where we need it most, in business models, financing, and technology.

Why Does Efficiency Need an Open Source Weights and Measures?

Weights and measures are some of the earliest tools used by humans. Early units of measurement came from parts of the body or people’s surroundings, such as a hand or a stone. Of course, this created as many problems as it solved because stones and hands come in different sizes. As civilizations developed, weights and measures were standardized both to make sure people didn’t get cheated and to enable more efficient markets.

When it comes to efficiency, things get complicated when you consider that we need innovation in the science of energy efficiency measurement to convince regulators and procurement departments that efficiency is a real resource, and we simultaneously need a standard measurement that enables markets to trade efficiency in a consistent, standard way.  

Putting a standard in place creates a challenge. How do we balance the market need for transparent, replicable, and consistent measurements so that performance risk can be managed to facilitate investment, while simultaneously advancing the state of the art in how savings are measured?

At first blush, this sounds impossible. Standardization and innovation seem on their face to be opposites. Standardization asks you to do the same thing everyone else is doing, while innovation demands that you do better. Both have enormous benefits, but one seems to come at the expense of the other.

Open source is the solution to deliver on both of these goals. Real success will come when we have thriving markets that encourage innovation on top of an open system of measurement that is available to all parties. Innovation will occur through collaboration, as the entire industry benefits from the confidence that savings are real and valuable.

Why Aren't “Methods” Enough?

Open and standard methods are an important layer in the effort to achieve consistent and replicable measurement.

Open Energy Efficiency (OpenEE) fully supports a methods-based approach. For the last four years, OpenEE has been leading the development of the CalTRACK standard, a collaboration that includes both the CEC and CPUC as well as all four IOUs, DNV-GL, EnergySavvy, NREL, and others. OpenEE and was the original contributor to the BPI / ACCA ANSI Quantification of Energy Efficiency standard that is underway. We believe both of these efforts represent great examples of how collaboration can lead to innovation.

Most EM&V is based on standard “methods.” Evaluators can choose from the Uniform Methods Project, ASHRAE 14, and PRISM, among others. However, it is clear that printed methods do not go far enough. The lack of consistent results, even when using the same methods, is endemic. As OpenEE has worked alongside state regulators, utilities, and EM&V consultants, it has become clear that significant variance exists even in the seemingly straightforward metric of weather normalized monthly savings based on the same methods. There are simply too many little details that come out in implementation - ask five engineers to calculate the savings from the same set of houses, and one can expect five different answers, and every one of them would be equally correct.

We need to stop letting the tail wag the dog. There is a market failure that is limiting scale in energy efficiency, and it’s not driven by a lack of innovation in measurement. Instead, the lack of confidence and replicability in our ability to measure is a much bigger barrier to investment and market innovation than any perceived lack of perfection.

Why Can’t We Just Test Proprietary Software?

Easier said than done.

Empirical testing of software accuracy sounds easy, but is in fact very complicated, expensive, and often produces less than reliable results.

First, one needs to start with the right answer. What is the gold standard that everyone is trying to achieve? Measurement starts with a standard. For a Kilo, there is literally an orb in a vault that all scales are tested against. In the case of measuring efficiency, the right answer to test against must be the open implementation that can be inspected by all parties.

Once we agree on the gold standard, then we need to identify a representative group of retrofit projects that cover the full range of ECMs, building types, and climate zones. The software could then run these projects and compare results to the open implementation of the standard.

This approach is certainly possible, and we have many examples from BESTEST to CalTEST (empirical testing of modeling tools - developed by OpenEE) that have attempted this approach. However, it is a considerable technical challenge to create and administer and has consistently generated disappointing results.

Even if we were able to create an equivalency process, it wouldn’t accomplish the goal of providing both standard outputs and supporting private innovation.

First, any “innovation” a private vendor adds to their calculations, will result in a different answer and will render that software out of compliance. For this reason, innovation must happen through open collaboration, which is the nature of standards setting and open source software.

Additionally, the cost of testing and the need to update software anytime there is an innovation in the standard creates a major barrier to innovation on methods. Every time consensus methods are updated, a new test must be generated, and all “equivalent” software tools must be updated and then re-validated.  This takes time and costs a lot of money, and creates resistance to change.

In short, retesting proprietary software against an open standard is expensive, slows innovation, and, most importantly, means that the market will inevitably be less confident in cash flows for energy efficiency.

Open Source Promotes Collaboration and Innovation

The OpenEEmeter project is working to be this open source standard. The OpenEEmeter is 100% open-source, which means that all of the code and documentation is available for everyone to see, make suggestions or submit improvements to the code. The OpenEEmeter is licensed under the MIT Open Source license that allows anyone to use the code for any purpose without restrictions.

Because the code is available for everyone to use and review, this effort is now powering not just utility programs and regulatory agencies but is also being used by a range of businesses, from program implementers to finance companies, technology providers, and contractors. To support distributed markets, the ability to measure also must be distributed and available to both buyers and sellers.

The OpenEEmeter platform can also be reviewed in detail and tested. The California Energy Commission, which was one of the original funders of the OpenEEmeter project, commissioned kW Engineering to review the code and test the platform against known models. NYSERDA similarly asked Taitem Engineering to review the OpenEEmeter methods, code, and results.

These efforts provided valuable input that has been integrated into the OpenEEmeter platform and is now available to all parties on the platform. This is the power of collaboration.

Open Source is Good Business

In fact, it's likely that most of the software that runs the computer or device you are reading this on right now is, in fact, open source. Using open source software is a way for governments and private entities to ensure that they are investing in public, not private, value, and to prevent vendor lock-in.

According to a 2016 survey of over 1,300 software companies, 65% of respondents contribute to at least one open source software project, and 1/3 had full-time resources dedicated to an open source development project. Ninety percent of responding firms said that open source development increased efficiency, interoperability, and innovation.

Open source does affect business model choices. Rather than competing to lock in markets into a single proprietary platform, vendors must compete on services, value, and solutions, driving innovation through competition.

Open Source is Good Public Policy

An open source approach will leverage collaboration across companies, states, and sectors. It also dramatically increases market innovation by enabling companies to provide a wide range of value-added services that build on a public and shared resource, without having to invest in re-inventing the wheel. This will also allow public investments in improving the state of the art in measurement to leverage the joint public investment through public open source tools that can deliver public benefit.

This is a trend in good government. In 2016 the White House required that any new custom source code developed “by or for the Federal Government” has to be made available for sharing and reuse by all federal agencies.

The State of California has similar policies to support open software and recently launched the California Department of Technology (CDT) California Innovation Lab. Developed using open source technology, the Lab resides in the state’s CalCloud as a place for government organizations to develop, build, and test open source technologies within its data center.

The City of San Francisco goes even further than the State, requiring that for all software purchases in excess of $100,000, agencies must consider open source solutions on equivalent grounds as proprietary software products.

In their Enterprise Information Technology Acquisition Policy, Massachusetts states that agencies “must consider as part of the Best Value evaluation all practical solutions that fulfill the requirements (e.g. in the case of software procurements, Applicable Agencies must consider, where available and practical, open standards compliant Open Source Software and Proprietary Software as well as open standards compliant Public Sector Code Sharing at the local, state and federal levels).”

In making a public policy decision it's important to consider the wide range of benefits and use cases at play and how can public investment and procurement of software supports the greatest public benefit.

Rather than competing to see who can build the best secret yardstick, the real innovation we need to unlock is competition and innovation to deliver efficiency at scale. Open source is the right choice as public policy to encourage markets and innovation that drive public benefits.

_______________________

CalTRACK Methods: The CalTRACK methods project is a multi-stakeholder process led by PG&E, including the CA IOUs, CEC, CPUC, and technical stakeholders, to develop a set of consensus methods for to calculate monthly and daily normalized metered savings.  docs.caltrack.org

OpenEEmeter: The OpenEEmeter is an open source platform, under an MIT License and without limitations on use, that was originally funded primarily by the California Energy Commission. The OpenEEmeter code leverages the methods developed through the CalTRACK process and includes an ongoing open source process to update, revise, and version the platform.  www.openeemeter.org

Open Energy Efficiency (OpenEE): OpenEE is a private company that has led the development of the core open source OpenEEmeter, and offers an enterprise supported SaaS version of the OpenEEmeter platform.  www.openee.io