By Ben Graf, P.E., M.SAME
Consider this scenario: A wing commander must decide what facility projects to prioritize for centralized funding. Staff engineers present the Building Condition Index (BCI) for several candidates, pulled from the BUILDER Sustainment Management System. The commander selects a missile launch facility in woeful condition as the top priority. Higher headquarters is concerned enough about the facility’s mission impact that a team is sent to the base to help develop the project. It quickly becomes clear, however, that while the facility has a few issues, its overall condition is not as dire as presented. The commander is frustrated and withdraws the project request, vowing never to trust the system’s data again.
Now consider an alternate version: In preparing to present candidate projects to the commander, engineers check each facility’s rating in the BUILDER Data Confidence dashboard. The missile launch facility has a low BCI, but the Data Confidence dashboard rates it as No Confidence. Digging deeper, the airmen realize that the electrical system is the only part of the facility that has been assessed and is therefore the only system contributing to the BCI. A facility assessment team is sent to complete the inventory, finding that, when all systems are assessed, the BCI is not nearly as low as before. The project is not presented to the wing commander, and the loss of faith in BUILDER is averted.
The first scenario has played out all too often across the U.S. Air Force. Because of BUILDER’s ability to track the condition of every facility down to individual components, the Office of the Secretary of Defense has mandated its use across the military services to project how conditions will degrade over time, and to predict the costs needed to sustain and repair facilities. The Air Force chose a decentralized approach, making individual craftsmen at the base primarily responsible for BUILDER inventory and assessment. While this has certain advantages, it can also result in inconsistencies, which have at times taught decision-makers the hard way that BUILDER’s power is only realized if the data used to populate it is accurate.
Today, the second version is not only possible but increasingly common, thanks to the release of the Data Confidence dashboard, developed by a team from the Air Force Installation & Mission Support Center and Air Force Civil Engineer Center.
BUILDER data is used for decisions at all levels of Air Force Civil Engineering on a daily basis, so even if some of the data is suspect, leadership cannot afford to discount all of it. The Data Confidence dashboard aims to identify where decision-makers can trust the data and where they should exercise caution. While expert BUILDER analysts can tackle this exact problem on a case-by-case basis, the team’s goal was to automate this expert skill to benefit the enterprise.
DEFINING METRICS
Development began by evaluating industry definitions of data confidence, settling on five relevant characteristics of BUILDER data that provide confidence: accuracy, completeness, credibility, currency, and validity. The team then identified 13 specific measures of these characteristics (measures that expert BUILDER analysts already use to intuit data confidence) and developed a tiered metric for each one, assigning a rating to each of High, Medium, Low, or No Confidence. The overall confidence in a facility’s BCI is assigned as the lowest of the 13 measures; for example, if 12 of the measures have high confidence and one has low confidence, the overall confidence is low.
The 13 measures attempt to provide a holistic evaluation of the confidence level in the BCI. Many measures focus on error types, such as a component’s install date being earlier than the building’s construction date. Some reflect compliance with Air Force guidance, including the requirement to provide inspection comments for any component with an inspected value below Green-minus (a condition index below 85). Still others act as barometers for inventory completeness, such as the ratio of the sum of component replacement values in BUILDER to the plant replacement value for that facility in the Air Force real property record. Haste, carelessness, and training deficiencies can result in data that might slip past any single measure. Examining the data from a wide variety of angles reduces this risk.
While the overall confidence is driven by the lowest rating (rather than a weighted average), the developers were still able to incorporate the relative importance of the measures by carefully tuning the unique tier structure used to assign each confidence rating. Measures deemed to have a smaller impact on BCI confidence were designed with lenient tier structures—indicating that a facility would need to be substantially deficient for one of these measures before it begins to drop the confidence rating. On the other hand, measures with significant impacts to confidence were given more aggressive tier structures.
VISUALIZING DATA
Once the BUILDER Data Confidence metric was developed, the team operationalized it in a user-friendly Tableau dashboard on the Air Force Data Lab Development Environment. Anyone with VAULT access (which requires a Common Access Card and an approved access request) can now explore the dashboard and use its contents to support better informed decision-making. Users have the ability to filter the Overview page’s list of 35,000 Air Force vertical facilities down to the installation, Category Code, or Mission Dependency Index tier they are interested in. Once they have homed in on a particular facility, users can jump over to the Drill Down page to see a detailed breakdown of the particular anomalies driving the calculated confidence level.
No data entry occurs in the dashboard. Even those without BUILDER access can use it to explore the intricacies of the database. If a user’s investigation uncovers data issues that need to be fixed, they must go into BUILDER to make the changes. Those adjustments then will appear upon the next weekly, automated data pull. Moreover, the BUILDER Data Confidence dashboard is connected to four other Facilities Activity Management dashboards for full spectrum visibility of Air Force vertical assets. These dashboards, and others in development, display the BCI Data Confidence rating immediately adjacent to the BCI, so that a decision-maker can see at a glance whether the condition can be relied upon, only having to jump into the Data Confidence dashboard if they need to delve into the rating’s drivers.
PROVIDING CONTEXT
The BUILDER Data Confidence dashboard is only a guide, not a verdict on the data. It does not have all the context a decision- maker uses, only what is contained within BUILDER. In addition, the confidence rating only refers to the likely accuracy of the BCI itself, not the confidence rating in a particular decision, such as whether to demolish, repair, or replace.
Still, the innovative tool can provide reassurance that the starting point for many decisions—the condition of the facility— is reliable, resulting in more confident decision-making, and, over time, more effective asset investments.
Ben Graf, P.E., M.SAME, is Senior Asset Management Data Scientist, Air Force Civil Engineer Center; ben.graf@us.af.mil.
[This article first published in the September-October 2022 issue of The Military Engineer.]
