Configuration Management Audits
A customer recently shared with us that a new-hire fresh out of engineering school asked, “Why do A&D programs need to perform so many reviews and audits; aren’t they obsolete or redundant in a world of continuous integration, AGILE development, model-based systems engineering, virtual prototyping, and digital twins?”
The answer from the perspective of an A&D project manager is no; audits are not redundant nor are they a relic of the analog past to be discarded in the hype of a digitally transformed future. Audits are the bastion against failures small and large such as hardware/software incompatibilities, data breaches, incorrect part substitutions, product performance deficiencies, and system disasters that become news headlines. In the complexity of increasingly digital world, audits have become more important, certainly not less.
A&D contractors are confronted by numerous categories of audits including business, regulatory, legal, and engineering. These can include financial, compliance, operational, security, delivery, product performance, data, and configuration management audits. In A&D contracts, Configuration Management audits are of two types: the Functional Configuration Audit (FCA) and Physical Configuration Audit (PCA).
In this first of two CMsights posts we will describe the process of continuous CM audits and two types of configuration audits. In Part 2 we will explore how CM software like EPOCH CM and PDMPlus support configuration audits.
Where Audits Fit in to the Product Development Lifecycle
The above illustration shows each of the steps in the management of a typical new product development (NPD) launch. The process flow may look logical but there are numerous traps, detours, and dead ends along the way as the devil is always hiding in the details. We will focus on the latter half of the NPD process.
Starting at Preliminary Design Review (PDR) the contractor’s allocation of the functional requirements to subsystems is evaluated to ensure the system will be operationally effective. Once a PDR is complete those allocations constitute what we call the Allocated Baseline. Critical Design Review (CDR) then takes a critical look at the detailed design against the allocations. Once completed in multi-unit procurements, the Configuration Items (CIs) are authorized for fabrication and test. Depending on design maturity there may be multiple PDRs and multiple CDRs until all systems and subsystems have been covered.
It looks very straight forward but looks are deceiving. As example, contract deliverable data is often identified with a specific due date associated with a program milestone. Program managers and systems engineers need to monitor this on an as released basis. (No company has the data pipeline bandwidth or the staff to hold deliveries until the specified date.) Data deliveries for documents like drawings, test procedures, test reports, analysis, inspection reports, and performance demonstration reports have to be released and delivered with precision. As a result there are multiple data deliverable pipes running in parallel at any one time which are often disconnected. Learn more about contract deliverables data management or CDDM.
Configuration Items and Configuration Audits
What often confuses A&D program managers and systems engineers alike is the relationship between Configuration Items and Configuration Audits. Going back to the fundamentals, we recall that all items under configuration control are subject to the five pillars of a sound CM implementation. CIs are subject to enhanced CM (often called distinct control) when it comes to Configuration Traceability (or Verification) & Audit.
Items not designated as CIs are also subjected to verification and audit activities. This is where continuous auditing becomes critically important. The continuous iterative process of audit, release, and delivery – instead of release, delivery, then audit – is a capability provided by the configuration management plan and change control system because:
Each change to an item under configuration control is audited prior to release in the CM system and delivery to the customer.
The customer then performs an incremental buy-off of the functional aspects of the CI as it is developed and tested.
CIs are items chosen for “distinct control” and subject to Functional Configuration Audits and Physical Configuration Audits.
The goal of these configuration audits is to provide the following:
Ensures that product design provides agreed-to performance capabilities
Validates integrity of product configuration information
Verifies consistency between a product and its product configuration information
Determines that adequate processes are in place to provide continuing control of the configuration
Provides confidence that product definition information is under configuration control
Ensures a controlled configuration is the basis for manufacture, installation, and maintenance instructions, training, spare and repair parts, etc.
Verifies that we know what we built including that part substitutions are known, and that assembly heritage is verified
Ties performance requirements to program documents that verify all technical requirements are met and documented
FCAs and PCAs are the two processes used to address these goals and thus verify that the end item delivered was vetted against requirements stated in the contract award. When the results of the FCAs and PCAs are combined they offer tangible proof that a controlled design, fabrication, test, and documentation system exists; and the product not only performs as required but you know what you manufactured and can replicate it.
FCA and PCA findings and observations can be positive or negative and they can be written against the customer or the contractor. Findings require a succinct statement of fact based on what is found or observed. All findings must be dispositioned before the audit is considered closed.
Functional Configuration Audit (FCA)
Department of Defense Directive 5000.1 (http://acqnotes.com/acqnote/acquisitions/dodd-5000) describes FCA as follows:
“4.3.3.9.2. System Verification Review (SVR) – The SVR (synonymous with Functional Configuration Audit) is a multi-disciplined technical review to ensure that the system under review can proceed into Low-Rate Initial Production and Full-Rate Production within cost (program budget), schedule (program schedule), risk, and other system constraints. Generally this review is an audit trail from the Critical Design Review. It assesses the system final product, as evidenced in its production configuration, and determines if it meets the functional requirements (derived from the Capability Development Document and draft Capability Production Document) documented in the Functional, Allocated, and Product Baselines. The SVR establishes and verifies final product performance. It provides inputs to the Capability Production Document. The SVR is often conducted concurrently with the Production Readiness Review.“
AcqNotes.com (http://acqnotes.com/acqnote/tasks/functional-configuration-audit-2) further states:
“A Functional Configuration Audit (FCA) examines the functional characteristics of the configured product and verifies that the product has met the requirements specified in its Functional Baseline documentation approved at the Preliminary Design Review (PDR) and Critical Design Review (CDR). It has to do more with systems engineering and program management that official auditing. The FCA is a review of the configuration item’s test and analysis data to validate the intended function meets the system performance specification. The FCA is normally performed prior to Low-Rate Initial Production (LRIP) and prior to or in conjunction with a Physical Configuration Audit (PCA). A successful FCA typically demonstrates that Engineering and Manufacturing Development product is sufficiently mature for entrance into LRIP. A FCA may also be conducted concurrently with the System Verification Review (SVR).”
The elements that are addressed during a FCA/SVR are:
Readiness issues for continuing design, continuing verifications, production, training, deployment, operations, support, and disposal have been resolved
Verification is comprehensive and complete
Configuration audits, including completion of all change actions, have been completed for all CIs
Risk management planning is/has been updated for production
Systems Engineering planning is updated for production
Critical achievements, success criteria and metrics have been established for production
The FCA process is often chaired by the customer. A factor to the success of the FCA is having someone who knows how to write findings and observations and who can tell the difference between the two. This generally falls on the contractor’s lead systems engineer. The parties should come to an agreement on the definition of a finding, what findings are considered to be major and minor, finding closure criteria, action items handling and assignments, and FCA completion criteria. If possible, the success criteria for all audits should be captured in the Statement of Work. This is generally accomplished via a coordinated FCA Audit Plan that is released and approved by the customer well in advance of the FCA event.
Physical Configuration Audits (PCA)
“A Guide to Auditing Defense Acquisition Programs Critical Program Management Elements” issued by the Office of the Deputy Inspector General for Auditing in September 1998 defines the PCA as:
“Physical examination to verify that the configuration item(s) (CI) “as built” conform to the technical documentation which defines the item. Approval by the government program office of the CI product specification and satisfactory completion of this audit establishes the product baseline. May be conducted on first full production or first low rate initial production (LRIP) item.”
AcqNotes.com (http://acqnotes.com/acqnote/tasks/physical-configuration-audit) states:
“The Physical Configuration Audit (PCA) examines the actual configuration of an item being produced and is conducted around the time of the Full-Rate Production Decision. It verifies that the related design documentation matches the Configuration Item (CI) as specified in the contract and confirms that the manufacturing processes, quality control system, measurement and test equipment, and training are adequately planned, tracked, and controlled. The PCA validates many of the supporting processes used by the contractor in the production of the item and verifies other elements of the item that may have been impacted or redesigned after completion of the System Verification Review. The PCA is also used to verify that any elements of the CI that were redesigned after the completion of the Functional Configuration Audit (FCA) also meet the requirements of the CI’s performance specification.”
The elements addressed during a PCA are:
The accuracy of the documentation in reflecting the production design
Validation of the supporting processes that the contractor uses in the production of the CI
Verification that any elements of the CI that were redesigned after the completion of the FCA also meet the requirements of the CI’s performance specification
The PCA determines that the item, product, or system was built right!
As with the FCA, the PCA also needs an audit plan. Unlike the FCA, the PCA is co-chaired by the contractor and the customer. Also unlike the FCA, the PCA should be limited to those major (Class I) changes that have been approved after CDR. It is not a recap of the entire design solution. Prior to the advent of PLM systems a PCA could last weeks and entailed dozens of customer representatives dismantling (as far as practical) a CI and comparing each piece against the drawings. Thankfully that is no longer true. At present, close coordination between the contractor and the customer’s on-site representatives allows for evaluation of subsystem assembly and integration events as they occur. After the successful completion of FCA and PCA the customer may declare that the CI design is a Product Baseline.
How Does a Configuration Management Plan and Configuration Management Software Support FCAs and PCAs?
We will address these two very important question in Part 2 of our CMsights post series on configuration management audits.
Until then, if your program or organization needs help with a review and validation of its configuration audit processes please contact the configuration management pros at CMstat with an email to information@cmstat.com.
Receive CMsights
Subscribe to CMsights News for the latest updates from CMstat on Configuration Management, Data Management, EPOCH CM, and EPOCH DM.
Request a Demo
See how EPOCH CM and EPOCH DM support industry standards and best practices in Configuration Management and Data Management.
