A Configuration Management Case Study - Part 1
There are many important applications of Configuration Management (CM) within the Aerospace & Defense industry outside of an OEM’s product engineering organization where rudimentary change control and configuration management may have been implemented as part of an engineering-centric PDM system.
It is widely understood that when CM is implemented effectively throughout an enterprise and extensively across its supply chain, there should be substantially more users of configuration-related processes and data who work downstream of engineering. These personnel can be found in manufacturing, logistics, test, compliance, tech pubs, field service, repair, and sustainment. The lack of consuming, updating, and repurposing of CM data in groups other than engineering is a warning sign that the full benefits of CM are not being realized.
Users in these organizations all share the common requirement to be able to easily access, interrogate, validate, update, enhance, share and secure in real-time the live configuration (“As-Configured”) status of products and equipment. This is especially the case for systems which may be deployed, maintained, and refurbished in the field, often for decades at a time.
The Long “As-Configured” Digital Tail
While CMstat references numerous CM uses on our website and in our CMsights posts, we are often asked for additional details and examples. Due to the nature of our customers’ defense-related work and to protect client confidentiality we are unable to share particulars about specific customer use cases.
Yet, what we can share is an amalgamation of CM use cases that is illustrative of all we have witnessed from years of work in consulting and deploying CM software tools like EPOCH CM and PDMplus CM. These examples include describing the customer’s business problems, CM deficiencies discovered, why existing tools were inadequate, requirements for a new solution including important CM functionality, and the results attained along with benefits to the customer.
We begin with the following business scenario created as a CM use training illustration that many readers with CM experience may recognize as being all too real. In this first part of our case study series you will learn:
What was the industry and who was the customer?
The illustrative customer is an Aerospace and Defense contractor responsible for the design, manufacture, installation, maintenance, and sustainment of ground test equipment deployed around the world in support of military aircraft operations. These systems include component parts manufactured by subcontracted suppliers in the U.S. and by international offset trade partners associated with the aircraft program.
The operation of military aircraft is supported by numerous ground support and test equipment
What were the business challenges and organizational problems?
The company had dozens of sites in the U.S. and abroad where their equipment had been deployed and maintained over a number of years. Each site had systems and components of varying production dates with different configurations and revision levels of hardware, software, firmware, product documentation, and maintenance records.
Because of all the different configurations in service, customer calls for routine maintenance, emergency repairs, or planned upgrades routinely took too long to perform, were error prone, and often required multiple site visits to complete. This led to equipment being inoperable and off-line for unacceptable periods of time.
The lack of accurate up-to-date configuration information made planning for site maintenance visits along with shipments of materials needed beforehand an exercise in guesswork. Often, maintenance personnel would arrive on-site only to find that the equipment configuration was not as expected or that they did not have the required records, tools, training, and replacement parts to perform the necessary repair. Another trip was required with often different personnel.
An example was the failure of an electronic component at one site which caused the customer to request a list of serialized units across all sites in which the faulty item, specified by a range of production dates, was installed. This included in-service installations, spare assemblies held in reserve, equipment awaiting refurbishment, and repurposed units used in training by the customer or other contractors. As a result, personnel had to conduct a manual inventory of all deployed units and collect serial numbers of affected components to allow HQ staff to identify suspect units.
The contractor’s field service engineers were frustrated that they were unable to respond in short order to such seemingly simple requests of the customer and to perform their work efficiently. Because repeat customer conversations and site visits were often required, the personnel rotated in were often different and communication of history was inadequate, leading to yet more delays. The contractor’s engineering services, quality, and logistics groups were equally exasperated from being called upon several times to assist with the same service work order.
Not surprising, the customer grew dissatisfied with the contractor’s performance and impact on their force readiness levels. Word of the customer’s dissatisfaction made its way up to senior managers. Upon investigation, not only was this understood to be a company reputation issue that threatened future business but also one of contract profitability that impacted financial performance and even the annual bonuses of executives.
Some managers after being briefed were surprised as they thought, and were once told, that their enterprise PLM solution with its supposed single source of truth accessible to all, and which integrated a legacy collection of engineering PDM software tools used by different program contracts, would have prevented all these problems.
Now they were once again being “sold” that what was really needed was yet another new massive digitalization program. This time to create individual product digital twins that would trace the digital thread over the entire lifecycle of each product from manufacture to disposal. Proponents of this investment argued it would lay the foundation for a new virtual MRO industrial metaverse that could be accessed by all stakeholders in the future.
“Will a massive program creating Digital Twins fix Configuration Management deficiencies or only duplicate them?”
However, many managers were skeptical, remembering the promises previously made about “Big PLM”, so they called in the CM subject-matter experts at CMstat for a preliminary evaluation.
What were the workflow problems, inefficiencies, and risks discovered?
After an initial investigation of the current as-is situation, a number of operational problems and performance gaps were confirmed and documented. These included:
Field technicians would indeed often go to a customer site and only then learn they did not have the correct information or materials on hand to perform the repair work or system upgrade.
Once on-site there was no trustworthy access to the live As-Configured data sets for each site’s installation.
In-service or As-Configured product structures, BOMs, and other data and documentation rarely resembled the As-Manufactured and As-shipped configurations on record by the HQ engineering staff and their PDM data.
Obsolete As-Built history records of serialized equipment contributed to shipping the wrong parts and spares ahead of a site visit.
Online tech pubs from engineering services rarely matched the on-site assembly and/or were not always available for live access in secure sites, resulting in personnel using their own unauthorized devices or personal uncontrolled copies of design documentation.
Since the inventory of on-site spares did not match what was needed in the field, there was an overreliance on using alternate, unauthorized, incompatible, or superseded parts.
Large Serialized Assembly Record sets were often obsolete or missing important data from suppliers of components, especially those from international partners.
Massive PDM data dumps of CAD files, change notices, effectivities, and service bulletins from the OEM’s engineering and component manufacturers were so overwhelming to assess what had changed that they were often ignored.
Service personnel did not have the ability or authority to correct or update the As-Deployed data once onsite, requiring after-the-fact change notices be routed back to HQ that was ill-prepared to track hundreds of serialized configurations.
For discovery of and replacement of defective parts, no one office had access to “where-used” information to research current configurations across the globe.
It was a cumbersome process to file an Unsatisfactory Change Notice Problem Report or request Approvals for Variances in reference to the defective part and potential replacement at other locations.
The communication of what worked and final dispositions back to HQ to capture best practices and lessons learned was so manual and time consuming that it was performed only occasionally.
The documentation, declarations and certifications supporting IP protection and ITAR compliance were at risk due to a lack of tracking and/or transparency.
Long-term archiving of records and data (LOTAR) was neglected even though some sites were expected to be in operation for over a decade due to the support of aging aircraft and their support equipment.
What was the initial assessment and recommendation?
The CM consultants reported upon and recommended the following top ten findings:
Management, due to a lack of understanding and appreciation for the role of CM outside product engineering, was under the mistaken impression that their organization was already performing an acceptable level of CM to support field operations and their customers.
Too few employees were sufficiently educated, trained, and experienced in CM to speak competently with authority to advise or warn managers before problems arose at customer sites.
Because of their investment in PLM/PDM with some CM functionality, it was erroneously thought that they did not have to prepare a formal CM Plan nor implement their own unique CM controls and processes outside of those available within the existing software tools.
The problems experienced can be expected to increase with greater frequency and disruption to aging As-Deployed equipment, especially as the most experienced personnel with institutional knowledge retire.
As-Maintained records must be meticulously constructed and maintained throughout the life of the program contract as a high priority. This should be facilitated by live access to and role-based review and editing of pertinent configuration information by maintenance personnel while on-site in real time.
To begin to remedy deficiencies, the customer should start with an in-depth formal CM Assessment and then develop a requirements summary followed by a CM Plan, all before evaluating any additional software tools or technology investments, especially something the size of a Digital Twin or Service Metaverse initiative.
The effort should not be led by an internal Engineering or IT representative, but instead driven by an independent unbiased CM consultant with participants from all the stakeholder groups involved who are given equal standing.
An executive at the V.P. or C-Level should provide oversight, support and funding outside of departmental silos to demonstrate how important the effort was to the overall business and contract retention.
During the project one or more senior employees should seek formal CM training from an organization, such as CMPIC or equivalent, so that continuous improvement can be monitored and performed internally.
Future product, project and program contract managers throughout the organization should be educated in the requirements for and role of CM in executing their responsibilities.
What were specific issues that the CM Assessment exposed?
After presenting the value of performing an assessment, a formal Configuration Management Assessment was conducted during site visits to the contractor’s U.S. HQ and field sites. The following CM specific issues were discovered and documented:
Over the years almost every serialized piece of installed equipment had developed its own unique configuration, maintenance history, service BOMs, and operational “personality.”
There were indeed many occurrences where the As-Maintained configuration was very different than the As-Built, As-Shipped, and As-Installed records.
There was no reliable process to resurrect the history of when exactly any of those configuration changes were made or recreate it from the data that was available.
No one office could easily produce records or gain access to views of data answering basic questions of status accounting of installations.
CM procedures were originally developed by product engineering without full understanding of the requirements of all the users of CM-related processes and data.
The engineering dominated Change Control Board (CCB) had little visibility to the work and needs of the field engineering support teams.
Multiple configuration Baselines were needed for each deployed system over a history that could span decades.
There was no standard definition of Configuration Items (CI) and parent/child relationships of some CI were missing, thus corrupting the As-Configured data.
Impact analysis of proposed changes was problematic and rife with inconsistencies that made it untrustworthy and largely unusable.
Change Control and Change Implementation processes and record-keeping at the sites was found to be ad hoc, incomplete or even altered after-the-fact before audits.
Unauthorized changes were made in order to “get it working” when the changes and parts as planned were unable to be implemented as documented.
Unified procedures for Status Accounting and Traceability were absent with business culture differences or even language issues rationalized as the reason.
Configuration Rules (CR) were insufficient or inconsistently applied for the complexity and number of roles involved including rules for where used and when used of spares and alternatives.
Routine or custom reports for management and the customer were difficult to prepare and reconcile differences.
Some sites maintained their own configuration notes in spreadsheets that proved unwieldy to maintain due to all the variations and exceptions.
Training procedures and operational manuals were out of synch with the release levels of what was actually installed.
Reviews and audits of quality control, instrumentation test, and compliance documentation as required by the customer contract for each site’s configuration were problematic.
Non-CM specialists and managers without formal training who needed on-demand access to the live status data had to call upon others for this information, causing further delays in often urgent situations.
Some components and documentation under configuration control were ITAR controlled yet some field personnel were foreign nationals not authorized to have such access.
There was no formal harmonized Configuration Management Plan developed with the needs of all the sites incorporated.
Existing PLM/PDM software tools and data were inadequate to support the CM functionality and workflows that field operations required.
Performance metrics were missing that could have been derived from more accurate CM process data, making continuous process improvements difficult.
What’s Next?
In future CMsights articles illustrating this example As-Configured CM case study we will share: why existing PLM/PDM software solutions were insufficient, elements of the recommended CM Plan, the requirements for a new CM solution, how a tool like CMstat EPOCH CM can meet those requirements, a demonstration of important CM functionality, obstacles planned for during the implementation, and the final results including benefits derived to the business.
To receive notice of the articles, register to receive our monthly CMsights news and then follow CMstat on LinkedIn.
Learn more about CMstat’s consulting services in Configuration Management Assessments, Planning, and Implementation HERE.
Receive CMsights
Subscribe to CMsights News for the latest updates from CMstat on Configuration Management, Data Management, EPOCH CM, and EPOCH DM.
Request a Demo
See how EPOCH CM and EPOCH DM support industry standards and best practices in Configuration Management and Data Management.
