Systems Engineering Management Plan 
Section 3.6 - Model Text

3.6 System test planning.

The objectives, scope, and type of system testing shall be the products of the engineering effort wherein all engineering specialities are integrated to define an effective and economical total system test program.

Whenever practicable, tests for different objectives shall be combined.

A Testability Program Plan shall be established early in the programme to identify and integrate all testability and test design management tasks, when applicable.

The aims of the test, evaluations, and trials shall be:

• Testing shall ensure that system and mission objectives will not be impaired adversely by improperly specified, designed, implemented, or maintained hardware and software;
• Quantitative requirements and tolerance shall be established for the technical and operational characteristics of the system elements and their capabilities;
• Tests on the system shall be systematic and qualitative using test cases in the most realistic way possible.

A Test Requirements Document (TRD) and/or Product Acceptance Test Specification (PAS for UUTs, etc.), shall initially be prepared to reflect the preproduction model of each UUT (Unit-Under-Test) and shall be written in True Values. This version of the TRD/PAS shall be complete when the configuration of the first preproduction model is established. The TRD/PAS shall be revised to reflect the configuration of the first production model (PBBS).

A TRD/PAS shall provide the information necessary to test the UUT in the most efficient manner possible and with a minimum of UUT interface. Sufficient tests shall be included so that all required performance characteristics can be verified. A TRD/PAS shall be provided for each Replaceable Unit (RU), RU Chassis, and RU subassembly. Product Test Schedules/procedures shall reflect the test equipments Indicated Values* which shall be the True Values minus the uncertainty of measurement. For further information, see MIL-STD-1519 and DEF STAN 00-52/3.

The product and maintenance requirements (SST) tests shall be specified in ATLAS (Abbreviated Test Language for All Systems) the internationally approved, disciplined, well structured and English like method for use in expressing test requirements in procedural form. (May have changed!).

Test data and metrics that are useful for Technical Performance Measurement (TPM) analysis shall be identified and integrated with program planning functions for maximum utility in updating and verifying the technical parameters being tracked.
See Metrics for details.

Verification of the acceptability and compatibility of human performance requirements, personnel selection, training, and man-machine interfaces of system procedural data shall also be integrated into the system/subsystem test program (UUTs, etc.).

As testing a system may be complex; consideration of testing using several different phases shall be adopted.

The suggested phases are:

• HWCI and CSCI product Testing;
• Prime/Critical item Integration Testing (RU, etc.);
• System/subsystem Integration and Performance Testing;
• Regression Testing;
• Quality Testing;
• Operational Testing (acceptance, trials, etc).

Testing in each phase shall be broken down into functional, stress, and load testing cycles. Many integration tests shall be performed from a functional test stand-point to ensure the functionality is correct. Integration testing shall also include stress and loading but shall be applicable in the regression and quality phases.

A System Test Plan shall be produced which shall contain the philosophy, strategy, resources, organization, evaluation and schedule for the HWCI development, equipment, and system/segment testing of the system. For CSCIs the individual CSCI Software Test Plan shall be referenced in the relevant CSCI Development Plan where applicable.

A Final Acceptance Test Plan shall be prepared when applicable and will contain the philosophy, strategy, resources, organization, evaluation and schedule for the final acceptance testing and any subsequent trials of the system.

Testing and trials: There is some ambiguity in the terms 'test' and 'trial'. In the production areas a test is defined as 'an inspection process in which a functional requirement is measured or observed' when stress or energy is applied to the item. A trial is thought of as a number of tests applied to a complete system or subsystem under conditions which are, for the design features under trial, reasonably representative of a real life.

Note: 'Trials and analysis' is a term used mainly in the UK the term 'Test and Evaluation' is used in other countries.

Adequate arrangements shall be made for significant tests to be witnessed by the appropriate acquirer representatives.

• 3.6.1 HWCI and CSCI testing.
• All HWCIs and CSCIs comprising the 'system' shall be tested individually against their development specifications agreed at their allocated baselines. Testing shall be concentrated on showing that the implemented software and hardware satisfies its specified functional and performance requirements in functional and physical configuration audits to establish the respective product baseline.
• 3.6.2 HWCI and CSCI integration testing.
• HWCI and CSCI integration testing shall test the validity and operation of the system/subsystem against the specific system/subsystem specification or equivalent specification. Testing shall be to the minimum functional requirements described in an 'Acceptance Test Procedure' prepared for each system item which when validated shall be used as a production acceptance test baseline for the specific item.
  Test Requirements shall be expressed in TRD or PAS form and described procedurally in ATLAS. Tolerance tiering shall be performed to establish the indicated values in the in the PAS/ATP test program schedule/procedure from the true (absolute) values minus the worst case uncertainties of the target test equipment.
• 3.6.3 System/subsystem performance and integration testing.
• System/subsystem performance testing against the system/subsystem specification or equivalent to ensure that the system/subsystem performs the functions desired prior to being put into production. System software testing shall be performed on all systems/subsystems, all software, firmware and interfaces of a distributed computing system. This shall assure that not only individual CSCIs but also that System Software and its associated hardware as a whole performs as specified.
  Engineering tests and qualification tests shall result in updated "Product" specifications, SVDs and Test reports.
  Development testing of prototypes and simulations shall be performed to establish or demonstrate whether performance, reliability, and maintainability are adequate. Tests should be performed to check the endurance, environment, and reliability are within the required limits.

• 3.6.4 Regression testing.
• Regression testing is performed on all incremental or "point" releases to the hardware and software to ensure problems are cured and that no additional errors have occurred, i.e., testing ensures that newly installed changes do not affect or impact on other functional areas.
• 3.6.5 Quality testing.
• Quality testing shall be performed continuously throughout the production cycle to ensure that the functional aspects of the system are as designed and that each individual system performs as intended.
  Quality Factors (Reliability, Maintainability, etc.,) for the system, hardware and computer software shall be the main items and data generated by statistical techniques (metrics) will be used and analyzed.
  A 'Systems Test Program Plan' with the subsequent production acceptance test specification/procedures shall be prepared and shall include each of the conceptual levels above.
• 3.6.6 Final acceptance testing.
• Final acceptance testing shall concentrate on system and subsystem testing in an operational environment. The testing shall be confined to areas that cannot be effectively simulated by performance testing to achieve the required quality in the production standard. The activities, objectives, organizations, and schedules shall be documented within a specific 'Test Plan' document.
  Evaluation trials shall be performed on systems that represent the final build standard (this is sometimes referred to as 'Beta' or pre-production testing).
  For discussion on Test and Evaluation principles.
  For discussion on Principles of Test Equipment Design.

Back to Home page MANAGING STANDARDS Home page

Please send any beneficial comments or identification of errors using the following form to: kenr@wysywig.airtime.co.uk

Indicated Values - The value of a measurement provided by a measuring instrument or measuring system. Test schedules are, unless otherwise stated, written in Indicated Values, i.e., an allowance has been made for the uncertainty of measurement.