Testing & Evaluation
As pointed out in the section, PTC Analyses, a major part of the requirements for the contents of the Product Safety Plan (PSP) is “validation and verification.” Validation involves a number of analyses that have been outlined in that section, verification involves data collection, performance evaluation and product refinement. The results of this process are also expected to be included in the PSP submitted to FRA for approval.
The testing usually is comprised of several steps:
Subsystem testing – each unit of the whole system is to be tested by itself. This is to verify that the subsystem unit is designed and built correctly. Software code for the unit is also checked to detect any deficiency.
Laboratory integration testing – in a laboratory environment, the subsystems units are assembled to be one system to verify the compliance of the system functionalities and design to the requirements before subjecting the system to the real operation. Requirements are first analyzed to design a comprehensive matrix of operational scenarios in this testing. Where a segment cannot be physically located in the laboratory, simulation will be provided to include that segment. One example is the locomotive and train. The operation of the train is very much a part of this verification testing. A simulation model with a real locomotive stand to input throttles and braking can be used to interact with other segments of the system. Dispatching system is usually also simulated in this environment. After the system passes the tests in this laboratory environment, there should be a confidence that most requirements, if not all, are met and the system should function as it is intended in all aspects.
Field testing - Upon successful laboratory test, the system can be put into testing in the real environment. A lot of the laboratory testing will be repeated in this field test since the system interacts with actual infrastructure which may not as controlled and predictable as in the laboratory environment, especially with the segments that are simulated. Some tests are actually repeated in various locations and various times so that the randomness and variability can be also tested. Certain testing can only be tested in the field. For example, actual train tracking with GPS and inertial navigation system is one of those. At this time, certain train operation is still in a control environment. Speed may be controlled to check the braking distance or the activation of alarm. Train orders may be artificially created for verifying the correct receipt and response in the controlling locomotive. Upon successful field testing, FRA may grant approval to elevate to the next level of testing in the revenue service.
Revenue service testing – The system is cut-in in this phase of testing in the actual train operation, after proper notification and training. Initially, even though technical and managerial personnel are not on-board of the locomotives or trains, monitoring of the system is still intense at the beginning and will be eased off as confidence is built. Quite often, the monitoring is performed remotely in shadow operation. The crew is specifically instructed to follow the existing method of operation. The enforcement function of the PTC system is not cut-in, but the system operation performs as if the trains are operated under PTC operation. Through the wireless communication network, the interaction among segments, system malfunctions and faults, and the general operational difficulties can be revealed and recorded in a log in a central office. The system is continued to be debugged and refined in this process. The enforcement function will then be turned on when no hazard is revealed after a certain monitoring period. The crew is instructed to operate the trains under the PTC method of operation. It is not unusual for this phase to last 6 months to 1 year before complete confidence is built for the successful development.