Our experts check the functionality of the software functions of future vehicle series on behalf of various OEMs. In this way, Da Vinci Engineering also carries out automated system tests on a communication module with a function for flashing over-the-air in the automotive industry. By guaranteeing that the Telematics Control Unit is working properly, it can be ensured that vehicles are kept up to date and continuously network with the surrounding infrastructure. The model series in which the Telematics Control Unit was used has now gone into series production. We would therefore like to shed light on the background and details of our successfully completed project for anyone who is interested.
The device under test was a “Telematics Control Unit” (TCU) with the following functions:
Technically speaking, the system is based on:
Da Vinci Engineering was commissioned by an OEM to produce a test specification and a test concept from a total of more than 3,000 individual requirements.
As a result, the automated system tests which were developed had to be carried out for the software versions which were delivered in accordance with the release schedule. The results, including the errors, had to be documented, and the elimination thereof had to be tracked and reported to the customer.
Testing took place of the features for
Two generations of the module were to be tested in parallel; In some cases, individual requirements differed, were removed or were added. Two expansion levels (basic version, extended version) of the modules were also to be tested in parallel.
As part of the feature roll-out and the project plan, we usually received a new version of the software every five weeks, or even weekly at peak times.
The anomalies were documented in the OEM's product lifecycle management system, and statistically reported to the customer after each test run. Da Vinci Engineering carried out progression tests in order to validate the error correction.
Test cases were initially developed and documented in a test specification. They were then carried out manually and adapted if necessary. Subsequently, they were programmed and validated and readjusted if necessary. All vehicle functions were simulated using residual bus simulation, which was generated by the CANoe Model Generation Wizard. When doing so, we had to make our own adjustments to the generated code, e.g. so that individual nodes could be switched off completely.
The serial interface on the processor was then contacted for the purpose of processor logging and access to the engineering interface. A backend connection via the GSM network was established for transmitting control information and downloading the package. The diagnostic responses from “flashable” ECUs in the vehicle network were then simulated. The SQL-based database records the results of the test cases and serves as the basis for all evaluations and statistics. The self-developed automation based on Vector CANoe, vTestStudio and C# allows operation and control of the entire test environment, including:
In order to perform the testing, the system was prepared for a test case (setting of preconditions). This was followed by placing a stimulus on the device (action) and the expected behaviour (expected result) was determined. The behaviour which was actually observed was documented and then evaluated. The results of test cases that were passed without errors were automatically entered into the database. Test cases with errors (failed) were identified and manually retested in order to ensure that the result is valid.
The CANoe version that used in conjunction with the RBS on the basis of adaptive AUTOSAR led to intensive use of the test PC's resources, and even termination of the simulation.
This hurdle was successfully overcome by means of the following two measures:
The automation of testing has many advantages, but also has occasional limitations. The dynamics of software development by the supplier is a factor which should not be underestimated in this case. If the contents of lines from the processor log are changed between two software versions, a human tester may still recognize the log line correctly, but the automated system no longer recognizes it and flags up an error. In some cases, an improvement in hardware performance between two sample versions leads to different timing behaviour, meaning that timeout criteria which have been laboriously worked out no longer apply, and have to be adjusted. The operation of the test configuration, consisting of CANoe, the automation and the supporting tools, is also not entirely trivial.
However, the advantages of automating tests such as repeatability, time savings and the improvement in the test depth and the test quality clearly outweigh the disadvantages. Our solution still offers potential for scaling to a larger environment in the near future.
Please click here to download the Automated system tests on a communication module with over-the-air flashing function in the automotive industry white paper.