Automated System Tests on a Communication Module with Function for Flashing Over The Air in the Automotive Industry
On behalf of various OEMs, our experts test the software functions of future vehicle series regarding their functionality. In the automotive industry, for example, Da Vinci Engineering also carries out Automated System Tests on a communication module with a function for flashing Over-the-air. By ensuring the proper functioning of the telematics control unit, we can ensure that the vehicles are kept up to date and continuously connected to the surrounding infrastructure. In the meantime, the series in which the telematics control unit is used has gone into series production. Therefore, we would like to take a closer look at the background and details of our successfully completed project.
Environment
The device being Tested is a „Telematics Control Unit“ (TCU) with the following functions:
- Automatically or manually triggered roadside assistance and emergency call
- Collection and transmission of telemetry data
- Remote diagnostics
- Remote flashing of control units including self-updating of the TCU
- Assistance functions (remote window opening, remote air conditioning, …)
- Internet in the vehicle
- GPS in the vehicle
- Additional services (weather, parking, fuel and charging station finder, live traffic, …)
Technologically, the system is based on:
- TCU with Ethernet interface into the vehicle
- Two-processor system
- LTE module
- Secure data and diagnostic communication in the vehicle
- Update functionality based on OMA (Open Mobile Alliance) DM (Device Management) specification
Task
Da Vinci Engineering was commissioned to develop a test specification and a test concept from the OEM‘s more than 3,000 individual requirements.
As a result, the developed system tests were carried out for the software states delivered according to the release schedule. The results, including errors, were documented, their elimination tracked and reported to the customer.
The following features have been tested:
- Remote Diagnosis, which enables the execution of scripts for the diagnosis of the ECUs in the vehicle
- Flashing Over-the-air, which provides the possibility to flash and code ECU software in the vehicle at the customer‘s site.
Two generations of the module were tested in parallel; in some cases, the individual requirements differed, were dropped or newly added. In addition, two expansion stages (basic, extended version) of each module were to be tested in parallel.
As part of the feature rollout and project plan, we typically received a new version of the software every five weeks, and even every week at peak times.
The anomalies were documented in the OEM‘s Product Lifecycle Management System and statistically reported to the customer after each test run. Da Vinci Engineering conducted progress tests to validate the bug fixes.
Concept
Test cases were first developed and documented in a test specification. They were then executed manually and adjusted if necessary. They were then programmed, validated and readjusted accordingly. All vehicle functions were simulated using restbus simulation generated by the CANoe Model Generation Wizard. In the process, custom adjustments to the generated code were necessary, for example, in order to be able to completely switch off individual nodes. Then the serial interface was connected to the processor to enable processor logging and access to the engineering interface.
For the transmission of control information and the packet download, a back-end connection was established via the GSM network. Subsequently, the diagnostic responses of „flashable“ ECUs are simulated in the vehicle network. The SQL-based database records the results of the test cases and serves as the basis for all evaluations and statistics. The specially developed automation based on Vector CANoe, vTestStudio and C# enables operation and control of the entire test environment including
- the CANoe restbus simulation
- the behavior of the backend via REST requests
- DUT diagnostics
- DUT interface
- the test sequence
- the documentation of the test results incl. the corresponding logs
Test execution
To carry out the test, the system was prepared for a test case (set preconditions). Then a stimulus was applied to the device (action) and the expected behavior (expected result) was determined. The observed behavior was documented and then evaluated. The results of successfully passed test cases (Passed) were automatically transferred to the database. Test cases with errors (Failed) were reported and manually retested to ensure that the result was valid.
Obstacles
The CANoe version used in conjunction with RBS based on adaptive AUTOSAR led to intensive use of the resources of the test PC up to and including the termination of the simulation.
This hurdle was successfully overcome with the following two measures:
- Disabling the use of SOME IP services of simulated nodes by other simulated nodes thanks to a patch in the Model Generation Wizard provided by Vector
- Removal of unneeded panels after the RBS has been generated
Benefits & potentials
There are many advantages to automating tests, but sometimes it reaches its limits. The dynamic in the software development by the supplier is a factor that should be carefully considered. If the contents of lines from the processor log are changed between two software states, a human tester may still be able to correctly recognize the log line, but the automated system already cannot and runs into error. In some cases, an improvement in the hardware performance between two sample states leads to a changed timing behavior, so that timeout criteria that have been laboriously worked out no longer apply and must be adjusted. The operation of the test configuration, consisting of CANoe, the automation and the supporting tools, is not entirely trivial either.
However, the advantages of test automation such as repeatability, time savings, and improvements in test depth and test quality clearly outweigh the disadvantages. Here, our solution still offers potential for scaling to a larger environment in the near future.
Download
To download the whitepaper Automated System Tests on a Communication Module with Function for Flashing Over The Air in the Automotive Industry please click here.