Shift-Left: Regression Testing

There are many different methods of testing your software. Regression testing is just one of many examples. However, they all share a common aspect, being that testing software brings along overhead costs in the form of time and tools. The most obvious reason for software testing is ruling out errors in the code, though there are other things we need to test for. We also need to compare the actual functionality of the device against its specifications. This aspect of testing is called requirement based testing.

It may be tempting to try and cut corners during this stage of development, saving resources which can be spent on other things (like the launch of the product), although it goes without saying that this could end up biting you in the behind. By limiting the number of tests per release, faults are likely missed at an early stage. Resolving these issues later on is more expensive, therefore it is advisable to start testing as early and often as possible. this is defined as shift left testing.

One way of ‘shifting left’ is to use regression testing, which is performed after each change made to the existing functionality of the software. These changes include bug fixes, software enhancements, configuration changes, and even substitution of electronic components. With regression tests you assure that future software updates do not affect the proper functioning of the product.

As the number of tests grow with each function added to the release, it is important to start testing early (shift left!) to be able to keep up in the long run and avoid problems during the eventual integration testing.

Testing software can be a real challenge, but you probably knew that already. Luckily, many tools and platforms help to save time by providing useful infrastructure for (visual) regression testing. Various test platforms assist in managing your tests, saving you a lot of time and stress. But what functionality should you consider before choosing such platform? That depends on who you are and what type of products you build, but here are some useful considerations:

Do you build safety critical code?
Does this code run on an embedded device?
Is it a real time system or a web hosted application?
What is the programming language used?

Software engineers and software architects prefer to be in control of the code including the test-software/scripts and focus on easy integration into their build/release pipeline & version control (SCM).

Test engineers and the QA Team will prefer a solution that helps them to translate requirements into simple tests allow easy refactoring to match new requirements and helps them to generate proper test documentation and make sure test reports are always up to date on each release with minimum manual effort.

Whether tests will be executed on the host or on Target make a huge difference. The first approach is much easier to manage, but if you build embedded devices, you will regret if your code is at least not partially tested on the actual system, how will you test I/O, Communication and Realtime behavior? Some write code that compiles and runs on the host and on the embedded device but what version of this software do you test and wouldn’t it be nice if you can migrate your tests from one platform to the other?

Does your platform allow you to focus on testing only a single feature at once and then execute the test in isolation? How about documenting these tests and, besides giving a Pass/Fail, providing test coverage details so you do know exactly when you have tested enough? If these tests can be re-used just like you re-use your code then the initial time spend to configure such a process will be in no relation to the Return on investment at the end of the project.

Shift-Left: Regression Testing

Regression testing

Using the right tools

Depending on your role in the team

On target or hosted

Make the right choice from the start