Draft for a refactoring and improvement of the test suite

11 Mar 2023


      Hello inkscapers,
I could contribute to the development of the code in my spare time if the
following idea makes sense and is of value. While talking in the #irc dev
channel and in the last developer meeting, it turned out that the test
suite in testfiles directory is somewhat messy and a refactoring could be
desirable for some reasons.
Here's my proposal. First of all, tests should be grouped in coherent and
standard categories.
A standard classification is:
end to end testing ---> integration testing ---> unit testing
This classification, translated into a dir tree would be:
----------------------------------------------------------
testfiles
|______end2end_tests
|      |_______gui_tests
|      |_______cli_tests
|
|______integration_tests
|
|______unit_tests
|
|______CMakeLists.txt
|
|______CTestCustom.cmake.in
|
|______README.md
[SUBDIR] gui_tests: empty for now, just add a TODO/FILL-ME note inside it.
[SUBDIR] cli_tests:
*) rendering tests are made with CLI (as the CMakeLists.txt confirms), so
they should be moved inside cli_tests. Therefore, move here svg files from
"rendering_tests" and move rendering_tests/test.sh as cli_tests/rendering.sh
*) leave here the current content of cli_tests
*) add an "images" dir containing all the graphic files
[SUBDIR] integration_tests: it will contain cpp or script files in the
form: operation1_operation2_..operationN.cpp (or .sh, or .py etc).
[SUBDIR] unit_tests:
a software unit can be meant both as:
    1) (see testfiles/src/curve-test.cpp) a self-referential "concept"
    2) (see testfiles/src/uri-test.cpp) as a class, or even a single
function, from the API.
The important thing is to separate unit tests from integration tests, which
are sequences of different tasks. From what I see, the "testfiles/src" dir
currently contains unit tests (if cpp files perform integration tests, they
should be moved to the testfiles/integration_tests dir)
In addition, do move lpe images here (inside an "images" dir). Add a note
on the README that lpe stuff is here (as it partially is, actually).
[FILE] CMakeLists.txt
[FILE] CTestCustom.cmake.in
[FILE] README.md: note: only ONE readme will be used for describing this
test suite. It will describe what the tests do and how to launch them
separately. It will also contain a very short mention - one or two lines -
of the model (e2e tests ---> integration tests ---> unit tests) used.
--------------------------------------------------------------------------
There are two possible concrete values/advantages in this new struct:
1) The reordered code would be much more readable, especially in the
perspective of finding and running on the fly a specific test for a
specific need or in the perspective of adding new ones, or deleting
obsolete ones.
2) Integration tests could significantly speed up bug fixing, other than
giving a more robust validation procedure of the released software. Let me
show an example.
An integration test could consist in the same steps described in a
submitted bug:
open a file ---> modify a curve ---> apply a filter ---> export to png
In the integration_tests dir, there could be, together with test files, one
(or more than one) file called "integration_test_template.cpp", which is
meant as an example, with basic operations done sequentially (open, modify,
filter, export etc.). Then, if another example called
"integration_test_custom.cpp" is created by the developer (but not added to
the source tree and added to .gitignore), it would contain code modeled on
the basis of "integration_test_template.cpp" and it would be automatically
compiled because as a part of the CMakeLists directives. In this way, we
have a ready-to-use environment for testing the API.
So, when a nasty bug is described together with its steps, a
"integration_test_custom.cpp" file can be created on the fly by using
chunks of code taken from "integration_test_template.cpp" in order to
replicate the steps.
What do you think? And do you think that the integration_tests template
would be really useful? In this last case, could you share an example (or
more than one) of a bug that can be managed with
"integration_test_custom.cpp" so as to have a useful case study?
If the idea appears good and collects approval, I can start making a fork
and a new branch for it on my local clone.

paolo fx

tavmjong＠free.fr

tags (0)

participants (2)