Draft for a refactoring and improvement of the test suite
Hello inkscapers,
I could contribute to the development of the code in my spare time if the following idea makes sense and is of value. While talking in the #irc dev channel and in the last developer meeting, it turned out that the test suite in testfiles directory is somewhat messy and a refactoring could be desirable for some reasons. Here's my proposal. First of all, tests should be grouped in coherent and standard categories. A standard classification is:
end to end testing ---> integration testing ---> unit testing
This classification, translated into a dir tree would be: ----------------------------------------------------------
testfiles |______end2end_tests | |_______gui_tests | |_______cli_tests | |______integration_tests | |______unit_tests | |______CMakeLists.txt | |______CTestCustom.cmake.in | |______README.md
[SUBDIR] gui_tests: empty for now, just add a TODO/FILL-ME note inside it.
[SUBDIR] cli_tests: *) rendering tests are made with CLI (as the CMakeLists.txt confirms), so they should be moved inside cli_tests. Therefore, move here svg files from "rendering_tests" and move rendering_tests/test.sh as cli_tests/rendering.sh *) leave here the current content of cli_tests *) add an "images" dir containing all the graphic files
[SUBDIR] integration_tests: it will contain cpp or script files in the form: operation1_operation2_..operationN.cpp (or .sh, or .py etc).
[SUBDIR] unit_tests: a software unit can be meant both as: 1) (see testfiles/src/curve-test.cpp) a self-referential "concept" 2) (see testfiles/src/uri-test.cpp) as a class, or even a single function, from the API. The important thing is to separate unit tests from integration tests, which are sequences of different tasks. From what I see, the "testfiles/src" dir currently contains unit tests (if cpp files perform integration tests, they should be moved to the testfiles/integration_tests dir) In addition, do move lpe images here (inside an "images" dir). Add a note on the README that lpe stuff is here (as it partially is, actually).
[FILE] CMakeLists.txt
[FILE] CTestCustom.cmake.in
[FILE] README.md: note: only ONE readme will be used for describing this test suite. It will describe what the tests do and how to launch them separately. It will also contain a very short mention - one or two lines - of the model (e2e tests ---> integration tests ---> unit tests) used.
--------------------------------------------------------------------------
There are two possible concrete values/advantages in this new struct:
1) The reordered code would be much more readable, especially in the perspective of finding and running on the fly a specific test for a specific need or in the perspective of adding new ones, or deleting obsolete ones.
2) Integration tests could significantly speed up bug fixing, other than giving a more robust validation procedure of the released software. Let me show an example. An integration test could consist in the same steps described in a submitted bug:
open a file ---> modify a curve ---> apply a filter ---> export to png
In the integration_tests dir, there could be, together with test files, one (or more than one) file called "integration_test_template.cpp", which is meant as an example, with basic operations done sequentially (open, modify, filter, export etc.). Then, if another example called "integration_test_custom.cpp" is created by the developer (but not added to the source tree and added to .gitignore), it would contain code modeled on the basis of "integration_test_template.cpp" and it would be automatically compiled because as a part of the CMakeLists directives. In this way, we have a ready-to-use environment for testing the API. So, when a nasty bug is described together with its steps, a "integration_test_custom.cpp" file can be created on the fly by using chunks of code taken from "integration_test_template.cpp" in order to replicate the steps.
What do you think? And do you think that the integration_tests template would be really useful? In this last case, could you share an example (or more than one) of a bug that can be managed with "integration_test_custom.cpp" so as to have a useful case study?
If the idea appears good and collects approval, I can start making a fork and a new branch for it on my local clone.
Looks good to me.
What I think is perhaps even more important is having good documentation on how to use the testsuite. For example, if a test fails, how does one run that test by itself?
----- Original Message ----- | From: "paolo fx" fxp9750@gmail.com | To: inkscape-devel@lists.inkscape.org | Sent: Saturday, March 11, 2023 5:30:13 PM | Subject: [Inkscape-devel] Draft for a refactoring and improvement of the test suite | | | | Hello inkscapers, | | I could contribute to the development of the code in my spare time if | the following idea makes sense and is of value. While talking in the | #irc dev channel and in the last developer meeting, it turned out | that the test suite in testfiles directory is somewhat messy and a | refactoring could be desirable for some reasons. | Here's my proposal. First of all, tests should be grouped in coherent | and standard categories. | A standard classification is: | | end to end testing ---> integration testing ---> unit testing | | This classification, translated into a dir tree would be: | ---------------------------------------------------------- | | testfiles | |______end2end_tests | | |_______gui_tests | | |_______cli_tests | | | |______integration_tests | | | |______unit_tests | | | |______CMakeLists.txt | | | |______ CTestCustom.cmake.in | | | |______README.md | | | [SUBDIR] gui_tests: empty for now, just add a TODO/FILL-ME note | inside it. | | [SUBDIR] cli_tests: | *) rendering tests are made with CLI (as the CMakeLists.txt | confirms), so they should be moved inside cli_tests. Therefore, move | here svg files from "rendering_tests" and move | rendering_tests/test.sh as cli_tests/rendering.sh | *) leave here the current content of cli_tests | *) add an "images" dir containing all the graphic files | | [SUBDIR] integration_tests: it will contain cpp or script files in | the form: operation1_operation2_..operationN.cpp (or .sh, or .py | etc). | | [SUBDIR] unit_tests: | a software unit can be meant both as: | 1) (see testfiles/src/curve-test.cpp) a self-referential "concept" | 2) (see testfiles/src/uri-test.cpp) as a class, or even a single | function, from the API. | The important thing is to separate unit tests from integration tests, | which are sequences of different tasks. From what I see, the | "testfiles/src" dir currently contains unit tests (if cpp files | perform integration tests, they should be moved to the | testfiles/integration_tests dir) | In addition, do move lpe images here (inside an "images" dir). Add a | note on the README that lpe stuff is here (as it partially is, | actually). | | [FILE] CMakeLists.txt | | [FILE] CTestCustom.cmake.in | | [FILE] README.md: note: only ONE readme will be used for describing | this test suite. It will describe what the tests do and how to | launch them separately. It will also contain a very short mention - | one or two lines - of the model (e2e tests ---> integration tests | ---> unit tests) used. | | -------------------------------------------------------------------------- | | There are two possible concrete values/advantages in this new struct: | | 1) The reordered code would be much more readable, especially in the | perspective of finding and running on the fly a specific test for a | specific need or in the perspective of adding new ones, or deleting | obsolete ones. | | 2) Integration tests could significantly speed up bug fixing, other | than giving a more robust validation procedure of the released | software. Let me show an example. | An integration test could consist in the same steps described in a | submitted bug: | | open a file ---> modify a curve ---> apply a filter ---> export to | png | | In the integration_tests dir, there could be, together with test | files, one (or more than one) file called | "integration_test_template.cpp", which is meant as an example, with | basic operations done sequentially (open, modify, filter, export | etc.). Then, if another example called "integration_test_custom.cpp" | is created by the developer (but not added to the source tree and | added to .gitignore), it would contain code modeled on the basis of | "integration_test_template.cpp" and it would be automatically | compiled because as a part of the CMakeLists directives. In this | way, we have a ready-to-use environment for testing the API. | So, when a nasty bug is described together with its steps, a | "integration_test_custom.cpp" file can be created on the fly by | using chunks of code taken from "integration_test_template.cpp" in | order to replicate the steps. | | What do you think? And do you think that the integration_tests | template would be really useful? In this last case, could you share | an example (or more than one) of a bug that can be managed with | "integration_test_custom.cpp" so as to have a useful case study? | | If the idea appears good and collects approval, I can start making a | fork and a new branch for it on my local clone. | | _______________________________________________ | Inkscape Devel mailing list -- inkscape-devel@lists.inkscape.org | To unsubscribe send an email to | inkscape-devel-leave@lists.inkscape.org |
participants (2)
-
paolo fx -
tavmjong＠free.fr