This page describes how to read results of your automated tests in your repository (note that the screenshots might be slightly different for your project).

When you open your course project, you will notice that there is either a green or red icon next to your last commit message representing state of your project.

Actually, it also has testing in progress state but that one will be quickly replaced by the final outcome.

Clicking on the icon will show detailed record of the actual tests.

The tests are split into groups that always capture topics related to a single lab.

Thus you can easily see your overall status and you can quickly check that you are ready for next lab.

Opening any of the groups will show you a transcript of actions that were performed by GitLab for you.

The important part starts with bin/tests.sh followed by text similar to the following.

1..5
ok 1 01/dayname.py - Submitted
ok 2 01/dayname.py - Correct Python script
ok 3 01/dayname.py - Module-ready
not ok 4 01/dayname.py - Works
  ---
  message: |
    ...
    -- Program output mismatch --
    output   : Hello, World
    expected : Friday
    --
  ...
ok 5 01/dayname.py - Some other test # skip No sense in testing this

This is a Test Anything Protocol (TAP) output that says that it executed five tests (1..5) and three of them were fine (ok 1 ...) while some failed (not ok 4 ...).

The message for passing test is rather short – it merely states the name of the test (01/dayname.py - submitted). We will use this format of the name for tests that merely check that the file is present in your project.

For a failing test, the message again contains the name of the test (01/dayname.py - Works) followed by further description of the failure. TAP uses comment-like syntax of prefixing the lines with extra details with the hash sign (#) or by indentation.

There we can see that the test expected some output (i.e. Friday) but our solution prints something else (Hello, World in this example).

Note that the fifth test is marked as okay even though the test was skipped. That is an unfortunate limitation of TAP protocol that skipped tests are not more visible but the reasoning is that skipped test is usually not considered a failure.

This should give you enough information of what went wrong. In this case, the program was written in Python so you can debug it yourself as you see fit on your own machine.

Please, do not use GitLab as your development platform. Always debug and implement your solution locally – GitLab cannot serve as a replacement for a missing installation of Python, Linux or other software on your machine. The jobs running the tests are executed on a shared machine – let others use the machine as well.

GitLab also sends you an e-mail when any of these automated actions fails (in GitLab terminology, these are Pipelines and Jobs). You can turn the notifications off too if you prefer to check the results manually.

The automated tests are active only for tasks from the current labs (e.g., two last labs). That is to lessen the load on the infrastructure and also to keep the list of jobs shorter.

But you can always run the tests locally via ./bin/run-tests.sh and specify which jobs to execute.

./bin/run-tests.sh            # Relevant tests (last and current lab)
./bin/run-tests.sh 01         # Tests from lab 01
./bin/run-tests.sh 04 05      # Tests from lab 04 and 05
./bin/run-tests.sh 06/backup  # Tests for 06/backup.sh