You will receive access to a new repository with the API designed by another team. Your task is to write tests for the public API and setup CI for the project. This task is done individually, which means that you should receive two sets of tests for your API.

API tests

The purpose of this task is to practice writing tests against a public API without access to the implementation (i.e. black-box style testing). Your tests should cover all the functionality provided by the library — review the API requirements from task 1, check whether the library provides some extra functionality and write the tests.

The character of your tests will depend on the API provided by the library. Libraries providing a declarative API may require setting up different situations using data classes and checking that the library had the expected effect after parsing a specific command line. Test for libraries providing an imperative interface may look just like a set of unit tests, because the API might provide you with a set of classes full of public methods.

Just keep in mind that you should test the API from the perspective of the client code. This means that your test classes should be in a different package (to make package private elements inaccessible), and should not rely on “friend” access (in languages where that applies).

Use the requirements to define the expected behavior in the tests. If the requirements do not clearly define some behavior, check if the API defines it (in the sample client code or in the documentation) and write tests according to that. If in doubt, define the expected behavior yourself using the tests. Do not forget to test for expected failures (i.e., error codes returned or exceptions thrown) too.

Because there is no implementation, your tests will (obviously) fail. While that can be somewhat discouraging, a failing test is also the primary driving force in Test Driven Development, because it makes the expected behavior explicit and thus makes it clear what needs to be implemented.

It is difficult to specify the number of tests that your solution needs to provide, but experience from past years shows that the quality of the solution clearly correlates with the number of tests. Solutions with 30+ test methods were markedly better than solutions with less than 20 or even less tests.

Use common sense to provide a reasonable set of tests that you believe would help your colleagues drive their implementation and also what you would expect the other students creating tests for your solution.

Finally, use a well-known unit testing framework that nicely integrates with the language; avoid inventing your own.

CI in GitLab

The second part of this task is to setup a simple CI for the project. Simply put, configure the project so that tests are executed automatically on each commit (push) to GitLab.

If you have never configured CI in GitLab, we suggest to consult the reference manual and examples for unit testing.

Suggestions

The following is a short summary of things to watch out for in your tests, because they keep appearing over and over in the submitted tests.

Naming

  • Test method names should reflect the situation being tested and the expected outcome.

    • StackTest.popFailsOnEmptyStack() is a better name than StackTest.testPop().

    • Avoid test prefix in the name if you language supports annotations, e.g., @Test (Java) or [Test] (C#), or similar.

    • Avoid should in the test method name, just describe the expected outcome.

    • Like any other names, they definitely should not be numbered.

  • Prefer the language of the problem, not the implementation. There are even libraries (e.g., catch for C++) where you don’t have to deal with test method names and just use plain English descriptions.

Coverage

  • Cover different situations in a systematic fashion, based on the requirements from task-1, and organize the tests accordingly.

  • Don’t forget to test robustness with respect to invalid inputs. The API should be explicit in how it reacts to such inputs — error handling IS part of design.

Implementation

  • Make the test code obvious for the reader.

    • The reader must be able to see that you are arranging the right situation, executing the right action, and performing the right asserts to determine the test outcome.

    • In other words, strive for the Arrange-Act-Assert (AAA) structure in your test code.

  • Keep tests small and test one situation at a time.

    • This also means that any testable conditions from the arrange phase should be tested using separate tests (not asserts). The assert phase should only concern the particular situation being tested.

  • Make test inputs “visible”.

    • Unlike non-test code, constant literals are tolerated and often preferable in tests, because they are part of the scenario being tested (e.g., invalid inputs, boundary conditions).

Exceptions

  • Declare expected exceptions using test annotations, don’t catch them.

    • Unexpected exception should crash (fail) the test.

    • If multiple statements may throw the expected exception, use the assert throws facility of your testing framework, which typically provides an assertThrows(() method which takes code to execute as a lambda and an exception type to expect. This may also indicate that the API does not provide exception types corresponding to the provided level of abstractions.

Parametrization

  • Consider using parametric (theory) tests for cases where everything but the input data changes and provide the data in a declarative way.

    • Strive to keep the tests obvious even when using parametric (theory) tests. This typically requires capturing the expected outcomes in the provided data. You can always comment on the data, or include an extra (dummy) string in the data containing a textual description.

  • Avoid code duplication (while keeping the code obvious).

    • Extract common parts into test setup phase or create parametric helper methods (i.e., build a “vocabulary”) to setup various scenarios. Just make sure the code remains readable.

Sources

  • Treat test code with the same care as your other (non-test) code.

    • Put the code in the right package, split the code into multiple files (if it makes sense), use white space (empty lines) to show boundaries between syntactic elements (methods), etc.

Submission

When you are satisfied with your solution to this task, make sure it is in the master branch of your task-4 repository and tag the commit using the task-4-submission tag.

Note: use the git tag task-4-submission command followed by git push --tags to tag the commit instead of using hash tags in commit messages (that’s not a Git tag).