Producing meaningful feedback

After your moulinette is applied on some students deliveries, the students that did not validate the exercise receive a feedback through the intranet. That message’s content is produced solely by your moulinette, and is the only way for students to understand why their delivery was rejected. Furthermore, technical assistants will also read that message when helping students. At some point, even you might have to read that message in order to debug your moulinette.

Therefore, it is crucial that the message is explicit enough. This tutorial will cover the tools at your disposal to achieve that goal.

An example blueprint

Let’s suppose we are starting from the following inspection step, which validates the my_strcmp function:

@quixote.inspector
def test_strcmp():
    strings = ["abc", "a", ""]
    for string in strings:
        ret_val = student_strcmp(string, string)
        expect_true(ret_val == 0, "incorrect return value")

    strings_pairs = [("bcd", "abc"), ("a", ""), ("abca", "abc")]
    for s1, s2 in strings_pairs:
        ret_val = student_strcmp(s1, s2)
        expect_true(ret_val > 0, "incorrect return value")

    strings_pairs = [("abc", "bcd"), ("", "a"), ("abc", "abca")]
    for s1, s2 in strings_pairs:
        ret_val = student_strcmp(s1, s2)
        expect_true(ret_val < 0, "incorrect return value")

Note

In the above code, the student_strcmp function calls the student’s function and retrieves it result.

The above inspector works, but does not provide much information. Here’s an example of a feedback it could produce:

incorrect return value
incorrect return value
incorrect return value

As you can probably tell, this is not a helpful feedback. Let’s see how we can improve it.

Structuring the tests

As specified by the official documentation, the my_strcmp function takes two strings s1 and s2, and must return:

• 0 if s1 is lexicographically equal to s2

• a strictly positive number if s1 is lexicographically superior to s2

• a strictly negative number if s1 is lexicographically inferior to s2

These different cases are implicitly shown in our tests, as we effectively test the function using:

• identical strings

• a lexicographically superior string for s1

• a lexicographically inferior string for s1

Making these test cases explicit would be of great benefit in our case, since it would not only improve the feedback messages, but also the readability of our moulinette’s implementation.

For that, Quixote provides the using() function, which encloses a test case inside of a Python with-block, like so:

with using("identical strings"):
    strings = ["abc", "a", ""]
    for string in strings:
        ret_val = student_strcmp(string, string)
        expect_true(ret_val == 0, f"incorrect return value")

with using("a lexicographically superior string for s1"):
    ... # perform the tests that use a lexicographically superior string for s1

...

Note

These blocks can be arbitrarily nested.

With these changes, our student’s feedback message will now look like:

Using identical strings:
  incorrect return value
  incorrect return value
  incorrect return value

By reading it, the student can now understand where the problem is.

Specifying expectations clearly

As you noticed, incorrect return value is not really explicit by itself: it explains what is wrong, but does not give any details regarding the incorrect value nor the expected value.

We should communicate what we expect more clearly, for example using sentences like “X should match Y” or “expected X to be Y”.

In the case of identical strings, we expect the return value to be 0. Let’s add that to the feedback:

with using("identical strings"):
    strings = ["abc", "a", ""]
    for string in strings:
        ret_val = student_strcmp(string, string)
        expect_true(ret_val == 0, f"expected a return value of 0, got {ret_val}")

Here is our updated message:

Using identical strings:
  expected a return value of 0, got 3
  expected a return value of 0, got 1
  expected a return value of 0, got 4

Adding teacher-only information

Providing helpful feedback to students is good, but we must be careful not to leak exact information about our tests. For example, in our moulinette, we cannot add the arguments passed to the student’s function to the feedback: otherwise, some students could just hardcode a solution that works only for these arguments.

On the other hand, teachers and assistants still might want to see detailed information and identify which cases failed (in order to help a student, or to debug an unexpected result). To achieve that, the using() function accepts a hidden keyword argument that can be set to True to hide the resulting message from students.

We can use that to safely display the test cases

with using("identical strings"):
    strings = ["abc", "a", ""]
    for string in strings:
        with using(f'"{string}" as argument', hidden=True)
            ret_val = student_strcmp(string, string)
            expect_true(ret_val == 0, f"expected a return value of 0, got {ret_val}")

After that change, the output will remain unchanged on the student side, but teachers will see the following message:

Using identical strings:
  Using "abc" as argument:
    expected a return value of 0, got 3
  Using "a" as argument:
    expected a return value of 0, got 1
  Using "" as argument:
    expected a return value of 0, got 4

Wrap-up

You now know how to ensure the feedback messages received by your students carry enough information for them to understand what they did wrong. You also know how to help your colleagues by providing useful debug information.