Writing a moulinette with a reference program

Sometimes, the best way to validate a student’s delivery for an exercise is to have an implementation of a correct solution, and compare its behavior with the one of the student’s code. In this technique, the correct implementation is called a reference implementation.

This tutorial will explain how such an implementation can be used as part of a moulinette.

The exercise

We will consider an exercise with the following instructions and reference implementation:

Write a Python program ``exercise.py``, that takes a number as parameter and prints its double.
If the given argument is not a number, you should display ``expected a number!``.
If there are too many or too few arguments, you should display ``expected one argument!``

#!/usr/bin/env python3

import sys

if len(sys.argv) != 2:
    print("expected one argument!")
else:
    try:
        print(int(sys.argv[1]) * 2)
    except ValueError:
        print("expected a number!")

Note

This reference program will be put in the resources directory of the moulinette, in a file named reference.py.

We will now jump right to the inspection phase, as the build and fetch phase are not particularly relevant to this tutorial.

Writing the inspection phase

In order to compare the delivery with our reference program, we can simply compare their output. We could execute the programs, collect each output and compare them manually, however that would be quite tedious.

To simplify that, Quixote offers the diff_exec() function which, as its name states, allows comparing the output of two commands using the diff utility.

That function takes the commands to execute, and an argument list to pass to them.

import quixote
from quixote import get_context
from quixote.inspection.diff import diff_exec

@quixote.inspector
def compare_output_with_reference():
    ref = f"{get_context()['resources_path']}/reference.py"
    student = f"{get_context()['delivery_path']}/exercise.py"

    # Test regular cases
    for i in range(0, 42):
        diff_exec(ref, student, arguments=[str(i)]).check("your output and ours differ")

    # Test edge cases
    diff_exec(ref, student).check("your output and ours differ")
    diff_exec(ref, student, arguments=["1", "2"]).check("your output and ours differ")
    diff_exec(ref, student, arguments=["abcd"]).check("your output and ours differ")

Note

When the diff command finds a difference, it exits with status 1, which is why we use check() to (in)validate the exercise.

Wrap-up

You now know a typical way of implementing a moulinette based on a reference program. As always, we encourage you to download the resuling blueprint to try it out and adapt it to your needs!