Validating an exercise

In our previous tutorial, we learned how to write a simple moulinette that compiles the student delivery into a program.

Here, we are going to go a little further and see how we can write a moulinette that checks whether the compiled program is fully conforming to what was asked. If it is, the student’s exercise will be validated, otherwise, it will be marked as failed.

Back to the blueprint

Before we go further, we are going to reuse our previous blueprint:

import quixote
from quixote import get_context
import quixote.build.apt as apt
import quixote.fetch.gitlab as fetch
from quixote.inspection.build import gcc

@quixote.builder
def install_gcc():
    apt.update()
    apt.install("gcc")

@quixote.fetcher
def fetch_delivery():
    fetch.gitlab()

@quixote.inspector
def compile_delivery():
    delivery_path = get_context()["delivery_path"]
    gcc(f"{delivery_path}/*.c", output_file="student")

Let’s assume the exercise we want to validate has the following instructions:

Write a C program which accepts exactly one parameter, and prints it back on its standard output,
followed by a newline.

We already have everything we need up to the compilation of the student delivery, so all we need to do is execute the resulting program and ensure it is fully conforming to the exercise’s requirements.

Validating the exercise (or not)

Using .check

Since C is a compiled language, the first case of failure we must consider is the fact that the student’s source files cannot be compiled. In order to fail if that happens, we must verify that the gcc() built-in executed successfully.

Just like many command-related built-ins, the invocation of gcc() returns a CompletedProcess object. These objects have a check() method, which can be used to ensure that the invoked command exited with a success status. Otherwise, the current step will be aborted with an error, causing the invalidation of the exercise.

@quixote.inspector
def compile_delivery():
    delivery_path = get_context()["delivery_path"]
    gcc(f"{delivery_path}/*.c", output_file="student").check("your delivery cannot be compiled")

Note

That method takes a message as parameter, giving context to the check.

By default, the stdout and stderr of the checked process are appended to the message in case of failure, in order to provide details about the failure.

Using assertions

Assertions are used to ensure that a certain condition is true, and abort the current step otherwise. This means that if an assertion fails, tests in the same step won’t be checked any further, and the student will not validate the exercise.

from quixote.inspection.check import assert_true, assert_equal
from quixote.inspection.exec import command

@quixote.inspector
def test_with_assertions():
    res = command("./student lalala")
    assert_true(res.return_code == 0, f"invalid exit status: {res.return_code}")
    assert_equal(res.stdout, "lalala\n", f"incorrect output for 'lalala': '{res.stdout}'")

Note

Python’s assert built-in can also be used as a way to perform assertions.

Note

We use the quixote.inspection.exec.command() to execute the student’s program. It allows running an executable and gathering its result into a CompletedProcess for further examination.

Using expectations

Finally, we can also use expectations, which can be seen as “soft” assertions. Unlike assertions, failed expectations do not abort the current step, but will still cause the invalidation of the exercise.

We can use expectations in order to collect information about every single failure rather than stopping on the first one.

from quixote.inspection.check import expect_true, expect_equal
from quixote.inspection.exec import command

@quixote.inspector
def test_with_requirements():
    res = command("./student lalala")
    expect_true(res.return_code == 0, f"invalid exit status: {res.return_code}")
    expect_equal(res.stdout, "lalala\n", f"incorrect output for 'lalala': '{res.stdout}'")

Wrap up

You now know the different ways to validate or invalidate an exercise using the inspection phase. Here, we used basic cases of the built-ins, however, don’t hesitate to explore the reference to learn about how they can be configured for your specific use cases.

Also, you can download the example moulinette for this tutorial here.