Validating an exercise
======================

In our :doc:`previous tutorial <writing_your_first_script>`, we learned how
to write a simple moulinette that compiles the student delivery into a program.

Here, we are going to go a little further and see how we can write a
moulinette that checks whether the compiled program is fully conforming
to what was asked. If it is, the student's exercise will be validated,
otherwise, it will be marked as failed.

Back to the blueprint
---------------------

Before we go further, we are going to reuse our previous blueprint:

.. code-block:: python

    import quixote
    from quixote import get_context
    import quixote.build.apt as apt
    import quixote.fetch.gitlab as fetch
    from quixote.inspection.build import gcc

    @quixote.builder
    def install_gcc():
        apt.update()
        apt.install("gcc")

    @quixote.fetcher
    def fetch_delivery():
        fetch.gitlab()

    @quixote.inspector
    def compile_delivery():
        delivery_path = get_context()["delivery_path"]
        gcc(f"{delivery_path}/*.c", output_file="student")

Let's assume the exercise we want to validate has the following instructions:

.. code:: none

    Write a C program which accepts exactly one parameter, and prints it back on its standard output,
    followed by a newline.

We already have everything we need up to the compilation of the student delivery, so all we need to do is
execute the resulting program and ensure it is fully conforming to the exercise's requirements.

Validating the exercise (or not)
--------------------------------

.. _using-check:

Using ``.check``
~~~~~~~~~~~~~~~~

Since C is a compiled language, the first case of failure we must consider is the fact that
the student's source files cannot be compiled.
In order to fail if that happens, we must verify that the :func:`~quixote.inspection.build.gcc` built-in executed successfully.

Just like many command-related built-ins, the invocation of :func:`~quixote.inspection.build.gcc` returns a
:class:`~quixote.inspection.exec.CompletedProcess` object. These objects have a
:meth:`~quixote.inspection.exec.CompletedProcess.check` method, which can be used to ensure that the invoked command
exited with a success status. Otherwise, the current step will be aborted with an error, causing the invalidation
of the exercise.

.. code-block:: python
    :emphasize-lines: 4

    @quixote.inspector
    def compile_delivery():
        delivery_path = get_context()["delivery_path"]
        gcc(f"{delivery_path}/*.c", output_file="student").check("your delivery cannot be compiled")

..

.. note::
    That method takes a message as parameter, giving context to the check.

    By default, the ``stdout`` and ``stderr`` of the checked process are appended to the message in
    case of failure, in order to provide details about the failure.
..


Using assertions
~~~~~~~~~~~~~~~~

**Assertions** are used to ensure that a certain condition is true, and
abort the current step otherwise. This means that if an assertion fails,
tests in the same step won't be checked any further, and the student will not validate the exercise.

.. code-block:: python
    :emphasize-lines: 7, 8

    from quixote.inspection.check import assert_true, assert_equal
    from quixote.inspection.exec import command

    @quixote.inspector
    def test_with_assertions():
        res = command("./student lalala")
        assert_true(res.return_code == 0, f"invalid exit status: {res.return_code}")
        assert_equal(res.stdout, "lalala\n", f"incorrect output for 'lalala': '{res.stdout}'")

.. note::
    Python's :code:`assert` built-in can also be used as a way to perform assertions.

.. note::
    We use the :func:`quixote.inspection.exec.command` to execute the student's program.
    It allows running an executable and gathering its result into a :class:`~quixote.inspection.exec.CompletedProcess` for further examination.

Using expectations
~~~~~~~~~~~~~~~~~~

Finally, we can also use **expectations**, which can be seen as "soft"
assertions. Unlike assertions, failed expectations do not abort
the current step, but will still cause the invalidation of the exercise.

We can use expectations in order to collect information about every single
failure rather than stopping on the first one.

.. code-block:: python
    :emphasize-lines: 7, 8

    from quixote.inspection.check import expect_true, expect_equal
    from quixote.inspection.exec import command

    @quixote.inspector
    def test_with_requirements():
        res = command("./student lalala")
        expect_true(res.return_code == 0, f"invalid exit status: {res.return_code}")
        expect_equal(res.stdout, "lalala\n", f"incorrect output for 'lalala': '{res.stdout}'")

Wrap up
-------
You now know the different ways to validate or invalidate an exercise using the inspection phase.
Here, we used basic cases of the built-ins, however, don't hesitate to explore the reference to learn about
how they can be configured for your specific use cases.

Also, you can download the example moulinette for this tutorial :download:`here <resources/validating_an_exercise/example.zip>`.