README.md

# Advanced teaching automation :robot: :mortar_board:

Leverage Renku and Gitlab to automate classroom management.

## Target audience

This repository is intended for technically inclined teachers who use Renku and/or Gitlab in their class and wish to automate repetitive tasks related to student and project management.

## Purpose

We showcase the potential of Renku + Gitlab through a few scripts covering common use-cases.
These scripts can be used as-provided, but are also meant to be extended and adapted by teachers for their specific needs.

## Setup

The repository is setup as a local python package.
You can install it with: `pip install -e .`

Most script below require a Gitlab API token. By default, you will be prompted for your token upon running the scripts, but you can also provide it as a plain text file with the `--token` option.

## Content

The following tasks are automated:

* [:email: Inviting students to the class group](#email-inviting-students-to-the-class-group)
* [:lock: Creating private student groups](#lock-creating-private-student-groups)
* [🗂️ Gathering all forks of a project](#%EF%B8%8F-gathering-all-forks-of-a-project)
* [💾 Cloning all forks of a project](#-cloning-all-forks-of-a-project)
* [:loudspeaker: Sending feedback](#loudspeaker-sending-feedback)


---
### :email: Inviting students to the class group

Creating a single group for the class provides a centralizd place to store all materials, student repositories and exercises. In some cases, one may also want to make this group private.

The manual process of inviting the whole class to the group can be automated using Gitlab's invitation API. Given a list of student email addresses (e.g. exported from moodle) and the URL to a an existing gitlab group, we provide a script to automatically send an invitation email to each student.

<details>
<summary> <b>Read more...</b> </summary>

script: [teach\_utils/invite\_students.py](teach_utils/invite_students.py)

**usage**:

Invite students to `class-group`:
`python invite_students.py emails.txt https://gitlab-instance.com/class-group`

</details>

---

### :lock: Creating private student groups

When students work on graded assignments, they usually have to work in private groups. The groups must be accessible to the teachers in order to grade the assignments.

One way to achieve this is to have the teacher create all private student groups (The teacher will therefore own the groups) and then invite students to their respective groups. We provide a script to automate the group creation and invitation process. It creates one private group for each student based on their information in a CSV file exported from Moodle.

<details>
<summary> <b>Read more...</b> </summary>

script: [teach\_utils/moodle\_to\_student\_groups.py](teach_utils/moodle_to_student_groups.py)

**usage**:

Create student (sub)groups inside `class-group`:
`python moodle_to_student_groups.py students_moodle.csv https://gitlab-instance.com/class-group`

</details>

---

### 🗂️ Gathering all forks of a project

Homework assignment can be provided by the teachers in the form of a repository containing instructions and depencencies. Students can then fork this project into their own private groups (e.g. created with the aforementioned script).

The teacher can then use the Gitlab API to keep track of student forks. We provide a script to gather metadata about all student-group forks of a given project, with the option to retrieve the last commit before a deadline. That script will output the metadata as a JSON structure containing the following fields for each fork:

<details>
<summary> <b>Read more...</b> </summary>

* `id`: The gitlab project ID of the fork
* `url`: The https URL to reach the fork
* `group`: The student group name
* `members`: The members of the group. For each member the following fields are provided:
  + `username`: The gitlab username
  + `name`: The full name (First_name Last_name)
  + `email`: The email address of the student
* `commit`: The hash of the last commit before the deadline, used for grading the assignment.
* `autostart_url`: A URL to directly start a Renku session at the last commit before the deadline.

> Note: the script assumes Gitlab commit timestamps and the deadline are in the UTC timezone.

script: [teach\_utils/collect\_forks.py](teach_utils/collect_forks.py)

**usage**:

Collect all group-owned forks of upstream-project, and write the JSON list to `forks.json`. Commits and autostart URLs will point to the last commit before 23h59 on March 15, 2022:
```sh
python collect_forks.py \
  --deadline "2022-03-15T23:59" \
  https://gitlab-instance.com/namespace/upstream-project \
  > forks.json
```

**sample output:***

```
[
    {
        "id": 379,
        "url": "https://renkulab.io/gitlab/demo-group/assignment.git",
        "visibility": "private",
        "group": "demo-group",
        "members": [
            {
                "username": "smcstudent",
                "name": "Student McStudent",
                "email": student.mcstudent@email.here
            }
        ],
        "commit": "0175894cf61fe820fdb84e5c52fc7ee0259a3c71",
        "autostart_url": "https://renkulab.io/projects/demo-group/assignment/sessions/new?autostart=1&commit=0175894cf61fe820fdb84e5c52fc7ee0259a3c71&branch=master"
    },
    ...
]
```

</details>

---

### 💾 Cloning all forks of a project

In some cases, the student projects may have to be cloned locally by the teacher. We provide a shell script to automate this process. It reads the JSON output from `collect_forks.py` and clones all the forks at the deadline commit into a target directory.

<details>
<summary> <b>Read more...</b> </summary>


script: [teach\_utils/clone\_forks\_from\_json.sh](teach_utils/clone_forks_from_json.sh)

**usage**:

Clone all forks listed in `forks.json` into `clone_dir`:

`./clone_forks_from_json.sh forks.json clone_dir`

Or reading directly from stdin:

```
python ./collect_forks.py https://gitlab-instance.com/namespace/upstream-project \
  | ./clone_forks_from_json.sh clone_dir
```

</details>

---

### :loudspeaker: Sending feedback

After grading student assignments, the teacher may want to send student groups their grades, as well as positive or critical comments on their solution. This can be automated as well, using the Gitlab issues API. We provide a script to read a csv file containing grades and comments for each student group and open issues in the corresponding repositories automatically.

<details>
<summary> <b>Read more...</b> </summary>

script: [teach\_utils/send\_feedback.py](teach_utils/send_feedback.py)

**usage**:

By default, this script opens issues in all forks listed in `forks.json`. For each fork, fields {grades}, {comments} in the issue description are filled using column values in a CSV file provided with `--feedback`. The issue description uses a default template defined in the script:

`python ./send_feedback.py --feedback grades.csv forks.json`

where grades.csv is:

```
id,grade,comments
379,6,great use of list-comprehension!
424,5.5,Missing imports
```

A custom template can also be provided, and fields will be replaced by values from columns with corresponding names in `--feedback`. One could also use this script to warn all groups that their fork is public for example:

```
echo ":warning: Your fork is public, please make it private!" > visibility.md

./collect_forks.py https://renkulab.io/gitlab/class/homework \
  | jq '.[] | select(.visibility == "public")' \
  | ./send_feedback.py --template visibility.md

```
> Note: `jq` is a tool to process JSON data. It can be used to reformat or filter the output of `collect_forks.py` in many ways. A number of useful one liners are [shown here](./docs/oneliners.md).

</details>

## Contributing

Contributions related to common use-cases, or improvements are welcome.
Testing is done with `pytest --doctest-modules` and can be run with `make test`.

## License

These scripts are provided under the [MIT license](./LICENSE).