1.12 Conclusion

We present λ, a system for autograding Snap! programs. The tool contains both a novel front-end student-centric autograder that presents immediate feedback to students, as well as a backend database that can serve as question repository. After successfully being used on edX, we've shown that the tools can be adopted in a traditional classroom. To accomplish we implemented the LTI protocol, so that λ may be used with as many LMSs are possible with the goal of reaching more students. Additionally, we've seen the benefits of having complete control of your infrastructure.

Our initial surveys have shown that while many students enjoy using an autograder, we must be careful not to replace instructional staff with technology. The stress that many students reported from the autograder suggest that there is a lot more human factors research that can be done about applying new technologies to the classroom. Still, we are confident that smart classroom policies which give students choice will continue to allow for the benefits of autograders without penalizing students who are uncomfortable with them. This does leave concern for online-only learning environments, but we see even a primitive autograder as a big step above nothing. Future interface enhancements and new features for studying will hopefully only add to the benefits students see while using the autograder.

Despite some successes, we know that there is a long way to go. We are still only able to grade a subset of all possible problems we could assign to students. There are many active research areas in techniques like automatic hint generation which we think will prove useful in the future. However, we also recognize that despite the promise, we need to spend to lower the barrier to entry for writing new content, both in terms of saving time and following pedagogical best practices. Fortunately, we think there is a clear path forward in this area.

Above all, we have shown that it is possible to build an autograder for a visual programming environment. Though these tools developed out of a need to scale teaching computer science, we believe that have the potential to greatly enhance traditional classrooms. As an area of active research we are excited to see the directions that autograding visual programming languages may take.