1.4 Introduction

In recent years, the field of computer science (CS), and the broader technology industry, have undergone significant changes, though not simply technical ones. Around the same time that enrollments started booming [Citation not found], the National Science Foundation (NSF) posted the original solicitation for Broadening Participation in Computing [Citation not found]. As a result of the efforts to improve diversity in CS, curricula such as Exploring Computer Science [Citation not found] (ECS), and AP Computer Science Principles [Citation not found] (AP CSP) have emerged. Many curricula have started to use visual programming languages (VPLs), also known as blocks-based languages, as their primary tool. As great as these environments can be, they often lack resources and tools found in more traditional environments (such as Python or Java). This lack of infrastructure can make courses more difficult to teach and scale, particularly to entirely online environments.

We aim to help students learning programming by:

  • Delivering the best resources to possible to completely remote students, and
  • Improving the capability and efficiency of in-person teaching assistants (TAs).

One example of missing resources is the capability to automatically evaluate code, and send the results to another process. Automatic evaluation and distributing the results are central components in an autograder. Without an autograder, it can be difficult (if not prohibitively expensive) to teach large online courses. During the 2015-2016 school year, we taught The Beauty and Joy of Computing [Citation not found] (BJC) as a series of four massive open online courses (MOOCs) on edX [Citation not found]. BJC is an AP CS Principles course that uses Snap! as its primary language. Without an autograder, we did not think that we could fairly give credit to students online. At Berkeley, BJC is offered as CS10, which could use an autograder for Snap!.

We developed λ as a system for autograding Snap! [Citation not found]. λ is composed of two main parts, the Ruby on Rails application server, and a Javascript application that augments Snap! with testing, analysis and logging capabilities. While the grading interface was developed for edX, we report on its use in the classroom and some recommendations for future development. We also hope that λ will be useful outside CS10, through the implementation of the Learning Tools Interoperability (LTI) protocol which should allow our system to be compatible with most Learning Management Systems. We have built additional features which should allow flexibility to integrate the autograder into a variety of classroom environments. These features are described in the chapter on implementation.

1.4.1 Snap!

Snap! is a blocks-based, drag-and-drop programming language inspired by MIT's Scratch [Citation not found], but adapted for university-level courses by including features such as first-class lists and custom functions (blocks). Snap! is implemented as a web application in JavaScript, which makes it compatible with mobile devices. Figure 1 shows a basic drawing script being executed.

An example Snap<em>!</em> program.
Figure 1: An example Snap! program.

1.4.2 The Beauty and Joy of Computing

The Beauty and Joy of Computing (BJC) is an introduction to computing designed to broaden participation among underrepresented groups. The primary language used is Snap!, and many of the exercises in the course have visual outputs. The course covers functions and abstraction, recursion, higher order functions (lambdas) and many other topics. Many of the examples and exercises in BJC have multiple correct paths to implementation, which can be a challenge for autograders to handle.

A typical example of BJC curriculum which includes graphical output.
Figure 2: A typical example of BJC curriculum which includes graphical output.

1.4.3 CS10

CS10 [Citation not found] is UC Berkeley's offering of BJC. Currently, CS10 is offered every semester to 200-300 students. Like many other introductory courses, CS10 relies heavily on laboratory sections as the primary method for students to learn to program. λ will be used in CS10 during lab sections to give students better feedback as they are working and to give credit for the assigned lab work. This year, we were able to trial the system for three of the 14 labs that students completed in Snap!.

1.4.4 Correctness and Effort

While designing the autograder, we considered two primary types of grading: correctness and effort. Correctness is simple: Does is given block (or script) do what the instructions say? Effort is more open-ended, but the goal is to be able reward students for trying to complete a problem, as well as to encourage experimentation with Snap!'s many features. In both these cases, we would like the autograder to be robust to different correct solutions, or the many different ways students can demonstrate effort.