UMIST EFFECTS Portfolio: Stephen Bostock
 

1.2 Computer Assisted Assessment: A brief review prior to some experiments

A naïve view of assessment is that we, as teachers, can simply measure the final learning outcomes of our students that are the result of their learning activities in a course. This view is inadequate for two main reasons.

• Assessment looms large in the student perspective of any course, largely determining their approach to learning and, therefore, their learning outcomes. This has been referred to as ‘assessment backwash’ (Biggs 1999, chapter 8). ‘Assessment always determines the actual curriculum’ (Ramsden 1992, 187) regardless of the ‘official’ curriculum we present to them.
• Assessment is an important way of students receiving feedback on their state of learning and it thus needs to be part of any effective course. Race (1995) lists ‘receiving feedback from other people’ as one of the four determinants of leaning. All assessment provides some feedback but the design of a good curriculum would promote its benefits.

The conclusion must be that assessment should be integral to course design: ‘Don’t bolt assessment on – tie it into course design.’ (Race 1995).

A distinction is usually made as to the purpose of assessment (e.g. Biggs 1999, 142):

• In formative assessment the results are feedback to students to help their learning. Feedback in various forms is essential to learning, and should and does occur in various forms in good teaching, even where it is not a formal part of the curriculum design, for example, questions at the end of lectures. Error is a source of correction and learning. Formative assessment can be self-assessment or peer-assessment as well as teacher assessment. It is necessary for the self-management of learning, which is generally one of the goals of Higher Education.
• In summative assessment the results of which are used to grade students at the end of a course. At this point, feedback is too late, a grade is recorded. Error in summative assessment is more a failure causing punishment than an opportunity to improve learning.

Oliver (1999), for example, has listed some additional purposes for assessment but they do not detract from the importance of this basic distinction. However, it is a mistake to assume that any one assessment is wholly formative or summative, their effect is often mixed. Two important curriculum decisions are what assessments should count for grading, and how feedback can be increased and improved.

To take formative assessment first, ‘it is generally recognised that learning depends on feedback to the learner and that providing quick and helpful feedback to students is extremely beneficial.’ (Falchikov 1995). However, with increasing numbers of students it becomes more difficult to provide timely or helpful feedback. Falchikov proposed the wider use of peer feedback marking, where other students provided information on the strengths and weaknesses of student work, and a grade. Her experience was that the student grades were close to those of teachers. When students were asked to evaluate the good and bad features of the system, good features were listed as fairness and feedback as an aid to learning. The lack of anonymity (in her system) was a problem, leading to a (small) tendency for marks to be high; an anonymous system would be better.

A common practical problem in curriculum design is that fair summative assessments of some of the learning outcomes we desire, the ‘higher’ functions such as analysis, evaluation and generalization, are time consuming for the teacher. It is easier and quicker to assess simple learning outcomes such as describing concepts and performing simple procedures. However, most courses have a range of learning outcomes, susceptible to a range of summative assessment methods purposes. So the question arises: Can computer based assessment have a useful role in the mix of assessment used in a course?

Biggs (1999, chapter 9) reviews the purposes and forms of assessment in HE. Learning objectives are usually a range from simple declarative or procedural knowledge (as demonstrated by, for example, the verbs describe, identify, do) to higher level abilities of ‘functioning knowledge’ (e.g. analyse, design, hypothesize). A common difficulty is assessing the higher forms of knowledge, which usually requires expert human judgement. Objective formats of assessment are those where there is one right answer. These are the easiest to computerise and the commonest is multiple choice questions (MCQs). "Theoretically (MCQs) can assess high level verbs. Practically, they rarely do." (page 175) They assess declarative knowledge, usually recognition. This may be valuable as long as they are part of a mix of assessment methods where other, higher, objectives are being assessed in other ways. But "when used exclusively, they send all the wrong signals" to students. Furthermore, MCQs encourage game-playing tactics, both by the student in answering them and the teacher in setting them, and this also undermines deep learning. So there are theoretical problems with MCQs even though they are the easiest assessment method to computerise.

Race (1995) provides a wealth of practical advice for improving both summative and formative assessment. The four below seem susceptible to being aided by computers in the courses being considered here.

• "Try to develop skills to help students share assessment tasks and criteria, rather than approaching assessment as a timed, competitive activity.
• Make feedback immediate (for example using computer feedback).
• Put students in control of their learning by getting them to set questions and to answer them.
• Use self and peer assessment."

Race (1999, 73) makes the point that students should be trained in the use of the assessments they will be given. Taking this further, and applying the principle from constructionism (Harel and Papert 1991) that ‘I do and I understand’, we can conclude that devising assessments should be an excellent learning activity for students. Given the learning objectives, setting multiple choice tests or examination questions, together with marking criteria, should require deep learning. After all, it requires the deep knowledge of the tutor who normally sets them. Having set the questions, there would be little need for students to take the assessments for formative purposes, although they could set questions for other students on different topics. A special case is where students are creating software as a learning task, and could create CAAs for other students to use. If some of these assessment questions were used for summative purposes, this would fulfil the third point of Race, above.

In the 1990’s interest in CAA has increased, driven by increasing student numbers needing assessing and increasing student access to computers. In 1994 Bull (1994a, 21) stated "the use of computer based assessment is in its early stages of development and its use is not yet widespread". That first Issue of Active Learning was devoted to CAA and various authors reviewed the issues involved, case studies, recent publications and the authoring software available. Bull (1994b) provided a more substantial acount.

After five years more experience, Brown, Race and Bull (1999) provided a more substantial compendium. They list three benefits that have been demonstrated for Computer Assisted Assessment (CAA):

1. reducing the load on hard-pressed teachers by automating appropriate parts of the task of marking students’ work
2. providing students with detailed formative feedback on their learning much more efficiently than is usually possible with traditional assessment
3. bringing the assessment culture experienced by students closer to the (computer based) learning environments with which they are increasingly familiar and confident.

Points 1 and 2 address the problems raised above, so we may look to CAA with some hope of success. Point 3 relates the fact that the student learning environment now commonly includes the use of computers (as productivity tools in their private work, providing access to networked information resources, or using software provided as part of the course content) yet assessment is still largely manual. Traditional exams, requiring handwriting under time pressure, will often be the first occasion needing this skill.

CAA for formative assessment is reviewed by Charman (1999) who states that "there is mounting evidence for the pedagogic advantages of CBA in providing feedback on student work in higher education." He summarises some of the evidence. Many studies have evaluated staff and student attitudes to the introduction of CBA/CAA, but he admits that few studies have evaluated the effects on learning as measured by summative assessment. The one example he gives, of a statistics module for geography students, is unconvincing: after introducing CAA for feedback there was a slight increase in student mean summative performance despite a slight decrease in their entry ability, but the variation around the means was large so the differences could not be significant statistically (Charman 1999, table 9.1). However, differences in student performance as a result of any pedagogic innovation are notoriously difficult to demonstrate. Amongst the factors affecting performance is student effort for a course, which can decline as the course design is made more effective. So while improved quality of learning (at no greater cost in staff time or resources) is the essential goal for the introduction of more student feedback, it is not likely to be measurable simply in terms of summative assessment grades, and other evaluation measures are likely to be more useful.

"The crux of succesful formative CBAs is the provision of online feedback". Charman (1999) summarises the advice on the nature of the feedback to be given.

• State clearly if the response was correct or not, explaining any errors
• Be as positive as possible
• Give time for students to read feedback
• Consider graphics in the feedback
• Keep the tone simple and friendly
• Consider feedback after each question or after all questions
• Give references to sources for improved understanding

Some of these may be determined by their availability in the software being used to contruct the feedback.

Most of the literature on formative CAA involves feedback being given automatically in response to the student answer. There are obviously limits on the type of feedback possible, given the low intelligence of the software compared to a human assessor. Other students may not be as capable as tutors but they are much more capable than most software. The advantages of peer review of student work were discussed above. Using computer networks to mediate this review process can make it more feasible, and add the advantage of anonymity. Robinson (1999) argues that the difficulties in principle of summative assessment by peers can be answered by anonymity and multiple reviewers (working much as peer review for journal publications) but she found very little relevant literature. She described two experiments in multiple anonymous peer review for formative and summative assessment combined, without the aid of computer support. The evaluation conclusions included the need for more than two reviews of each piece of work, and the need for student training and practice in reviewing. No software exists to support this type of activity and she specifies what it would entail. However, a mix of manual administration and standard web forms could achieve much of what is needed.

As discussed above, the type of learning outcomes that can be assessed objectively limits the use of CAA for summative purposes. Although in theory some higher level verbs could be assessed, this would at least involve more time and effort in developing the questions. Although MCQs could be part of a mix of assessments, their use would need careful explanation to the students if their backwash was not to prompt surface learning strategies. On balance, summative assessment with CAA should be approached with caution.

In conclusion, Harvey and Mogey (1999) summarise the problems or objections to the introduction of CAA and their solutions. Their final advice on ‘what should be the first steps’ is ‘don’t be too ambitious … start off with a small and manageable project … build a thorough evaluation programme into the pilot’. My practical intentions for my teaching in the Computer Science Department are therefore:

• to provide several web MCQs for formative purposes
• to organize learning activities where students design examination questions and to encourage some students to create web MCQs as a part of coursework
• to use a web MCQ as a first summative assessment of basic knowledge and as a driver for study
• to use the web as a medium for anonymous student peer review of coursework.

References

Biggs, J. 1999 Teaching for quality learning at University, Open University Press

Brown, Race and Bull (eds) 1999 Computer Assisted Assessment Kogan Page

Bull, J. 1994a Computer based assessment: some issues for consideration Active Learning 1, December, 18-21

Bull, J. 1994b Using technology to assess student learning TLTP Project ALTER, University of Sheffield, ISBN 1 86889 091 8

Charman, D 1999 Issues and impacts of using computer-based assessments (CBAs) for formative assessment, 85-84 in Brown, Race and Bull (eds) 1999 Computer Assisted Assessment Kogan Page

Falchikov, N 1995 Improving feedback to and from students, 157-166 in Assessment for Learning in Higher Education ed. P Knight, SEDA/Kogan Page, London.

Harvey, J and Mogey, N 1999 Pragmatic issues when integrating technology into the assessment of students, 7-20 in Brown, Race and Bull (eds) 1999 Computer Assisted Assessment Kogan Page

Harel, I and Papert, S 1991 Constructionism Norwood NJ: Ablex

Race, P. 1995 What has assessment done for us – and to us? 61-74 in Assessment for Learning in Higher Education ed. P Knight, SEDA/Kogan Page, London.

Ramsden 1992 Learning to teach in Higher Education, 1992, Routledge

Martin Oliver A framework for evaluating the use of educational technology, http://www.unl.ac.uk/tltc/elt/elt1.htm

Stephen Bostock December 1999