Computer Assisted Assessment - experiments in three courses

Stephen Bostock, May 2000

Some thoughts on assessment

1. The naïve view is teach and then measure the final learning outcomes, but

• (Summative) Assessment backwash. ‘Assessment always determines the actual curriculum’ (Ramsden 1992).
• (Summative and formative) Assessment provides feedback and thus promotes learning.

2. So ‘Don’t bolt assessment on – tie it into course design.’ (Race 1995) Two important curriculum decisions are what assessments should count for grading and how to relate learning activities to them, and how feedback can be increased and improved.
3. Formative assessment: ‘it is generally recognised that learning depends on feedback to the learner and that providing quick and helpful feedback to students is extremely beneficial.’ (Falchikov 1995). Worsening S/S ratios makes that more difficult. Possible solutions are

Self assessment from computer based multiple choice quizzes?
Peer assessment ? Anonymity and replication helpful.
4. Summative assessment: Learning objectives are usually a range from simple declarative or procedural knowledge (as demonstrated by, for example, the verbs describe, identify, do) to higher level abilities (e.g. analyse, design, hypothesise). So we need a mix of assessment tools. It is easier to assess the simple learning outcomes with objective tests (one right answer- e.g. multiple choice) and therefore with CAA. Fair assessments of ‘higher’ outcomes, such as analysis, evaluation and generalisation, need a human assessor in practice.

"When used exclusively, (MCQs) send all the wrong signals" to students (Biggs 1999) but can computer assisted assessment be a valid part of the mix of assessment?
5. Advice by Race (1995) includes

• "Try to develop skills to help students share assessment tasks and criteria, rather than approaching assessment as a timed, competitive activity.
• Make feedback immediate (for example using computer feedback).
• Put students in control of their learning by getting them to set questions and to answer them.
• Use self and peer assessment."
• And Race (1999, 73) "students should be trained in the use of the assessments they will be given."

6. Can CAA help? Brown, Race and Bull (1999) list benefits that have been demonstrated for CAA.

• reducing the load on hard-pressed teachers by automating appropriate parts of the task of marking
• providing students with detailed formative feedback much more efficiently
• bringing the assessment culture experienced by students closer to their (computer based) learning environments.

7. Peer review: other students may not be as capable as tutors but they are much more capable than most software. Robinson (1999) argues that the difficulties of principle of summative assessment by peers can be answered by anonymity and multiple reviewers (working much as peer review for journal publications). Using computer networks to mediate this review process can make it more feasible, and add the advantage of anonymity. No software exists to support this type of activity. However, a mix of manual administration, email and standard web forms could achieve most of what is needed.
8. Summative CAA should be approached with caution. Although MCQs could be part of a mix of assessments, their use would need careful explanation to the students if backwash was not to prompt surface learning.
9. Harvey and Mogey (1999) summarize the problems of CAA. Their advice on first steps is ‘don’t be too ambitious … start off with a small and manageable project … build a thorough evaluation programme into the pilot’.

Conclusion: it seemed worth trying

• to provide web MCQs for formative purposes, to improve feedback on learning
• to organize learning activities where students design examination questions and to encourage some students to create web MCQs as a part of their coursework
• to use a web MCQ for a summative assessment of basic knowledge and therefore as a driver for learning activity
• to use email and the web to administer anonymous student peer review of coursework, for formative and summative purposes.

CAA in three Computer Science courses
Semester 2, 1999/2000

a) The Business Information Systems module on the MSc in IT (35 students, one tutor).

A one semester module with a weekly 4 hour 'seminar' session plus some tutorials and practicals. The seminars included lectures, case studies done in small groups, and other exercises. Assessment is by two pieces of coursework plus a 2 hour exam. Much of the content is discursive and conceptual.

Seven formative multiple choice tests related to lecture topics were placed on the module web site, with 50-80 questions, each with one right answer from four. They were generated by software accompanying one of the course texts, from a bank of questions. Students were encouraged and reminded to use them after lectures. The tests did not record results but provided a score and the correct answers.

Also, more emphasis was placed on assessment in class: some of the MCQs were used in class, displayed on the screen. 10 or 20 questions were answered and marked by fellow students. Other activities included writing examination questions from the learning objectives and lectures, and writing outline answers to past exam questions.

b) Subsidiary module 'The Internet' (376 students, one tutor).

A one semester module with a weekly one hour lecture and 13 parallel 2 hour practicals, with 12 demonstrators in practicals. Assessment by 9 practical tasks marked by demonstrators as pass/fail and a final report for a grade. The learning objectives are information and communication skills.

Last year the task involving an online tutorial over two practicals was identified as a problem area. Some students found making notes from it tedious, although most preferred it to lectures. A quiz was firstly intended as a better way of getting students to use the tutorial. In the first two practicals a multiple choice 'Practice Quiz' allowed repeated tries and its feedback pointed students to places in a web tutorial for the answers, thus setting the task goals of discovering the answers. With one or more correct answers from 4 or 5, it was created using JavaScript with CALnet. An identical 'Assessed Quiz' was then completed as the second practical task, using a web form that emailed answers to the demonstrators for marking, and to the student and tutor.

c) The Multimedia and the Internet module on the MSc in IT (38 students, two tutors).

A one semester course with two 2 hour sessions per week plus practicals. The learning objective is to be able to use various web technologies to develop content based applications. Assessment by a 2 hour exam plus one piece of coursework developing a content-rich web application (hypermedia or tutorials), the content of which was a module topic and it may include a quiz.

Email and web forms were used for peer review of coursework. Students put a prototype application on their web sites and filled in a form to give its URL. By email they were told which four applications of other students they were to review formatively, and their secret reviewer's number. On a web form they gave marks and comments on several criteria, which were sent anonymously to the authors and to the tutors. The reviewer number allowed tutors to monitor the process, which was single-anonymous: reviewers know authors but authors did not know reviewers. Double anonymity would be difficult as the URLs include the username. At the submission deadline, web sites were frozen and then summative peer reviews of the same applications were done, sending marks only to the tutors. The four marks per author were compiled and moderated by the tutors.

Evaluation

a) The Business Information Systems module - MCQs

Voluntary anonymous module evaluation forms were emailed; 11 were returned.

1. How many times did you use the online quizzes?

0 to 3 with median of 1 for all 7 quizzes in total.
2. Did you find the online quizzes useful?

An even spread in 5 categories from 'not at all useful' to 'very useful'.
3. Did you find the classroom quizzes useful?

Some spread of answers but 9 out of 11 said 'moderately useful' or more.

Voluntary anonymous Post-it evaluations were used at the end of many sessions, asking 'What was useful in the session?' and 'What was less useful and could be improved?'. For example, of 8 returned for the Decision Support Systems session, the classroom quiz was listed as useful by 3, while practice exam questions listed as useful by 6. (Module evaluation returns also gave strong support for use of past exams in class. An exercise to write their own exam questions produced little in the way of realistic questions but they enjoyed it.)

Conclusion: Classroom quizzes and past exam questions were popular. Online quizzes were less well used but liked by some. Therefore, continue to support them on the web, and have more regular mini-MCQs in class. So CAA is helping a few but mostly the formative assessment-related exercises during classes were most valued.

b) Subsidiary module on The Internet

Did the students pass the assessment? With most of the data entered (8/5/00) , only 5 were late submissions, and 2 failed. So it was largely completed on time and to standard.

What did the students think of it? A Web form was used to evaluate the use of the web quiz and the tutorial. A summary is below, from the spreadsheet summary.
 

Generally it was well received. Some found having a right answer of more than one selection confusing, and the negative marking confusing. The quiz could be improved by removing negative marking and maybe prompting how many selections are needed in each answer.

c) The Multimedia and the Internet module on the MSc in IT

For the formative assessments of the prototypes, 38 students were each asked by email to do 4 or 5 evaluations each. Each reviewer gave a mark and text criticism for each of 5 criteria. Only three did none. Four prototypes were not adequately evaluated and were then evaluated by the tutor as if by an anonymous student reviewer.

After final submission, all students were emailed again to point them at a summative assessment web form. These were less well completed, because some students disappeared to revise for exams. A truncated marks table for the summative assessments is below. Each of 22 reviewers gave a mark to each of 4 or 5 assignments for each of 5 criteria.
 

Three applications could not be found, two students were away with permission, 14 did no reviews and one did them non-anonymously! The main problem was student participation that meant that the mean number of evaluations was only 2.3 per author, with one zero and several single assessments. The range of marks across assessors for any one category was 15% on average, and the range across the overall marks for one piece of work was 11% on average. So all of the marks will have to be moderated by tutors. No saving in staff time there!

Student evaluation by email of the peer review process got 16 replies, summarized below.

1. How many formative reviews did you receive?

2. Were the formative peer reviews of your prototype done professionally?

3. How useful were they in improving your work?

4. Are you happy with summative reviews being used as part of your assessment as long as it is tutor-moderated?

5. What was the best part of the review process?

6. What was the worst part of the review process?

7. Do you think we should do it next year? Why or why not?

Conclusion: formative text evaluation was valued, as was seeing other students' work. Better done a little earlier. There were some anxieties about summative marking as some students did not do a professional job. Poor rates of summative marking and poor consistency between markers means that full moderation will be necessary.

Administratively, it was time-consuming allocating reviewers, sending emails, and reminding students who did not do it on time. A student project is developing a web application to administer peer reviewing.

References and sources

Biggs, J. 1999 Teaching for quality learning at University, Open University Press

Brown, S., Race, P. and Bull, J. (eds) 1999 Computer Assisted Assessment Kogan Page

Bull, J. 1994a Computer based assessment: some issues for consideration Active Learning 1, December, 18-21

Bull, J. 1994b Using technology to assess student learning TLTP Project ALTER, University of Sheffield, ISBN 1 86889 091 8

Charman, D 1999 Issues and impacts of using computer-based assessments (CBAs) for formative assessment, 85-84 in Brown, Race and Bull (eds) 1999 Computer Assisted Assessment Kogan Page

Falchikov, N 1995 Improving feedback to and from students, 157-166 in Assessment for Learning in Higher Education ed. P Knight, SEDA/Kogan Page, London.

Harvey, J and Mogey, N 1999 Pragmatic issues when integrating technology into the assessment of students, 7-20 in Brown, Race and Bull (eds) 1999 Computer Assisted Assessment Kogan Page

McKenna, C. and Hesketh, I. 2000. A review of online resources for Computer Assisted Assessment. Educational Developments, 2.

Race, P. 1995 What has assessment done for us – and to us? 61-74 in Assessment for Learning in Higher Education ed. P. Knight, SEDA/Kogan Page, London.

Ramsden, P. 1992 Learning to teach in Higher Education, 1992, Routledge

Robinson, J.M. 1999 Computer assisted peer review. 95-102 in Brown, S., Race, P. and Bull, J. (eds) 1999 Computer Assisted Assessment Kogan Page

Twomey, E., Nicol, J. and Smart, C. Computer Assisted Assessment, using computers to design and deliver objective tests, CTI Biology factsheet 4, 1998.

Oliver M. A framework for evaluating the use of educational technology, http://www.unl.ac.uk/tltc/elt/elt1.htm

Stephen Bostock May 2000