Computer assisted assessment workshop

Stephen Bostock

1. Some initial thoughts on assessment

'Students can avoid bad teaching: they can't avoid bad assessment' (Boud 1994)

'Assessment methods and requirements probably have greater influence on how and what students learn than any other single factor.' (Boud 1988)

'Assessment always determines the actual curriculum' (Ramsden 1992)

'Don't bolt assessment on - tie it into course design' (Race 1995)

1.1. A naive view of assessment is bolt-on: plan, teach, then assess. This is wrong for at least 2 reasons.  

• assessment backwash (Biggs 1999) means that student behaviour is determined by it, not by our teaching
• formative and summative assessment provide feedback and so are helpful to learning

1.2. Feedback

Student learning would be improved by more feedback - early, informative, positive. How can we do that without more resources?

1.3. Variety

 More variety of assessment methods is fairer to the variety of skills being assessed and variety of students being assessed.

1.4. Objective tests

Objective tests are seen as fair, but harder to write for 'higher order' learning outcomes (HOTS - higher order thinking skills)

1.5. Staff time

Without more staff time for marking/feedback, can we use 

1. automatic (computer) marking/feedback 
2. student peer marking/feedback (mediated by computer network)
3. self marking/assessment

most effort has gone into a, recently some into b. Structuring c could also improve student work, providing advice and criteria for self assessment prior to peer or tutor assessment.

1.6 Values

Assessment is a power relationship. We want students to become independent lifelong learners with 'key skills' like evaluation, so they must take responsibility for their own learning. Assessment should be criterion-based, objective, valid, reliable. Therefore we should involve them as partners in assessment, to encourage objectivity, responsibility and self assessment. Peer assessment also provides practice in assessment.

2. How might CAA help? 

2.1. Computer/web based formative quizzes could provide feedback on demand. Should we monitor if they are used, or student performance? This means objective tests. 

Example: Quizzes in MSc IT, Business Information System module
Look at example formative quizzes at bottom of page
http://www.keele.ac.uk/depts/cs/Stephen_Bostock/bisv3/index.html

2.2. Summative objective testing as part of an assessment mix. Easier for low level learning outcomes. But 'when used exclusively, (multiple choice questions) send all the wrong signals' to students (Biggs 1999).

Example: test in 1999/00 subsid. CS, Internet module - Look at tutorial and summative quiz at
http://www.keele.ac.uk/depts/cs/Stephen_Bostock/Internet/wk1.htm

2.3. Mediating anonymous student peer review - assignment submission, delivery to marker, collection of marks/feedback, sending feedback to author, storing data for tutor, analysing data for moderation.

Example: PROMT software being developed (L&T innovation project 1)
http://www.keele.ac.uk/depts/aa/landt/projects/index.htm

2.4. Costs and benefits: most likely worthwhile with large numbers of students (100+?), with introductory material or simple outcomes, with material that is stable, and re-using existing question banks. Payback in staff time for initial development of objective tests: 1 or 2 years?

3. How can it be done?

3.1. Web page quizzes (client-side processing)
web pages with Javascript can mark objective tests and put feedback on the screen - but all answers and feedback are in the web page source code so cannot be secure, cannot send data to tutor of a store, and can be re-used by a 'Refresh' command so number of tries is unlimited.

e.g. CALnet and Hot Potatoes are free (links on Stephen Bostock's web site)

3.2. Server-side processing
programs on the web server can mark the answers, provide  feedback as another web page, and store answers for grading. e.g. 

• CASTLE at Leicester University provides a free service for creating and marking quizzes
• QuestionMark Perception provides facilities for formative quizzes and summative tests.
• FrontPage has a free add-in, but needs a server with FrontPage extensions to work.

4. Some sources and resources

1. The CCA Centre http://caacentre.lboro.ac.uk/
   published Blueprint for computer aided assessment
   and has Frequently Asked Questions (about CAA) on the web site
2. Links at:
   Stephen Bostock, Keele: http://www.keele.ac.uk/depts/cs/Stephen_Bostock/at.html#eval  
    
3. Computer Assisted Assessment: experiments in three courses.
   S. Bostock 2000, staff development workshop. 
   http://www.keele.ac.uk/depts/Stephen_Bostock/docs/caa-ktn.htm
4. Computer Assisted Assessment by Twomey, Nicol & Smart, CTI, 2 page Primer, 1999
   http://www.business.ltsn.ac.uk/cti/primers/assess.pdf
5. Issues and Impacts of using computer-based assessments (CBAs) for formative assessment, 1999 by Dan Charman, 85-93 in Computer-Assisted Assessment in Higher Education eds Sally Brown, Phil Race and Joanna Bull, Kogan Page/SEDA
6. A Review of Online Resources for Computer-Assisted Assessment by Colleen McKenna and Ian Hesketh, Educational Developments 1.2, April 2000 by SEDA at  www.seda.ac.uk
7. Peer assessment: principles, a case, and computer support, Stephen Bostock, presentation at LTSN-ICS workshop on CAA, April 2001
   http://www.keele.ac.uk/depts/Stephen_Bostock/docs/ltsnicsapril2001.pdf
8. Student peer assessment, Stephen Bostock Dec. 2000
   http://www.keele.ac.uk/depts/cs/Stephen_Bostock/docs/bostock_peer_assessment.htm
9. Computerized  peer assessment, Phil Davies, Innovations in Education and Training International 37, 4, 346-355, 2000 
10. CALnet to develop web tutorials and quizzes 
    www.webecon.bris.ac.uk/calnet/manual/manual.htm
11. CASTLE to develop web quizzes
    http://www.le.ac.uk/castle/