Learning Technology by Stephen Bostock
 You are here: Keele University > Learning Technology home > documents

UMIST EFFECTS Portfolio: Stephen Bostock

1.3 A review of software tools for generating Web assisted assessment

(Note - some figures are files at their original web sites and could therefore disappear if these sites are not available.)

 

 

As a member of the editorial board of Educational Developments I commissioned 'A Review of Online Resources for Computer-Assisted Assessment'  by Colleen McKenna and Ian Hesketh (2000) of the CAA Centre, University of Luton.  This was restricted in length and confined itself to online materials but nonetheless includes links to, and comments on, the commonest tools. Unless another source is given, all the tools reviewed in 3. below are in their review.

The need

Before deciding on criteria, the task for the tool must be clear. There are three possible tasks: selecting tool(s) for the experiments in my own courses in 1999/00,  selecting tool(s) for wider use within the institution, and a broader review of tools useful for HE. The immediate task assumed here is the first option, but some comments will be made on the second. The courses and CAA involved were
  1. Business Information Systems module in the MSc in IT.

    2. Self-assessment quizzes were wanted to provide student feedback on learning basic concepts.
  2. The Internet module on the Subsidiary Computer Science.

    3. A self assessment quiz which would give feedback as hints on where to find the answers in an online tutorial and a summative test that would count as an assessed task.
  3. Multimedia and the Internet module on the MSc in IT.

    4. Peer reviews of prototype web applications were planned, for formative and summative purposes.

Evaluation Criteria

Evaluation must take place against criteria. As I discussed in An evaluation of three authoring tools (1994) there are three broad categories of criteria for authoring tools in general:
    1. Functionality asks 'What can it do?' This aspect is emphasised in many published evaluations.

      2. In this case, multiple choice question structure was appropriate or adequate, except that peer review (a) must accommodate free text answers. The tools needed, therefore, should have the functionality to generate MCQs with or without feedback to the student, and with or without storing answers centrally for the tutor.
    2. The author interface, in a broad sense, asks 'How does it allow the author to do it?' What sort of experience is expected of the author? Does it imply certain instructional designs?

      3. In this case the author is quite flaxible about interface, although a familiar one would save time to learn.
    3. Wider questions concern accessibility, support, cost, documentation, supported hardware platforms, run-time licenses, institutional  and industry standards.

      4. In this case, the cost must be nothing or negligible. The software must be presented through the web, as all three courses provide other web materials, but the software must not require installing on central web servers, which is not possible for teaching staff at Keele, without discussion with Information Services and in the time available. This will be a particular problem for assessments saving data for the tutor or peers.

Some CAA tools

1. Virtual Learning Environments

VLEs (or Managed Learning Environments) were reviewed by myself in January 2000 (Bostock 2000). Other reviews can be found off  my Web Based  Learning page. One feature that the traditional VLEs (e.g. WebCT, LearningSpace) have is objective testing. If testing was all that was required, then a VLE would be more costly and/or complex than necessary. However, for an institution it may be cost effective to adopt a VLE and thus gain many online facilities at once, including testing. This is unlikely at Keele; distance learning is not an important part of the strategy, and the number of people prepared to try a VLE is small. In this case, stitching together separate online facilities through the web is a safer option and in line with our gradual appoach to implementing IT in teaching.

2. Authoring tools for learning applications

Another type of  tool is a generic authoring tool capable of generating a quiz or test for the web. For example, in commercial training Macromedia's Authorware 'is the leading visual rich-media authoring tool for Web and online learning'. It allows easy generation of quizzes. No doubt very capable,  it's expensive at $2,700. Also, the authoring interface based on flowlines, which may be intuitive for some but I find cumbersome, personally.

example Authorware interface for authors

In contrast, CALnet is free within HE. It was developed by the Institute for Learning and Research Technology (ILRT) at the University of Bristol. CALnet generates tutorials but it can also generate multiple choice quizzes as HTML pages with Javascript. They do not require server programs, can give feedback to the user, but cannot store answers for marking or analysis.

3. Tools specifically for generating assessments

CASTLE

McKenna and Hesketh list seven tools within HE and four commercial tools for generating. CASTLE at Leicester is straightforward to use but cannot store answers for analysis.  The output is attractive, creating a quiz and processing student answers are both done at the Leicester site (and so an Internet connection is needed), and it is free. Overall, it is an attractive option as long as no records need to be kept.

Hot Potatoes

from the University of Victoria, Australia, is a suit of programs that generate tests of various types, but again cannot store answers. The software can be downloaded for free and is run on a PC. The web pages generated use Javascript for interactivity but cannot store answers.

TACO

at UCL is an institution-wide assessment system that gets its functionality from Java programs on the web server. This means that it can store all student answers for later analysis. The browsers for students and authors need not be capable of running Java, but it also means that it needs central installation and support. So it is not suitable for this project. Webtest at Herriot-Watt University seems similarly capable but also requires programs on the server.

Question Mark

is the leading commercial supplier of assessment software in the UK. It supports most assessment and management facilities that can be imagined but requires central installation for the web version (Perception), needs an NT server, and costs serious money. On all three grounds it was not suitable.

O'Brien and Exam 97

All the systems reviewed so far are content-free; the author must write his/her own questions, or possibly suscribe to a question bank with an institutional system. For the BIS module, one of the text books recommended for purchase, O’Brien (1999), has a password-protected web support site for tutors which includes a MCQ generating program which can be freely downloaded: 'Exam 97'.  While this can only generate several types of tests, it does come with a question bank for every chapter in the book. The program allows the author to select questions personnally or at random on a topic, and then generates a web page with Javascript providing the interactivity. It cannot store answers, but requires no central installation, just a Javascript enabled browser. Furthermore, the program allows the author to put their own questions into the bank, and add instructions and additional reference materials to the test pages. All-in-all, very useful for free!
 

HTML and CGI

The tools described avoid the need for author/tutors to use low level coding such as HTML or cgie scripts. I already have some knowledge of these, so they presented a feasible option. Although web authors at Keele cannot install their own CGI (or other) programs on the web server, two programs are provided to deal with forms: cgiemail and cgifile. The former will email data entered into a form to a list of addresses. The latter will send the data to a text file that must be called 'incoming.txt', so that each form needs to be in a folder with its own incoming.txt file. Such a file can be imported into Excel, and this can be made easy and reliable by formatting the data sensibly. The formatting is done by a slamm control text file written by the web author to accompany a particular form. (See Practical Exercises  for a course on Web Page Design and Development with HTML or the CWIS authors guide for more details.) An assessment web page written as a form can use these scripts to send student answers either to the tutor and others, or to a data file for later marking (but not both).

Some other systems

Some other CAA systems were not suitable due to cost or the need for institutional support. The TRIADS project of three Universities can generate impressive graphical questions but requires Authorware for authoring. The Clyde Virtual University Assessment engine requires institutional support, as does Test and Learn from Bristol.

Conclusion

The generic authoring systems or tailored versions of them like TRIADS could be used where the tool is already in place, otherwise the cost of purchase and of staff training time is prohibitive. The multi-function VLEs could provide CAA if one were already in place, but CAA alone could not justify the cost in support that they all require, even where purchase is free to HE.

The specific CAA systems described, and others, fall into two types:

  • Systems which generate stand-alone web pages using Javascript (or less commonly, Java) within a web browser to prodice interactivity to give immediate feedback as marks and comments to students. The authoring tool could be a PC application or a web application on a web server accessed by the author through a browser.
  • Systems which are installed on a web server and which therefore can be used through older browsers or browsers where Java and Javascript are switched off.
The former type is convenient for generating formative quizzes. The test answers are embedded in the Javascript within the page HTML source so they are not secure for summative testing, neither do they store data centrally as part of a student record. Of those described, CALnet and Hot Potatoes were free and were preferred on the quality of their interfaces and reliability of source. But several others would produce comparable material. Exam97 is unique in providing a data bank suited to one of the courses bing used in this project.

The latter type is needed to produce summative tests on a large scale, but they require installation and support on a web server and so are suitable for an institution-wide initiative, not a small experiement such as this project. However, small scale summative tests can be prduced with 'vanilla' HTML forms that use standard cgi scripts to send answers to markers by email, or store them in a text file for recovery by the tutor.

References

J A O’Brien, Management Information Systems: Managing Information Technology in the Networked Enterprise, 4th
edition, Irwin, Chicago, 1999.

Colleen McKenna and Ian Hesketh, 2000, A Review of Online Resources for Computer-Assisted Assessment, Educational Developments 1.2 (April 2000)

S Bostock 2000

Keele University Home | Learning Technology Home | email Stephen Bostock

Stephen Bostock asserts his moral right to be acknowledged as the author of documents on this site, unless another author is identified.  Copyright remains with Keele University, or the author.  The views expressed in this site are those of the author and do not necessarily represent those of Keele University.
 Last edited: November 22, 2006