Evaluation checklist

based on J.Machell & M.Saunders, 'Evaluating Training Software: MEDA'  1991, CSET, Lancaster University

The INTERNAL STRUCTURE of a training software package is the general structure which enables its various components to be linked together.

QUESTIONS

A Is its division into modules appropriate?

B Are the objectives for each module explained clearly?

C Do various procedures ensure the coherence of the whole (a common theme, a central

Idea)?

D Are there organising principles which enable the content to be structured?

INTERACTIVITY between two partners implies that each of them is able to act and react.

A training software package will be termed interactive if it reacts in a way which is varied and adaptable according to the differing responses of its users and if it enables the latter to affect the way in which the training software proceeds

QUESTIONS

CAN THE USER:

A STOP the program and EXIT at will?

B ask for HELP?

C Obtain basic instructions at all times?

D take time over a difficult section?

E jump a section?

F see the point reached in the program and what is left to do?

G return directly to the place left at the end of a previous session?

H control the speed of presentation?

I control the quantity of information?

J control the number of problems set?

K control the number of examples given?

L choose one specific exercise out of several suggested ones?

M choose one specific experiment out of several suggested ones?

N choose the data on which to work?

O easily correct or delete mistakes if not sure of a response?

P use a menu to select parts of a program
 

WITH REGARD TO THE SOFTWARE, AFTER A CHOICE BY THE USER…

A can it display different messages?

B can it move onto the next sequence?

C can it miss out sequences?

D can it move backwards?

E can it select different routes according to the level of difficulty?

F can it select different routes according to their content?

G can it select different routes according to the learning strategies proposed?

H can it suggest going to the trainer for help?

I can it provide adapted differentiated feedback?

J can it take into account an individual's particular way of working?

K can it help the trainee

• by prompting with external clues
• by a series of approximations (ie by accepting response which are partly correct and progressively nearer to the desired response)

SCREEN MANAGEMENT is concerned with the use of space and the way in which information is displayed on the screen. The aim is to make interaction between user and software easier by presenting information appropriately. Screen layout is extremely important in terms of motivating the user as well as in structuring information. The following aspects are involved: the organisation of TEXT ON THE SCREEN, graphics, use of colour, use of sound and the lay-out of the screen page.

QUESTIONS

Does the presentation of the text on the screen enable the user to read it through in a systematic way?

A Are important words within a paragraph emphasised (eg by the use of CAPITALS, underlining…..)?

B Is appropriate use made of capital?

C Is the importance of certain paragraphs emphasised?

D Is supplementary information correctly laid out (eg at the bottom of the screen so that reading is not interrupted)?

E Does the screen background permit easy reading if the text ?

F Is there a change of page when new information is being presented?

G Is the spacing between lines and letters appropriate?

The following aspects are involved: the organisation of text on the screen, GRAPHICS, use of colour, use of sound and the lay-out of the screen page.

Quality of the graphics:

A Are the graphics appropriately positioned on the screen page?

B Are the images unambiguous?

C Is there access to an illustration when necessary?

D Do the illustrations tie in with the message?

E Do the number of illustrations used match the number implied in the text?

F Do the movements on the screen correspond to the normal movements of the eyes when reading?

G Are the outlines precise enough?

The following aspects are involved: the organisation of text on the screen, graphics, USE OF COLOUR, use of sound and the lay-out of the screen page.

A Is colour used to draw attention to important points?

B Are combinations of colours which clash avoided?

C Is there sufficient contrast between the colours of the text or illustrations and the colour of the background?

D Are the usual colours used to denote certain messages (eg red for stop or danger)?

The following aspects are involved: the organisation of text on the screen, graphics, use of colour, USE OF SOUND and the lay-out of the screen page.

QUESTIONS

Use of sound:

A Does the trainee control the sound?

B Is sound used appropriately to draw attention (eg bleeps to signify the return of control to the user)?

The following aspects are involved: the organisation of text on the screen, graphics, use of colour, use of sound and the lay-out of the screen page.
 

QUESTIONS

Quality of the screen page:

A Is there some variety of screen pages?

B Is there adequate progression from one screen to another?

C Can several screens be superimposed?

D Is use made of a variety of window displays?

E Is sufficient time to read and assimilate allowed between two screen pages?

F Is it possible to control the speed?

G Is wasted time avoided?

H Are signs used to draw attention to important points (eg arrows)?

The LEGIBILITY of the text relates to how easy it is to read or decode.

It enhances the possibility of the training software being pleasurable to read and operate.

QUESTIONS

Text:

A Is the vocabulary appropriate to the anticipated educational standards of the user?
B Is the length of sentences appropriate to the anticipated educational standards of the user?

C Are the sentences structured correctly?

D Are the printing devices (eg height, width etc) and the characters appropriate?

E Do the colours of the characters contrast well with the background?

F Are colour combinations suitable and varied?
 
 

Graphics:

G Do they suit the text (eg do the labels on the graphics use the same words as the text)?

H Is the size appropriate?

I Are they shown for the right length of time?

J Are they pitched at the right degree of complexity for the target users?
 
 

The RESPONSE ANALYSER consists of all the operations which the product uses to deal with responses expressed in ordinary language.

Its quality depends on the extent and variety of responses which it is capable of interpreting.

QUESTIONS

A Is there provision for rounding-up numbers?

B Are several ways of expressing the same numerical value acceptable?

C Is the required unit specified?

D Is it acceptable if the numerical response in units is expressed in a different way?

E Is allowance made for syntactical equivalents (eg spelling mistakes, inversion)?

F Is allowance made for semantic equivalents (eg Synonyms)?

G Is allowance made for the present of one or several words?

H Is allowance made for the absence of one or several words?

I Is allowance made for the absence or present of words in a given order?

J Is allowance made for the absence or presence of words in a given list?

K Is allowance made for the use of inaccurate or slang expressions?
 

The CONTENT is the subject matter covered by the product. It might be designed to train in areas of physical abilities or in the learning of attitudes or concepts.

QUESTIONS

A Is the content precise?
B Is the content valid in relation to the training situation (eg one should check that simulations correspond to the working situation)?

C Is the content up-to-date?

D Is the content exhaustive?

E Is the content pitched at the trainees' level (eg is the degree of abstract thought appropriate)?

F Is the content progressive?

G Is a clear structure discernible in the training software?

H Does it contain an introduction (eg its relation to prior knowledge)?

I Does it provide partial synthesis at various stages?

J Are essential points given prominence?

K Does it contain appropriate examples?

FEEDBACK is the information given to the user about the validity of his response.

Is it correct? Complete? In what respect?

Adaptive feedback is an important element of interactivity.
 
 

QUESTIONS

A Is the feedback appropriate to the age of the target users?

B Is the feedback appropriate to the level of the target users?

C Is the feedback appropriate to the ethos of the organisation?

D Does it follow on directly from correct responses?

E Is it varied in its form?

F Is the evaluation of an incorrect answer neutral (or does it provoke a feeling of anxiety, fear or rejection on account of its aggressive or mocking tone)?

G After an incorrect response is feedback specific (ie does it say in what respect or why the response was incorrect)?

The important TECHNICAL CHARACTERISTICS are both those of THE EQUIPMENT on which the product runs and those of the training software support materials. It is important that the program is simple to operate and for technical assistance to be provided.
 

QUESTIONS

A Is information available on the type and make of computer?

B Is information available on the memory capacity used?

C Is information available on the layout required? (eg on a network? Autonomous? Supplementary card?)

D Is information available on the necessary peripherals (eg printer, light pencil, touch - sensitive screen, audio-visual piloting)?

E Is this data compatible with the lay-outs available at the training site?

The important TECHINCAL CHARACTERISTICS are both those of the equipment on which the product runs and those of the training SOFTWARE SUPPORT materials. It is important that the program is simple to operate and for technical assistance to be provided.

QUESTIONS

Training software

A Is information available on the support material (diskette, tape etc)?

B Is information available on the operating system required?

C Is this data (eg A & B) compatible with the lay-outs available at the training site?
 

The important TECHNICAL CHARACTERISTICS are both those of the equipment on which the product runs and those of the training software support materials. It is important that the program is simple TO OPERATE and for technical assistance to be provided.

QUESTIONS

Operation

A Do the commands have obvious abbreviations which are easy to remember?
B Are the instructions after a display sufficiently clear so that the trainee always knows he is to do (wait, press a certain key etc)?

C Are functions provided (eg help, calculator etc.)?

D Does the structure of the sub-programs appear clearly in the form of a menu for the trainee to use?

E Are all these conventions compatible with the operation of the other programs in use at the training site?

F Is access to the disk fast?

G Is loading time acceptable?

H Are the methods of using the software described?

The important TECHNICAL CHARACTERISTICS are both those of THE EQUIPMENT on which the product runs and those of the training software support materials. It is important that the program is simple to operate and for TECHNICAL ASSISTANCE to be provided.

QUESTIONS

Technical assistance:

A Is there provision for the product to be installed?

B Is there provision for a breakdown and maintenance service?

C Is there provision for putting the product into operation?

COMPUTER PROTECTION is concerned with the measures taken to ensure against improper copying of the training software

QUESTIONS

A Is there provision for protective devices (keys etc) for the diskettes?

B Does the user have a back-up copy available?

C Are damaged diskettes replaced?

D Is it possible to work on the training site with several work diskettes?

E Is information available on limitations on the number of workstations?

F Must a diskette be kept in the disk drive for the software to be used?

The VALIDATION of a product is concerned with the operations carried out in order to demonstrate that it corresponds well to the training objective which led to its development. Particularly important to the study re the following: the extent to which the content is representative of the particular field of study, whether the results obtained tie in with the objectives, whether it fits well into the training scheme etc.,

QUESTIONS

A Has the content been validated by specialists?

B Are the results of trial available which show that the product is capable of meeting its objectives?

C Has the product been subjected to validation procedures by the trainers at the training site?

EVIDENCE OF THE USERS PROGRESS are present if the trainer or the user retains information about the work session.

QUESTIONS

A Does the user or the trainer record the outcomes of a session?

B Does the user or the trainer keep a record of the learning process experienced?

C Does the user or the trainer record the point reached?

D Will the user or the trainer have a diagnosis available?

E Will the user or the trainer have a list of suggested further activities?

ADAPTABILITY is that quality of the product which can be developed by a trainer in order to make it as effective as possible in terms of achieving its objectives.

QUESTIONS

A Can the trainer modify accompanying documentation?

B Can the trainer modify the subject matter and objective?

C Can the trainer modify the given examples?

D Is it possible to stop where one likes in the training program?

E Can one re-start from the same point?

F Is it possible to use the product over different periods of time (eg can it be used either for an hour or half an hour)?

G Can the trainer modify the amount of freedom given to the user?