Meeting of Evaluators of Phase II Projects
Wednesday, 11 June 1997 at 10.45am
The Board Room, Craiglockhart Campus, Napier University
Present
Jean Ritchie (Chairperson)
Child Health Medi-CAL - Neil Hamilton, Aberdeen University and Wilhelmena Hols-Elders, the Open University
Clyde Virtual University - Allison Littlejohn, Strathclyde University
FINESSE - Colin Allison, St Andrews University
MANTCHI - Margaret Brown and Stephen Draper, Glasgow University, Sandra Foubister, Napier University
Clydenet Web Server - Mike Kavanagh, Glasgow University
National Learning Network for Remote Sensing - Iain MacLaren, Paisley University and Gwen Bayne, Dundee University
SUMSMAN - Fiona Campbell, Napier University, Jean Cook, Glasgow Caledonian University, Kirsty Davidson, Judy Goldfinch and Tom Scott, Napier University, and Neil Pitcher, Paisley University
TALiSMAN - Colin Milligan, Heriot-Watt University
In attendance: Anne Spalding
Apologies: Cliff Beevers (SUMSMAN); David Tait and John Nixon (National Learning Network for Remote Sensing)
1. Introduction
J Ritchie welcomed everyone to the meeting and noted that all of the phase 2 projects, along with TALiSMAN, were represented. Each project was invited to indicate briefly their evaluation plans.
1.1. FINESSE
The management group have taken the view that the most useful evaluation and feedback is likely to come from specialists in Business Finance education. Two external specialists have been invited to act as independent monitors with a remit to provide feedback and evaluation. Links with the Scottish Business Education Group (SBEG) will also be exploited to provide a third source of independent feedback and evaluation. However, the management group are also keen to stay aware of other approaches to evaluation.
1.2. Child Health Medi-CAL
Wilhelmina Hols-Elders of the Open University in Edinburgh is the evaluator for this project.
1.3. Clyde Virtual University
A Littlejohn reported that her post is to integrate multimedia teaching throughout Strathclyde University. She is likely to be assisting in evaluating some of the projects within Clyde Virtual University. CVU2 is now developing courseware that will directly replace existing traditional teaching modules in four subject areas. A prototype Product Design Engineering module was fully evaluated by M Sclater of the Glasgow School of Art in June with advice from Margaret Brown of Glasgow University. She used a variety of evaluation techniques such as questionnaires and structured interviews. The evaluation methods developed will be applied to the other subject areas when materials are first tested with students in these subjects.
1.4. The Clydenet Web Service
This project provides a web server for other UMI projects in Glasgow. No evaluation was undertaken on server installation. M Kavanagh would like evaluation of his role in liaison and communication in the Clydenet area. He is involved in developing web-based conferencing software for MANTCHI.
1.5. MANTCHI
M Brown indicated that evaluation had taken place by informal interviews, student evaluation and questions.
S Draper reported that evaluation on MANTCHI involves collaboration with the four universities involved (Glasgow Caledonian, Glasgow, Heriot-Watt and Napier) and the focus is on tutorials. The subject of human computer interaction involves use of the MAN in teaching collaboration. MANTCHI concentrates on tutorials rather than primary exposition of information which would have clearly defined learning outcomes. Questions that might be asked include ‘what are tutorials for?’ and the rationale of tutorial provision: ‘why give a tutorial?’, ‘why give a tutorial in a particular form?’. Use is made of Answergarden software, and there is extensive educational evaluation.
1.6. SUMSMAN
The first SUMSMAN newsletter was handed out. Two types of software are being used - Mathwise, and probability and statistics software. The effectiveness of the materials will be assessed, investigating the student experience at Heriot-Watt, Napier and Edinburgh University by questionnaires, interviews with staff and students, and by observation. Kirsty Davidson, the project leader for the development of the probability and statistics software, indicated that formative evaluation is being used on students (about 500 a year) and staff (through seminars). Different ways of integrating multimedia materials in courses were being investigated.
1.7. TALiSMAN
This project runs some courses once only; they are trying to find out whether running courses for a short period is effective. TALiSMAN is looking for behavioural or attitude changes in staff. Some use is being made of confidence logs.
1.8. National Learning Network for Remote Sensing
The importance of evaluation was stressed. Two courses with discrete, stand-alone multimedia modules are run, which are customisable to each institution’s presentation. Evaluation exists at all stages to ensure the materials are attractive and useful, and to allow for maximum uptake by other institutions. Evaluation enables quick, direct feedback to developers; independent, critical evaluators are used. Field testing in classes is carried out, and some evaluation of communication tools such as email and bulletin boards will need to be done. Components of evaluation include questionnaires, pre and post-testing of students’ knowledge and observation. Project workers are in place, learning objectives have been produced and so a framework exists. Evaluators are John Nixon (Paisley), David Tait (Glasgow), Gwen Bayne (Dundee) and a PhD student (Paisley).
2. Reasons for carrying out an evaluation
J Ritchie distributed a list of some reasons for carrying out an evaluation. Further discussion followed on this subject at a later stage and this is reported in section 6.
3. Presentation on European Metereological Education and Training (EuroMET)
I MacLaren presented a summary of the EuroMET project, a major collaboration between 23 institutions across Europe and Quebec funded under EC Framework IV, to produce web-based training materials for professional meteorologists and students across Europe. The functional aspect of evaluation has already been carried out, and technologies refined. On the educational aspect, the effectiveness of materials as a primary means of learning has to be identified. Educational evaluation has been integrated into the development phase of materials and testing takes place at prescribed dates. Additional funding is available for a one-year demonstration phase to test courses in a variety of institutions. Two thousand students will give feedback. Evaluation strategy is a core aspect of the whole project, evaluation teams having been established independently of developers.
Having gone through one evaluation phase, standard approaches are ongoing, such as questions, pre- and post-testing of knowledge and skills, interviews and video-taped learning sessions. At the courseware development stage, modules were released in April, and will be again in July and October 1997. Each evaluation partner tries a number of approaches and individual evaluators’ reports provided. Excel-based questionnaires are used. This method enables quick evaluation of all modules by all evaluators, more detailed evaluation and a revision of questionnaires after each evaluation event.
The URLs for EUROMET, its evaluation strategy and the Institute of Educational Technology’s site on evaluation methods and procedures for studying learners’ use of media are given below.
http://euromet.paisley.ac.uk/eval.html
http://www-iet.open.ac.uk/iet/PLUM
Most of the materials developed will go into the public domain, though some met agencies abroad are run as semi-businesses. The Java applets have copyright, and I MacLaren will check on whether the style guide is to be made public.
If disagreement occurs, each consortium member has a vote, so there would be a majority decision.
Project leaders have made a secondary application for funding so that the materials, which allow just-in-time training, can be maintained and developed.
The National Learning Network for Remote Sensing project has been guided by EUROMET.
4. Presentation by S Draper
A paper was handed out, giving details of the MANTCHI project and identifying evaluation papers, most of which are available on the web.
Two distinctions can be made between evaluation - that of evaluating teaching and learning, and the actual project. The four types are summative (measuring and comparing effects), formative (finding design bugs), illuminative (discussing context issues) and integrative (formative evaluation of overall delivery to integrate the technology into its context).
On project evaluation, an external assessor forces reflection. Consideration can be taken of anything still used after the project ends such as dissemination, transfer or organisational change. Outputs can be compared to the original promises in the grant application, SHEFC goals in general or new goals resulting from the project, ie unexpected good outcomes.
MANTCHI had an assessor after four months of project set-up.
At this point the meeting broke for lunch, when some useful, informal discussion took place.
5. Presentation by W Hols-Elders
W Hols-Elders is the evaluator of Child Health Medi-CAL. The project’s web site can be seen on www/abdn/medicine/index/htm and its email address is schg@abdn.ac.uk. The evaluation strategy will be similar to the work undertaken for the FaTMAN project on advanced computer support for learning through broadband technology, and two papers on this were distributed. One of the key benefits of evaluation is that student needs will be met. As an evaluator has an influence on development, design problems can be picked up at an early stage.
N Hamilton indicated that functional evaluation is being carried out separately on the Mpeg2 video project across the MANs.
6. Presentation by J Ritchie
J Ritchie had drawn up a list of some of the reasons for carrying out an evaluation, and this is included as Appendix 1. She invited discussion about the list, and suggested that evaluation might be focused on some of the following topics:
a. materials: educational effectiveness; field testing; usability etc; leading on to revision
b. tutorials across the network: process; impact of virtual tutorial sessions
c. integration into courses
d. use of technology: eg feedback speed more important than bandwidth; can lecturer in a videoconference hear audience reactions? does he or she need to?
e. what is left at the end of the project?
f. instant feedback v reflective feedback
g. unexpected successes or failures. The design of VC studios.
She agreed to set up an email discussion group for evaluation issues. J Ritchie would like to make evaluation results known to other institutions and asked projects to think about putting evaluation reports on the UMI web site.
7. Videoconferencing
Details of a TALiSMAN videoconferencing seminar on 3 July, aimed at academic staff, were circulated. J Ritchie undertook to find out what videoconferencing suites were up and running, and to inform the group. Information on this is available at:
http://superbook.ed.ac.uk/userlist.html
and information about how to book studios, contact names etc, is included in the guide available at:
http://superbook.ed.ac.uk/smoverview.html
8. Next meeting
It was hoped that the next meeting could be held by videoconference. Those present commented on how useful they had found the event, and the main meeting closed at 2.45pm. Some discussion in small groups continued.