Clyde Virtual University -
Niall Sclater, University of Strathclyde (NS)

FINESSE -
Rosa Michaelson, University of Dundee (RM)

MANTCHI -
Sandra Foubister, Napier University (SF)
Julian Newman, Glasgow Caledonian University (JN)

National Learning Network for Remote Sensing -
Iain MacLaren, University of Paisley (IMacL)
Gwen Bayne, University of Dundee (GB)

SUMSMAN -
Judy Goldfinch, Napier University (JG)
Tom Scott, Napier University (TS)

Jean Ritchie (Chairperson) (JR)
Kim Underwood (UMI Administrator)

Apologies: Margaret Brown (MANTCHI)

1. Introduction
The meeting was a follow on from a similar meeting of evaluators a year ago.

 2. Evaluation of UMI
JR described the evaluations that are being planned for all parts of UMI.

2.1 UMI phase 1

• JR plans to write a report presenting lessons to be learned from UMI 1, in particular the strand 2 projects.

• JR is gathering information about use of Strand 2 projects in teaching

 
• Dr Ian Smith (UKERNA) will look at the larger Strand 1 (hardware) projects and write a report

 
• TaLiSMAN will be evaluated by a team appointed by SHEFC

 
• At some point SHEFC will undertake an evaluation of the whole UMI

2.3 UMI 2
Each UMI phase 2 project has its own programme of evaluation. It is likely that SHEFC will conduct a separate evaluation. Questions likely to be asked will focus on use of the network, success in teaching, and whether project materials or methods have been, or can be, taken up outside the project consortia.

2.3.1 Questions about network use might ask what tools were used (eg. ftp, www, telnet, usenet, videoconferencing); how much bandwidth was needed; and measures of network use.

2.3.2 Users of ftp might record the size of files transferred and the frequency of transfer.

2.3.3 Projects using the web would give the URL of the project's home page, whether resources were made available, the number of visitors to the site, and use of Java. Web sites should have 'counters'. JR will put one on the UMI web site.
                                                                                                                        Action JR

2.3.4 Videoconferencing evaluation should include information on participants, sites and purpose of meetings (eg. administrative meetings or teaching sessions).

2.3.5 If other communication tools such as Usenet and IRC are used, projects might record how many participants, how many discussion topics, success of Chat sessions.

2.3.6 Teaching success is the subject of projects' own evaluation. The UMI 2 bid process specified that projects would use materials as part of credit-bearing courses.

2.3.7 The take up of UMI 2 materials outside the consortia, within the same subject area but at different HEIs, or in different subject areas probably at the same HEI, would be seen as a sign of success.

2.3.8 SHEFC will also ask about dissemination activities, including publications, conference presentations, and success in reaching audiences other than teaching and learning specialists.

2.4 TS asked what the predicted time scale is for these aspects of evaluation. JR replied that UKERNA should finish their hardware evaluation within the next couple of months; TaLiSMAN evaluation will take place within the next 3 - 4 months. Evaluation of the UMI as a whole will be conducted at some stage in the future.

3. Evaluation activities of UMI 2 projects
Each project presented a summary of their evaluation plans.

3.1 SUMSMAN
SUMSMAN’s evaluation aims to:

• evaluate individual experiences of staff and students
• evaluate sharing - benefits and problems
• evaluate how the MANs are being used in this project

• student questionnaires
• student focus groups
• student confidence logs
• student assessment results
• staff questionnaires - people who did use a shared resource
• staff questionnaires - people who did not use a shared resource
• staff focus groups
• matching of syllabi to CBL content and investigation of how this affects take up of the CBL
• interviews with key staff
• TS visiting people involved, talking to them etc.

There has so far been a low response from staff surveys and more staff surveys and interviews are to be conducted. Longer term surveys of use of materials and the monitoring of database use will also be included in the evaluation of the SUMSMAN project. SUMSMAN will be presenting a paper at LTDI’s VALUE conference, to be held on 15th May. Conference proceedings will be available on the SUMSMAN web page.
                                                                                                                         Action JG

3.2 FINESSE
RM reported that FINESSE evaluation will cover three dimensions:

• finance education content
• open/ distance learning
• distributed CBL model

• student questionnaires
• feedback from the Scottish Business Education Group
• students comments to lecturers
• viewing the 'notebook' set up for student communication

Dissemination activities had included:

• a paper to the Scottish Business Education Group, CTI Accountancy
• FINESSE has been described in various talks, in the context of CBL and also at accountancy conferences
• more papers and talks are planned
• FINESSE’s main WWW site may be opened up and possibly provide a limited functionality guest portfolio facility (FINESSE has to pay a large fee for up-to-the-minute online financial data for use in their portfolio management game)
• brochure and improved presentation of WWW site
• presentation by RM on 5th June at Scottish Business Education Group conference

• shared applications in support of a core curriculum
• MPEG1 digital video
• Java, HTML etc
• videoconferencing

• two trials have been undertaken with the new materials, one at University of Aberdeen and one at University of Dundee
• students were asked 'is it easy to navigate?' As a result of evaluation, the user interface has been improved
• students were asked if they liked CBL; most said ‘yes’, but the students wanted access to references as they worked through the exercises so they could check their answers as they went. Students preferred CBL on their own during university hours. Students commented that speed of delivery is very important and were satisfied with the speed of delivery over the MANs between Aberdeen and Dundee
• One problem encountered was the need for upgrading of the University of Dundee computer labs.

3.4 National Learning network for Remote Sensing
IMacL reported three aspects to evaluation:

• Technical (basic functionality, network configuration, student access/ availability)
• Educational (comparison with other methods, mode of implementation)
• Overall Project Success (uptake of materials)

The Remote Sensing project will also carry out evaluation of the following related issues:

• videoconferencing evaluation (questionnaires after each session, interviews, consideration of costs/ resource implications/ booking and problems with organising)
• comparison with EuroMET materials (some are being used in the same courses)
• use of bulletin board/ email to contact subject specialists

Further evaluation work will include:

• evaluation in a wider range of courses
• detailed monitoring beyond the end of the project (PhD studentship/ ICTE group) to examine continued usage and wider dissemination
• more extensive videoconferenced lectures scheduled for the coming academic year and research seminars for summer
• testing and evaluation of virtual labs
• publication of evaluation results planned and conference paper submitted

3.5 Clyde Virtual University
CVU is developing courses and course materials in

• social work
• chemistry
• business
• design engineering

• look at technology
• understand communication protocols that needed to be adopted
• see if the design process itself was hindered or enhanced
• find out whether there are any added benefits

• pre and post case study questionnaires
• pre and post case study interviews
• confidence logs
• qualitative assessment of shared workspace

Responses from students included:

• gained a lot, a valuable learning experience
• learnt other skills such as team work
• problems included technology collapsing and incompatibility between platforms
• felt computer experience and confidence had increased
• collaboration helped the learning process
• improved interpersonal skills
• improved debating skills

• data will be gathered on the relative usefulness of the different learning sources available
• how the technology fits in to the total learning experience
• evaluation will look at getting a better understanding of the learning process using this technology

4. Discussion
NS suggested that UMI 2 projects put together a joint publication on how to evaluate MAN based learning, including what each project has done with regard to evaluation of their own project. The paper could include chapters on individual projects own experiences, recommendations and conclusions. There could be two separate papers, one dealing with videoconferencing and one with teaching. JR said she would coordinate this/these communal paper(s), starting with an email discussion.
                                                                                                                       Action JR

It was suggested that another meeting be held about mid June to hear more about what projects are doing with regard to evaluation and to further discuss the idea of a joint paper. JR will organise this meeting.
                                                                                                                             Action JR