Assessment in 2008: looking forward

Gales are howling, trains in chaos, so it must be January and time to look ahead to what 2008 has in store…

The final release of QTI v2.1 should be out this spring, and it’ll be interesting to see what uptake is like.  This will be the most stable and mature version of the specification to date, supported by a long public draft stage and a number of implementations.  Angel Learning are a significant commercial early adopter, and other vendors are bound to be looking at their experiences and whether Angel’s embracing of the specification has an impact on their own customer demand for QTI 2.1. 

Other significant implementors of 2.1 are the JISC Capital Programme projects which will be concluding around March.  AQuRate offers an item authoring tool, Minibix provides support for a range of item banking functions while ASDEL is an assessment delivery engine which supports both standalone use and integration with a VLE.    These projects should deliver quality resources to the community which will provide a firm foundation for use of the specification.  There was a sneak preview of these projects at our last SIG meeting.

Talking of SIG meetings, dates for the next two meetings can now be confirmed. 

On 19 February there will be a joint meeting with the CETIS Educational Content SIG in Cambridge.  This meeting will cover a range of shared concerns such as new content related specifications such as Common Cartridge and Tools Interoperability, and innovative approaches to educational material and assessment.  Information about this meeting and online registration will be available very soon.  This will be preceded by a workshop hosted by the Capital Programme projects discussed above.

The focus shifts from assessment as content to assessment as process with another joint meeting on 1 May in Glasgow.  This meeting will be a joint meeting with the CETIS Portfolio and Enterprise SIGs and will offer an opportunity to explore some of the shared issues in these domains.  Again, information on the event will be available on the mailing lists, on this blog and on the website in due course.

Another event of note is the annual International Computer Assisted Assessment Conference on 8 and 9 July at Loughborough.  The call for papers is already out, with submissions due by 29 February.  As always, this should be a lively and important event in the CAA calendar.  Alt-C 2008, Rethinking the Digital Divide, will be held in Leeds on 9 – 11 September; again, the closing date for submissions is 29 February.  There’s also a regularly updated list of non-CETIS assessment related events on the wiki.

And what about the trends for eassessment in 2008?  The results of Sheila’s poll, with a strong emphasis on Web 2.0 technologies and possibilities, do seem to reflect to some extent the comments on the last meeting’s evaluation forms which suggested increasing interest in innovative technologies, signficant concern with transforming and enhancing the assessment experience and direct engagement with teaching and learning rather than the more abstract issues of standards and specifications for their own sake.  It will be interesting to see how the more ‘traditional’ XML-based QTI v2.1 fares in the light of the increasing popularity of mashups and web services in 2008.

Assessment SIG meeting, 26 September 2007

Academics and developers met in Glasgow recently to participate in the most recent Assessment SIG meeting. The very full agenda covered a range of topics, both technical and pedagogic, and presentations led to some lively discussions.

Myles Danson of JISC opened the day by presenting JISC’s views and priorities for eassessment, as well as pointing to some future work they will be undertaking in the domain.

Yongwu Miao of the Open University of the Netherlands discussed work undertaken by the TENCompetence Project, with a particular focus on the relationship between IMS QTI and IMS Learning Design and the work they have done in this area. Dick Bacon of the University of Surrey and the HEA discussed the relationship between different varieties or ‘dialects’ of QTI, exploring some of the implementation and interpretation issues that hinder or break interoperability between systems nominally implementing the same version of the specification. CAL Consultant Graham Smith pleased the audience with news that a new Java version of his QTI demonstrator will be available shortly with updated support for QTI 2.0 items, which should help in the identification and resolution of implementation problems.

Martin Hawksey of the University of Strathclyde presented the work of the Re-Engineering Assessment Practices project. With a focus on real world assessment experiences, including an impressive collection of case studies exploring the impact of transformation within assessment practices, the REAP project was of particular interest to participants. Also of great interest, and perhaps unsuprisingly sparking the greatest amount of debate, was the exploration of ‘Assessment 2.0′ presented by Bobby Elliott of the Scottish Qualifications Authority. Bobby looked at ways in which Web 2.0 technologies can be used to enhance and modernise assessment in ways which can engage and appeal to increasingly digitally literate learners.

The day also featured several demonstrations of tools under development. Niall Barr of NB Software demonstrated his current work, an assessment tool which utilises the IMS QTI, Content Packaging and Common Cartridge specifications, while Steve Bennett of the University of Hertfordshire demonstrated MCQFM, a JISC-funded tool which provides a simple text-based format for converting and editing items between formats. Two more JISC projects closed the day. AQuRate, presented by Alicia Campos and David Livingstone of Kingstone University, is an elegant item authoring tool while ASDEL, presented by Jon Hare of the University of Southampton, is an assessment delivery tool which builds on the R2Q2 project to provide a fuller test tool. A third project, Minibix (University of Cambridge) on item banking, is working closely with AQuRate and ASDEL.

Links to presentations (via slideshare), project websites and other information can all be found on our wiki: http://wiki.cetis.org.uk/JISC_CETIS_Assessment_SIG_meeting%2C_26_September_2007.

Massively Multi Learner

The HEA Information and Computer Science subject centre recently ran a workshop, ‘Massively Multi Learner’, on learning in multi user virtual environments which I was fortunate enough to be able to attend.

Perhaps inevitably, the presentations on the day were heavily skewed towards Second Life, a fact that I was glad to see the organisers themselves acknowledged as not necessarily ideal.  Unfortunately, Carl Potts, who had been scheduled to speak on learning within guilds in World of Warcraft, was unable to attend, but Laz Allen of TPLD (standing in for Helen Routledge) provided a non-SL and more game-orientated perspective on emerging technologies.  Of particular interest was the emphasis in this presentation on the assessment of game-based learning and of gaming activities, through reflection and debriefing, and through the logging and interpretation of ingame activities with reference to an identified set of skills.  Unlike commercial off-the-shelf games (COTS) and other resources such as SL, games specifically designed for learning can offer a more effective balance of learning objectives, subject matter content and gameplay, with assessment – often itself highly innovative – integrated from the outset.

The rest of the presentations all referenced SL to a greater or lesser extent.  I hugely enjoyed Aleks Krotoski‘s work on social networking in virtual worlds, in particular her identification of 75 avatars (“they know who they are”) who form “the feted [fetid?] inner core of Second Life”.  Unlike either single-player or MMO games, MUVEs such as SL are inherently socially orientated rather than goal-orientated; ‘success’ doesn’t come necessarily from accumulation of in-game objects or from PvP or PvE pwnage but from occupying key, extremely powerful positions within social networks.  As an infrequent and rather ‘resistive’ SLer, I feel strongly that the lack of scaffolding within SL, in contrast to the carefully balanced quest structure in games such as WoW which directs players through the game world and encourages casual grouping, makes social relationships within SL disproportionately important.

Other presentations explored some of the many purposes to which SL is being put.  Dave Taylor of the National Physical Laboratory discussed some very exciting international collaboration which has been taking place in the Space Island cluster, while Peter Twining demonstrated the Schome island pilot on the teen grid which is trialing SL as a learning space for a group of ‘gifted and talented’ learners.  Jeremy Kemp discussed Sloodle, an integration of SL and Moodle which uses mashups to connect the two systems.  The integration of SL and Moodle also offers the potential for resolving accessibility issues around SL by offering meaningful real time alternatives to inworld communications.

The final three speakers had all integrated SL closely into their teaching practice.  Mike Hobbs of Anglia Rushkin University described scripting tasks undertaken by second year Computing Science students to create learning resources used to explain computing concepts to first year students, while Annabeth Robinson (well known in SL as AngryBeth for her creative and practical objects) described the options her Design for Digital Media students had for woriking in SL and particularly for using it as a tool for machinima.  Mike Reddy provided an entertaining end to the day, looking at various ways in which Second Life can be integrated into a range of courses.