Integrated Assessment – IMS Webinar

Posted on by

On Monday night I attended IMS’s webinar on ‘Integrated assessment products and stategies: gauging student achievement and institutional performance’.  This was the first IMS webinar I’d attended, and I found it a useful session.  Over 80 people participated on Horizon Wimba for the session.

Rob Abel, CEO of IMS, introduced the session by describing integrated assessment as assessment which is designed into and throughout the learning experience.  He discussed the outcomes of a recent survey on satisfaction with elearning tools which showed that tools for quizzing and assessment had the highest satisfaction ratings amongst users; of the 88 products surveyed, Respondus came top in terms of user satisfaction.  This is a consequence both of the usefulness and maturity of this category as well as the availability and quality of tools available.

Rob also suggested that ‘standards are a platform for distributed innovation’, which is a nice phrase, although one of the criticisms often made of QTI is that it isn’t innovative.  It’s hard to see, however, how true innovation could be standardised.

Neil Allison (BlackBoard Director of Product Marketing), Sarah Bradford (eCollege Vice President of Product Management) and Dave Smetters (Respondus President) all spoke briefly about how their tools could be used for integrated assessment. 

Neil illustrated how BlackBoardcan ‘make assessment easier and more systematic’ by integration with other elements of the VLE such as enterprise surveys, portfolios and repositories.  One comment I found particularly interesting was an outcome from a December 2005 Blackboard Survey of Priorities in Higher Education Assessment, which found that portfolios are used in 86% of public institutions but only 43% of private, while interviews and focus groups are used by 78% of private institutions and only 48% of public.  I’ve tried to find this online without success; it’s referenced in slide 20 of the presentation.

Sarah noted that eCollegeusers are using Respondus and Questionmark’s secure browser to assess their learners.  Her talk focused on the eCollege outcome repository or database, which is linked to their content manager, stressing the importance of a good tagging system.  The eCollege Learning Outcome Manager addresses some of the problems for usage data management for quality assurance, an important issue given the current interest in item banking.

Dave’s talk was most wide-ranging, looking not only at the highly popular Respondus assessment authoring and management tool but at some of the wider issues around integrated eassessment.  He referenced research which found that only between 13 – 20% of courses with an online presence have one or more online assessment as part of that course – yet market research consistently shows that online assessment capabilities are one of the most appealing elements in drawing users to esystems.  As he said, once the system is in place, ‘reality kicks in': online assessment takes work, effort and time, raises difficulties in converting or creating content, and raises fears of the potential for cheating.  He argued that only a very small number of students have the desire to cheat, yet the impact can affect an entire class.  Students themselves like a secure assessment environment that minimises the possibilities for cheating.  Locked browsers are a big issue for Respondus at the moment; security of online assessments is also addressed by BS7988 which is currently being adopted by ISO.

Colin Smythe, IMS’s Chief Specification Strategist, provided a brief survey of the standards context for integrated assessment.  He noted that all specifications have some relevance for assessment, citing Tools Interoperability, Common Cartridge, ePortfolio, Content Packaging, LIP, Enterprise and Accessibility.  He also posted a useful timeline (slide 69) which shows that QTI v2.1 is scheduled for final release in the second quarter of 2007, to be synchronised with the latest version of Content Packaging.

He also said that Common Cartridge provides ‘for the first time content integration with assessment'; how much this will be adopted remains to be seen but IMS are marketing it quite forcefully.

There was time for a short question and answer session at the end.  I asked about the commitment of the vendors to QTI 2.1 and the use of QTI 2.1 in Common Cartridge.  The Common Cartridge specification uses an earlier version of QTI partly because there were some migration issues with 2.0 which have been resolved through transforms in 2.1, and also because IMS ‘didn’t want to force the marketplace to adopt a new specification’.  As Rob says, interoperability requires the ‘commitment of the marketplace’, and it would be useful to know what commitment these vendors have to the newer version. 

The session concluded with a reminder about the Learning Impact 2007 conference being held in Vancouver on 16 – 19 April 2007, which should be of interest to many.

Comments – and apologies

Posted on by

Apologies to those who had left comments on my earlier post – I’d set them up to be moderated before posting (to avoid the usual pr0n and poker stuff) and blithely forgot to actually see if there were any.

Will try harder in future.

And thank you for commenting, it’s great to have some discussion on these issues!

Web 2.0 (TM)

Posted on by

A little behind the times as always, but I learned yesterday that CMP and O’Reilly have registered Web 2.0 as a ‘service mark’, effectively attempting to prevent anyone from organising any kind of live event which uses the term ‘Web 2.0′ in the title without their express permission.  They chose to assert their ownership of it by launching a ‘cease and desist’ demand at an Irish Web 2.0 Half Day Conference, the organiser of which had received a ‘best of luck with your event’ message three months earlier from, er… Tim O’Reilly.

Reactions on the company’s own blog were less than positive.

Googling “Web 2.0″ produces 28.7 million hits, yet I’ve never been very convinced by it, tending to side with those that think it’s not much more than a trendy buzzword that doesn’t actually say all that much.  Perhaps this will mean that we have to be a bit more thoughtful about what it is that we’re actually doing and how we should describe it.

Now if only we’d thought to call the new service CETIS 2.0…

Learning in Second Life?

Posted on by

Having wandered around Second Life for the past couple of weeks in search of anything more appealing than casinos, cybersex and endless pretty, deserted islands, I finally got organised enough last night to take part in my first formal educational event in world, a presentation by Dr John Bransford on ‘virtual environments as ways to reorganize thinking about research and education’.

Perhaps the best way of summing up my feelings about the session is ‘interested frustration’.  

My avatar arrived at the venue a few minutes before the event was due to start, and then spent most of those few minutes trying to work out how to fly to a seat and sit down without wildly over shooting the ampitheatre.  Quite a number of other people were there – apparently eight-eight in total – and they were shouting to each other through the chat screen, discussing various research projects and where they were from.  There are no native voice capabilities in SL, and although there are various workarounds using Teamspeak or Skype, the organisers had decided the event should be text only.  That’s fair enough: a system shouldn’t require external add ons to make it useable.

What would have helped, however, was for the presenter to have had some pre-prepared text from which he could copy and paste rather than attempting to type live.  The combination of the length of time it took for each section of text to appear, and the sheer number of typos – an average of more than two per line – made it difficult at times for me even as a native English speaker to follow.  There were also some problems when the speaker went out of the range of where his text could be ‘heard’, and those of us sitting on the far side of the arena missed a little of the presentation.  This was quickly addressed by the organisers, however.

The presentation included a video (requires Quicktime) of a purported educational use of SL, where a group collaborates to solve a basic maze.  Two main points struck me about this: firstly, that the group were using Teamspeak or a similar tool, and that the task would have been far more difficult if they’d had to rely on SL’s text facility only, particularly as speed was an issue.  Secondly, I find it difficult to see how SL could be used for teaching abstract subjects that don’t have the kind of performative element that a science experiment might.  While there are definite advantages and cost savings available from SL in the longer term for many of the same reasons as simulations, I struggle to see how it could be used to transform the way in which subjects like English Literature or Linguistics are taught.

A very useful Q&A session followed the talk.  Moon Eggplant contrasted the potential for collaborative activity within SL with the guild or clan system found within many MMORPGs.  That’s a very good point: games such as World of Warcraft offer players highly structured quests and challenges with clearly defined goals and which often require excellent collaboration amongst up to forty highly-skilled players at once.  WoW, however, has a huge team of full-time professionals working on developing content, not a luxury open to many HEIs and FEIs.  Buddy Sprocket raised the possiblity of ‘massively multi-author collaborative worlds’ and this was very warmly welcomed.

At first, I was disappointed that the event had been very much like a real world lecture, with us all sitting around ‘listening’ to a presenter and watching a brief video.  But then it occurred to me that it was the fact that this event was being held at a set time and place, and the fact that we all had some kind of visible, tangible presence within the event, that made it much more immediate and meaningful than it would have been to listen to a podcast or read a conference paper.  Having been a distance learning tutor, and now being a distance learning student, the opportunity SL gives for genuine engagement with colleagues shouldn’t be underestimated.  I’ve taught using conference calls, and regularly participate in them, and tend to find them tiring and difficult.  The addition of visual components, whether webcam streaming as in Breeze or visual markers like SL avatars, makes a genuine difference.  As was pointed out during the session, SL does offer an excellent method of ‘combining learning by experience and learning by description’ and discussion.

There are some issues with using SL in education.  The system requirements require a fairly high-end system, even though the graphics themselves aren’t all that impressive.  Additionally, in-game currency is purchased using real world money, and the complex economy developing within the game means that pretty much everything worth having comes at a real world price.  I’ve made the decision on various principles not to buy any money for my avatar, which means that she’s drifting rather aimlessly through the world rather than having any real identity or purpose within it.

Fortunately, the organisers had reversed their original decision to charge L$10 (ten Linden, or in world, dollars, equivalent to about 7c) for the event to avoid griefing; I wouldn’t have been able to participate had they charged for it, and in the event, there was no griefing at all of which I was aware, despite the event being reasonably well-publicised in advance.

Towards the end of the session, it was suggested that SL offers ‘one of the most powerful ways we know that might radically change the nature of education': so far, it remains to be shown how.