On Monday night I attended IMS’s webinar on ‘Integrated assessment products and stategies: gauging student achievement and institutional performance’. This was the first IMS webinar I’d attended, and I found it a useful session. Over 80 people participated on Horizon Wimba for the session.
Rob Abel, CEO of IMS, introduced the session by describing integrated assessment as assessment which is designed into and throughout the learning experience. He discussed the outcomes of a recent survey on satisfaction with elearning tools which showed that tools for quizzing and assessment had the highest satisfaction ratings amongst users; of the 88 products surveyed, Respondus came top in terms of user satisfaction. This is a consequence both of the usefulness and maturity of this category as well as the availability and quality of tools available.
Rob also suggested that ‘standards are a platform for distributed innovation’, which is a nice phrase, although one of the criticisms often made of QTI is that it isn’t innovative. It’s hard to see, however, how true innovation could be standardised.
Neil Allison (BlackBoard Director of Product Marketing), Sarah Bradford (eCollege Vice President of Product Management) and Dave Smetters (Respondus President) all spoke briefly about how their tools could be used for integrated assessment.
Neil illustrated how BlackBoardcan ‘make assessment easier and more systematic’ by integration with other elements of the VLE such as enterprise surveys, portfolios and repositories. One comment I found particularly interesting was an outcome from a December 2005 Blackboard Survey of Priorities in Higher Education Assessment, which found that portfolios are used in 86% of public institutions but only 43% of private, while interviews and focus groups are used by 78% of private institutions and only 48% of public. I’ve tried to find this online without success; it’s referenced in slide 20 of the presentation.
Sarah noted that eCollegeusers are using Respondus and Questionmark’s secure browser to assess their learners. Her talk focused on the eCollege outcome repository or database, which is linked to their content manager, stressing the importance of a good tagging system. The eCollege Learning Outcome Manager addresses some of the problems for usage data management for quality assurance, an important issue given the current interest in item banking.
Dave’s talk was most wide-ranging, looking not only at the highly popular Respondus assessment authoring and management tool but at some of the wider issues around integrated eassessment. He referenced research which found that only between 13 – 20% of courses with an online presence have one or more online assessment as part of that course – yet market research consistently shows that online assessment capabilities are one of the most appealing elements in drawing users to esystems. As he said, once the system is in place, ‘reality kicks in': online assessment takes work, effort and time, raises difficulties in converting or creating content, and raises fears of the potential for cheating. He argued that only a very small number of students have the desire to cheat, yet the impact can affect an entire class. Students themselves like a secure assessment environment that minimises the possibilities for cheating. Locked browsers are a big issue for Respondus at the moment; security of online assessments is also addressed by BS7988 which is currently being adopted by ISO.
Colin Smythe, IMS’s Chief Specification Strategist, provided a brief survey of the standards context for integrated assessment. He noted that all specifications have some relevance for assessment, citing Tools Interoperability, Common Cartridge, ePortfolio, Content Packaging, LIP, Enterprise and Accessibility. He also posted a useful timeline (slide 69) which shows that QTI v2.1 is scheduled for final release in the second quarter of 2007, to be synchronised with the latest version of Content Packaging.
He also said that Common Cartridge provides ‘for the first time content integration with assessment'; how much this will be adopted remains to be seen but IMS are marketing it quite forcefully.
There was time for a short question and answer session at the end. I asked about the commitment of the vendors to QTI 2.1 and the use of QTI 2.1 in Common Cartridge. The Common Cartridge specification uses an earlier version of QTI partly because there were some migration issues with 2.0 which have been resolved through transforms in 2.1, and also because IMS ‘didn’t want to force the marketplace to adopt a new specification’. As Rob says, interoperability requires the ‘commitment of the marketplace’, and it would be useful to know what commitment these vendors have to the newer version.
The session concluded with a reminder about the Learning Impact 2007 conference being held in Vancouver on 16 – 19 April 2007, which should be of interest to many.