Bridging the tool development gap

Posted on by

A post this morning on the WebPA discussion list raised an issue that I’ve long felt has a negative impact on the uptake of JISC project outputs: how to support the use of tools produced by JISC projects in an institutional environment that is not interested in supporting them.

WebPA is a great project success story, being adopted by a number of institutions and winning a Bronze Award at the 2008 IMS Learning Impact awards.  It provides an innovative approach to peer assessment evaluation, allowing individual marks for each participant in a piece of group work.  The system generated a lot of interest when they presented at a CETIS event last year.

The poster has identified WebPA as a possible tool to support his teaching, but says:

Unfortunately there is little if any chance of the application ever being hosted on my University servers, I won’t even waste my time trying to get this on their radar […] I should say that while I am not completely IT illiterate I am not going to install the application myself since this is well beyond my personal skill level.

This highlights what I feel is a gap in the project lifecycle: bridging the support gap between the production of useable tools and enabling those outputs to be used in real educational contexts.  Although some lecturers have a high level of technical confidence and competence, this absolutely cannot be expected for the vast majority, and there seems to be a lack of support for those who are keen to use these innovative tools but lack the confidence or expertise to do so.  How do we encourage institutions to be willing to broaden their horizons and support those lecturers who wish to use what they feel are the best tools for their teaching practice?  The poster references commercial companies which host open source systems such as Moodle, but what about newer systems that lack the wide uptake that make providing support and hosting services commercially attractive?

So who should be responsible for supporting projects after the end of their formal funding period, and supporting lecturers and institutions in using these tools?  We’ve addressed this issue before in relation to supporting emerging developer communities in an open source model, but what about tools that are ready for use in actual teaching practice?

Say what you see

Posted on by

poll

Something that’s always puzzled me is how often I hear the acronym CETL pronounced in what seems to me to be the ‘wrong’ way, so I ran a quick Twtpoll to find out what people thought the ‘right’ way should be.

 I was surprised to see so many vote for ‘kettle’, although @Lawrie pointed out that that is how it was pronounced by the minister who launched the scheme which may well explain it.  There was also a vote for ‘rhymes with beetle’ from @dkernohan (there’s always one :p ), although he didn’t specify whether it should be ‘seetle’ or ‘keetle’.

Nicest story of all was also from @dkernohan :)

As a linguist, I’d argue that it should be ‘settle': ce- in English is almost invariably pronounced ‘se’, and the sound in the word it’s short for (‘centre’) is also pronounced with an ‘s’.  It’s very interesting to see how widespread the anomalous pronunciation of CETL as ‘kettle’ actually is.

Many thanks to everyone who voted!

Navigating through the competences maze

Posted on by

Relativity - M C Escher

Around 35 delegates struggled through Wednesday’s sweltering heat and the baffling mysteries of Manchester Metropolitan University Business School’s internal layout to discuss a range of issues around competences for learning, assessment and portfolio.  Delegates represented a wide range of knowledge and expertise, from novices looking to find out ‘what it’s all about’ to experienced practitioners and developers.

It was an impressively international turn out, with delegates from Norway, Greece, Austria, Spain and Belgium joining the UK contingent, mainly representing the iCoper project which is exploring the linking of assessment with competences.  Assessment interests were also represented by the University of Southampton, who are working on the automatic construction of statements of competency from QTI XML, exploring the underlying modelling of competencies for machine processing.  The majority of delegates came from a strong (e)portfolio background, with interests in the movement of information into and out of eportfolios.  JISC and CETIS participants also highlighted the relevance of this work to JISC’s Curriculum Design projects.

The morning session featured a number of short presentations (all presentations from the day can be found here) on competences requirements in the field of medical education, an area which is relatively advanced in the use of competence frameworks.  Claire Hampshire (MMU), Julie Laxton (ALPS CETL), Karen Beggs (NHS Education for Scotland) and Jad Nijjar (iCoper and Synergetics) covered a range of topics, including the desire for non-hierarchic representations, the management of massive amounts of data, and addressing the various points in a student’s career at which  information can move between one system and another.  The ownership of data in portfolios, including competency information, is an ongoing issue that still is not clear, with at least three actors involved: the data subject, data controller and data processor.  Three main points of interoperability were identified: across time (for example, undergraduate to postgraduate), across specialities (for example, from psychiatry to gynecology), and from elearning experiences to portfolios.

After coffee, Paul Horner (Newcastle University), Shane Sutherland (PebblePad), Dave Waller (MyKnowledgeMap) and Tim Brown (NHS Education for Scotland) delivered short presentations on various tools for handling competence information.  One key issue that emerged from this session was the strong need for a specification to enable the sharing of profiles between systems: while evidence can be exported as HTML, entire profiles cannot be moved between systems except in unwieldy formats such as .pdfs.  Interoperability is needed for both import and export.  There is a noticeable move away by customers from monolithic approaches towards using a variety of (Web 2.0) tools, and developers are working on building open APIs to support this. 

What struck me most from both sessions was the way in which developments around eportfolios and competence recording are very firmly rooted in actual teaching and learning practice, with requirements emerging directly from real-world practice and tool developments directly benefiting teachers and learners.

In the afternoon the meeting split into four groups, ostensibly to work on identifying and representing information structures for a purported competences specification.  In practice, my group spent most of our time discussing widely around the whole area of competences, eportfolios and assessment, but as a newbie in this field I found this hugely helpful.  Overall conclusions from the groups identified the following requirements and issues:

  • ability to transfer information between different tools and systems
  • transition
  • curriculum progression pathways
  • relationship between competences and evidence versus qualifications
  • repeatable pattern of description at the core
  • fairly simple structure
  • identifiers for defining authority
  • a definable core structure enables extension for extra semantics
  • able to express the relationship between a learning object and skills, competences and knowledge
  • collection of outcomes
  • architectural issues: data is created and needed in many locations instead of at a central point
  • competences are highly context dependent

The meeting concluded with asking delegates what they want CETIS to focus on in taking forward work on competences.  Suggestions included:

  • development of a data model
  • business case for interoperability
  • look beyond HE/FE to workplace standards, particularly in the HR domain
  • look for connections to the HIRA progress reports due out by November
  • look at what has failed so far in order to learn from past experiences
  • look at defining competences in such a way that a specification can be combined with XCRI
  • have loosely defined competences that can be moved between systems
  • need a high level map of the competency domain in comparison with curriculum description and learning objects.

CETIS will be looking at how best we can take this work forward and, as always, we very much welcome input and suggestions from our community – please feel free to leave comments here, follow up via the wiki or contact Simon or me!

More on MCQs

Posted on by

There’s been an interesting discussion over the last couple of days on the Computer Assisted Assessment JISCMail list around delivery of multiple choice questions. 

The question of how long should be allowed for multiple choice questions produced a consensus of around ‘a minute per question plus a wee bit’ for a ‘typical’ MCQ, but that difficulty level or the use of negative marking or more sophisticated questions would impact on this.  Sandra Gibson cited research by Case and Swanson which suggests that

good students know the answers and … select the right one in very little time (seconds), poor students try and reason out the answers which takes longer. It depends how long you want to give the poorer students to try to work it out, which then impacts on the validity, reliability and differentiation of your assessment.

Discussion broadened to cover the issue of sequential delivery, i.e. when a candidate is unable to return to questions and revise their response once they have moved on to the next question in the test.  There were some compelling educational arguments in favour of this, for example, a series of questions building on or even containing the answers to previous questions; and less satisfactory justifications such as technical limitations on delivery software.    Fascinatingly, a number of posters reported the same (sadly anecdotal) finding that where students revise their response, the likelihood is that they’ve changed a correct answer to a wrong one.   It was also noted that tests which do not permit candidates to revise their responses required less maximum time than those that do.

It’s a good discussion that’s still going on, so well worth following or contributing to!

Assessment when ready failing pupils?

Posted on by

The Guardian is reporting that Single Level Tests, the replacement for the controversial Sats exams which have been piloted over the last eighteen months, are plagued with ‘substantial and fundamental’ problems.  The exams, which allow pupils to take the exams ‘when ready’ at any age between seven and fourteen as part of the larger personalisation agenda, produced what the Guardian calls ‘extraordinary results’, with primary school pupils consistently outperforming those in secondary school in certain areas.

This variation in performance across age groups is explained by the fact that the tests themselves are based on the primary school curriculum, which younger pupils have freshly been taught while older pupils have forgotten much by the time they sit the tests.  This is a fundamental flaw in these tests which raises a number of questions around the area of assessment when ready and assessment on demand; it is ironic that a system intended to recognise individual needs and abilities could actually undermine individual performance.

IMS withdraw QTI v2.1 draft specification

Posted on by

Over the last few days a new notice has appeared on the IMS Question and Test Interoperability webpage in place of the QTI v2.1 draft specification:

The IMS QTIv2.1 draft specification has been removed from the IMS website. Adequate feedback on the specification has not been received, and therefore, the specification has been put back into the IMS project group process for further work.

QTI v2.1 was under public review for more than 2 years and did not achieve sufficient implementation and feedback to warrant being voted on as a final specification. Therefore it has been withdrawn for further work by the IMS membership. IMS cannot continue to publish specifications that have not met the rigors of the IMS process.”

IMS GLC has convened a set of leading organizations to take the lead on this new work – which will be considered to be in the CM/DN draft phase in the IMS process.  Therefore, we are very encouraged and hopeful that a new version will be available in due time, possibly a QTI v2.2, along with the necessary conformance profiles. However, we cannot assume that it will be a linear evolution from QTI v2.1.

Until that time the only version of QTI that is fully endorsed by IMS GLC is v1.2.1, that is supported under the Common Cartridge Alliance: http://www.imsglobal.org/cc/alliance.html . While QTI version 2.0 has been voted on as a final specification by the IMS members, it’s deficiencies are well known and IMS does not recommend implementation of it.

This was clearly completely unexpected, not only for us at CETIS but also amongst a number of commercial and academic developers who have been working with the specification as can be seen by posts to the technical discussion list hosted by UCLES.  In particular, I’d encourage you to read Wilbert’s response on behalf of CETIS.

Concerns from the developer community addressed a number of the issues raised in IMS’s statement.  In response to the claim that ‘adequate feedback on the specification has not been received’, several commentators argued that this is because of the high standard of the specification; while the suggestion that ‘QTI v2.1 … did not achieve sufficient implementation … to warrant being voted on as a final specification’ sparked the addition of a number of implementations to Wikipedia’s QTI page.

There is agreement that work will progress on the basis of the public draft, so it is still perfectly possible that the outcome will be a mildly amended version of the public draft with some small profiles.

CETIS will be following this up, and will of course keep you all informed about progress.  In the meantime, we’d be very keen to hear any thoughts or comments you have, although I would encourage you to sign up for both the UCLES list and the official IMS QTI list to ensure your voice is heard as widely as possible; it would be most beneficial for the wider QTI community I feel for discussion to be focused in one place, i.e. the UCLES list.

VIEWS of the future

Posted on by

Yesterday I attended the first VIrtual Education Worlds Scotland (VIEWS) Forum meeting, hosted by the JISC Regional Support Centre Scotland South and East and facilitated by the irrepressible Kenji Lamb from the JISC RSC Scotland N&E.  There were about fifteen of us in total, from a wide range of institutions and organisations, with developers, educators, support services and the simply curious all represented.

I really enjoyed this event, and particularly welcomed the conscious decision of the organisers to focus on virtual worlds other than just Second Life, with demonstrations of Open Sim and Metaplace featuring on the agenda.  The discussion sessions covered a range of issues facing educators and developers trying to work with virtual worlds, with a few topics in particular seeming to stand out:

  • Lack of support from institutional IT departments for VW-related activities, even to the point of refusing to unblock the ports necessary to actually run them.  Here at Strathclyde, for example, we’re in the fortunate position that Second Life runs fine on the wired network, but (as we found at January’s joint event with Eduserv) the ports necessary for voice to run are still blocked.  Some institutions apparently blame JISC/Janet policies for this, but the inconsistent application of these supposed policies suggests that there might be other reasons for this…
  • Monitoring and evaluation of in-world activities particuarly for the purpose of summative assessment, and tie-in to other systems such as BlackBoard and Moodle to support this and other educational and administrative functions.
  • The relatively steep learning curve of SL in particular, especially when contrasted with the high level of useability of alternatives such as Metaplace and commerical games.
  • Age-related issues.  SL is currently restricted to over-18s only, as is Metaplace, at least during its beta phase.  This is a problem nationally for FE, and a significant issue for Scottish HE given the number of students entering university after their Highers at age 17 rather than at 18 after Sixth Year as is more common elsewhere.  Open Sim is an obvious solution for this, but the relative obscurity of it and other VWs compared with SL mean that it’s often not an obvious answer for people who are just begining their exploration of how VWs can be used in education.  There’s a lot of talk at the moment about Linden Labs merging the adult and teen SL grids which may eventually overcome this, but in the short term it can place apparently insurmountable barriers to adoption.
  • Even the basic processes of registering and selecting an avatar can be problematic.  Registration for multiple SL accounts from a single location requires advance ‘whitelisting’ of the IP range with Linden Labs, updated every six months, to prevent blocking after just a handful of accounts have been created.  Students can be asked to create their account from home in advance of the session, but this has its own problems: students may not have access to a computer capable of running SL, may not get around to doing so, and may require the support of an experienced user that can be provided in a lab session but not at home (no matter how good the documentation prepared for the class may be).  Third-party registration sites can allow mass registrations, but pre-made accounts don’t allow students to select their own avatars which may reduce engagement and identification and therefore the effectiveness of using VWs in the first place.

All present agreed that the RSCs should lead support of the VIEWS Forum, at least in these early stages.  Plans for the future of the Forum include events showcasing other alternatives to SL, a shared space for discussion and knowledge exchange, and the development of a training package for staff and students including items such as a getting started guide, etc.

So what can CETIS do to support this work?  Should it, even?  To me there’s no doubt that this is an area with which we need to engage, but I’m not sure what exactly we can do to meet the needs of our communities while allowing this new and exciting field to develop and mature.  So please, let us know!  Either here, on Twitter, or by getting in touch offline, I’m really keen to know what we can do to help :)

Will tweet for money (revisited)

Posted on by

Ten months ago I mentioned Andrew Baron’s attempt at selling his Twitter account on eBay, and the idea seems to have resurfaced today.  As Baron points out, his last attempt to do so caused a lot of debate and provoked an intriguing mix of hurt feelings and fascination.  I’ll definitely be following progress again this time round, but I still think it’s a publicity stunt :)  With Twitter becoming such a massive pheonomenon and adoption constantly increasing, and with users becoming far more accustomed to it’s use as a publicity tool (although one that is not without its downsides), I’m eager to see how the reaction to Baron’s move plays out this time.

PS Someone did actually try to sell 867 5309 on eBay, but it was pulled :(

A rose by any other name…

Posted on by

New year, same old stories, as Tuesday’s Guardian recycled last year’s claims by Deborah Taylor Tate of the US’s Federal Communications Commission that games like World of Warcraft are so addictive that college students are dropping out of their courses to devote themselves to them.  There’s a rather more balanced response today from Aleks Krotoski pointing to the likelihood that an innate predisposition towards excessive behaviour coupled with poor parenting or poor self-control result in behaviour which appears addictive.

Students have always dropped out, so perhaps we should have learned by now to look for the underlying causes rather than how they manifest.  When I was teaching, the most common causes of students dropping out were financial, such as being obliged to take on increasing amounts of part-time work in order to be able to afford to attend university – a nightmarish, catch-22 situation that government policy actively encourages.  Illness, either their own or a family member’s, or simply being completely unsuited to their course and having very little interest in the subject matter or faith in the mythical graduate employment market were also recurring factors.  Too many students are pushed into higher education straight from school by parents and other social factors, rather than waiting until they as individuals are in the best place to benefit most from higher education, while others are pushed (or push themselves) into academic rather than vocational courses because academic snobbery is allowed to take precedence over common sense.

This same snobbery appears to dictate what forms of excessive behaviour are and are not considered problematic: avid reading and extensive involvement in athletics, for example, are approved and indulged, while avid television viewing and extensive involvement in gaming are frowned on.  Ultimately, however, the book worm and the couch potato are the same creature, displaying the same basic behaviour in different forms.  Surely it’s time to stop blaming the way in which an individual’s unhappiness, discontent or psychological makeup manifest, and to try to address the actual causes of those truly undesirable feelings and problems.