The Problem with “Evaluating Standards”

Posted on by

I’ve just uploaded an attempt, “Evaluating Standards – A Discussion of Perspectives, Issues and Evaluation Dimensions” (MS Word), to say in more than a few words why “Evaluating Standards” is easier to say than to do. For most of the issues there are no easy answers but I have tried to make some suggestions for a heuristic approach inspired by the Neilsen and Molich approach to usability. I’d like to acknowledge Scott Wilson for contributing his insight into what makes a good standard.

A Good Showing for the UK as IMS Learning Impact Finalists

Posted on by

IMS recently posted the list of finalists for the 2009 “Learning Impact Awards”.

Well done to all and in particular, to be a little partisan:

  • Glow - Scotland’s National Intranet – RM and Learning Teaching Scotland
  • Racing Academy – University of Bath and Lateral Visions
  • CURVE - Equella, The Learning Edge Europe and Coventry University
  • LEXDIS - University of Southampton
  • MrCute Moodle Repository System – Worcester College of Technology
  • SIMulated Professional Learning Environment (SIMPLE) – University of Strathclyde and University of Strathclyde Law School

All but the first have been supported by JISC. Hurrah for the finalists and for JISC!

I’m sure they will have an excellent time in Barcelona at the conference, where the finalists will be judged. Best wishes to them and watch out for the announcement of next year’s competition.

Open Educational Resources and the Zachman Framework

Posted on by

The theme of Open Educational Resources (OER aka Open Educational Content, OEC) is clearly a topic of current interest but, I believe, often discussed from different perspectives. If we are to discuss OER, and specifically if are to discuss projects, programmes or initiatives around OER, do we need a structure to make more clear the perspectives? Does the Zachman Framework help?

The Zachman Framework, devised by John Zachman about 20 years ago, is most commonly associated with the field of Enterprise Architecture and generally viewed as a 2-dimensional grid. Do not be put off by the “Enterprise Architecture” label; the Zachman Framework is really a way of understanding and structuring the components of a description of a system (in the general sense of the word). A quote from Zachman:

“The Zachman Framework is not a methodology for creating the implementation (an instantiation) of the object. The Zachman Framework is the ontology for describing the Enterprise. The Framework (ontology) is a STRUCTURE whereas a methodology is a PROCESS. A Structure is NOT a Process. A Structure establishes definition whereas a Process provides Transformation.”

The Zachman Framework, as it is described, is rather industrial in its terminology so some of the descriptions of the meanings associated with the cells in the grid need a bit of re-phrasing to be applicable to OER as do row headings like “technician” (would “custodian” or “support” be better?). For each cell, though, I can imagine a re-description consistent with the concepts of the Zachman Framework.

My proposition is that the Zachman Framework is useful if, for each row/column intersection in the grid, we ask questions like: “do we have anything to say about this aspect” or “is this an aspect that I should be considering deeply or is it someone else’s concern”. Similarly, if discussing doing something under the category of OER: “which cells does our discussion cover” or, if developing a coordinated programme, “did we miss anything”.

As the whole point of the Framework is about thinking at an “Enterprise” level, although that term is somewhat inappropriate to OER, it seems like a resonable approach to asking whether a set of planned interventions are consistent with achieving the changes we seek across the “Communication Interrogatives” (columns) and the “Reification Transformations” (rows). To diverge again from the industrial/manufacturing heritage of the Framework, maybe we should think about the “Enterprise” from more of a social viewpoint and ask, when considering the Operations row, “what is is about an OER-related intervention that seeks to change the capabilities and intentions of the various stakeholders?” I think sociologists would use the terms “structure and agency” here.

What do you think?

Grand Challenges at the CETIS 2008 Conference

Posted on by

On the 25th of November I facilitated a session entitled “Grand Challenges in HE and FE”. The first half of the session was given over to a structured activity, “Future Backwards”. This activity, which worked well except that the groups should have been a bit smaller than 6, and the products created by the groups are available on the CETIS wiki. The second half of the session was devoted to free discussion. What follows is my account of the broad flow of the discussion, bringing out stated or implied challenges, “grand” or otherwise. It is unclear what effect the Future Backwards process had on the discussion but this process does dwell on the key events that influenced the current state and imagined future unreasonably bad and good states and the discussion often came back to questions of “how we got here” and the challenges of avoiding the problems our path to here seems to have been beset with.

I will use “participant” to refer to someone who participated in the discussion. Audio files for a recording of the discussion are also available on the CETIS wiki.

One non-specific challenge was identified as how to get to a point where decision-makers had well-grounded strategic understanding. In its various aspects, the current state of affairs seemed to have been influenced by reactivity, pulls in different directions and “tail steering”. We should not accept this for the future. One participant suggested that no-one really knew whether universities were cities or factories, references back to Andrew Feenberg’s keynote. Feenberg contrasted an industrially-inspired view of education with a “place of cosmopolitan interactions [the city] and enhanced communication¦It is not dedicated to the rigid reproductions of the same, the ‘one best way’, but to the flexible testing of possibilities and the development of the new- not hierarchical control but unplanned horizontal contacts; not simplification and standardization but variety and growth of the capabilities required to live in a more complex world.” (Transforming Technology, 2002). A different dimension of strategic understanding relates specifically to technology: what can it do, what can it not do and what could it do (sometimes unanticipated and often unintended consequences).

An specific example of lack of strategic understanding of technology was cited as the implementation, effectively a realisation of the concept, of the VLE. Participants contended that the implementation of VLEs had largely not been to serve an institutional strategic purpose but, driven by a supply of funds made available by a government that believed IT had benefits, more because that is what everyone was doing. There is something similar here to the “no-one got sacked for buying IBM” adage.

A great danger was perceived, arising from the consequences of society’s understanding of education. If what society is prepared to give is support for “here is the content, off you go, its up to you..” and the richer experience with social and tutorial input is reserved for those that can pay, what future for the country? The non-specific challenge is how to avoid going down this path and the fear was that we do not have the strategies to guide us. One specific challange was suggested: to direct people’s attention to what learning should be about by developing good communication tools embodying our best knowledge of how people learn. One participant noted that it was curious that a change of technology from books to ICT seems to have affected people’s judgement on the limitations of the value of “content”.

The media was criticised as comprising an obstacle to progress on this non-specific challenge. From a realistic perspective, we cannot expect the media to work with more than stories. Stories attract people. Stories spread. The voice of the expert is lost until it plays a part in the hindsight of a story. Our naive experience of education as learners couples with a flawed, shallow and unreflective portrayal by the media. The media could have a really important role to play but no participant presented a strategy to effect the change.

Three specific challenges were outlined, challenges relating to JISC and CETIS’s innovation remit but not entailing the development of specific technology. The questions to be asked are:

  1. what are the principles for implementing a technology to achieve results (learning results, but defining the measures is implicitly part of the challenge too)?
  2. how should JISC use return on investment (again, defining this is part of the challenge) to guide the programmes it delivers in the pursuit of “innovation”?
  3. what are the factors for successful use of technology?

In summary, the participants in the Grand Challenges session seemed to feel that they had a realistic view on the education and technology landscape are were generally rather dissatisfied with where we are now. Strategic understanding is key.
Many specific points have been missed from the above account, generally when they referred back to an idea mentioned earlier in the discussion. I believe I have conveyed the general flavour above. If you want more, listen to the recordings (page with links).

SOA and Spaghetti Bolognese

Posted on by

What does eating Spaghetti Bolognese and service-orienting your IT systems have in common? (Please indulge me in a little frivolity.)

  • You won’t find many restaurants selling Bolognese Sauce that you can combine with a bowl of spaghetti you made yourself and Parmesan from Tesco.
  • Factorising out a service is like pulling one strand of spaghetti from the bowl; you know it must be possible but everything is just so tangled.
  • When you do get a strand separated it is tempting to eat it in a way that makes a mess.
  • It can’t be done elegantly.
  • Chopping it up small works but isn’t very clever.

“Spaghetti code” is already a well-used programmers’ term.

Joining Dots at the IMS September 2008: Learning Design

Posted on by

Last week (15-18 Sept) was the IMS Quarterly meeting, hosted by JISC, in Birmingham (UK). It was a rather unusual meeting as all of the sessions were open to non-members. As usual, however, it concluded with a “summit” day where various interesting people shared their ideas. I’m sure everyone joined a different set of dots (For readers not familiar with the culture I belong to, “joining dots” is concisely explained on wikipedia). For me, the shape of the animal that is the role of IMS Learning Design (LD) became more clear. Actually I think it might be a family.

During the week there were several demonstrations of current generation LD tools and it is certainly true that these are an order of magnitude more usable than the first round of tools. These days you don’t have to know or understand the IMS specification, either its conceptual model or the technical details, to use the tools. Gilbert Paquette showed us TELOS, which is impressive and takes a graphical approach to visualising the workflows. Dai Griffiths and Paul Sharples showed two products of the TENCompetence project: ReCourse, a mixed graphical and tabular LD authoring tool that hides complexity and Wookie, a widget-based approach to providing the “services” (forum, chat, voting etc). Fabrizio Giorgini showed work from the PROLIX project, oriented towards work-place staff development, where Giunti have extended their eXact Packager to include a graphical LD editor.

In spite of the substantial progress demonstrated by the above software, we will still hear even tech-savvy academics exclaim “impressive but I can’t see how I’d ever use it” (not a real quote) or “I keep feeling that IMS LD was a solution looking for a problem and havent yet seen anything that solves any problems I have in learning & teaching.” (David Davies). I don’t think we can address this by talking about LD. Rather we need to talk about talking-about LD.

Dai Griffiths made some observations about LD that, if you will indulge me in continuing my dot-joining metaphor, I think pointed out which way up the paper is. He said: “The history and multiple uses of the specification mean that it is a complex artefact with many perspectives on it.” He produced a diagram (following) to expand on this point.

Dai Griffiths Multiple Uses of LD

For me, the diagram did more than expand on the point: it gave me an indication of a profitable way of reducing the complexity of our discussions by being clear that there is more than one way to perceive LD. Unless we can move discourse onto a more differentiated set of conversations, I believe we will not be able to really get anywhere with LD or, indeed, make much progress in dealing with the challenges the creators and proponents of LD believe it can address.

The situation of LD is not unique and there has been some interesting work exploring the concept of “enactment” in relation to Ecological Modelling Language conducted under the Comparative Interoperabilty Project. In this work Miller and Bowker say: “Jane Fountain invites us to distinguish between an ˜objective technology “ that is to say a set of technical, material and computing components such as the Internet “ and an ˜enacted technology “ that is to say the technology on the ground as it is perceived, conceived and used in practice, in a particular context.”

So, what are we to do? Where should we start?I speculate that we should begin by clarifying what LD is. Bill Olivier had opened the “summit” with some reflections on the work of IMS and JISC in support of interoperability and thoughts on where we as a community could profitably work in the future. He described the work of IMS on data models as being more akin to domain modelling and I think this may be a good insight. Domain models are necessarily rather more abstract and application agnostic than most people care to deal with. I think they are fundamentally models of “objective” rather than “enacted” technology. I believe we should accept and embrace this and conclude that LD is a language for technologists to coordinate the creation of artefacts that are the subjects of the different differentiated discourse I referred to earlier.

There are two parallels with the eFramework to be made here, but it would be a diversion to go into detail. It too has a “history and multiple uses” and consequentially there are many perspectives. The second parallel is that one of the purposes of the eFramework initiative is to enable dialogue within and across domains through the emergence of an explicit vocabulary appropriate to a service-oriented approach.

As a candidate for one differentiated conversation, I suggest picking up on another pearl from Dai Griffiths: you can consider LD to be about provisioning a learning environment. “Provisioning” is a bit of a jargon term for setting-up-what-you-need. If you start a new job, you expect a number of facilities to be provided: desk, computer, security card, payroll, user id, staff handbook …. The equivalent provisioning of a learning environment entails the marshalling of resources, conversation (forum/chat) and other “tools”, assignment to groups etc. It is online classroom management of a sort. Let us now have a conversation about this “thing” that you can drop into a virtual learning environment that magically does all of the provisioning for e.g. a 2 way online debate. Its a “wizard”: just add a few Word docs, choose how groups are assigned and its done. This isn’t a new use case; I discussed something very similar with Bill Olivier 5 years ago. I do think, though, that this is closer to the language of enactment.

Is this an application profile? i.e. is it a definition of which data elements to use, vocabularies and extensions. Not exactly: an application profile may emerge as a necessity but it would be prudent to be clear what the application is first and that entails discourse in the language of enactment not the domain model. As a closing aside, I would like to stress that I see an “application profile” as being a quite opinionated work; it should be for a purpose.

I am conscious that this is a some-what under-developed argument, probably with numerous errors and certainly with leaps of faith but I think it is time to expose my thinking out loud for criticism and to leave this piece definitely un-concluded…

The presentations referred to above are available on the web, linked from the agenda.

JISC Hosts IMS Quarterly Sept 15-18

Posted on by

JISC is hosting the September IMS Quarterly meeting from September 15th to 18th in Birmingham (UK). Usually these meetings are for IMS member representatives but this time the sessions are all open to the public (registration required).

Whereas there will be some work on interoperability specifications at the meeting, most of the sessions are far less hard-core but no less important in dealing with strategy and requirements:

  • Learning Object Discovery and Exchange
  • Question and Test Interoperability
  • Common Cartridge (including interoperability testing and demos)
  • Portfolio
  • Learning Design
  • Service orientation

The agenda has full details.

Lots of CETIS should be there and we’ll be pleased to welcome old friends and new.