There is, of course, no simple recipe, no cookie-cutter template and perfection is an unattainable… but there are some good examples.

The Signals Project at Perdue University is among the most celebrated examples of analytics in Higher Education at the moment so I was intrigued as to what the person behind it would have to say when I met him just prior to his presentation at the recent SURF Education Day (actually “Dé Onderwijsdagen 2012“; SURF is a similar organisation to JISC but in the Netherlands). This person is John Campbell and he is not at all the slightly exhausting (to dour Brits) kind of American IT leader, full of hyperbole and sweeping statement; his is a level-headed and grounded story. It is also a story from which I think we can draw some tips on how to do analytics right. These are my take-home thoughts.

Analytics = Actionable Intelligence

Anyone who has read my previous blog posts on analytics will know I’m rather passionate about “actionable insight” as a key point about analytics so I was naturally pleased to hear John’s similar take on the subject. We vigorously agreed that more reports is not what we need. If you can’t use the results of analysis to act differently it isn’t worth the effort. The corollary is that we should design systems around the people who need to take action.

Take a Multi-disciplinary Approach

Putting analytics into practice (at scale) is not “just” addressing IT or a statistical matters but requires domain knowledge of the area to be addressed and an understanding of the operational and cultural realities of the context of use. John stressed the varied team as a means to taking this kind of rounded approach. Important actors in this kind of team are people who understand how to influence change in organisational culture: politics.

You do still need good technical knowledge to avoid false insights, of course.

Take Account of “User” Psychology

The people who use the analytics – whether driving it or intended to be influenced by it – are the engine for change. This is really pointing out aspects of a multi-disciplinary approach; think soft systems, participatory design, and a team with some direct experience as a teacher/tutor/etc.

Signals has several examples, all elementary is some respects but significant by their presence:

• teaching staff trigger the analysis and can over-ride the results (although rarely do);
• it is emphasised to students that signals is NOT about grades but about engagement;
• there are helpful suggestions given to students in addition to the traffic-light and, although these come from a repertoire, the teachers have a hand in targetting these.

Start Off Manually

OK, a process based on spreadsheets and people manually pushing and pulling data between databases and analysis software is not scalable but this can be an important stage. Is it really wise to start investing money and reputation in a big system before you have properly established what you really need, what your data quality can sustain, what works in practice?

This provides opportunity to move from research into practice, to properly adapt (rather than blindly adopt or superficially replicate) effective practice from elsewhere,etc. A manual start-off helps to expose limitations and risks (see next point).

KISS

The old adage “keep it simple stupid” (a modern vernacular expression of Occam’s razor) is not what John actually said, but he got close. Signals uses some well established and thoroughly mainstream statistical methods. It does not use the latest fancy predictive algorithms.

Why? Because fancy treatments would be like putting F1 tyres on a Citroen 2CV: worse than pointless. The data quality and a range of systematic biases* means that the simpler method and a traffic-light result is appropriate technology. John made it clear that quoting a percentage chance of drop-out (etc) is simply an indefensible level of precision given the data: red, amber and green with teacher over-ride is.

(*- VLE data, for example, does not mean the same across all courses/modules, teachers)

Be Part of a Community

OK… I liked this one because it is the kind of thing that JISC and CETIS has been promoting across all of their areas of work for many years. Making sense of what is possible, imagining and realising new ideas works so much better when ideas, experiences and reflections are shared.

This is why we were pleased to be part of the first SoLAR Flare UK event earlier this week and hope to be working with that community for some time.

Conclusion

Many have, and will, attempt to replicate the success of Signals in addressing student retention but not all will succeed. The points I mentioned above are indicative of an approach that worked in totality; a superficial attempt to replicate Signals will probably fail. This is about matching an appropriate level technology with organisational culture and context. It is innovation in socio-technical practice. So: doing analytics right is about holism directed towards action.

The views above include my own and not necessarily John Campbell’s.