Category Archives: analytics

Student Progression: Smartcard Bursaries and the Students FIRST Project

Posted on by

Photo of a brass compassThe JISC funded Students FIRST Project has been improving the use of bursary schemes for purchasing learning materials and other services at the University of East London and Anglia Ruskin University, in conjunction with AMOSSHE and John Smith’s Booksellers.

Challenges

The Students FIRST project pulled together a group of technologies – financial information app, bursary on a smarcard, and social messaging tools (texting) – to help improve progression and retention. However, there were some challenges in this approach:

  • technologies, such as smartcard or mobile apps, may be used in a “scattergun” approach and need to be part of a strategic service delivery
  • staff can be unwilling to engage with new technologies; for example, because they don’t want to remember additional logins
  • staff must be trained in the use of any new technologies, but it can be difficult to find the resource to do so.

Benefits

Access to the bursary is staggered according to student progression; i.e. a student must progress to the second year of their studies in order to receive the second installment. Students can then purchase from a list of products specified by their institution, such as books, art materials, nursery fees, campus accommodation, etc. Almost 74% of students surveyed at one university found that the bursary was beneficial. Other benefits include:

  • establishing a clear link between the spend on books and academic achievement
  • a targeted bursary encourages students to achieve and progress
  • such a bursary also equalises opportunity across the student body; for example, one student said “I have access to books that otherwise I wouldn’t be able to own and progress further”.

Recommendations

A collaborative approach to this project was taken with a mix of educational and commercial providers and this gave the project team the opportunity to draw up guidance materials on working across different sectors. Recommendations include:

  • taking a service design approach can help you to understand student needs, expose failpoints in service delivery, and build collaborative relationships between departments/institutions and providers
  • sharing data between institutions can be a cause for concern, so consider alternatives such as using separate hard drives to store data
  • actively engage with technologies with which students are familiar, such as mobile apps, to encourage engagement.

Further Information

If you would like to find out more about this project, the following resources may help:

Student Retention: Student Dashboard at University of Southampton

Posted on by

Photo of a brass compassThe JISC funded Southampton Student Dashboard Project at the University of Southampton has been aggregating data from across a number of systems and presenting it in a single place, so that pastoral tutors can provide better informed support for students.

Challenges

Data held by institutions is not always easy to access. For example:

  • data held across a number of systems only offers a partial or disjointed view of information that may be relevant to staff and students
  • multiple systems require multiple log-ins; for example, Student Services at Southampton need to switch between four different applications in order to amalgamate student information
  • selecting the data that tutors might need to see on a dashboard can be contentious; for example information from Finance or Student Counselling services.

Benefits

In common with other projects that have focussed on using data to identify “at risk” students, the project team identified the following benefits:

  • providing a complete view of an institution’s student data enables staff to identify any early signs of problems and possible non-progression
  • improving access to data can encourage the organisational culture to be more innovative and transparent
  • by allowing a small set of data to be shown to pastoral tutors, it is expected that this will generate requests for more data to be included.

Recommendations

Encouraging people to open up access to data can be challenging. Issues of data access and organisational culture can be difficult to handle, so try to:

  • manage change carefully to ensure that all stakeholders are engaged, especially those who have power to implement change and those who have influence over opinion in the institution
  • identify champions in each group of stakeholders, who will help drive through changes
  • find out what data is held by each stakeholder and how it is accessed (some of it may be paper-based) as this can help determine how that data could be accessed in a dashboard; it can also expose information that some stakeholders didn’t even know existed.

Further Information

If you would like to find out more about this project, the following resources may help:

Student Retention: Project fulCRM at Roehampton University

Posted on by

Photo of a brass compassProject fulCRM, funded by JISC, at Roehampton University has brought together two existing processes, one technical and one manual, in order to reduce the number of students leaving their course early.

Challenges

The initial challenges faced by the project team included:

  • over 2000 mitigating circumstances requests were made each year by students in the University, which were all handled manually
  • lack of consistency in the way in which mitigating circumstances were handled, which was a cause for student concern
  • data formats that are not designed to integrate with other applications.

Benefits

Two existing processes (a spreadsheet-based Early Warning System of lack of attendance and other indicators and the manual process of applying for mitigating circumstances) have been improved by taking a technological approach. This has resulted in a number of benefits for staff and students alike, including:

  • improved communication by ensuring that students are kept informed via e-mail of the progress of the mitigating circumstances process
  • flagging up of students who need additional support
  • automated data collection which has replaced complex and time-consuming activities and reduced staff workload.

Recommendations

Students now feel that the technological improvements provide them with a safety net and they will also be able to view their performance alongside their tutor. However, arriving at this stage has not been easy and the following suggestions may help:

  • in order to gain acceptance across a number of disparate Departments, ensure that any system can cater for the creation of different student performance indicators
  • if data (such as data relating to the number of times a student accesses a VLE) is held in the cloud, accessing that data and moving it across to private university servers may pose a security risk, therefore additional servers with secure links direct to the VLE may need to be built
  • do not underestimate the amount of manipulation required to access complex data which is often housed in various disparate systems.

Further Information

If you would like to find out more about this project, the following resources may help:

Student Retention: Learner Analytics at Loughborough University

Posted on by

Photo of a brass compassThe JISC funded Pedestal for Progression project at Loughborough University has focussed on bringing together information from their in-house Co-Tutor dashboard and other systems to help tutors be pro-active in their response to “at risk” students.

Challenges

The project team found that having access to the data is not enough, there needs to be a balance between automated and human interventions, for example:

  • educational data mining is complex and will be unsuccessful if used solely to identify “at risk” students without human intervention processes being designed
  • incorporating more student-centred approaches into the development of exisiting systems (such as VLEs, student data systems, attendance recording systems etc) can be challenging
  • changes in processes and the introduction of new software systems can be seen as threatening to staff.

Benefits

The project team looked at both staff and student needs by:

  • using service design techniques to help identify learner needs, such as flexibility of study, planning of deadlines for coursework, and employability, for example
  • providing more and better quality learner information to help a tutor be more active in identifying and supporting “at risk” students; for example teaching sessions where attendance is taken electronically are designated as being “critical” or not, so e-mails are sent only to students missing “critical” sessions
  • developing a standardised process for managing placements across three departments in order to reduce student anxiety around the process.

Recommendations

The relationships that students have with staff are vital to the enhancement of the student experience, but the relationships that staff have within in the organisation are also important. For example:

  • provide staff with access to learner data to help them identify any human interventions that need to be made
  • use the student voice to influence improvements
  • sincere management buy-in is critical, otherwise an institution’s organisational structure will remain a barrier no matter what improvements are suggested.

Further Information

If you would like to find out more about this project, the following resources may help:

Student Retention: Engagement Analytics at University of Derby

Posted on by

Photo of a brass compassThe University of Derby has been scoping early indicators (engagement analytics) for spotting students at risk of withdrawing in their SETL (Student Experience Traffic Lighting) project.

Challenges

Institutions generally hold a vast array of data about students, often in different systems which are not always interoperable. The data challenges experienced in this project include:

  • predisposing factors, such as responsibility as a carer, means that students are more likely to withdraw from their studies, however it’s not always possible to capture this type of data as it’s not generally held in any IT system
  • there is little interoperability between different data systems; for example, the data required to populate an engagement dashboard is held in at least seven different systems at the University
  • each student is an individual who brings with them individual challenges to succeeding at and engaging with higher education

Benefits

Scoping out the type of data to be included in a dashboard of core engagement data:

  • means that staff would be able to view a student’s level of engagement with the institution; for example, linking data on absences, access to the library and the VLE (Virtual Learning Environment) could help a tutor see if the student was still engaging in the course, even if they were absent due to illness
  • has produced a change of thinking in the way students at risk of withdrawing will be supported at the University; i.e. it will be more proactive than reactive
  • has helped staff identify key points in the student lifecycle where students are most likely to be at risk of withdrawal.

Recommendations

Engagement analytics goes beyond the hard data recorded in learning analytics, because:

  • it’s dangerous to make decisions about student engagement based solely on a set of data, as understanding the context of the data is important and developing the relationship between the tutor and student is essential
  • both staff and students find it useful to have their own customisable engagement dashboards
  • soft data that can’t always be found on institutional systems should also be recorded and considered.

Further Information

If you would like to find out more about this project, the following resources may help:

Using Data to Improve Student Retention: More Questions than Answers?

Posted on by

The “Issues around Using Data to Improve Student Retention” session at the JISC CETIS Conference 2012 examined some of the issues around using data to predict students at risk of failure and provide some examples of possible solutions to ensure that the institution does not fail such students.

The session was divided into five mini-sessions run by some of the JISC Relationship Management Programme projects. Here is an overview of some of the solutions being trialled by the projects and the challenges they face.

Loughborough University’s Pedestal for Progression project is using Co-Tutor to collect attendance data. Student attendance at lectures and personal tutor meetings is recorded, so it’s possible to see at a glance whether a student is turning up or if they dislike Monday mornings. The system gives automatic flags and provides a personal tutor with a red, amber or green visual of attendance (traffic lighting). As a result of introducing registers, attendance has gone up from 65.54% in 2004/5 to 70.26% in 2010/11.

Roehampton University’s fulCRM project is being run in parallel with attendance data recording within the Psychology Department. Fingerprint readers have been installed in all lecture theatres to record student attendance. Although there have been debates by staff about the “big brother” aspects of this approach, it would appear that students aree less worried as many are already used to doing this in school and in any case, fingerprint attendance seems to make students think that it (and they) really count and that someone is taking an interest in them. As the Department is fairly small, data was being kept on an Excel spreadsheet and a traffic light system used. The fulCRM project is now in the process of pulling automated data feeds from the attendance monitoring system into a student performance module.

Using a traffic light system, 18 students were identified as “red” just after the first semester (mostly male or part-time students with personal problems, or who had to travel long distances, etc) so extra support was put in for these students. In 2010, 8.3% of at risk students terminated and 19.8% didn’t have enough credits to go to Year 2. In 2011, since the traffic lighting system was put in place, this was reduced to 5.2% of at risk students terminating and 9.34% without enough credits to go on to Year 2. Of the 19 students with red flags, 16 have now gone on to Year 2 following retakes and extra support (only 3 actually left). In the past, a lot of students would have just drifted off course.

During the course of their project, the Southampton Student Dashboard at the University of Southampton project team has met with resistance from data owners, service providers, ethics authorities and faculty administrators, who cite a number of reasons why various types of data (from photos to student grades) can’t be used. They currently have a simple dashboard that shows picture of tutees, directory info, whether coursework has been handed in, and attendance. This particularly helps staff who need to let the Borders Agency know that foreign students have been seen and have attended classes. However, most information isn’t available in the rest of the University because of data protection, which is seen as protecting staff from doing any work or taking any risks.

The University of Derby’s SETL (Student Engagement Traffic Lighting) project is also using a traffic lighting approach and has been looking at some of the softer areas off engagement analytics. The team has produced a dartboard diagram showing the primary (e.g. attendance monitoring), secondary (e.g. sickness) and tertiary (e.g. student complaints) indicators of risk. They have also produced a “Withdrawal Calendar” to ascertain whether there are any key dates for withdrawal – there are. At Derby, a Progress Board meets twice a year to decide whether students are meeting the academic requirements and can progress on their course. These key times when students are likely to withdraw.

The ESCAPES (Enhancing Student Centred Administration for Placement ExperienceS) project at the University of Nottingham is exploring how an ePortfolio can enhance student engagement whilst students are on placement. Previous project work has already shown that both students on placement and staff found that ePortfolio tools, such as Mahara, have helped them to keep in touch with each other. Students have also valued the feedback aspect as it helps them to feel more motivated.

Some of the challenges, questions, and issues that have arisen during these projects can be grouped into the following areas:

Technical

  • Using the relationship management systems, such as Co-Tutor, may result in a bigger workload for staff, which may then manifest as a lack of engagement with the system.
  • There may not be any integration with other systems, such as e-mail, so staff have to copy and paste any e-mail from students into the system, or it may be difficult to exchange data between systems.
  • Not all data may be captured electronically but may still be paper-based.
  • It can be expensive to roll out pilot data monitoring solutions across the whole institution.

Human Aspects

  • Where interactions are recorded, such as personal tutor meetings, staff may feel that they have less freedom about where that interaction takes place, i.e. it may now have to take place in the tutor’s office rather than a neutral space such as a café.
  • Institutions need to make it clear to students that such systems are there to help them.
  • Should students be forced to attend classes?
  • The behaviour of departments who hold such data can be difficult. Data owners may be distributed. People hold their data very close to their chests and don’t want to share. Perhaps the most difficult bit is managing the soft human interface.

Data Privacy/Ethics

  • Who should be allowed to see staff data?
  • Who monitors the monitors?
  • Can use of data in this way be seen as “Big Brotherish”?
  • How quickly should staff intervene if a student is “red-flagged”?
  • Is it ethical to pre-load a system with at-risk demographics, e.g. part-time, male students?
  • What directory information should be available to everyone on a university intranet? Should photos be included (it can be useful for staff to add a name to a face)? What information shouldn’t be included?
  • Should grade history be confidential?
  • Students already share their data informally on Facebook, so why should the institution get involved?
  • Is it OK to use student data for research?
  • Who are the stakeholders involved in holding the data? What about contracts with work based learning students?
  • Sensitive handling of data and how it’s being collected is necessary.
  • Who should access what data? Should whoever needs the data be able to access it? What is legitimate? Data access needs to be considered before any opt-in box is checked. Explicit permission is needed from the student. Should anyone have access to depersonalised data as long as it’s properly depersonalised?
  • What about control of access for student? Student might be overwhelmed with information if everyone has access to the student’s data.
  • What about access by parents or other interested parties?
  • What is the “right data” to collect and analyse? For example, a student might have patchy attendance but may interact very well with the tutor and have a good assessment. Whoever acts on the data provided needs to know the student well.
  • What are the softer engagement analytics? What aspects of the student lifecycle can be harnessed? How do we capture this soft data? It needs to be about the social life as well as the academic life.
  • How early do we need the data and what would those data sources be? The data might come too late, i.e. the predisposing factors may already be in place before the student starts.

The session brought a lot of issues to light, along with some real privacy/ethical concerns, but it also highlighted that even a small change (such as keeping a register) can make a difference to student retention. However, we need to remember that students are not just collections of bits and bytes to be analysed and examined for trends. We are human with all the foibles, idiosyncrasies and circumstances that make us unique.