8. Evaluation

---

0. Overview | 1. Student Profiles | 2. Discipline Contexts | 3. Media Choices | 4. Intended Learning Outcomes | 5. Assessment | 6. Learning Teaching Activities | 7. Feedback | 8. Evaluation

---

It may seem strange to design our evaluation structures before we have even recruited students onto our programmes. We first need to understand the distinction between assessment, feedback, and evaluation. It is then important to explore both the evaluation of learning experiences, including the robustness of your course design, and evaluation for learning, which I will refer to as in-class evaluation for consistency.

Basic Concepts

This stage explores five basic concepts that underpin the evaluation of learning.

1. Distinguishing between Evaluation, Feedback and Assessment
2. Measuring Student Performance versus Teacher Performance
3. In-Class evaluation versus Post-Completion evaluation
4. Learning Gain
5. Progression: Access, Retention, Pass Rates, Grades, Completion and Destination

Distinguishing between Evaluation, Feedback and Assessment

The language used in higher education is often inconsistent and less than helpful. American literature often refers to the evaluation of learning rather than the British English use of assessment for learning. It is important to watch out for these linguistic differences as you review international literature. Although it is rare to conflate the terms evaluation and assessment in a UK context, it is not uncommon to confuse evaluation and feedback. See the table below for clarification.

In addition to the table above, here are three working definitions.

• Evaluation is all communication designed to elicit evaluative comments provided by students on their learning experiences, and/or from academic peer review and self-reflection on our teaching practices.
• Feedback is all communication designed to support a student’s future learning capabilities.
• Assessment is all communication designed to enable the student to evidence their ability to meet a defined learning outcome.

We should try to be consistent in our language and use feedback only when concerned with all communication that students receive, designed to advance their learning, rather than what they give. This would greatly enhance students’ comprehension of assessment and feedback, their ‘academic literacy’.

Measuring Student Performance versus Teacher Performance

Module evaluation reports are a common feature of higher education. It is important to distinguish the sources of data used to establish ‘performance’. Both will feature in a comprehensive evaluation of a course or programme. These are often produced as an annual programme or module review.

Both staff and student performance are generally measured differently. There is evidence captured during the course itself, which is captured at the end of the course delivery, and that which is collected later, after the course has been completed. Some of these data sources are detailed here in the table below.

Staff performance is measurable during the course itself. The individual themselves can organise in-class evaluation activities with their own students (see below) and routinely record personal reflections on their own performance. This could take the form a commentated micro-teaching recording or keeping a learning journal. In addition, there is a range of external perspectives on teacher performance. The most common are peer observations, either for developmental or management purposes and teaching scores.

At the end of the course, or before and after marking assessments, it is a good idea to make a teaching reflection. Take the four questions featured in the in-class evaluation process (see below) as a basis.

After a course is completed, it is common for students to rate the faculty’s performance. Such scores are almost ubiquitous despite being a somewhat flawed mechanism that fails to take into account cohorts’ prior learning, competence, and various factors that impact student evaluation, such as gender and ethnic bias. Nonetheless, most institutions persist and have moved to standardise and centralise data collection, using centrally administered evaluation software, so that adjustments and interpretations can be made. 

Better mechanisms for quality enhancement include reviewing, marking, and moderation, as well as routinely peer-reviewing feedback offered to students throughout the learning process and on final credit-bearing assessments. Peer reviewing our ability to assess and provide feedback is important to continuously enhance our capability to ‘teach’ in the classroom. The final external measurement is provided by external examiners and reviewers. These are normally made in incredibly diplomatic language and are unlikely to ‘name names’, but they do serve to contextualise other data sources. External examiners are also a common source of measurement data on student performance.

Student Performance can also be evaluated during, at the end of, and after a course.

Student Performance during a course is ordinarily measured by using all of the techniques for feedback throughout dealt with in Stage 7 of the 8-SLDF. Faculty use this feedback as an evaluative instrument. Increasingly, there is also a range of learning analytics available for those studying entirely online. The picture is mixed for those undertaking blended provision. Whilst in-class evaluation is designed to guide the teaching practitioner towards enhancing the student experience, it also provides insights into student performance.

Student performance can also be evaluated through the lens of the assessment tasks designed to support students to evidence their learning against the outcomes. Self-declared performance indicators are usually included in end-of-module questionnaires.

After a course has been completed, evaluations based on students’ perspectives are also possible through analysis of the grades awarded, the progression of individual students and cohorts and the number of students that withdraw from their studies.

None of these factors on its own tells anyone anything about the quality of learning design or its delivery.

In-Class evaluation vs Post-Completion evaluation

Course designers need to decide in advance where they anticipate the enhancement opportunities for our course will be and design in-class and post-course evaluation instruments to capture them. Most UK institutions’ NSS and end-of-course evaluation processes do not generate actionable data. We can design some of our own.

Most of you will be familiar with the concept of ‘end-of-course’ evaluations. They provide data on students’ prior experience in a specific course. Aggregated with other course results, one can build a picture of the student experience across a specific programme. Most UK institutions collect data using the current national survey instrument, the National Student Survey (NSS). Most Universities also centralise data collection processes to ensure consistency and common adjustments. Increasingly, data is collected online and processed immediately so it can be re-presented to students and faculty.

It is also important to ensure that we have efficient and effective in-course evaluation techniques in mind to provide an opportunity to enhance the course as it is underway. We need to avoid making knee-jerk adjustments to a module that appears not to be working. In-course evaluation needs to be appropriately positioned within a course, with sufficient time and preparation provided. This invaluable evaluative process allows faculty to capture students’ experience of their learning whilst the course is still running. This has various advantages:

• It allows faculty to make minor adjustments to suit specific cohorts
• It gives students’ the feeling that they are listened to
• It transfers some of the responsibility for learning back to the student

This technique is described as Small Group Instructional Diagnostics or SGID for short (Seldin, 1999). Here, we describe it simply as an In-Class Evaluation. There are two models:

• A colleague who does not teach the current students undertakes an in-class discussion, eliciting responses to four set questions, and then writes a short report for those students and their tutor. This means sacrificing 30 minutes of your class time.
• The Tutor distributes a paper-based questionnaire with the same four questions (or a digital equivalent), collects the responses, and provides feedback to the cohort at their next encounter.

Timing

The timing of such an intervention is important. Ideally, it will be early enough in a module for both students and tutors to benefit from the intervention. But, it also has to be grounded on ‘genuine’ learning and teaching activities, so there is no point in doing it immediately after a typical induction week, for example. I’d suggest about week 3 or 4 of a 10-week module, aiming roughly for a third of the way in.

Questions

The order and structure of the questions are important. It ensures that students’ evaluative comments focus on their learning rather than on your teaching. You also want students to provide actionable responses.

The set questions are:

1. What is helping and supporting my learning in this course?
2. What is stopping me from learning in this course?
3. What could the tutor do differently to improve my learning in the course?
4. What could I do differently to improve my learning in this course?

Feedback

What is vital is that students know and understand that their evaluative comments have been read and acted upon. That does not mean that everything a student says must be honoured or implemented, but it should be acknowledged. The easiest way to do this is to summarise the points made against each question, aggregate the data, ensure it is anonymised, and then return it to the student with your considered responses.

For example, I ran an online module (focused on reflective and professional practice), and a small but vocal minority insisted that I provide the PowerPoints used to illustrate my webinars in advance. I declined, but it did give me an opportunity to show that the PowerPoints were designed not as stand-alone learning resources but as visual stimuli for debate.

On another occasion (a contemporary issues module), a majority of the cohort wanted access to all the readings in advance. I consented to provide those, but made it clear that I reserved the right to substitute literature later in the course if something changed.

What is important is that you see the responses as valuable evaluative data about your practice without actually asking students to grade you or directly comment on your performance.

Design Tip: Build this process into your module design and allow time in class for students to complete it. Normally, a paper-based activity should take 3-5 minutes to explain and 10 minutes for students to complete.

Learning Gain

A highly disputed term in educational circles is ‘Learning Gain’. It is hardly a new concept, dating back even before the contemporary fascination with ipsative assessment (Stage 5) and progressive models of education (Hughes, 2017). In brief, learning gain is the ability to measure the progress made, or distance travelled, between a student’s command of a discipline at one point in time and a later point in time. So, by way of example;

1. An experienced legal secretary, Chris, with 10 years of practical experience, undertakes learning activities to complete a conveyancing course. In reality, they have already gone through this process dozens of times over the previous years, so the 10-week module has done nothing more than reconfirm prior knowledge. They scored 90% on the assessment for this task. Their learning gain is minimal.
2. A student, Sam, with an undergraduate law degree and theoretical knowledge but no practical experience in conveyancing, undertakes the same learning. They find it challenging; they face lots of new terminology and practical contexts. They acquire significant new knowledge and new skills. They scored 60% in the assessment. Their learning gain is significant.

Traditionally, we have assessed students based on their ability to perform within set assessment tasks. On that basis, Chris, above, clearly outperforms Sam. However, if we are to ask: ‘Who learnt the most?’ ‘Who received the most learning gain?’ We would come to a different conclusion.

Increasingly, university funders and quality assurance agencies are concerned not only with the ability of excellent students who enter University to leave with excellent results, but also with how below-average students are supported to become above-average graduates.

The concept of learning gain is now forcing its way onto the design agenda. How can we design learning that allows students to measure their own metacognitive and skill progression throughout a course and, importantly, across a programme? I believe all well-designed learning necessarily facilitates the analysis of learning gain.

Some design suggestions that a design team might want to consider:

• Benchmarking or diagnostic assessments are placed early on in a course and repeated at the end.
• Support the reflective processes through e-portfolio completion with structured reflections on learning gain throughout
• Design a synoptic assessment that includes statements of prior knowledge and learning gain integrated into reflections. Some of the US literature refers to this as ‘Capstone’ learning or assessment.

Design Tip: Coordinate the assessment models between all modules within a programme and draw synoptic assessments across modules. Is there room in your programme for a ‘synoptic’ module?

Progression: Access, Retention, Pass Rates, Grades, Completion & Destination.

Another inconsistent use of language concerns different quantitative measurements used to evaluate learning effectiveness. Falling broadly under the category of ‘Progression’ these denote the relative success of individual students. Relative in that they are frequently measured against internal institutional norms and external benchmarks.

The language is contested, and you will see various representations of these interdependent issues. The important thing is that a design team should discuss the kind of data expected to be generated later in the reporting process and ensure that their design facilitates the easy capture of this data wherever possible.

Next Steps

Here are a few questions you should consider when forming your design team. I suggest you review these each time you meet, not at the end of the process.

• Do the Design Team (and the delivering Tutors) share a common language regarding feedback and evaluation?
• Do the Design Team (and delivering Tutors) share a common frame of reference with respect to the performance mechanisms in place for your module and programme?
• Have you made provision for meaningful in-class evaluations in your design? Have tutors been adequately prepared to respond to evaluative comments?
• Has the Design Team considered the possibility of being asked to report on Learning Gain for their students?
• Have the Design Team and the delivering Tutors been fully briefed on the likely progression measurement environment in which their course or programme will emerge?

References

Ferrell, G. (2013). Changing assessment and feedback practice. Retrieved August 29, 2018, from https://www.jisc.ac.uk/guides/changing-assessment-and-feedback-practice

Hughes, G. (2017). Ipsative Assessment and Personal Learning Gain: Exploring International Case Studies. Springer.

Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090

Seldin, P. (1999). Changing Practices in Evaluating Teaching: A Practical Guide to Improved Faculty Performance and Promotion/Tenure Decisions. Wiley.

---