0. Overview | 1. Student Profiles | 2. Discipline Contexts | 3. Media Choices | 4. Intended Learning Outcomes | 5. Assessment | 6. Learning Teaching Activities | 7. Feedback | 8. Evaluation
Knowing our intended learning outcomes (ILOs) enables us to design meaningful assessments that provide opportunities for students to evidence their learning against those ILOs. It is important to initially identify which outcomes across different domains of learning can be combined through assessment. This allows us to manage the assessment load for both faculty and students, whilst ensuring all ILOs are assessed. Using taxonomy circles, we can then draft marking rubrics for the appropriate level that represent all the guidance that individual assessors and students need to guide their practice.
Before we explore the different forms of assessment that are available to learners, we should begin with some fundamental questions.
Philosophical Construct
This is a (tongue-in-cheek) representation of how we, in the West, have come to understand assessment. We are, however, unwittingly informed by our cultural conditioning, which sees ‘perfection’ as close to unattainable. Here’s a Judeo-Christian perspective on ‘divine assessment’:
This profound cultural influence is also one of the reasons why our past mark is (generally) 50%, lowered by convention to 40% for undergraduates, and why we have uneven grade boundaries. This meant that, historically, marks were awarded on a bell curve; if everyone scored highly or poorly within a cohort, the curve was adjusted to ensure the grades remained appropriately distributed.
In the above model, ‘normal’ people are likely to converge in the middle. It’s the reason that until very recently, ‘norm-referenced’ assessment was the norm (!). This meant that everyone was assessed against their peers; they could never be better than a pre-determined ‘best’, and the majority were expected to be ‘average’.
A final point on the underlying philosophies of assessment is to encourage you to reflect on the conventions that exist in your discipline and institution. The illustration above shows how grade boundaries have been widely adopted across the UK higher education sector. Institutions will vary, but this is a broadly accurate profile. The first line shows the percentage marks assigned, the second row a typical Master’s-level grade descriptor, and the third row a typical undergraduate degree classification.
How is it possible that 40% of the grades at the lower end represent failure, 30% at the top the highest possible grade and everything else compressed into that middle 30%? This is not to propose an alternative approach, only to encourage you and your design team to reflect on your current practice before embarking on designing fresh assessments.
Reminder of Terminology
For those new to designing assessments, here are a few definitions to support your course team discussions.
Forms of Measurement
The first is to be aware of how your students’ assessments will be measured. There are three dominant forms of measurement (grading)
I do not believe that norm-referenced is appropriate in higher education, and so I do not intend to spend time on it. I acknowledge that students often want to know how well they are doing relative to others, but it is more helpful for them to understand how they measure up against the criteria. There is some impact here, with the introduction of learning analytics and student dashboards, which I have published about elsewhere.
Your course or programme reporting after the assessment has been marked will generate a graph that may indeed look like a bell curve, but that is no reason to ‘mark to the curve’. Instead, I believe that for the majority of higher education, we should be examining criterion-based assessment. Where there are professional competency frameworks that overlay the design, it may also be appropriate to provide ipsative assessment forms too.
Reliability and Validity
Two key concepts to bear in mind as you design an assessment. These are validity and reliability.
Validity is a measure of the degree to which the assessment accurately assesses the desired outcomes (ILOs) of the module. Is the assessment measuring what it aims to measure?
Reliability is a measurement of the consistency or replicability of the assessment. If I assess multiple cohorts over time, will I get broadly the same results? Below are four illustrations of potential grade patterns.
- A – The assessment produced consistent results (irrespective of the range of marks achieved) and the assessment is constructively aligned to the ILOs so that it is assessing what it is supposed to assess.
- B – The assessment is designed to assess the ILOs but yields varied responses, making duplication across cohorts problematic.
- C – The assessment produces consistent and replicable results, but does not assess the ILOs. (a pitfall of using Multiple Choice Questions in the wrong context)
- D – The assessment is not assessing the ILOs for the module and is also producing hugely varied responses.
It is important to try and design with these principles in mind.
Objectivity and Subjectivity
Another issue course designers need to remember is the distinction between objectivity and subjectivity in assessment. This is fraught with difficulties. At first glance, it appears preferable to always be objective in assessing another’s skills or knowledge, but much of higher education’s skills and capabilities cannot be objectively measured. Explore the lists below:
| Objective | Subjective |
| Where a single correct response is possible. | Where multiple interpretations of facts, circumstance or concepts exist. |
| Can be ‘machine-marked’ without human interaction to produce a grade. | Requires ‘subjective’ interpretation by the assessor to arrive at a grade, hopefully, guided by meaningful criteria (see Rubrics below) |
| Uses ‘selected response’ or ‘structured responses’ | Uses ‘free-response’ or ‘constructed response’ |
| Includes Multiple Choice Questions, matching and alternative-choice items | Include short-answer and essay questions, performance, presentations and so on. |
| Works well to assess knowledge and comprehension. Facts and unequivocal contexts in applications of facts. | Works well to assess interpretations of factual knowledge, skills acquisition and capabilities in relation to varied contexts. |
Types of Assessment
Now that we have reviewed some of the fundamental mechanics of assessment, we are better positioned to ask, ‘Why do we assess?’ Arguably, we assess for some widely-agreed purposes; namely, to ensure learning has taken place, to award earned qualifications and to ‘benchmark’ individuals against criteria assigned by the discipline or a profession. To conclude this quick overview, let us identify the most commonly encountered assessment types in higher education.
| Type of Assessment | Function | Purpose |
| Diagnostic (sometimes ‘Needs’) | Establishes a baseline in existing knowledge or skills to structure future learning approaches. | Allows learning needs and support mechanisms to be identified |
| Ipsative (sometimes’ Benchmarking’) | Allows a student to measure their own performance over time. | |
| Formative | Provides learners with timely feedback and feed-forward on their learning progress | Impacts on current learning engagement |
| Summative (sometimes ‘Final’) | Provides learners with an opportunity to produce evidence of achievement against the defined module outcomes for the learning | Provides progression and certification mechanism |
| Synoptic (sometimes ‘Capstone’) | Provides learners with an opportunity to evidence of achievement against the defined programme outcomes for the learning, integrating learning acquired across different modules | Provides progression across levels and certification mechanisms. Holistic. |
What is being assessed?
Now that we have reviewed some of the basic concepts in assessment in higher education, we can address some of the practical issues before exploring the ‘optimal’ design approaches.
Clearly, there are many university programmes where the assessment, outcomes, and sometimes even the content are dictated by external bodies. Rather than abdicating our responsibility as learning designers, this is a call to better understand how to articulate the relationship between the intended learning outcomes of a course, how it is being assessed, and what students are experiencing as learning.
Let’s begin by identifying the knowledge, skills, or attributes we are trying to assess. Explore this table.
- Do you agree with its suggestions?
- Pause and think of exceptions where you have reliably and validly assessed defined skills using alternative methods.
| ILO Domains of Learning (4/8-SLDF) | What is being assessed | Suggested assessment method (very short) |
| Metacognitive (personal epistemology) | Declarative knowledge, answers that can be stated verbally. This includes specific facts, principles, trends, criteria, and ways of organizing events. [Metacognition] |
|
| Cognitive (intellectual skills) | Factual and procedural knowledge, knowledge of what is and how to do things. |
|
| Affective (professional skills / values) | Ethical skills, professional behaviours and attitudes, beliefs and feelings [Epistemology] |
|
| Psychomotor (manual skills) | Technical skills (software or specialist tools) |
|
| Interpersonal (communication skills) | Communication ability, verbal and written [Cultural Values] |
|
In many programmes, there is pressure now to assess a range of skills and behaviours beyond subject knowledge. The challenge is to design assessments that allow students to demonstrate a range of skills (across various domains) through a single assessment. Remember, all module outcomes must be assessed and passed, and each outcome needs to be assessed only once.
Drafting an assessment framework is an iterative process. Ideally, one designs the assessment at the same time as one writes module ILOs, ‘tweaking’ them to give them depth and flexibility. Once one has established that the best way of assessing several ILOs might be through a case study, for example, the case study could change for each cohort without needing to rewrite the ILOs or change the learning and teaching content and activities. Next, we’re going to remind ourselves how we do that.
Reminder: Constructive Alignment
Let’s briefly revisit the concept of Constructive Alignment (see 4/8-SLDF), which is the idea that what is taught (the learning activities) is directly related to the student’s ability to evidence (assessment) their achievement of the intended learning outcomes (ILOs).
Students are taught through their educational experience that final grades matter, so it is natural that they become fixated on the final assessment. A transparent design that closely aligns with the ILOs in the teaching activity and assessment will necessarily engage students in a broader and deeper understanding of their learning journey.
Given that all ILOs in a module must be passed, it is important that you are realistic in how you intend to allow a student to evidence their achievement of them, in other words, how you plan to assess them. Ideally, a course should be designed with the ILOs first, then an assessment strategy, and finally the learning activities and associated content.
Clearly, in an externally dictated curriculum, it becomes important to be able to ‘interpret’ external guidelines into the language of higher education to gain the advantages of a transparent, constructively aligned course.
Choosing Assessment
In an ideal world, we would design assessments in tandem with the ILOs for a course, with direct reference to the ILOs for a programme, and all within a carefully mapped out assessment regime.
In such a set of circumstances, the guidance is self-evident:
- Assess each ILO once
- Assesses combinations of ILOs across a range of different domains (skill sets)
- Choose an appropriate assessment form that reflects the skills being assessed
- Schedule assessments throughout a module to support student progression. Remember that ‘summative’ is not synonymous with ‘final’. So-called ‘summatives’ do not have to come at the very end of a module.
So how do we go about designing (or re-designing) an assessment that is constructively aligned? Below is an example of a 30 credit (300 student hours) postgraduate (UK Level 7) law course, in which;
- All course intended learning outcomes (there are 7) are assessed once.
- There are two distinct assessments (balancing the load throughout the module)
- Each assessment can be reworded without changing the essential structure (for plagiarism avoidance)
- In the course specification, this assessment would appear as two case studies, both focused on the ability of students to explain the law in real-life contexts.
The message here is that combining outcomes from different domains can help avoid over-assessing students. By ‘coding’ your iLOs by domain, you can generate interesting and engaging assessments.
It is important that whatever assessment is designed can be managed by the assessors! Acknowledging whatever degree of subjectivity is necessary within the discipline (see above) it is then incumbent o the course design team to provide guidance as to how to interpret, grade and feedback to assessments.
Marking and Designing Assessment Rubrics
Marking can sometimes be tedious. It needn’t be. If students are properly guided to generate well-structured evidence, it can be a fascinating and engaging process. In the following example, a marking rubric has been prepared for the second of the two example assessments above.
Note that the rubric used by the assessor is also shared with students at the very beginning of their course. This is important. This transparency removes any mystery from the assessment process. This means that when you design your course, you should already have designed the guidance markers that you will use. If you are marking to the ILO, you can provide feedback on work against each specific ILO using the description of the expected threshold content in the table.
Note also that this should be the ONLY marking guidance needed. There should be no need for different guidance for students and assessors, and certainly no need for further guidance to assessors. While the assessment briefs (the ‘questions’) do not have to appear in your module and programme validation documents, I suggest your marking rubrics should.
Examine the rubric above (click to enlarge) and consider how extensive the modifications to the rubric would need to be (if any) as the assessment brief changes, while remaining aligned with these same ILOs!
Final Questions for your Course Team to reflect on…
| How valid is your assessment? Are your ILOs meaningfully structured and are they being assessed? Are students able to evidence attainment against the ILOs? |
| How reliable is your assessment? Are you able to modify the details of the assessment whilst retaining aligning to the ILOs and a consistent rubric? |
| How much assessment is required? How many ILOs can or should be assessed by an individual piece of assessment? |
| How many domains are reflected in your ILOs that can be combined is assessments? |
| How creative can you be in aligning your assessment and ILOs to external guidelines, standards or competency frameworks |
| Undertake a mapping exercise of all module ILOs and assessments within a programme, exploring the weighting and timing of submissions. |
In the next stage of the 8-SLDF, we explore learning and teaching activities
References
Handley, K., Bryant, R., Rust, C., O’Donovan, B., & Price, M. (2013). Assessment Literacy: The Foundation for Improving Student Learning. Oxford Centre for Staff and Learning Development.
Pokorny, H. (2016). Enhancing Teaching Practice in Higher Education. (D. Warren, Ed.). SAGE Publications Ltd.
Stevens, D. D., Levi, A. J., & Walvoord, B. E. (2013). Introduction to Rubrics: An Assessment Tool to Save Grading Time, Convey Effective Feedback, and Promote Student Learning. Stylus Publishing.