Have you got some time for professional development over the holiday period? Or do you have colleagues or design teams working on course designs over the holiday period?
Anyone who has ever tried to assess or teach to poorly learning outcomes, and then tried to defend their practices or results, will tell you that getting it right at the offset saves a huge amount of effort and heartache.
Intended Learning Outcomes are the foundations of any sound well-aligned course and programme design. Being able to create effective well-structured learning outcome is a valuable skill required of all learning designers, faculty and quality officers.
I have created a short, self-study, course hosted on a Moodle instance. The full course will take between up to 10 hours at a leisurely pace but is designed to allow you to navigate your way through it as you please. You are welcome to dip in and out. The course complements the book ‘Writing Good Learning Outcomes and Objectives’. (https://www.amazon.com/dp/0473657929/)
Join the free course entitled ‘Designing Effective Intended Learning Outcomes’ at https://sijen.net/courses
This post is a shortened version of a new resource posted about designing effective ILOs available here.
ILOs are the detailed explanations, written in language the students will understand before beginning the module or programme, as to what they will be able to DO when they have successfully completed the learning.
Many quality assurance structures, institutional or external, require Programme and Module Specifications contain details of the ‘intended learning outcomes’ (ILOs) of the programme of study. ILOs serve to provide students with a ‘checklist’ of the types of skills, attributes, abilities or competencies they should be able to evidence through successfully completing the module or programme.
Intended Learning Outcomes should not be seen as a straight-jacket for faculty. Rather, if they are well written, they should provide scaffolding for creativity in teaching and assessment.
Most teachers can identify any number of unintended learning outcomes, depending on the character of the cohort, the changing context in which learning takes place or the emergent nature of the discipline. However, the ILOs are the facets of learning that will be assessed. They should be written knowing that these are the capabilities that will be assessed, not the content knowledge.
Read more for guidance on how to structure effective ILOs that cover all dimensions of learning.
We hear much about the changing world of work and how slow higher and professional education is to respond. So in an increasingly competitive global market of Higher Apprenticeships and work-based learning provision I began to take a particular interest in students’ ‘graduateness’. What had begun as an exploratory look for examples of intended learning outcomes (ILO) with ’employability’ in mind ended up as this critical review published in an article entitled ‘Graduate Competencies, Employability and Educational Taxonomies: Critique of Intended Learning Outcomes’ in the journal called Practice and Evidence of the Scholarship of Teaching and Learning in Higher Education.
I randomly identified 20 UK institutions, 80 undergraduate modules and examined their ILOs. This resulted in 435 individual ILOs being taken by students in current modules (academic year 2014-2015) across different stages of their undergraduate journey (ordinarily in the UK this takes place over three years through Levels 4,5 and 6). This research reveals the lack of specificity of ILOs in terms of skills, literacies and graduates attributes that employers consistently say they want from graduates
The data in the table below from the full paper which describes the post-analysis attribution of ILOs to domains of educational objectives (see paper for methodology) which I found rather surprising. The first surprise was the significant percentage of ILOs which are poorly structured, given the weight of existing practice guidance and encouragement for learning designers and validators (notably from the UK Higher Education Academy and the UK Quality Assurance Agency). Some 94 individual ILOs (21.6%) had no discernible active verbs in their construction. 64 ILOs (14.7%) did not contain any meaningful verbs so could not be mapped to any educational domain. This included the infamous ‘to understand’ and ‘to be aware of’. So as a result only 276 ILOs (64%) were deemed ‘well-structured’ and were then mapped against four domains of educational objectives.
Table 8. Post-analysis attribution of ILOs to Domains of Educational Objectives
Level 4
Level 5
Level 6
Total
Knowledge(Subject Knowledge)
14
5
11
30
Cognitive (Intellectual Skills)
46
91
61
198
Affective(Professional Skills)
1
4
1
6
Psychomotor(Practical/Transferable Skills)
12
18
13
43
No Verbs
35
32
27
94
Not classifiable
23
30
11
64
Totals
131
180
125
435
Remember what I had been originally looking for were examples of ILOs that represented skills that the literature on employability and capabilities suggested should be there. These could have been anticipated to be those in the affective or psychomotor domains.
So it was rather surprising to see that of the 64% of the full sample that was codeable, sizeable percentage were cognitive (45.4%), a relatively small percentage fell into the psychomotor domain (9.8%), even less into the knowledge domain (6.8%) and a remarkably small number could be deemed affective (1.4%).
I say remarkable because the affective domain, sometimes detailed as personal and professional skills, are very much the skills that employers (and most graduates) prize above all else. These refer to the development of values and the perception of values, including professionalism, inter-personal awareness, timeliness, ethics, inter-cultural sensitivity, and diversity and inclusivity issues.
Apparently despite all the sterling work going on in our libraries and career services, employment-ready priorities within programmes and modules in higher education, are not integrated with teaching and learning practices. I suggest that as a consequence, this makes it difficult for students to extract, from their learning experience within modules, the tangible skill development required of them as future employees.
There is an evident reliance by module designers on the cognitive domain most commonly associated at a lower level with ‘knowing and understanding’ and at a higher level as ‘thinking and intellectual skills’. The old favourite ‘to critically evaluate’ and ‘to critically analyse’ are perennial favourites.
There is much more to the picture than this single study attempts to represent but I think it is remarkable not more attention is being paid to the affective and psychomotor domains in module creation.
More analysis, and further data collection will be done, to explore the issue at programme level and stage outcomes (is it plausible that module ILOs are simply not mapped and unrelated and all is well at programme level). I would also be interested to explore the mapping of module and programme ILOs to specified graduate attributes that many institutions make public.
I go on in the full paper about the relative balance of different ILOs in each of the domains depending on the nature of the learning, whether it is a clinical laboratory module or a fieldwork module or a literature-based module.
The reason I think this is important, and I have written here before, that this important (it is about semantics!), is that students are increasingly demanding control over their choices, their options, the shape of their portfolios, their ‘graduateness’, and they need to be able to identify their own strengths and weaknesses and make meaningful modules choices to modify the balance of the skills acquired in a ’practical’ module compared with those in a ‘cerebral’ one. I conclude that the ability to consciously build a ‘skills profile’ is a useful graduate attribute in itself…. which incidentally would be an affective ILO.
Full citation:
Atkinson, S. 2015 Jul 9. Graduate Competencies, Employability and Educational Taxonomies: Critique of Intended Learning Outcomes. Practice and Evidence of the Scholarship of Teaching and Learning in Higher Education [Online] 10:2. Available: http://community.dur.ac.uk/pestlhe.learning/index.php/pestlhe/article/view/194/281
Whilst the majority of writings and reflection concerning the use of taxonomies of educational objectives remain focused around the cognitive domain, typified by Bloom (1984), there is a growing attention being paid to the affective domain, particularly in professional education. Bloom’s now famous research project which resulted in Book 2 of the ‘Taxonomy of Educational Objectives: the affective domain’ led by Krathwohl, which has been much neglected, applied scantily (and often erroneously) to practice (Krathwohl, Bloom, & Masia, 1999).
Often described in terms of feelings or emotions I think it more useful to regard the affective as being a question of degree of acceptance or rejection of values, of the evolution of value structures. It is certainly true that having one’s values challenged can result in emotional disturbance (Mezirow, 2000) but I think there is a danger to using language that describes the domain in terms of emotions, not least because it scares off academics!
In my interpretation of the affective domain, illustrated above as a circular representation, I have retained Krathwohl’s five divisions but choose to use active verbs rather than the passive. So receiving becomes to receive, responding becomes to respond, valuing becomes to value, organization becomes to organize, and the final division originally entitled characterization becomes to internalize.
Original
Atkinson
Descriptor
Sample Verbs
Receiving
To Receive
Ability to learn from others
Sense, experience, concentrate, attend, perceive
Responding
To Respond
Ability to participate responsibly , respectively as appropriate to the context
Perform, contribute, satisfy, cite
Valuing
To Value
Ability to associate personal and collective values with contextual experiences and express value judgements
Justify, seek, respect, persuade
Organization
To Organize
Ability to structure, prioritize and reconcile personal and others’ value systems
Clarify, reconcile, integrate
Characterization
To Internalize
Ability to articulate one’s own values and belief systems and operate consistently within them
Conclude, internalize, resolve, embody
The reason I think it helpful to think about values rather than emotions is that clearly most of higher and professional education, is concerned by changing not just how much or what students know, or even how they know and apply that knowledge, but with how they ‘feel’ about knowing. At its simplest we seek to instil a love of learning and a passion for the subject. In professional education we also seek to instil our professional values into the learning process. Whilst it is clearly very difficult to evaluate the emotional impact that learning has on students there are ways of providing formative assessment to support affective developments.
For those in the Academy who are seeking to merely perpetuate their academic DNA in their students, the latters’ changes in values may not mean a great deal to them. For those of us who teach in order to make the next generation better than we are, better able to adapt to the ever-changing world that they face outside of the Academy, then having an interest in our students affective development is extremely important.
I have argued elsewhere that the relative weighting given in learning design to the domains depends largely on the subject and the context of learning. Clearly there will be foundational modules in a degree programme in which knowledge domain learning will be dominant. I would expect much that is done in an undergraduate degree to be concerned with the cognitive domain, clearly an ability to analyse, evaluate, synopsise and synthesise represent the higher-order skills we expect from graduates on graduation. I have also argued elsewhere that psychomotor skills are also worthy of being part of higher education. But it occurs to me that much of the learning opportunity offered in our current universities neglect and equally important set of skills.
Almost all employers agree that they want to attract applicants who share their values. These oft cited idealized values are in fact widely held; the ability to work well with others, to be an effective communicator, to be an effective listener, to work independently, to take the initiative. It seems unrealistic to expect students to necessarily acquire such skills without being guided through the learning process and taught to identify their own development. And it is fair to say that certainly in the United Kingdom sector a huge amount has been added to the curriculum, with employability strategies and planning personal development (PDP) initiatives, that students do not wont for opportunities.
But I maintain that we should ingrain in our students the values we expect them to demonstrate through the disciplines themselves, not bolt them on and relieve the academics from their repsonsability. To my mind it makes sense to write intended learning outcomes to encapsulate a range of affective outcomes and align learning and teaching activities to rehearse those skills with our students.
Why not include alongside an intellectual skills outcome (cognitive) that states “by the end of this module you will be able to critique at least three different perspectives on (whatever the subject is)” another outcome, this time an affective one that says “by the end of this module you will be able to reconcile two contrasting, and contentious, perspectives on (whatever the subject is)”. There is nothing touchy-feely about the second outcome but it focuses on the students value structures, supporting their ability to structure and reconcile personal value systems in contrast to those held by others. Critiquing sounds very higher education, but to be able ‘to reconcile’ is a much needed skill in the workplace.
Bloom, B. S. (1984). Taxonomy of Educational Objectives Book 1: Cognitive Domain (2nd edition). Addison Wesley Publishing Company.
Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1999). Taxonomy of Educational Objectives Book 2/Affective Domain (2nd edition). Longman Pub Group.
Mezirow, J. (2000). Learning as transformation : critical perspectives on a theory in progress. San Francisco: Jossey-Bass.
Four ‘Domains’ of educational objectives represented in a circular form
I think being able to visualise things is important. Faculty and learning designers need to be able to see Intended Learning Outcomes (ILOs) take shape and mant find existing lists are uninspiring. It’s not uncommon for faculty and instructional designers to get tired and weary of ILOs; they can feel restrictive, repetitive, formulaic and sometimes obstructive. In previous posts I’ve tried to suggest that the bigger picture, the challenges of effective 21st century university level learning design, make them not only useful, but also essential. If you don’t agree, don’t bother reading. I’m not going to try and persuade you. If you think there’s some truth in the argument and you want to engage with ILOs to make your teaching more focussed, your students increasingly autonomous and your graduates equipped with meaningful evidence, then I hope I have something worthwhile sharing and will welcome your thoughts.
My argument is that a module (a substantial unit of a full years undergraduate study), and the programme of which is part, should have clearly articulated outcomes in four domains:
Knowledge and understanding – or the knowledge domain
Intellectual Skills – or the cognitive domain
Professional Skills – or the affective domain
Transferable Skills – or the psychomotor domain
I’m suggesting one SHOULD expect to see a different distribution of ILOs between the outcomes in these domains depending on the focus of the module and the level of study. One might expect to see a second year anthropology module on ‘theoretical perspectives’ emphasising cognitive outcomes and a module being studied alongside it on ‘research design and techniques’ emphasising affective and psychomotor outcomes. One might reasonably expect to see more foundational ‘knowledge and understanding’ outcomes in the first year of a programme of study, and more ‘cognitive’ outcomes at the end of the programme. The lack of this ‘designed articulation’ in many modules undermines their value to the student and ultimately to faculty.
The basic principle is that an outcome should be assessable. Lots of great stuff can happen in your teaching and students’ learning that DOESN’T need to be assessed. It can be articulated in the syllabus, it just isn’t a measured outcome. A student should be able, at the end of this course of study (module or programme), to evidence that they have attained the intended learning outcomes. This evidence has been assessed in some way and the student is then able to point to the ILOs amassed throughout their programme and say “I can demonstrate that I learnt to DO this”.
Representing Taxonomies
There has been a significant shift in the language we now use from the original work in the 1950s by Bloom and colleagues. The passively descriptive language of Bloom’s Taxonomy has become the active language of Anderson and Krathwohl (Anderson & Krathwohl, 2001). The taxonomies have moved from Evaluation to Evaluate, from Analysis to Analyse. This is significant in that the emphasis has moved away from describing what the focus of the teaching is supposed to be, to the demonstrable outcomes of the learning.
The illustration above consists of four visual ‘wheels’ that I have used to discuss learning outcomes with faculty in the context of module and programme design at Massey University in New Zealand and at the LSE and BPP University College in the United Kingdom. These visual representations were inspired by work done elsewhere, on the cognitive domain in particular. The first documented example of this circular representation I have been able to find is attributed to Barbara Clark in 2002, but a great many people have since represented Bloom’s original, and the revised, cognitive domain in this way.
The circular representation has the higher level terms at the centre, proto-verbs if you will, surrounded by a series of active verbs that articulate actions an individual might undertake to generate evidence, of their ability to represent to proto-verb. The circular visualisation also serves to create a more fluid representation of the stages, or divisions, in the proto-verbs. Rather than a strict ‘step-by-step’ representation where one advances ‘up’ the proto-verbs, one might consider this almost like the dial on an old telephone, in every case one starts at the ‘foundational’ and dials-up though the stages to the ‘highest’ level. Each level relies on the previous. It may be implicit that to analyse something, one will already have acquired a sense of its application, and that application is grounded on subject knowledge and understanding. So the circle is a useful way of visualising the interconnected nature of the process. Most importantly in my practice, it’s a great catalyst for debate.
The circular representations of the domains and associated taxonomies also serve to make learning designers aware of the language they use. Can a verb be used at different levels? Certainly. Why? Because context is everything. One might ‘identify’ different rock samples in a first year geology class as part of applying a given classification of rocks to samples, or one might identify a new species of insect as part of postgraduate research programme. The verb on its own does not always denote level. I talk about the structure of ILOs in a subsequent post.
Structure of the circular representations of Educational Taxonomies
More recent representations have created new complex forms that include the outer circle illustrated here. I’ve found these rather useful, in part because they often prove contentious. If the inner circle represents (in my versions) the proto-verbs within our chosen taxonomies, and the next circle represent that active verbs used to describe the Intended Learning Outcomes (ILO) AND the Learning and Teaching Activities (TLS), the outermost circle represents the evidence and assessment forms used to demonstrate that verb. Increasingly I’ve used this to identify educational technologies and get faculty thinking more broadly about how they can assess things online as well as in more traditional settings. The outermost circle will continue to evolve as our use of educational technologies evolves. In Constructive Alignment one might reasonably expect students’ learning activity to ‘rehearse’ the skills they are ultimately to evidence in assessment (Biggs & Collis, 1982; Boud & Falchikov, 2006) and the forms to enable that are becoming increasingly varied.
Re-visioning Taxonomies
One of my favourite representations of the relationship between the knowledge dimension and the cognitive domain is from Rex Heer at Iowa State University’s Center for Excellence in Learning and Teaching (http://www.celt.iastate.edu/teaching/RevisedBlooms1.html accessed ). It’s an interactive model that articulates the relationship, as Anderson and Krathwohl saw it, rather well. My own interest, as we look to effective ILOs, is to separate out the knowledge dimension as a subject or knowledge domain and have faculty articulate this clearly for students, before reconnecting to the other domains. A process I’ll talk about subsequently.
Here are my four ‘working circles’ using adaptations of taxonomies from Anderson and Krathwohl (Knowledge and Understanding, and Cognitive), Krathwohl et al (Affective) and Dave (Psychomotor). I have adapted the Knowledge Dimension of Anderson and Krathwohl to do two things; to describe the dimension in terms of active verbs rather than as a definition of the nature of the knowledge itself, and I have incorporated a stage I believe is under represented in their articulation. I have added the ability to ‘ contextualise’ subject knowledge between the ability to specify it (Factual) and the ability to conceptualize (Conceptual). I have also rearticulated the original ‘Metacognitive’ as the ability to ‘Abstract‘. This will doubtless need further work. My intent is not to dismiss the valuable work already in evidence around the relationship between a knowledge dimension and the cognitive domain, rather it is to enable faculty, specifically when writing learning outcomes, to identify the subject, discipline or knowledge to be enabled in more meaningful ways.
These images are provided as JPG images. If you would like me to email the original PowerPoint slides (very low-tech!) so that you can edit, amend and enhance, I am happy to do so. I only ask that you enhance my practice by sharing your results with me.
I hope these provoke thought, reflection and comment. Feel free to use them with colleagues in discussion and let me know if there are enhancements you think would make them more useful to others.
Cognitive Domain – Intellectual Skills
Cognitive Domain – Intellectual Skills
Affective Domain – Professional and Personal Skills
Affective Domain – Professional and Personal Skills
Psychomotor Domain- Practical, Technical and Transferable Skills
Psychomotor Domain- Practical, Technical and Transferable Skills
Knowledge Domain – Subject and Discipline Knowledge
Knowledge Domain- Subject or Discipline Skills
The next post will illustrate the usefulness of these visualisations in drafting Intended Learning Outcomes with some examples.
………………………………………………………………………………………
Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing : a revision of Bloom’s taxonomy of educational objectives. New York: Longman.
Biggs, J. B., & Collis, K. F. (1982). Evaluating the Quality of Learning: Structure of the Observed Learning Outcome Taxonomy. Academic Press Inc.
Boud, D., & Falchikov, N. (2006). Aligning assessment with long‐term learning. Assessment & Evaluation in Higher Education, 31(4), 399–413.
