Lärare: Viggo Kann and Anna-Karin Högfeldt Handledare: Svea Ekelin Högskoleverkets kvalitetskonferens i Umeå 2007 Högskoleverkets kvalitetskonferens i Umeå 2007 2007-10-11 2007-10-11 LH216V Utveckla lärandet med betygskriterier, Träff1 Lärare: Viggo Kann and Anna-Karin Högfeldt Handledare: Svea Ekelin viggo@nada.kth.se; akhog@kth.se Viggo Kann: Målrelaterade betygskriterier kräver modifierad examination Viggo Kann: Målrelaterade betygskriterier kräver modifierad examination 1 1

Dag 1 12.15-12.35 Välkomna och introduktion 12.35-12.50 Varför betygskriterier? 13.00-14.00 Exempel på betygskriterier, Viggo Kann 14.15-14.45 Prova på! 14.45-15.00 Kursinformation, webbregistrering och miniutvärdering

Betygskriterier/Grade Descriptors / Grading Criteria / Grading Rubrics Högskoleverkets kvalitetskonferens i Umeå 2007 Högskoleverkets kvalitetskonferens i Umeå 2007 2007-10-11 2007-10-11 Betygskriterier/Grade Descriptors / Grading Criteria / Grading Rubrics Prestationsnivå i relation till förväntade lärandemål /Levels of achievements in relation to the intended learning outcomes Viggo Kann: Målrelaterade betygskriterier kräver modifierad examination Viggo Kann: Målrelaterade betygskriterier kräver modifierad examination 3 3

Vad är det? Obs att detta är ett utdrag ur hel kurs för att visa på tanken med kriterierna! Grading criteria for the Advanced Pavement Engineering Analysis and Design course (AH2905). Example with one ILO ILO1: Carry out mechanical analyses of pavement structures, report the analyses results to the client, formulate and motiative assumptions and approximations used A B C D E [B criterion included] using out of the course material; implications of using different analysis strategies are evaluated Note that this is a narrowed example with only one ILO [C criterion included] with alternative analysis strategies proposed based on the course material [D criterion included] for a more involved technical problem [assessment task] With analysis approach, assumptions and approximations justified based on the course material With a given hint on analysis approach; assumptions and approximations thoroughly documented

Varför använda betygskriterier? Högskoleverkets kvalitetskonferens i Umeå 2007 Högskoleverkets kvalitetskonferens i Umeå 2007 2007-10-11 2007-10-11 Varför använda betygskriterier? Diskutera möjligheter, utmaningar och hinder. Utgå gärna från olika perspektiv …studenterna …kursansvarig/examinator …lärare i andra kurser / programansvariga / GA / studierektorer… …avnämare/potentiella arbetsgivare Redovisa på tavlan efteråt! Viggo Kann: Målrelaterade betygskriterier kräver modifierad examination Viggo Kann: Målrelaterade betygskriterier kräver modifierad examination 5 5

Högskoleverkets kvalitetskonferens i Umeå 2007 2007-10-11 2007-10-11 När betygskriterier tillämpas brukar det ofta leda till förändringar på examinationen och lärandemålen Betygskriterier behöver vara integrerat med examinationen och bedömningen. Annars blir de lätt ett spel för gallerierna (gesture politics). Viggo Kann: Målrelaterade betygskriterier kräver modifierad examination Viggo Kann: Målrelaterade betygskriterier kräver modifierad examination 6 6

Välj ut ett lärandemål från en kurs Välj ut ett lärandemål från en kurs. Diskutera med 1-2 pers hur det här lärandemålet kan uppnås/bedömas i olika nivåer (om du har A-F). Om du har P/F, kan du gå djupare och diskutera specifika kriterier för att säkerställa att lärandemålet är uppfyllt. Prova på!

Taxonomies for learning Bloom, Feisel-Schmitz, SOLO, … Taxonomies for learning

Feisel-Schmitz Technical Taxonomy Judge: (värdera) To be able to critically evaluate multiple solutions and select an optimum solution Solve: (lösa problem) Characterize, analyze, and synthesize to model a system (provide appropriate assumptions) Explain: (förklara) Be able to state the outcome/concept in their own words Compute: (räkna typtal) Follow rules and procedures (substitute quantities correctly into equations and arrive at a correct result, Plug & Chug) Define: (återge) State the definition of the concept or is able to describe in a qualitative or quantitative manner

SOLO taxonomy Biggs p. 79

Bloom’s taxonomy [Bloom 1956]

Bloom’s revised taxonomy in 2 dim. Krathwohl (2002) The Cognitive Process Dimension The Knowledge Dimension 1. Remember 2. Understand 3. Apply 4. Analyze 5. Evaluate 6. Create A. Factual Knowledge B. Conceptual Knowledge C. Procedural Knowledge D. Metacognitive Knowledge Different forms of learning Different types of knowledge

The Knowledge Dimension The Cognitive Process Dimension The Knowledge Dimension 1. Remember 2. Understand 3. Apply 4. Analyze 5. Evaluate 6. Create A. Factual Knowledge Aa. Knowledge of terminology Ab. Knowledge of specific details and elements B. Conceptual Knowledge Ba. Knowledge of classifications and categories Bb. Knowledge of principles and generalizations Bc. Knowledge of theories, models, and structures C. Procedural Knowledge Ca. Knowledge of subject-specific skills and algorithms Cb. Knowledge of subject-specific techniques and methods Cc. Knowledge of criteria for determining when to use appropriate procedures D. Metacognitive Knowledge Da. Strategic knowledge Db. Knowledge about cognitive tasks, including appropriate contextual and conditional knowledge Dc. Self-knowledge

The Cognitive Process Dimension 1. Remember 1.1 Recognizing The Knowledge Dimension 1. Remember 1.1 Recognizing 1.2 Recalling 2. Understand 2.1 Interpreting 2.2 Exemplifying 2.3 Classifying 2.4 Summarizing 2.5 Inferring 2.6 Comparing 2.7 Explaining 3. Apply 3.1 Executing 3.2 Implementing 4. Analyze 4.1 Differentiating 4.2 Organizing 4.3 Attributing 5. Evaluate 5.1 Checking 5.2 Critiquing 6. Create 6.1 Generating 6.2 Planning 6.3 Producing A. Factual Knowledge B. Conceptual Knowledge C. Procedural Knowledge D. Metacognitive Knowledge

1-5: ILOs X: Learning activity E: Examination

Kursinformation Kurs-PM: https://kth.instructure.com/courses/287/pages/kurs-pm?module_item_id=6433

Inlämn.uppg 1a. Deadline före träff 2 https://kth.instructure.com/courses/287/assignments Inlämn.uppg 1a. Deadline före träff 2

Uppgift 1 – Designa betygskriterier för din kurs Följa gärna arbetsgången nedan. Summera i en kort rapport som även beskriver kursen kortfattat. 1. Börja med att skriva om lärandemålen om det behövs. 2. Om du bara har/tänker ha ett enda examinationsmoment i kursen och vill skriva holistiska betygskriterier: hoppa till steg 5! 3. Bryt ner lärandemålen i delmål så att varje delmål examineras på samma sätt i (den tänkta examinationen av) kursen. Notera hur varje delmål examineras.

5. Utforma betygskriterier för E (godkäntnivån) för varje delmål. 4. Gör en matris som visar vilka mål som examineras var (se ADK-exemplet). 5. Utforma betygskriterier för E (godkäntnivån) för varje delmål. 6. Markera vilka delmål som inte behöver ha kriterier på högre nivå än E. 7. Titta igenom gamla examinationsuppgifter och fundera över vad som brukar krävas för olika betyg. Vad borde krävas? 8. Utforma betygskriterier för A-nivån för varje delmål. Om du anser att ett delmål inte behöver ha kriterier på högre nivå än till exempel E så skriver du kriterier för den högsta nivån som används! 9. Utforma betygskriterier för C-nivån. 10. Skissa på hur betygskriterierna ska användas i examinationen. Kan flera delmål examineras med samma uppgift? Behövs flera uppgifter för att examinera ett delmål? Behövs olika uppgifter för olika betygskriterier?

Miniutvärdering

Vi ses på träff 2! Dag 2 studie om betygskriterier, betygskriterier i din egen kurs, testning och granskning av betygskriterier, sammanvägning av betyg, fallgropar och tips

Analyzing and ”fixing” the learning outcomes – this final part of the slides is not included in the lecture but can be useful when analyzing your ILOs. [All steps for the design of grading criteria are found in instructions for Assignment 1, and details/examples will be given by Viggo]

What is Intended Learning Outcomes (ILOs)? “Objectives that describe intended learning outcomes as the result of instruction are usually framed in terms of (a) some subject matter content and (b) a description of what is to be done with or to that content.” (Krathwohl, 2002) Example: The student should be able to develop and implement algorithms with data structures and analyze them with respect to correctness and efficiency

The objectives define the course To the course teachers To the study program To the students For mobility and comparison between institutions For employers

Criteria for the formulation of learning outcomes RESULT ORIENTED OBSERVABLE HINTING AT THE LEVEL OF UNDERSTANDING AND REALISTIC BASIS FOR ASSESSMENT, PLANNING OF TEACHING, FOLLOW-UP AND QUALITY ASSURANCE

A. RESULT ORIENTED Not a learning outcome, but important in course description The course gives an overview of the development of history of technology from the 70’s to today, and the students have the possibility to specialize within a chosen area Possible aspects to consider for criteria To whom? Somebody with non-technical background? In what ways? How many important steps to present? What needs? What parts of society? Deep investigation or a wide approach? Examples on ILOs Students should be able to Summarize the development of the history of technology from 1970 to 2014 Relate the needs of society to technical solutions

An ILO is result oriented when The student is the subject, not the course When you’re not telling what you are doing within the course, but as a result of the course

There is nothing wrong with wanting the students to understand There is nothing wrong with wanting the students to understand. But how do we know that the students understand? B. OBSERVABLE The students shall understand different processes for product development BUT, you can keep these specifications to the different levels in the grading criteria! A possible learning outcome, depending on what you mean with understand The student shall be able to describe, compare and critically examine different product development processes as well as their properties

An ILO is OBSERVABLE when It describes what observable performancies the students should show as reasonable ”proofs” for understanding

C. Hinting at the level of understanding and Realistic Pretty vague on what level we are talking about The student must be able to describe properties of the material Many of these aspects can be used in the grading criteria instead of the ILO The student shall be able to - explain the properties of materials in relation to chemical binding - choose the appropriate materials for a given construction with regard to function, conditions of operation, economical and environmental factors A possible learning outcome, depending on what you mean

The level of understanding The context and the work carried out The contributions the course makes to the study program, progression and red thread

Taxonomies for learning Bloom, Feisel-Schmitz, SOLO, … Taxonomies for learning

Feisel-Schmitz Technical Taxonomy Judge: (värdera) To be able to critically evaluate multiple solutions and select an optimum solution Solve: (lösa problem) Characterize, analyze, and synthesize to model a system (provide appropriate assumptions) Explain: (förklara) Be able to state the outcome/concept in their own words Compute: (räkna typtal) Follow rules and procedures (substitute quantities correctly into equations and arrive at a correct result, Plug & Chug) Define: (återge) State the definition of the concept or is able to describe in a qualitative or quantitative manner

SOLO taxonomy Biggs p. 79

Bloom’s taxonomy [Bloom 1956]

Bloom’s revised taxonomy in 2 dim. Krathwohl (2002) The Cognitive Process Dimension The Knowledge Dimension 1. Remember 2. Understand 3. Apply 4. Analyze 5. Evaluate 6. Create A. Factual Knowledge B. Conceptual Knowledge C. Procedural Knowledge D. Metacognitive Knowledge Different forms of learning Different types of knowledge

The Knowledge Dimension The Cognitive Process Dimension The Knowledge Dimension 1. Remember 2. Understand 3. Apply 4. Analyze 5. Evaluate 6. Create A. Factual Knowledge Aa. Knowledge of terminology Ab. Knowledge of specific details and elements B. Conceptual Knowledge Ba. Knowledge of classifications and categories Bb. Knowledge of principles and generalizations Bc. Knowledge of theories, models, and structures C. Procedural Knowledge Ca. Knowledge of subject-specific skills and algorithms Cb. Knowledge of subject-specific techniques and methods Cc. Knowledge of criteria for determining when to use appropriate procedures D. Metacognitive Knowledge Da. Strategic knowledge Db. Knowledge about cognitive tasks, including appropriate contextual and conditional knowledge Dc. Self-knowledge

The Cognitive Process Dimension 1. Remember 1.1 Recognizing The Knowledge Dimension 1. Remember 1.1 Recognizing 1.2 Recalling 2. Understand 2.1 Interpreting 2.2 Exemplifying 2.3 Classifying 2.4 Summarizing 2.5 Inferring 2.6 Comparing 2.7 Explaining 3. Apply 3.1 Executing 3.2 Implementing 4. Analyze 4.1 Differentiating 4.2 Organizing 4.3 Attributing 5. Evaluate 5.1 Checking 5.2 Critiquing 6. Create 6.1 Generating 6.2 Planning 6.3 Producing A. Factual Knowledge B. Conceptual Knowledge C. Procedural Knowledge D. Metacognitive Knowledge

1-5: ILOs X: Learning activity E: Examination

E. BASIS FOR ASSESSMENT, PLANNING OF TEACHING, FOLLOW-UP AND QUALITY ASSURANCE

What work must the students do in order to reach the desired outcomes? What should the students be able to do as a result by the course? Learning outcomes Constructive alignment [Biggs] Activities Assessment What work must the students do in order to reach the desired outcomes? What should the students do in order to prove that they have reached the learning outcomes?

Time to analyze your ILOs RESULT ORIENTED OBSERVABLE HINTING AT THE LEVEL OF UNDERSTANDING AND REALISTIC BASIS FOR ASSESSMENT, PLANNING OF TEACHING, FOLLOW-UP AND QUALITY ASSURANCE