Task Performance Indicator (TPI) för Malmö högskola www.mah.se April 2016 Metodansvarig: Gerry McGovern, www.customercarewords.com Kvalitetsansvarig: Fredrik Wackå, www.wpr.se Testledare: Michael Gustafson, www.knowit.se

“Task Performance Indicator” – scenariotestning Metoden “Task Performance Indicator” – scenariotestning

Mätningen Testpersonen sitter vid sin dator och får uppgifter att utföra eller frågor att hitta svaret på. Användningen observeras, noteras och spelas in. Två separata syften: Få mått på hur sajten faktiskt används. Effektivitet – tidsmätning Kvalitet – framgång eller inte Attityd – förtroende för hittad lösning Lära oss vad som bör göras bättre för att förenkla användningen.

Resultatens trovärdighet I den bransch som utvecklat testmetoderna (användbarhet/”usability”) betraktas 5 tester vara tillräckligt för att avslöja 80 % av bristerna i det som testas. Vi har gjort nästan tre gånger så många.

Resultatens trovärdighet vs. syftena Mycket hög trovärdighet i användningsmönstren. Tidsmätningen ger en god fingervisning. Tolka den principiellt, inte exakt.

Sammanlagda resultat, alla uppgifter.

Task Performance Indicator (TPI) TPI = 100% minus – % of Failures including Time Limits – % of Disasters – % (Time Penalty for Successes) – % (Confidence in Disaster Penalty) – % (Non-confidence in Success Penalty) Task Performance Indicator ranges from 0 to 100% Rating categories and score criteria Excellent (80-100) Minor Issues (50-59) Good (70-79) Major Issues (35-49) Fair (60-69) Critical Issues (0-34)

Time Penalty – Example of impact on TPI score Longer task completion times reduce the TPI Excellent (80-100) Good (70-79) Fair (60-69) Minor Issues (50-59) Major Issues (35-49) Critical Issues (0-34) Task completion time as multiple of target time e.g. 3 = 3 times as long as the target time

Overall Task Performance Fair (60-69) TPI = 60

Overall Task Performance Fair (60-69) TPI = 60 * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

Overall task outcome summary (157 trials) WR Disasters – People thought they were successful but they were seriously wrong WM Wrong / Minor – People found only a partial minor impact wrong solution to the task TL Time Limit – People took longer than the 5 minute time limit GU Gives Up – People became frustrated and gave up WS Wrong System Error – System was in error ## Success – Number indicates the time to complete the task (in seconds) SK Skipped task

Success Rate by Task – Ordered by Success Rate

Task Performance Indicator (TPI) by TPI Score Fair (60-69) TPI = 60

Overall Task Performance Indicator Fair (60-69) TPI = 60

TPI Performance Peer Comparison

Action Priority Fair (60-69) TPI = 60

Overall Task Confidence for Successes

Task Confidence for Successes by Confidence

Strategies Used and Transitions – Search versus Navigation All Completions Started with Search but changed to Navigation prior to Completion Started with Navigation but changed to Search prior to Completion 157 Trials

Strategies Used and Transitions – Search versus Navigation All Completions – by Task

Strategies Used and Transitions – Search versus Navigation Successes Only Started with Search but changed to Navigation prior to Completion Started with Navigation but changed to Search prior to Completion 98 Successful Trials

Strategies Used and Transitions – Search versus Navigation Successes Only – by Task

Completion Times by Task – Successes Only

Completion Times by Task – Successes, Failures, Disasters

Resultat per uppgift

9 Forskning Performance Critical Issues (0-34) TPI = 10 Hitta vilken forskning som är publicerad av Malmö högskola kring barns utveckling. * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

9 Forskning TPI Breakdown Critical Issues (0-34) TPI = 10

9 Forskning Completion Time Results Critical Issues (0-34) TPI = 10

9 Forskning Confidence Confidence in Disasters Critical Issues (0-34) TPI = 10 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 22%

9 Forskning Strategies Used Critical Issues (0-34) TPI = 10 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

9 Forskning Strategies Used Critical Issues (0-34) TPI = 10 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 3 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Forskning Christine Vallberg Roth är det första som dyker upp i Google vid sökning. Många väljer “Forskare I alfabetisk ordning”och får en lista på namn. Oklart om man vet vad MUEP är. Jag tror att man inte tänker på att man ska hitta en publikation och därför går man inte in under ”Sök publikation”. Man använder INTE browsersök! Hittade rätt: Forskning => Sök publikation => Muep: Söker på ”barns utveckling” 2 st Hittade nästan rätt: Bibliotek => Databaser a-ö => Tillbaks till Bibliotek => Publiceringar i vänster meny => Klickar på länk till MUEP 20:17 05NS Exempel på felaktigt svar : Christine Vallberg Roth

8 Exjobb Performance Critical Issues (0-34) När är nästa CV workshop? TPI = 22 När är nästa CV workshop? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

8 Exjobb TPI Breakdown Critical Issues (0-34) TPI = 22

8 Exjobb Completion Time Results Critical Issues (0-34) TPI = 22

8 Exjobb Confidence Confidence in Disasters Critical Issues (0-34) TPI = 22 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 23%

8 Exjobb Strategies Used Critical Issues (0-34) TPI = 22 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 13 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

8 Exjobb Strategies Used Critical Issues (0-34) TPI = 22 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 3 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Exjobb Söker man på ”CV workshop” missar man helt Future finder. 404 på toplänkarna i sök.

Användningsmönster: Exjobb Finns inte i kalendern (några kollade där). En som hade rätt sökte på CV Workshop datum och kom till http://www.mah.se/gps/praktik => Karriarservice => Future Finder. Sökning på Google ”cv workshop mah” ger även rätt ingång på FF (event)

6 Hitta person/kontaktuppgifter Performance Critical Issues (0-34) TPI = 34 Hitta kontaktuppgifter till en forskare som forskar om EU och Europafrågor. * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

6 Hitta person/kontaktuppgifter TPI Breakdown Critical Issues (0-34) TPI = 34

6 Hitta person/kontaktuppgifter Completion Time Results Critical Issues (0-34) TPI = 34

6 Hitta person/kontaktuppgifter Confidence Critical Issues (0-34) TPI = 34 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 43%

6 Hitta person/kontaktuppgifter Strategies Used Critical Issues (0-34) TPI = 34 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

6 Hitta person/kontaktuppgifter Strategies Used Critical Issues (0-34) TPI = 34 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 6 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Hitta person/kontaktuppgifter Denna funkar om man kan browsersök. En deltagare hittade rätt genom att skanna/skumma texten vid listning av Ämnen Jag såg för övrigt en forskare från Mah som uttalade sig om Europafrågor på Nyheterna, Inge Eriksson.

10 Bostad Performance Major Issues (35-49) TPI = 37 Hitta den sida där du ska registrera dig för att stå i kö till en studentlägenhet på Studeum i Västra hamnen. * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

10 Bostad TPI Breakdown Major Issues (35-49) TPI = 37

10 Bostad Completion Time Results Major Issues (35-49) TPI = 37

10 Bostad Confidence Confidence in Disasters Major Issues (35-49) TPI = 37 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 43%

10 Bostad Strategies Used Major Issues (35-49) TPI = 37 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

10 Bostad Strategies Used Major Issues (35-49) TPI = 37 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 6 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Bostad Förstår inte att man ska in på en annan sajt. Namnet Studeum finns inte på Boplats Syd så man missar det vid sökning där. Första sökträffen om man söker på Studeum leder till den engelska siten. En del går Start => Student => Ny student. Där finns en länk i högerspalt men den missas. En del går in på rikshem för att det står på sidan. Även om en del läste att studenter måste anmäla sig på Boplats Syd förstår man inte det. Man verkar tänka att man registrerar sig på en viss bostad

5 Antagningsstatistik Performance Minor Issues (50-59) TPI = 50 Vilken poäng krävdes på högskoleprovet för att i andra urvalet bli antagen till Fastighetsförmedling på Malmö högskola hösten 2015? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

5 Antagningsstatistik TPI Breakdown Minor Issues (50-59) TPI = 50

5 Antagningsstatistik Completion Time Results Minor Issues (50-59) TPI = 50

5 Antagningsstatistik Confidence Minor Issues (50-59) TPI = 50 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 50%

5 Antagningsstatistik Strategies Used Minor Issues (50-59) TPI = 50 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

5 Antagningsstatistik Strategies Used Minor Issues (50-59) TPI = 50 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 7 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Antagningsstatistik Söker man på Antagningsstatistik så får man inte uhr.se i sökträffen. Dock är första träffen bra om man förstår att man ska klicka på den sökträffen (vilket alla inte gjorde) De som lyckades snabbast gick via antagning.se. Några hittade till antagning.se via mah.se, några gick direkt till antagning.se Jag trodde att problemen skulle uppstå vid filtreringen på uhr.se men det gick bra. Det svåra var att ta sig dit.

Användningsmönster: Antagningsstatistik Många gick in på programmet Fastighetsförmedling men får där ingen länk vidare till uhr.se.

4 Hitta program eller kurser Performance Minor Issues (50-59) TPI = 55 Vilka heltidsprogram finns på Malmö högskola hösten 2016 för den som är intresserad av att arbeta inom sjukvården och kommer direkt från gymnasiet? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

4 Hitta program eller kurser TPI Breakdown Minor Issues (50-59) TPI = 55

4 Hitta program eller kurser Completion Time Results Minor Issues (50-59) TPI = 55

4 Hitta program eller kurser Confidence Minor Issues (50-59) TPI = 55 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 64%

4 Hitta program eller kurser Strategies Used Minor Issues (50-59) TPI = 55 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

4 Hitta program eller kurser Strategies Used Minor Issues (50-59) TPI = 55 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 9 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Hitta program eller kurser Här användes lite olika strategier, ibland använder man sig av drop downen, ibland sökte man i sökfältet.

11 Hitta e-bok (reserv) Performance Fair (60-69) TPI = 68 Hitta en e-bok i ämnet ”globalization” och börja ladda ner den/läsa den. * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

11 Hitta e-bok (reserv) TPI Breakdown Fair (60-69) TPI = 68

11 Hitta e-bok (reserv) Completion Time Results Fair (60-69) TPI = 68

11 Hitta e-bok (reserv) Confidence Fair (60-69) TPI = 68 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 79%

11 Hitta e-bok (reserv) Strategies Used Fair (60-69) TPI = 68 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

11 Hitta e-bok (reserv) Strategies Used Fair (60-69) TPI = 68 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 11 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Hitta e-bok Olika GUI för e-böcker

Användningsmönster: Hitta e-bok Man förväntar sig nog att enbart få träffar på e-böcker när man klickar på länken e-böcker. Online-ikonen är inte självklar men ingen katastrof heller.

7 Lediga jobb Performance Good (70-79) TPI = 77 Vem ska du prata med om du vill veta mer kring den lediga tjänsten som ”kommunikatör inom internkommunikation”? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

7 Lediga jobb TPI Breakdown Good (70-79) TPI = 77

7 Lediga jobb Completion Time Results Good (70-79) TPI = 77

7 Lediga jobb Confidence Good (70-79) TPI = 77 No Data Confidence in Disasters Disaster Rate = 7% Non- Confidence in Successes Success Rate = 79%

7 Lediga jobb Strategies Used Good (70-79) TPI = 77 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

7 Lediga jobb Strategies Used Good (70-79) TPI = 77 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 11 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Lediga jobb Flera har gått till medarbetarportalen men kommit rätt till slut. Antingen via sök “Lediga jobb” eller via navigering. En hittade rätt via sidfotslänken. En hittade inte alls utan skulle ha ringt växeln.

3 Utbildningsplan Performance Excellent (80-100) TPI = 84 Vad läser du under den första kursen på termin 3 i programmet ”Visuell Kommunikation” * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

3 Utbildningsplan TPI Breakdown Excellent (80-100) TPI = 84

3 Utbildningsplan Completion Time Results Excellent (80-100) TPI = 84

3 Utbildningsplan Confidence Excellent (80-100) TPI = 84 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 86%

3 Utbildningsplan Strategies Used Excellent (80-100) TPI = 84 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

3 Utbildningsplan Strategies Used Excellent (80-100) TPI = 84 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 12 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Utbildningsplan Här sökte man oftast på Visuell kommunikation och hamnade rätt. En person hittade till den PDF som ligger i programmets sökresultat och hittade svaret där.

12 Studieformer (reserv) Performance Excellent (80-100) TPI = 90 Hitta en förklaring till vad det innebär att studera på deltid. * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

12 Studieformer (reserv) TPI Breakdown Excellent (80-100) TPI = 90

12 Studieformer (reserv) Completion Time Results Excellent (80-100) TPI = 90

12 Studieformer (reserv) Confidence Excellent (80-100) TPI = 90 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 93%

12 Studieformer (reserv) Strategies Used Excellent (80-100) TPI = 90 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

12 Studieformer (reserv) Strategies Used Excellent (80-100) TPI = 90 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 13 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Studieformer Man väljer Är student => Ny student och hittar sen inte (till en början)

2 Hitta litteratur Performance Excellent (80-100) TPI = 90 Var kan du fysiskt hitta boken ”Smått och gott om vetenskapliga rapporter”? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

2 Hitta litteratur TPI Breakdown Excellent (80-100) TPI = 90

2 Hitta litteratur Completion Time Results Excellent (80-100) TPI = 90

2 Hitta litteratur Confidence Excellent (80-100) TPI = 90 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 92%

2 Hitta litteratur Strategies Used Excellent (80-100) TPI = 90 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 13 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

2 Hitta litteratur Strategies Used Excellent (80-100) TPI = 90 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 12 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Hitta litteratur Möjligtvis att man inte förstod frågan.

1 Hitta schema Performance Excellent (80-100) TPI = 100 I vilken lokal hålls ditt nästa utbildningstillfälle/föreläsning? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

1 Hitta schema TPI Breakdown Excellent (80-100) TPI = 100

1 Hitta schema Completion Time Results Excellent (80-100) TPI = 100

1 Hitta schema Confidence Excellent (80-100) TPI = 100 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 100%

1 Hitta schema Strategies Used Excellent (80-100) TPI = 100 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 5 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

1 Hitta schema Strategies Used Excellent (80-100) TPI = 100 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 5 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

Användningsmönster: Hitta schema Inga problem. Dock går man lite olika vägar. Via Its Learning x2 Via Google: Schema Mah x2 Via Kronox Film 5: 35:40 Via Google Schema Mah Film 8: 21:28 Via Its Learning Film 10: 06:20 Via Kronox

Task Performance Indicator (TPI) för Malmö högskola www.mah.se April 2016 Metodansvarig: Gerry McGovern, www.customercarewords.com Kvalitetsansvarig: Fredrik Wackå, www.wpr.se Testledare: Michael Gustafson, www.knowit.se