Task Performance Indicator (TPI) för Malmö högskola

En presentation över ämnet: "Task Performance Indicator (TPI) för Malmö högskola"— Presentationens avskrift:

1 Task Performance Indicator (TPI) för Malmö högskola www.mah.se
April 2016 Metodansvarig: Gerry McGovern, Kvalitetsansvarig: Fredrik Wackå, Testledare: Michael Gustafson,

2 “Task Performance Indicator” – scenariotestning
Metoden “Task Performance Indicator” – scenariotestning

3 Mätningen Testpersonen sitter vid sin dator och får uppgifter att utföra eller frågor att hitta svaret på. Användningen observeras, noteras och spelas in. Två separata syften: Få mått på hur sajten faktiskt används. Effektivitet – tidsmätning Kvalitet – framgång eller inte Attityd – förtroende för hittad lösning Lära oss vad som bör göras bättre för att förenkla användningen.

4 Resultatens trovärdighet
I den bransch som utvecklat testmetoderna (användbarhet/”usability”) betraktas 5 tester vara tillräckligt för att avslöja 80 % av bristerna i det som testas. Vi har gjort nästan tre gånger så många.

5

6 Resultatens trovärdighet vs. syftena
Mycket hög trovärdighet i användningsmönstren. Tidsmätningen ger en god fingervisning. Tolka den principiellt, inte exakt.

7 Sammanlagda resultat, alla uppgifter.

8 Task Performance Indicator (TPI)
TPI = 100% minus – % of Failures including Time Limits – % of Disasters – % (Time Penalty for Successes) – % (Confidence in Disaster Penalty) – % (Non-confidence in Success Penalty) Task Performance Indicator ranges from 0 to 100% Rating categories and score criteria Excellent (80-100) Minor Issues (50-59) Good (70-79) Major Issues (35-49) Fair (60-69) Critical Issues (0-34)

9 Time Penalty – Example of impact on TPI score
Longer task completion times reduce the TPI Excellent (80-100) Good (70-79) Fair (60-69) Minor Issues (50-59) Major Issues (35-49) Critical Issues (0-34) Task completion time as multiple of target time e.g. 3 = 3 times as long as the target time

10 Overall Task Performance
Fair (60-69) TPI = 60

11 Overall Task Performance
Fair (60-69) TPI = 60 * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

12 Overall task outcome summary (157 trials)
WR Disasters – People thought they were successful but they were seriously wrong WM Wrong / Minor – People found only a partial minor impact wrong solution to the task TL Time Limit – People took longer than the 5 minute time limit GU Gives Up – People became frustrated and gave up WS Wrong System Error – System was in error ## Success – Number indicates the time to complete the task (in seconds) SK Skipped task

13 Success Rate by Task – Ordered by Success Rate

14 Task Performance Indicator (TPI) by TPI Score
Fair (60-69) TPI = 60

15 Overall Task Performance Indicator
Fair (60-69) TPI = 60

16 TPI Performance Peer Comparison

17 Action Priority Fair (60-69) TPI = 60

18 Overall Task Confidence for Successes

19 Task Confidence for Successes by Confidence

20 Strategies Used and Transitions – Search versus Navigation All Completions
Started with Search but changed to Navigation prior to Completion Started with Navigation but changed to Search prior to Completion 157 Trials

21 Strategies Used and Transitions – Search versus Navigation All Completions – by Task

22 Strategies Used and Transitions – Search versus Navigation Successes Only
Started with Search but changed to Navigation prior to Completion Started with Navigation but changed to Search prior to Completion 98 Successful Trials

23 Strategies Used and Transitions – Search versus Navigation Successes Only – by Task

24 Completion Times by Task – Successes Only

25 Completion Times by Task – Successes, Failures, Disasters

26 Resultat per uppgift

27 9 Forskning Performance
Critical Issues (0-34) TPI = 10 Hitta vilken forskning som är publicerad av Malmö högskola kring barns utveckling. * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

28 9 Forskning TPI Breakdown
Critical Issues (0-34) TPI = 10

29 9 Forskning Completion Time Results
Critical Issues (0-34) TPI = 10

30 9 Forskning Confidence Confidence in Disasters
Critical Issues (0-34) TPI = 10 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 22%

31 9 Forskning Strategies Used
Critical Issues (0-34) TPI = 10 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

32 9 Forskning Strategies Used
Critical Issues (0-34) TPI = 10 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 3 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

33 Användningsmönster: Forskning
Christine Vallberg Roth är det första som dyker upp i Google vid sökning. Många väljer “Forskare I alfabetisk ordning”och får en lista på namn. Oklart om man vet vad MUEP är. Jag tror att man inte tänker på att man ska hitta en publikation och därför går man inte in under ”Sök publikation”. Man använder INTE browsersök! Hittade rätt: Forskning => Sök publikation => Muep: Söker på ”barns utveckling” 2 st Hittade nästan rätt: Bibliotek => Databaser a-ö => Tillbaks till Bibliotek => Publiceringar i vänster meny => Klickar på länk till MUEP 20:17 05NS Exempel på felaktigt svar : Christine Vallberg Roth

34 8 Exjobb Performance Critical Issues (0-34) När är nästa CV workshop?
TPI = 22 När är nästa CV workshop? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

35 8 Exjobb TPI Breakdown Critical Issues (0-34) TPI = 22

36 8 Exjobb Completion Time Results
Critical Issues (0-34) TPI = 22

37 8 Exjobb Confidence Confidence in Disasters
Critical Issues (0-34) TPI = 22 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 23%

38 8 Exjobb Strategies Used
Critical Issues (0-34) TPI = 22 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 13 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

39 8 Exjobb Strategies Used
Critical Issues (0-34) TPI = 22 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 3 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

40 Användningsmönster: Exjobb
Söker man på ”CV workshop” missar man helt Future finder. 404 på toplänkarna i sök.

41 Användningsmönster: Exjobb
Finns inte i kalendern (några kollade där). En som hade rätt sökte på CV Workshop datum och kom till => Karriarservice => Future Finder. Sökning på Google ”cv workshop mah” ger även rätt ingång på FF (event)

42 6 Hitta person/kontaktuppgifter Performance
Critical Issues (0-34) TPI = 34 Hitta kontaktuppgifter till en forskare som forskar om EU och Europafrågor. * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

43 6 Hitta person/kontaktuppgifter TPI Breakdown
Critical Issues (0-34) TPI = 34

44 6 Hitta person/kontaktuppgifter Completion Time Results
Critical Issues (0-34) TPI = 34

45 6 Hitta person/kontaktuppgifter Confidence
Critical Issues (0-34) TPI = 34 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 43%

46 6 Hitta person/kontaktuppgifter Strategies Used
Critical Issues (0-34) TPI = 34 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

47 6 Hitta person/kontaktuppgifter Strategies Used
Critical Issues (0-34) TPI = 34 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 6 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

48 Användningsmönster: Hitta person/kontaktuppgifter
Denna funkar om man kan browsersök. En deltagare hittade rätt genom att skanna/skumma texten vid listning av Ämnen Jag såg för övrigt en forskare från Mah som uttalade sig om Europafrågor på Nyheterna, Inge Eriksson.

49 10 Bostad Performance Major Issues (35-49)
TPI = 37 Hitta den sida där du ska registrera dig för att stå i kö till en studentlägenhet på Studeum i Västra hamnen. * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

50 10 Bostad TPI Breakdown Major Issues (35-49) TPI = 37

51 10 Bostad Completion Time Results
Major Issues (35-49) TPI = 37

52 10 Bostad Confidence Confidence in Disasters
Major Issues (35-49) TPI = 37 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 43%

53 10 Bostad Strategies Used
Major Issues (35-49) TPI = 37 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

54 10 Bostad Strategies Used
Major Issues (35-49) TPI = 37 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 6 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

55 Användningsmönster: Bostad
Förstår inte att man ska in på en annan sajt. Namnet Studeum finns inte på Boplats Syd så man missar det vid sökning där. Första sökträffen om man söker på Studeum leder till den engelska siten. En del går Start => Student => Ny student. Där finns en länk i högerspalt men den missas. En del går in på rikshem för att det står på sidan. Även om en del läste att studenter måste anmäla sig på Boplats Syd förstår man inte det. Man verkar tänka att man registrerar sig på en viss bostad

56 5 Antagningsstatistik Performance
Minor Issues (50-59) TPI = 50 Vilken poäng krävdes på högskoleprovet för att i andra urvalet bli antagen till Fastighetsförmedling på Malmö högskola hösten 2015? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

57 5 Antagningsstatistik TPI Breakdown
Minor Issues (50-59) TPI = 50

58 5 Antagningsstatistik Completion Time Results
Minor Issues (50-59) TPI = 50

59 5 Antagningsstatistik Confidence
Minor Issues (50-59) TPI = 50 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 50%

60 5 Antagningsstatistik Strategies Used
Minor Issues (50-59) TPI = 50 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

61 5 Antagningsstatistik Strategies Used
Minor Issues (50-59) TPI = 50 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 7 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

62 Användningsmönster: Antagningsstatistik
Söker man på Antagningsstatistik så får man inte uhr.se i sökträffen. Dock är första träffen bra om man förstår att man ska klicka på den sökträffen (vilket alla inte gjorde) De som lyckades snabbast gick via antagning.se. Några hittade till antagning.se via mah.se, några gick direkt till antagning.se Jag trodde att problemen skulle uppstå vid filtreringen på uhr.se men det gick bra. Det svåra var att ta sig dit.

63 Användningsmönster: Antagningsstatistik
Många gick in på programmet Fastighetsförmedling men får där ingen länk vidare till uhr.se.

64 4 Hitta program eller kurser Performance
Minor Issues (50-59) TPI = 55 Vilka heltidsprogram finns på Malmö högskola hösten 2016 för den som är intresserad av att arbeta inom sjukvården och kommer direkt från gymnasiet? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

65 4 Hitta program eller kurser TPI Breakdown
Minor Issues (50-59) TPI = 55

66 4 Hitta program eller kurser Completion Time Results
Minor Issues (50-59) TPI = 55

67 4 Hitta program eller kurser Confidence
Minor Issues (50-59) TPI = 55 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 64%

68 4 Hitta program eller kurser Strategies Used
Minor Issues (50-59) TPI = 55 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

69 4 Hitta program eller kurser Strategies Used
Minor Issues (50-59) TPI = 55 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 9 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

70 Användningsmönster: Hitta program eller kurser
Här användes lite olika strategier, ibland använder man sig av drop downen, ibland sökte man i sökfältet.

71 11 Hitta e-bok (reserv) Performance
Fair (60-69) TPI = 68 Hitta en e-bok i ämnet ”globalization” och börja ladda ner den/läsa den. * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

72 11 Hitta e-bok (reserv) TPI Breakdown
Fair (60-69) TPI = 68

73 11 Hitta e-bok (reserv) Completion Time Results
Fair (60-69) TPI = 68

74 11 Hitta e-bok (reserv) Confidence
Fair (60-69) TPI = 68 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 79%

75 11 Hitta e-bok (reserv) Strategies Used
Fair (60-69) TPI = 68 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

76 11 Hitta e-bok (reserv) Strategies Used
Fair (60-69) TPI = 68 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 11 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

77 Användningsmönster: Hitta e-bok
Olika GUI för e-böcker

78 Användningsmönster: Hitta e-bok
Man förväntar sig nog att enbart få träffar på e-böcker när man klickar på länken e-böcker. Online-ikonen är inte självklar men ingen katastrof heller.

79 7 Lediga jobb Performance
Good (70-79) TPI = 77 Vem ska du prata med om du vill veta mer kring den lediga tjänsten som ”kommunikatör inom internkommunikation”? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

80 7 Lediga jobb TPI Breakdown
Good (70-79) TPI = 77

81 7 Lediga jobb Completion Time Results
Good (70-79) TPI = 77

82 7 Lediga jobb Confidence
Good (70-79) TPI = 77 No Data Confidence in Disasters Disaster Rate = 7% Non- Confidence in Successes Success Rate = 79%

83 7 Lediga jobb Strategies Used
Good (70-79) TPI = 77 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

84 7 Lediga jobb Strategies Used
Good (70-79) TPI = 77 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 11 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

85 Användningsmönster: Lediga jobb
Flera har gått till medarbetarportalen men kommit rätt till slut. Antingen via sök “Lediga jobb” eller via navigering. En hittade rätt via sidfotslänken. En hittade inte alls utan skulle ha ringt växeln.

86 3 Utbildningsplan Performance
Excellent (80-100) TPI = 84 Vad läser du under den första kursen på termin 3 i programmet ”Visuell Kommunikation” * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

87 3 Utbildningsplan TPI Breakdown
Excellent (80-100) TPI = 84

88 3 Utbildningsplan Completion Time Results
Excellent (80-100) TPI = 84

89 3 Utbildningsplan Confidence
Excellent (80-100) TPI = 84 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 86%

90 3 Utbildningsplan Strategies Used
Excellent (80-100) TPI = 84 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

91 3 Utbildningsplan Strategies Used
Excellent (80-100) TPI = 84 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 12 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

92 Användningsmönster: Utbildningsplan
Här sökte man oftast på Visuell kommunikation och hamnade rätt. En person hittade till den PDF som ligger i programmets sökresultat och hittade svaret där.

93 12 Studieformer (reserv) Performance
Excellent (80-100) TPI = 90 Hitta en förklaring till vad det innebär att studera på deltid. * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

94 12 Studieformer (reserv) TPI Breakdown
Excellent (80-100) TPI = 90

95 12 Studieformer (reserv) Completion Time Results
Excellent (80-100) TPI = 90

96 12 Studieformer (reserv) Confidence
Excellent (80-100) TPI = 90 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 93%

97 12 Studieformer (reserv) Strategies Used
Excellent (80-100) TPI = 90 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 14 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

98 12 Studieformer (reserv) Strategies Used
Excellent (80-100) TPI = 90 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 13 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

99 Användningsmönster: Studieformer
Man väljer Är student => Ny student och hittar sen inte (till en början)

100 2 Hitta litteratur Performance
Excellent (80-100) TPI = 90 Var kan du fysiskt hitta boken ”Smått och gott om vetenskapliga rapporter”? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

101 2 Hitta litteratur TPI Breakdown
Excellent (80-100) TPI = 90

102 2 Hitta litteratur Completion Time Results
Excellent (80-100) TPI = 90

103 2 Hitta litteratur Confidence
Excellent (80-100) TPI = 90 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 92%

104 2 Hitta litteratur Strategies Used
Excellent (80-100) TPI = 90 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 13 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

105 2 Hitta litteratur Strategies Used
Excellent (80-100) TPI = 90 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 12 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

106 Användningsmönster: Hitta litteratur
Möjligtvis att man inte förstod frågan.

107 1 Hitta schema Performance
Excellent (80-100) TPI = 100 I vilken lokal hålls ditt nästa utbildningstillfälle/föreläsning? * All Failure times treated as 300 seconds or 6 x Target Time (whichever is greater)

108 1 Hitta schema TPI Breakdown
Excellent (80-100) TPI = 100

109 1 Hitta schema Completion Time Results
Excellent (80-100) TPI = 100

110 1 Hitta schema Confidence
Excellent (80-100) TPI = 100 No Disasters Confidence in Disasters Disaster Rate = 0% Non- Confidence in Successes Success Rate = 100%

111 1 Hitta schema Strategies Used
Excellent (80-100) TPI = 100 SS Site Search (same as Search) SE Search External – usually Google First Strategy Used for All Completions SE Search External – usually Google 5 Trials Final Change of Strategy Used for All Completions N→S Navigation changing to Search S→N Search changing to Navigation

112 1 Hitta schema Strategies Used
Excellent (80-100) TPI = 100 SS Site Search (same as Search) SE Search External – usually Google Last Strategy Used for Successful Trials SE Search External – usually Google 5 Successful Trials Final Change of Strategy Used for Successful Trials N→S Navigation changing to Search S→N Search changing to Navigation

113 Användningsmönster: Hitta schema
Inga problem. Dock går man lite olika vägar. Via Its Learning x2 Via Google: Schema Mah x2 Via Kronox Film 5: 35:40 Via Google Schema Mah Film 8: 21:28 Via Its Learning Film 10: 06:20 Via Kronox

114 Task Performance Indicator (TPI) för Malmö högskola www.mah.se
April 2016 Metodansvarig: Gerry McGovern, Kvalitetsansvarig: Fredrik Wackå, Testledare: Michael Gustafson,