Modelling brain activity understanding center- embedded sentences Center for Languages and Literature.

En presentation över ämnet: "Modelling brain activity understanding center- embedded sentences Center for Languages and Literature."— Presentationens avskrift:

1 Modelling brain activity understanding center- embedded sentences Bengt.Sigurd@ling.lu.se Bengt.Nilsson@neurol.lu.se Center for Languages and Literature LUND UNIVERSITY Bengt.Sigurd@ling.lu.se Bengt.Nilsson@neurol.lu.se

2 A MULTI center-EMBEDDED sentence THE RAT THAT THE CAT THAT THE DOG CHASED BIT FLED THE RATN1FIRST NOUN THAT RELATIVE PRONOUN THE CATN2 THATRELATIVE PRONOUN THE DOGN3 CHASEDV1 FIRST TRANSITIVE VERB BITV2 TRANSITIVE VERB FLEDV3 INTRANSITIVE VERB

3 Why multiple center-embedded sentences are difficult 1.The brain has to process long noun-verb patterns, e.g. N1 N2 N3 V1 V2 V3, and combine N,V (as subj,pred,obj) correctly 2. Several (pending) subjects introduced without immediate predicates are difficult. 3. Several predicates at the end cause problems: which predicate belongs to which subject. 4. Every subj-pred split by another subj-pred increases the difficulty. The total difficulty can be predicted in a computerized grammar.

4 A PET study, Stroemsvold et al (Brain and Language,1996; see fig.) showed that a sentence with a center-embedded (subject-pred splitting) clause causes greater brain activity and takes longer time than a right-branching sentence: The juice that the child spilled stained the rug (ca 3400 msec) The child spilled the juice that stained the rug (ca 3000 msec)

5

6

7 SWETRA grammar fragment for Om dig har jag då du, då jag, etc sent(S,[B,P]) --> pp(A,B),ppdefs(A,S,P). % pp + ppdefekt sats ppdefs(A,[subj(N),pred([V,Vi]),advl(A2)],[Vx,Nx,Ax,Vp]) --> auxfin(V,Vx),np(N,Nx),advb(A2,Ax),ppdefpart(A,Vi,Vp),{diff(D),D2 is D+3,retract(diff(D)),assert(diff(D2))}. /* regel som visar functionell representation och frasrepresentation före pilen. Difficulty increased by 3 */ pp([om,dig],pp([om,dig])) --> [om,dig]. auxfin(har,vfin(har)) --> [har]. np(jag,pron(jag)) --> [jag]. np(du,pron(du)) --> [du]. advb([C,B],[Cf,F]) --> sub(C,Cf),bs(B,F).

8 Demo för Om dig har jag då du, etc enligt Swetra grammar. Difficulty (D) is given retractall(diff(_)), assert(diff(0)), sent(F, X, [om, dig, har, jag, då, du, då, jag, nös, skrattade, fått, en, dålig, uppfattning], []), diff(D) F = [subj(jag), pred([har, [fått, obj(n([en, dålig, uppfattning, [om, dig]]))]]), advl([då, [subj(du), pred(skrattade), advl([då, [subj(jag), pred(nös)]])]])], X = [pp([om, dig]), [vfin(har), pron(jag), [sub(då), [pron(du), [sub(då), [pron(jag), vfin(nös)]], vfin(skrattade)]], vp([vpart(fått), np([en, dålig, uppfattning])])]], D = 5

9 Tree diagram generated by computerized grammar showing phrase structure of long multi center- embedded sentence

10 SWETRA grammar extract for Råttan som katten, etc. Calculates difficulty based on weights fixed to some rules sent([subj(Np), pred(V)],[np(Nx),vi(Vx)]) --> np(Np,Nx),vfin(V,Vx). sent([subj(Np),pred(V),obj(No)],[np(Nx),vi(Vx),np(Nf)]) -- >np(Np,Nx),vtfin(V,Vx),npg(No,Nf). vfin(pep,vi(pep)) --> [pep]. np(N,Nf) --> n(N,Nf). n(råttan,n(råttan)) --> [råttan]. n(katten,n(katten)) --> [katten]. n(hunden,n(hunden)) --> [hunden]. np([N,Rs],[n(N),rels(Rf)]) --> n(N,Nf),[som],rdef2(N,Rs,Rf),{diff(D),D2 is D +2,retract(diff(D)),assert(diff(D2))}. % Difficulty increased by 2

11 DEMOS for SWETRA grammar S shows functional roles, F shows categories The theoretical difficulty (D) is also calculated 1: retractall(diff(_)), assert(diff(0)), sent(S, F, [råttan, som, katten,bet, pep], []), diff(D), print(D) S = [subj([råttan, [subj(katten), pred(bet), obj(råttan)]]),pred(pep)], F = [np([n(råttan), rels([som, n(katten), vt(bet)])]),vi(vi(pep))], rels([som, vt(bet), n(katten)])]), vi(vi(pep))], D = 2 2: retractall(diff(_)), assert(diff(0)), sent(S, F, [råttan, som, katten,som, hunden, jagade, bet, pep], []), diff(D) S = [subj([råttan, [subj([katten, [subj(hunden), pred(jagade),obj(katten)]]), pred(bet), obj(råttan)]]), pred(pep)], F = [np([n(råttan),rels([som, [n(katten), rels([som, n(hunden), vt(jagade)])], vt(bet)])]),vi(vi(pep))], D = 4

12 Program (WMPARSE) modelling the use of restricted working memory and operations which assign functional roles to the categories N,V, that recognized. The total distance betwen subj-pred, pred-obj in the clauses is calculated func1(X,Y) :- X=[N,that,V1,V2], % 1 itr rel clause. % Boy that ran ran n(N),vi(V1),vi(V2), % gram categories Y=[subj(N),pred(V2),subj(N),pred(V1)],print(3),nl. % Dist:N-V2:2+ N-V1:1. Total:3 func2(X,Y) :- X=[N1,that,N2,that,N3,V1,V2,V3], % 2 tr obj rel clauses n(N1),n(N2),n(N3),vi(V3),vt(V1),vt(V2), Y=[subj(N1),pred(V3),subj(N2),pred(V2),obj(N1),subj(N3),pred(V1),obj(N2)],print(16),nl. % Boy that cat that dog chased bit ran % Dist: N1-V3: 6 + N2-V2:3 +V2-N1:5+V1-N2:2. Total =16

13 Demo till programmet WMPARS, which models the working memory and the operations on it func1(X, Y) No.1 : X = [dog, that, dog, bit, ran], Y = [subj(dog), pred(ran), subj(dog), pred(bit), obj(dog)] func2(X, Y) No.1 : X = [dog, that, dog, that, dog, bit, bit, ran], Y = [subj(dog), pred(ran), subj(dog), pred(bit), obj(dog), subj(dog), pred(bit), obj(dog)] func2([boy, that, dog, that, cat, chased, bit, ran], Y) No.1 : Y = [subj(boy), pred(ran), subj(dog), pred(bit), obj(boy), subj(cat), pred(chased), obj(dog)]

14 Program (NEXSUCC) som förstår ”råttan som katten som hunden bet jagade pep” i meningen att det drar/skriver ut ingående satser successivt. nex(X,Y) :- % drar ut innersta sats ur ordrad med två relativa som append(I,[N1,som,N2,V|F],X), % söker efter mönstret N1, som,N2,V. I är en initial del,F är en final del, eventuellt [] n(N1),n(N2),v(V), % kollar att N1 och N2 är substantiv och V är verb print([N2,V,N1]),nl, % skriver ut sats med N2 som subjekt, N1 objekt och V predikat append(I,[N1],I2), % lägger N1 till den initiala delen I, bildar I2 append(I2,F,Y). % lägger ihop I2 med F till en ny sträng på vilken regeln kan tillämpas igen och ev. finna en ny innersta sats nex(X,Y) :- % drar/skriver ut intransitiv sats X=[N1,V], n(N1),v(V),R=[N1,V],print(R),nl,Y=R.

15 Demo till successive nex (NEXSUCC) program The program first extracts the innermost clause (hunden jagade katten), then the next innermost (katten bet råttan) and last (råttan pep). nex([råttan, som, katten, som, hunden, jagade, bet, pep], X), nex(X, X2), print(X2), nl No.1 : X = [råttan, som, katten, bet, pep], X2 = [råttan, pep] [hunden, jagade, katten] [katten, bet, råttan] [råttan, pep]

16 Eye movement studies Several image studies (PET and fMRI) showing that the processing of complex sentences induces hemodynamic responses specifically in the opercular portion of Broca’s area have supported the hypothesis of a load on verbal working memory (vWM). The concept of WM may be based on several models but WM is usually considered to have a limited capacity of 7 +/- 2 units (”the magical number 7). Center-embedded sentences seem useful for the study of vWM. In order to further evaluate the possible load of center-embedded sentences on WM including attentional and visual mechanisms we have undertaken a preliminary study of eye movements during reading of different types of center-embedded sentences. The study was carried out at Humlab, Lund University. We acknowledge the assistence of Richard Andersson. Method: Ten sentences, five grammatically correct, five incorrect were presented to four ”naive” students. Eye movements were registered during the time the persons took to reach the judgement correct/incorrect.

17 Circles mark fixation points during reading - longer time bigger circle. The lines between circles are saccadic movements. Green circles mark fixation points during the first second. The blue circle marks the end of eye movements. The correct sentence ”Av segling…” (literally: Of sailing had Karl as he as he was young lived in the countryside no experience) with preposed preposition phrase ”Av segling” was difficult and marked as incorrrect by all subjects. Note the great eye activity over ”då han då han”.

18 The attention scan diagram shows the time the eyes have focused different parts of the sentence.The middle of the sentence with the beginnings of the adverbial clauses and the two subjects han have clearly attracted the greatest attention.The colour scale goes from blue to red.

19 The correct sentence ”Bollen som Karl…” (The ball that Karl as Per missed took hit the goal) was marked as correct by most subjects. The correct sentence ”Glaset som...”(The glass that the woman who owned the dog dropped split) was marked as correct by all subjects.

20 As opposed to the eye movement pattern during reading of common text our recordings show high antero- and retrograde activities with generally more than 20 saccades/fixations per sentence. The correct sentence ”Glaset som kvinnan…” (The glass that the woman who owned the dog dropped split) with only two pending subjects (glaset, kvinnan) was understood and marked as correct by all persons. Sentences like ”Bollen som Karl då Per…” (The ball that Karl as Per missed took hit the goal) with three pending subjects (bollen, Karl, Per) were marked as correct in half the cases. The sentence ”Av segling…” (literally: Of sailing had Karl as he as he was young lived in the countryside no experience) with three pending subjects (Karl, he, he) and the pending prepositional phrase ”Av segling” belonging to ”erfarenhet” was the most difficult and marked as incorrect by all persons. The incorrect sentences were identified in almost all readings. The findings indicate that an increase in syntactic complexity by more pending constituents and subj-pred splits induce a marked increase of working load as evidenced by eye movements and incorrect judgements.

21 References Baddeley, A.D. (1986) Working memory. New York: Oxford University press Chomsky, N. (1957) Syntactic Structures. The Hague:Mouton Gibson, E. (1998) ”Linguistic Complexity: Locality of Syntactic Dependencies”. Cognition, 68.1-76 Gibson, E., Desmet, T., Grodner, D., Watson, D. and Ko, K. (2005) ”Reading Relative Clauses in English”. Cognitive Linguistics, 16.313-53 Gouvea, A. C. (2003). ”Processing Syntactic Complexity: Cross-Linguistic Differences and ERP Evidence”. University of Maryland, College Park Karlsson, F. (2007) ”Constraints on mutiple center-embedding of clauses”, J. Linguistics 43, 365-392 Miller, G. & Chomsky, N. (1963) ”Finitary models of language users”. In: Luce, R. D. et al (eds) Handbook of mathematical psychology. New York: Wiley Roll, M., J. Frid & M. Horne (2007) ”Measuring Syntactic Complextity in Spontaneaous Swedish Speech”. Language and Speech 50:2, 227-245 Sigurd, B. (1978) ”Referent Grammar (RG). A generalized phrase structure grammar with built-in referents”. Studia Linguistica 41:2, 16-35