MeasurementofConversationalInteractivityinCOMPSComputerMediatedProblem-SolvingDialogues MichaelGlassValparaisoU. michael.glass@valpo.edu JungHeeKimNorthCarolinaA&TStateU. jungkim@ncat.edu MicaylaGoodrumNorthCarolinaA&T mjgoodru@aggies.ncat.edu Abstract ThispaperreportsonexperimentsinmeasuringthegenerallevelofconversationalinteractivityinCOMPSproblemsolvingdialogues.COMPSisaputermediatedproblemsolvingenvironmentforstudentcollaborativeexploratorylearning.Theprimarymodeofinteractionistypeddialogue.Weanticipatethatputerwillprovideastatusdisplaytoaidtheinstructor,whoiseffectivelylookingovertheshouldersofthestudentsastheywork.Towardthegoalputermonitoringofconversationquality,wehaveanalyzeddialogueturnsforInitiateandResponddialoguemovesasprescribedbyConversationAnalysistheory.Manyofourdialoguesarequiteinteractivebythismeasure.putertaggingofindividualturnsasInitiateorRespondhasproveddifficult.Hereweshowwhatmakessuchtaggingdifficultinourproblem-solvingenvironment.Wealsoproposethatthereareshallowmeasuresofoverallinteractivitythatmaycorrelatewithhowmuchthestudentsareresponding,withouttheneedtocorrectlytagindividualdialogueturns. Introduction ThegoaloftheCOMPSprojectistoprovideputeraidedinstrumentforcollaborativelearningofconceptsthroughproblem-solvingdialogue[Desjarlais,Kim,andGlass,2012;Kimetal.,2013].Thestudentsmainlyengageintyped-chat,thoughforsomeproblemsCOMPShasspecificproblem-relatedaffordancesforthestudentstomanipulate.COMPSshowstheinstructortheconversationsinrealtime,permittingtheinstructortointervene. AnunusualfeatureoftheCOMPSonlinechatenvironmentisthatstudentstypesimultaneously.Theycanseeeachother’mentsastheyaretypedinrealtime.Thisaddsaninteractivedimensionthatevenspokenlanguagedoesnotsupport,sincestudentschatsimultaneouslywithoutinterruptingeachother. Weanticipatethatputerwillprovideastatusdisplaytoaidtheinstructor,whoiseffectivelylookingovertheshouldersofthestudentsastheywork.Theinstructorwillbeinformedofeachgroup’sdegreeofcooperativebehaviorandprogresstowardsolvingtheirproblem. Thispaperillustratesannotateddialoguescollectedus- KelvinBryantNorthCarolinaA&TStateU. ksbryant@ncat.edu MelissaDesjarlaisValparaisoU. melissa.desjarlais@valpo.edu ThomasMartin NorthCarolinaA&
T . tmartin2@aggies.ncat.edu ingCOMPSwhilestudentsweresolvingproblemsrelatedtoaJavaSwingprogram.Weannotatedthesedialoguesusingourownschemethatdescribesthesocialstyleofthecontribution:confidentassertion,suggestion,askingaquestion,etc.OneaspectofourcodingschemecorrespondstoExchangeStructurefromConversationAnalysis,whereeachturnofdialogueisanalyzedasinitiatinganewdialoguesegment(I)orrespondingtotheinitiateturn(R).Althoughsomeexchangestructureanalysesalsorecognizeafollow-upcategory(F),ourownannotationsrecognizeonlytheIandRcategories.FromtheseR-andI-annotateddialogueswehavedocumenteddifferentstylesofgroupinteraction[Kimetal.,2013;Glassetal.,2013]. Theoriesofstudentcollaborativelearningsuchasgroupcognition[Stahl,2006]andknowledgeco-construction[Zhou,2009]presupposethatstudentsarerespondingtoeachothers’utterances.Fromourannotatedtranscripts,weseethatCOMPSconversationshaveahighpercentageofrespondmoves.Beingabletomachine-tagindividualturnsasrespondorinitiatewouldbeasteptowardjudgingwhethercollaborationwashappeninginoursessions. InthispaperwediscusstheissueofbuildingmachineclassifierstorecognizewhethereachturnrepresentsanIorRdialoguemove.Oureffortshavebeeninformedbyeffortstoclassifytransactivity[Roséetal.,2008;Aietal.,2010].TransactivityisaclassificationofRdialogueturnsspecializedforcollaborativelearning.Howeveroureffortsatbuildingclassifiershavenotbeenessfultodate.Wediscussherepossiblereasons,andsuggestdirectionsforfuturework. Wealsodiscussanotherpathtowardrecognizingwhetherstudentconversationscontainahighlevelofinteraction:countingtheeasilyrecognizablephenomenathaturinconversationwhenpeoplearerespondingtoeachother.Thesedialoguephenomenaareindependentoftheparticulardomainunderdiscussion.Weproposethatmeasuringthegenerallevelofinteraction,withouttaggingindividualIandRdialogueturns,mightbesufficienttogivearoughmeasureofquality.Inthispaperweidentifyputer-tabulateseveralofthesephenomenaurring withinCOMPSdialogues,suggestingthatageneralmeasureofinteractivitymightbepossible. Background Thelearningtask.ThedataforthisefromasecondsemesterJavaprogrammingclassatNorthCarolinaA&
T.Theprotocolwasasfollows.Duringlabperiod,studentsloggedintotheCOMPSwebpageingroupsof3.TheysolvedproblemsinunderstandingaJavaSwinggraphicaluserinterface.Theproblemswerepresentedtothestudentsonpaper,paniedbyapictureoftheGUIwithponentsnumbered.ThenatureofthetaskwastounderstandandarticulatetheJavasoftwarestructurethatnecessarilylaybehindtheinterfacetheywereseeing.Forexample,theyneededtodecidewhichoftheponentscouldbeanonymousinthecode,whicheventlistenersmustbepresentinordertosupportthedesiredbehaviors,andwhatisthevisibilityofinstancevariablesincertainJavaclasses.ThequestionsexercisetheirabilitytounderstandJavaSwing. Thestudentswereinstructedetoanagreementonanswers.Onestudentwouldtaketheanswertotheprofessorforfeedbackinperson,thenreturntothegrouptofinishthediscussion.Thisprocesscontinuedforeachproblemuntilallproblemswereunderstoodbyallmembersofthegroup. TheoreticaljustificationforusingCOMPSforthiskindoflearningtask.Thestudentskillsthatarethefocusofthisprojectareorientedtowardunderstandingandmanipulatingconcepts.ThisiswhatSkemp[1987]calls“relationalunderstanding,”plementtotheinstrumentalskillsofprogrammingthatarethebreadandbutterofelementaryprogrammingclasses.COMPSexercises,suchasthisone,arefocusedonlearningthingsthatarehardtomeasure.Thisorientationguidestheconstructionofourexercises,inparticularhavingetosharedagreement,tellingthemtheanswers,andhavingthemreconciletheirunderstandingwiththegivenanswers. Thereisalsoresearchshowingthatcollaborativeactivityisadesirablepedagogicalapproachspecificallyforcreatingconceptualunderstanding[Tchounikineetal.,2010].Keytoengenderinglearningisdialoguethatengagesindomainreasoning,suchasexplaining,negotiating,orinferring[Stahl,2006].Justifying,arguing,andsimilarknowledge-engenderingdialoguemoveswerenotableintheVirtualMathTeamdialogues[Zhou,2009]. Collaborativediscourseisalso,intheory,afertileapplicationforputerstoanalyzestudentknowledgeandbehavior.Whenstudentthinkingisnaturallyexpressedintheconversationitismadeavailableforputertofindit.Workingingroupsforcesstudentthinkingoutintheopen,forexampleasobservedbyKoschmann[2011].Inadditiontoreasoningtogether,con- versationalparticipantsmunicatetheirlevelofunderstandingtoachievegroundingandtosatisfydiscourseobligations[ClarkandBrennan,1991].Thereisnoneedforputertoaskspecialassessmentquestions,forexample,becausestudentthinkingisvisible. Theconstructrepresentinginteractivity.Todeterminewhetherastudentconversationisinteractive,weproposetolookfortransactivity.Ineducationaldialogueanalysis,adialoguemoveistransactiveifa)itrespondstoanotherdialoguemove,andb)contributestoknowledgebuilding.Ifwecanidentifybymachineintwoseparateproceduresthatastudent’sutterancea)respondstoanotherstudentandb)isontask,wewillhaveapproximatelyidentifiedatransactivecontribution.Inthispaperwearelargelyconcernedwithtaggingthefirstaspect,whetheraturnrespondstoanother. TransactivityappearsinWeinbergerandFischer’s[2006]fourdimensionalframeworkforgroupcognition.Transactivityisthe“socialmode”dimension:itcategorizesinwhatwaysinterpersonalprocessesareatworkintheconstructionoftheanswerwithoutaddressingtheknowledgeorreasoning.Thecategoriesoftransactivecontributionare:externalization(simplycontributing),eliciting,quickconsensusbuilding,integration-orientedconsensusbuilding,andconflict-orientedconsensusbuilding.Thesecategoriesseemtobeonascaleoflesstransactivetomoretransactive[Teasley,1997;WeinbergerandFischer,2006].Wehypothesizethatforthepurposesofassessingaconversation,simplymeasuringthedegreeoftransactivitycouldbeuseful.Itmaynotbenecessarytospecificallyidentifytheabovedifferentvarieties. AsawaytowardannotatingtransactivityweturntothelinguisticdisciplineofConversationAnalysis(CA).CAanalyzestheexchangestructureofadialogue,dividinguptheturnsintothreetypes:initiate(I),respond(R),andsometimesfollowup(F).Thesebasicstructuralunitsofdialoguearetheworkhorseforanalyzingphenomenasuchasturn-taking(howpeoplearbitratewhowillspeaknext),socialloafing(whoisnotparticipating,orbeinglazy),andpowerrelationships[Wells,1999].Followupissometimesomitted;theseturnscanbethoughtofasadditionalresponses. ConversationAnalysisbelongstothestructuralistbranchoflinguistics;itisconcernedwithobservablesfirst(whethersomebodyisresponding),notwhatfunctionisbeingplishedorwhatthespeaker’sintentionis.Inthisaspectannotatinginitiateandrespondissimilartoanalyzingtransactivity. Thereisacaveat:I/R/Fcanbehardtoanalyzeinconversationswheretherearemorethantwoparticipants.Whenthereisonlyoneotherpersondecidingwhichstatementapersonmightberespondingtoiseasier.Also,inamany-partyconversationasinglestatementmightelicitseveralresponsesfromdifferentparticipants. DataandManualAnalysis Weconducted17COMPSproblem-solvingdialoguesovertwosemesterswiththeJavaSwingproblem.StudentswereintheGeneralEngineering165classatNorthCarolinaA&T,thesecondsemesterofelementaryprogramming.Statisticsonthedialoguesare: Sessions:17Dialogueturns:1827Turnspersession:107Mean/medianduration:50min/52minMin/maxduration:26min/67minThreeofthesedialogueshavebeenextensivelyannotatedbyhand.Theannotationcategorieshavebeenrevisedsinceourearlierwork[Kimetal.,2013]toa)moreuratelymatchthejudgmentsoftheannotators,b)includeconversationanalysisIorRvariantsofmostcategories.TheannotationcategoriesareinTables1and2.Figure1(atend)showsanextractofannotateddialogue.InFigure1,dialogueturnsmarked‘<<’arenotcategorizedasIorR.Theannotationswithahyphen‘-’suffixareI,theotherannotationsareR.Thefollowingannotationcategoriesweredevisedbyourstudentannotatorsaftertheyandpreviousstudentshadsomeexperiencewiththecategoriesoftransactivityoutlinedabove.Essentiallythedifferencebetweenmonlyusedtransactivitycodesandourcodesisthatinourcodestheperceivedaffectofthespeakersubstitutesforthesocialconstructionofreasoning.Forexample,thecodersfelttheycouldmorereliablydistinguishwhetheraspeakerwasbeinghesitantorconfident,asopposedtodistinguishingwhetheracontributionwasmoreintegration-orientedorconflict-oriented. Table1:Modeofparticipation:responsecategories. Response RAstatementthatreferstoonemadeearlier AcknowledgementContradictory Definitive Suggestion GroupWork ACosigningonamessage/definitive//suggestion CResponsethatdisagreeswithamessage DResponsethatconfidentlygivesasolution SAlessconfidentpossiblesolution GGroupworkingtogether Question QSomeoneaskingforclarificationorstatingconfusion Table2:Modeofparticipation:Initiatecategories. Initiate I-Generalstartofanewthread Definitive D-Asureanswertoaquestionorproblem Suggestion S-Aless-sureanswertoaquestionorproblem Question Q-Arequestforfeedback/statementofconfusion GroupWorkG-Groupworkingtogether Ifessiveturnsrespondtoeachotherorbuildoneachotherserially,weannotatethemasastringofresponses.Inotherwords,turni+2canrespondtoturni+1whichrespondstoturni.ThisdiffersfromconventionalConversationAnalysispracticewhichwoulddividetheseintoanumberofInitiate/Respond/Followupexchangesegments.Onemotivationforthisdepartureisthenatureofmulti-partyconversation.Intwo-partyconversation,itispossibleto(somewhatarbitrarily)declarethatanewsegmenthasstarted.Inmulti-partyconversationsstudentsBandCmaybothrespondtoA,orCmayrespondtoBwhorespondedtoA.Itesimpossibletoisolateinitiaterespondpairswithoutassigningtworolestooneturn.Forexample:BrespondstoA,whileB’ssameturnsimultaneouslyinitiatestoC.Motivatedinpartbythatkindofcase,wechangedtheprotocoltoadmitserialresponseturns.Thisisalsomoreinlinewithhowtransactivityisusuallyannotated. OverlappedtypingpresentsanotherdifficultyinannotatingIandR.Figure2(atend)illustratesoverlappeddialogue,specificallyturns5and6fromtheFigure1transcript.
1.Time2:21:Astartstotype"Labels1,2,3,4,5,and14canbeinstantiated...
2.Time3:16:Btypes:“whatabout6and7?
”
3.Time3:48:Afinishestyping:“...thesedonothave tobechanged."NoticethatBstartedaskingaquestionafterAstarted.InspectionrevealsthatBwasprobablyrespondingtoA.ButBalsofinishedfirst.B’sresponsetoAthusursastheearlierdialogueturninthetranscript. ObservationsfromManualAnalysis Annotationofthedialoguesrevealspatternsofgroupinteractionandgroupcognitivefunctioning.OnephenomenonthatisillustratedintheFigure1segmentisthatstudentCistheprimaryinitiatorandservestosetthegoalstructureoftheconversation.OtherstudentslargelyrespondtoC’s agenda.Thisisanexampleofapatternweoftensee[Kimal.,2013;Glassetal.,2013]whereonestudenttakestheroleofmetacognitiveregulatorforthegroupcognitiveprocess.Thisregulatorstudentisnotnecessarilytheonewhocontributesthemosttothesolution.Twoofthethreeintensivelyannotatedtranscriptsillustratethispattern,visibleinTable3.StudentBinbothsessions1and2hadthelargestfractionofturnsinthesethree-partyconversations.Mosttellingly,inbothdiscussionslargepercentagesofstudentB’sturnswereI(initiate).StudentB(markedwith*)wasdrivingtheconversationalagenda,initiatingstatementsintotheconversationthattheothertwostudentswererespondingto.Session3didnotfollowthispattern.Session3wasalsounusualinthatparticipantCjoinedlateintheconversation;itwasatwo-partydialogueformuchofitsduration.Wedonothaveenoughtwo-partydialoguestosaywithconfidence,butanecdotallyitseemsthattwopartydialoguesdonotusuallyfollowthesamepatternofonepersonsettingthegoals. Table4showsthenumbersofI,Randoff-taskturnsineachofthethreecodedsessionsandintotal.TheratioofR/(I+R)isaresponsivenessindex:highernumbersmeanthestudentsarerespondingmoreandinitiatingless.Thelessontonoteisthatourstudentsareindeedmostlyontaskandmostlyrespondingtoeachother. Thenumberofquestionturnsmayalsobeindicativeofgroupinteractivebehavior.Inourcodingscheme,questionturnscanbeeitherrespondingorinitiating.Butanyquestion(exceptpossiblyarhetoricalone)isasignofstudentsengagingwitheachother.Table5showsthenumbersofquestions,withIandRquestionsaggregatedtogether. Table3:Patternofonestudentcontrollingagenda.CountsofIandRturnsonly,off-taskturnsomitted. SessStuTurnsStu’spct.ofPct.ofStu’s no. allI+
R turnsthat turns areI
1 A 2325% 17% B* 4852% 63%
C 2123% 10%
2 A 2824% 21% B* 5849% 52%
C 3227% 34%
3 A 2740% 52%
B 3146% 32%
C 913% 11% Table4:FractionofResponsivenessandOn-taskTurns. SessNo.123Total IROfftaskOfftaskpct. 3656 88% 4771 1713% 2542 2829% 108169 5316% R/(I+R)pct.61%60%63%61% Table5:FractionofQuestionTurns. SessNo.123 Total Q1927854 Q/(I+R)pct.21%23%12%19% I/RClassifier Inordertomeasurewhetheradialogueturnistransactiveweneedtoidentifywhethertheturnisa)respondingtoanotherpersonandb)on-taskaddressingepistemicknowledge-building.Wearebuildingclassifierstoidentifyinitiateandrespondingcategoriesfirst. Usingthehand-annotatedtranscriptswetriedtotrainWekaJ48decisiontreeclassifierstorecognizeIvs.Rdialogueturns.Intheseexperimentseachtrainingcaserepresentedonedialogueturn.Eachcasecontainedthefollowingfeatureset: Thelengthofthedialogueturn.Presenceorabsenceofeachofaboutmon words,chosenforurringwithhighfrequencyinthetranscripts.Presenceofadiscoursemarkerwordwithinthefirstfivewordsoftheturn,chosenfromasmallsetofdiscoursemarkers,e.g.“so.”Presenceofoneofadozenvocabularywordsspecifictotheproblemdomain,e.g.“JPanel.”Presenceofaquestionmark.Predictedclassvariable:eitheracodefromTables1and2,orInitiate/Respond/neither.Decisiontreesweretrainedandtestedontheapproximately300annotatedturns.Thedecisiontreesoftenovertrainedorpickedspuriousfeaturessuchasincidentalvocabularywords.Thustheydidnotholdupwhenappliedtoheld-outtestdata.Anothersetofexperimentsincorporatedtiminginformationasfeatures,usingthesameclassifiermethods.When- everparticipantpletedachatmessage(bypushingtheenterkey),paredA’smessagetothemostrecentmessagesofparticipantB.Thisgeneratedfourtimedifferencesforeachrecord: A-start-typing–B-start-typingA-end-typing–B-end-typingA-end–B-startA-start–B-end.Inathree-participantconversation,putingtimedifferencesAvs.BandAvs.Cdoublesthenumberofcases.Onesetofcasescontainsthedelta-timesforAvs.B,thesecondsetisidenticalexceptfordelta-timesAvs.C.Mostofthefeaturesintheduplicatedrecords,e.g.sentencelength,discoursemarkers,andclassvariable,remainthesame.Thedelta-timefeaturealsosometimesrevealedcasesofsimultaneoustyping.Forexampleconsiderturn6vs.turn5inFigure2.Turn5isthe“earlier”turnbecauseitendedearlier,thereforethe“later”turn6isevaluatedasapotentialresponseto5.However6startedbefore5.ThedeltatimeA-start–B-startisthusnegative,indicatingoverlap.Whencareistakentoremoveduplicaterecordsandidentifywhichturnisrespondingtowhichotherparticipant,J48pruneddecisiontreesutilizingthedelta-timefeaturesaremorerobustthantheearlierclassifierexperiments.Applyingthetrainedtreestoheld-outdataworksreasonablywell. ResultsandDiscussion Results.Noneoftheexperimentswerenotablyessful.Usingthenon-timingfeatures,typicalgoodresultsusing 10-foldcross-validationwerekappaagreementofabout0.45withhumanraters,andFscoresof0.7onidentifyingtheIandRlabels. Whendelta-timeswereavailableasfeaturesforclassifiertrainingtheuracywasaboutthesame.Kappaagreementwiththehumanratersremainedinthelowendofthe0.4–0.5range,andFscoresremainedatabout0.7. Thebestdecisiontreesusingtimingfeatureswerenotstartling.IfAstarted61ormoresecondsafterBended,AwasmostlikelynotaresponsetoB.ButgiventhatAstartedlate,ifA’sstatementwaslongitwasalittlemorelikelyaresponsetoB. Introducingdelta-timesisanimprovementinclassification.Eventhoughclassificationuracywasnotimproved,thetimingfeaturesarepotentiallydomain-independent.TheclassifiertrainedontimingfeaturesmightworkforallourCOMPSdialoguesinthreedifferentclasses.Whereasaclassifierthatusesvocabularymightworkonlyfortheparticularproblemorstudentpopulationitwastrainedon.Thefactthatcross-foldvalidationtendedtodegradeuracyinclassifiersusingthewordfeaturesmorethanitdegradedclassifiersusingtimingfeaturesis anotherindicationthattimingfeatureswillholdupbetterwithlargerandmorediversedatasets. Generatingtimedifferencerecordsagainsteveryotherparticipantintheconversationprovedtobeamethodologicalproblem.Itbiasestheclasslabels.AsingleIturnisrepresentedbytworecordsinthedatasetwithnearlyidenticalfeaturespredictingthesameclassvariable.Theresultisastrongtendencyfortheclassifiertopredictthedoubledcases. Comparisontootherresults.OtherresearchersachievemoderatelybetterKappabetween0.5and0.6,e.g.[Roséetal.,2008]workingwithonlinechatdiscussionsand[Aietal.,2010]workingwithtranscribedclassroomdiscussion.Inbothcasestheclassvariablewastransactivity.TheywereabletoboostKappaagreementto0.7usingseveralstagesofclassification.Itisinstructivetoanalyzesomeofthedifferencesbetweentheirclassifiersandours.Inadditiontothefeatureswementionedabove,e.g.vocabularywordsandlengths,theseresearchersderivedfeaturessothatonecase(oneannotateddialogueturn)wouldincludefeaturescontrastingthatdialogueturnagainstpreviousturns.Thesederivedfeatureswere:
1.LSA(latentsemanticanalysis)parisonsofthewordsinthecurrentturntoa)thewordsinthepreviousturn(usuallyanotherspeaker),b)twoturnsback,andc)threeturnsback.
2.Typeofspeaker(studentorteacher),typeofspeakerforpreviousturn,whetherthespeakeristhesamepersonasforthepreviousutterance.
3.Changeic:whetherichasshiftedinthepreviousutterance. Italsoappearedthattheirchatdatadidnotincludeoverlappedsimultaneoustyping. Discussion.ExaminingFigure1showswhy,webelieve,ourclassifiershavenotbeenessfultodate.Amainissueisthatmuchofthedialoguedoesnotcontainconcepts,studentsinsteadrefertomultiple-choiceanswersbylettersandtonumbereditemsontheSwingGUI.Theconceptsandobjectivesbeingreasonedaboutarenotsituatedwithintheconversation.Theletterandnumberreferencesaremonthansentencescontainingrecognizablereasoninginthedomain.Asevidencethatplicatesthetask,wenotethatitisnotpossibleforahumanannotatortotellwhethertwopeoplearediscussingthesameconceptwithoutapictureoftheGUIandthemultiple-choiceanswershandyforreference. Asecondarysourceplexityisthetypingoverlapproblem.InadditiontothekindsoftiminganomalyillustratedinFigure2,weseestudentssometimesneglecttopressenter.Everybodycanseewhattheytypedwithoutit.Weseestudentspauseinthemiddleoftyping,waitforotherstudentresponses,thenpickupagain,effectivelyputtingtwodialogueturnsinonechatmessage. TheWayForward1: MoreandBetterFeatures Aprioritytaskistofindshallowfeaturesthatshouldcorrelatewitheither1)studentsrespondingtoeachotheror2)studentsreasoningic.Thesearetheponentsofatransactivecontribution.Wewillusethesefeaturestoseeiftheclassificationtaskcanbeimproved. Wewillalsotrytousesimpledetectionandcountingofthesefeaturestoderiveatransactivityindexthatcorrelateswithhumanjudgment.Thisisdiscussedbelow. Featureswehaveextractedfromthetextbutnotyetappliedtomachinelearningexperiments. Discoursemarkers.Usinganexpandedcatalogofdiscoursemarkers[Alemanyetal.,2005],weseediscoursemarkersstart10%ofthe1800turnsfrom17sessions.Discoursemarkersmightindicatethatreasoningorargumentationishappening.Inadditiontothefixedlexicon,weaddedsomediscoursemarkersrecognizedbyregularexpressions(e.g.“soooo...”). Problemdomainvocabulary.Thesewordsareanindicationthatstudentsarediscussingicathand.WeexpandedthevocabularyofproblemdomainwordsfortheJavaSwingGUIproblem.Thenumberofturnsthatarenowrecognizedasincludingdomain-specificwordsis20%,paredto7%inthemachinelearningexperimentsdescribedabove. Task-relatedvocabulary.Thesearewordsrelatedpletingthetaskbutnotpartofthedomainunderdiscussion.Forexample,themultiple-choiceanswerlettersandthelabelsontheponentsoftheJavaSwingscreenshotaretask-relatedwordswhichwecanrecognize.Ifastudentsays“Ithinkwecanruleoutbandc”,thatstudentisdiscussingthetaskathand.30%ofturnscontainareferencetoamultiple-choiceansweroraponent. Overlappedtyping.Amongthe1800turns,47%exhibitoverlappedtypingwhereseveralpeopleareenteringanewmessagesimultaneously. Emoticons.Peopleputemoticonsintotheirchatdialoguepreciselybecausetheyareinteractingwithotherpeople.Emoticonsexpressaffectivestate.Emoticonsurinonly1%ofourcorpusforthisproblembuttheyaremuchmoreprevalentinotherCOMPSexercisesusingadifferentstudentpopulation. Pronouns.Inasimilarvein,thepresenceof2ndpersonpronounsand1stpersonpluralpronounscouldbeindicativeofinteractivediscourse.16%ofturnscontainsuchapronounwithinthefirst10words. Otherfeaturestoexplore:Otherexpressionsofaffect.Theoriesofaffectgenerallyholdthatpeopleexpressaffectinorderforotherpeopletosenseit.Expressionsofaffect,therefore,maybeindica- tionsofsocialprocessesatwork.Weproposethatthepresenceofsuchwordsmightbeausefulfeature. Moreuseoftimingoverlap.Wemaysplitaturnintotwointheeventofalengthypause,treatingthetwopartsasdifferentdialogueturns.Especiallyifotherpeopleweretypingduringthepause,itislikelythatthetwopartsserveasdistinctdialogueturns.Wemayusetimingdifferencestotrytoidentifycandidateturnsastargetsofresponseinawaythatdoesnotduplicaterecords. parisons.Wecantrytheparisonsandothermeasurementsonessiveturnsthatotherresearchershavefoundfruitful. TheWayForward2:Adifferentstyleofmeasurement. Wewillexploremeasuringtheinteractivityofadiscussionwithoutlabelingeachindividualturnastransactiveornot.Someofthefeaturesmaybythemselvesbeindicativeofstudentsinteractingwitheachother,e.g.emoticons,closetimingandoverlaps,andpronouns.Otherfeaturesmaybeindicativeofstudentsengaginginreasoning(discoursemarkers),ofengagingtheproblem(domainvocabulary)andofengagingthetask(taskvocabulary).Simplymeasuringthedensityofthesefeaturesmightprovesufficienttoevaluatethequalityofastudentproblem-solvingdiscussion.Thismeasurementcouldbeappliedinrealtimetotheentirediscussionstartingfromthebeginning,ortoaslidingwindowofmostrecentdialogueturns. Forpurposeoftrainingputerizedformulaforthismeasurement,wewillusethemanuallycodedcorpustoindependentlyassessoveralltransactivity.Wewillstartbycountingthefractionofinteractiveturnsinourannotations. Acknowledgments Thankyoutoourhard-workingstudentsatNorthCarolinaA&TandValparaisoUniversities.ThisworkissupportedbytheLockheedMartinCorporationundertheprogramofComputerScienceUndergraduateResearcherstoNorthCarolinaA&TStateUniversityandbytheNationalScienceFoundationunderawards0633953toNorthCarolinaA&TStateUniversityand0851721toValparaisoUniversity. References Ai,Hua,MariettaSionti,Yi-ChiaWang,andCarolynPensteinRosé.2010.Findingtransactivecontributionsinwholegroupclassroomdiscussions.InKimberlyGomez,LeilahLyons,andJoshuaRadinsky(eds.),Proceedingsofthe9thInternationalConferenceoftheLearningSciences(ICLS'10).InternationalSocietyoftheLearningSciences,vol.1pp.976–983. Alemany,LauraAlonso,IreneCastellónMasalles,andLluısPadróCirera.2005.RepresentingDiscourseforAutomaticTextSummarizationviaShallowNLPtechniques.UnpublishedPhDthesis,UniversitatdeBarcelona.LexiconofdiscoursemarkersinCatalan,Spanish,andEnglish.RetrievedMarch,2014fromhttp://russell.famaf.unc.edu.ar/~laura/shallowdisc4summ/ Clark,HerbertH.,andSusanE.Brennan.1991.GroundinginCommunication.InLaurenB.Resnick,JohnM.Levine,andStephanie.D.Teasley(eds.),PerspectivesonSociallySharedCognition.AmericanPsychologicalAssociation,pp.127–149. Desjarlais,Melissa,JungHeeKim,andMichaelGlass.2012.COMPSComputerMediatedProblemSolving:AFirstLook.ProceedingsoftheMidwestAIandCognitiveScienceConference(MAICS2012),Cincinnati. Glass,Michael,JungHeeKim,MelissaDesjarlais,andKelvinS.Bryant.2013.COMPSComputer-MediatedProblemSolvingDialogues.(Posterabstract)Proceedings:Computer-SupportedCollaborativeLearning(CSCL2013),Madison,WI,volII,pp.257–258. Kim,JungHee,MelissaDesjarlais,KelvinBryant,andMichaelGlass.2013.ObservationsofCollaborativeBehaviorinCOMPSComputerMediatedProblemSolving.ProceedingsoftheMidwestAIandCognitiveScienceConference(MAICS2013),NewAlbany,IN,pp.71–77. Koschmann,Tim.2011.Understandingunderstandinginaction.JournalofPragmatics.vol.43no.2,pp.435-437. Rosé,Carolyn,Yi-ChiaWang,YueCui,JaimeArguello,KarstenStegmann,ArminWeinberger,andFrankFischer.2008.Analyzingcollaborativelearningprocessesautomatically:Exploitingtheadvancesputationallinguisticsputer-supportedcollaborativelearning.InternationalJournalofComputer-SupportedCollaborativeLearning.vol.3no.3,pp.237–271. Skemp,Richard.1987.ThePsychologyofLearningMathematics.Hillsdale,NJ:Erlbaum.Chapter12. Stahl,Gerry.2006.GroupCognition.MITPress. Tchounikine,Pierre,NikolRummel,andBruceM.McLaren.2010.ComputerSupportedCollaborativeLearningandIntelligentTutoringSystems.InR.Nkambo,
J.Bourdeau,&
R.Mizoguchi(eds.),AdvancesinIntelligentTutoringSystems.Springer.Chapter22,pp.447–463. Teasley,Stephanie.1997.Talkingaboutreasoning:Howimportantisthepeerinpeercollaboration?
InL.B.Resnick,RSaljo,
C.Pontecorvo,andB.Burge(eds.),Discourse,Tools,Reasoning:EssaysonSituatedCognition.Springer. Wells,Gordon.1999.DialogicInquiry.CambridgeUniversityPress. Weinberger,ArminandFrankFischer.2006.Aframeworktoanalyzeargumentativeknowledgeconstructionputer-supportedcollaborativelearning,Computers&Education,vol.46no.1,pp.71–95. Zhou,Nan.2009.QuestionCo-ConstructioninVMTChats.InGerryStahl(ed.)StudyingVirtualMathTeams.Springer.pp.141–159. TurnStu
1 C
2 C
3 B
4 C
5 B
6 A
7 B
8 A
9 C 10C 11A 12B 13C 14B 15B16C17C18B19C Text heypeopleokayquestiononeI'mreadingitdoeitherofyouknowwhatthequestionisevenasking?
idon’twhatabout6and7?
"Labels1,2,3,4,5,and14canbeinstantiatedanonymously.Becausethesedonothavetobechanged."thatmakessense6and7cannotbeinstantiatedanonymouslybecausethesevalueshavetochange.okay.Imlostwhereareyouguysgetttingthisfromthebackgroundinformation?
It'sonthesecondpage.discriptionohhhmowiseethanksforproblemtwoIknowtheactionlistenerinterfaceneedstobeimplementedisthereanyothers?
andactionListenerithinkthosearetheonlytwowhawasthefirstone?
imentmouse Annotation< 3:16 3:26 Turn6Turn5 Figure2:OverlappedTypingofResponse.