DisclosureAuthorizedPublicDisclosureAuthorizedPublicDisclosureAuthorized

PublicDisclosureAuthorizedPublic

EDUCATION

WORKINGPAPER

No.10|February2024

ValidatingTeachingObservationTools:

AContent-BasedApproachforPLAYandTeachPrimary

DiegoLunaBazalduaandEmmaCarter

Educationoperations

SUPPORTHUB

THEWORLDBANK

IBRD.lDAIWORLDBANKGROUP

?2025InternationalBankforReconstructionandDevelopment/TheWorldBank

1818HStreetNW

WashingtonDC20433

Telephone:202-473-1000

Email:AskEd@

Internet:

/en/topic/education

ThisworkisaproductofthestaffofTheWorldBankwithexternalcontributions.Thefindings,interpretations,andconclusionsexpressedinthisworkdonotnecessarilyreflecttheviewsofTheWorldBank,itsBoardof

ExecutiveDirectors,orthegovernmentstheyrepresent.

TheWorldBankdoesnotguaranteetheaccuracy,completeness,orcurrencyofthedataincludedinthisworkanddoesnotassumeresponsibilityforanyerrors,omissions,ordiscrepanciesintheinformation,orliabilitywith

respecttotheuseoforfailuretousetheinformation,methods,processes,orconclusionssetforth.The

boundaries,colors,denominations,links/footnotesandotherinformationshowninthisworkdonotimplyanyjudgmentonthepartofTheWorldBankconcerningthelegalstatusofanyterritoryortheendorsementor

acceptanceofsuchboundaries.ThecitationofworksauthoredbyothersdoesnotmeantheWorldBankendorsestheviewsexpressedbythoseauthorsorthecontentoftheirworks.

NothinghereinshallconstituteorbeconstruedorconsideredtobealimitationuponorwaiveroftheprivilegesandimmunitiesofTheWorldBank,allofwhicharespecificallyreserved.

RightsandPermissions

Thematerialinthisworkissubjecttocopyright.BecauseTheWorldBankencouragesdisseminationofitsknowledge,thisworkmaybereproduced,inwholeorinpart,fornoncommercialpurposesaslongasfullattributiontothisworkisgiven.

Anyqueriesonrightsandlicenses,includingsubsidiaryrights,shouldbeaddressedtoWorldBankPublications,TheWorldBankGroup,1818HStreetNW,Washington,DC20433,USA;fax:202-522-2625;e-mail:

pubrights@.

Coverdesign:MarianneSiblini

ValidatingTeachingObservationTools:AContent-BasedApproachforPLAYandTeachPrimary

DiegoLunaBazalduaandEmmaCarter

TableofContents

Abstract 3

Keywords 3

Acknowledgements 4

ExecutiveSummary 5

1.Studyobjective 6

2.Descriptionofobservationtools 6

3.Method 10

4.Results 12

a.Contentvalidityresults:ContentmatchingofTeachPrimarybehaviorsandPLAYitems 12

b.Concurrentvalidityresults:RelationshipamongTeachPrimaryandPLAYscores 19

c.ConstructvalidityresultsforPLAY:InternalstructureforthePLAYitems 19

5.Conclusion 22

References 23

Annexes 25

AnnexA:PLAYConstructsofsupportforchildren’sengagementinlearning 25

AnnexB:TablecomparingthemainfeaturesoftheTeachPrimaryclassroomobservationtooland

PLAYtoolkit 30

AnnexC:Overviewofexperts 31

AnnexD:FullmappingresultsforPhase1 32

AnnexE:MappingresultsforPhase2 47

AnnexF:CorrelationofTeachPrimaryandPLAYscores 52

AnnexG:CorrelationofPLAYitems,item-totalcorrelationsandvisualrepresentationsofinternal

structure 53

AnnexH:ExploratoryFactorAnalysissolution 58

Abstract

Thisstudycomparestwotoolsusedtoobserveandassesshowteacherssupportstudent

learning:theLEGOFoundation’sEngagetool,formerlyknownasPLAY,andtheWorldBank’s

TeachPrimarytool.Thegoalwastoevaluatetheirsimilaritiesandoverlapintheirmeasurementofteachingpractices.Agroupofnineexpertsreviewedbothtoolsindetail,carefullymatching

itemsfromPLAYtothoseinTeachPrimary.Resultsindicatethataboutone-thirdoftheitems

closelymatched,showingthatPLAYisbuiltonasolidframeworktocaptureengagingteaching

practices.Anotherthirdoftheitemsdidnothaveaclearmatch,meaningthateachtool

separatelycapturesuniqueaspectsofteachingthattheotherdoesnot.Thestudyalsolookedathowwellthetwotools’scoresalignusingrealclassroomdatafromEthiopia,Peru,andSierra

Leone.Theresultsshowedasmallpositiverelationshipbetweenthetwotools,meaningtheymeasurerelatedbutdi?erentaspectsofteaching.ThestudyalsofoundthatthewayPLAYis

structuredmayneedsomeadjustmentstobetterre?ecttheskillsitisdesignedtomeasure.Bycombiningexpertopinionswithdataanalysis,thisstudyprovidesstrongevidencethatboth

toolsarevaluablebutservedi?erentpurposes.The?ndingshighlighttheimportanceofusingmultipleapproachestoassessteachingquality.Thestudyalsoo?ersrecommendationsfor

improvingtoolimprovementandsuggestsnextstepsonhowclassroomobservationtoolscanhelpsupportimprovedteachingandlearningworldwide.

JEL:I25-EducationandEconomicDevelopment

Keywords

Classroomobservations,teachingquality,engagingteaching,socio-emotionallearning,teachingpractices,contentvalidity,constructvalidity.

Acknowledgements

ThisdocumentwasdevelopedbyDiegoLunaBazaldua(SeniorEducationSpecialist;HEDGE),

EmmaCarter(Consultant;HEDGE).TheteamworkedundertheoverallguidanceofHalilDundar(PracticeManager,HEDGE).LauraGregory(SeniorEducationSpecialist;HEDGE),SergioVenegas(Economist,HEDGE)andMaryamAkmal(Economist,HEDGE)providedoverallsupportinthe

di?erentstagesofthisstudy.

PeerreviewersincludeAliHasanAnsari(SeniorEconomist,HAWE3),HumaKidwai(Senior

EducationSpecialist,HAEE1),EzequielMolina(SeniorEconomist,HLCED),andLauriPynnonen(SeniorEducationSpecialist,HEAED).AdditionalvaluableinputswerereceivedfromAmer

HasanandHarshilKumarSahai,andothermembersoftheEducationGlobalPracticewho

participatedindiscussionsonthepaper.Thenextlistofinternationalexpertscontributedtocontentreviewsofthetools:AbbieRaikes,CarolinaMeloHurtado,CarolinaMoreiraVasquez,DawitTiruneh,ElizabethHentschel,JonathanSeiden,LenkaJanikBlaskova,RabeaMalik,andTadesseTeferi.

ThisworkissponsoredbytheFoundationalLearningCompact(FLC)TrustFund.

ExecutiveSummary

Engagingteachinghasemergedasakeyapproachtofosteringchildren’ssocio–emotionaland

cognitiveskills.Becauseoftheimportanceofembeddedengagingactivitiesintheteachingandlearningprocess,LEGOFoundationandtheWorldBankhavecollaboratedtoadvanceamoreevidence–basedapproachtoenhancingholisticteachingandlearning.Thesee?ortsemphasizepracticesthatfocusonengagement–basedandsocio–emotionallearning.Thisreportpresents?ndingsfromasigni?cant

componentofthepartnership,focusingoncomparingtheLEGOFoundation'sEngage,formerlyknownPlayfulLearningAcrosstheYears(PLAY)

1

,classroomobservationtoolwiththeWorldBank’sTeach

Primaryclassroomobservationinstrument.Thegoalwastoassesstheextentofoverlapbetweenthesetools,bothqualitativelyandquantitatively,providingcontentandconcurrentvalidityevidenceforthenewlydesignedPLAYinstrument.Thisstudyalsoaimedtoenhanceunderstandingofthetechnical

propertiesofbothtools,emphasizingsynergiesanddistinctions.

Methodologically,thequalitativeaspectofthestudyfollowedtheDelphimethod,involvingnineexpertsfromdiverseinternationalbackgrounds.Theyconductedtwophasesofcontentreview.Phase1includedadetailedcontentcomparisonanditem–to–itemmatching,withrationalesprovidedformatchingdecisionsmade.Phase2focusedonreviewingambiguouscases—itemsthatdidnotclearlymatchonPhase1.Inaddition,quantitativeanalysisassessedthecorrelationbetweenTeachPrimaryandPLAYaggregatescorestoevaluateconcurrentvalidity,usingdatafromEthiopia,Peru,andSierraLeone.

KeyqualitativeresultsshowedthatoverathirdoftheitemsinthePLAYtoolstronglycorrespondedwithbehavioralitemsfromtheTeachPrimaryinstrument,providingevidenceoftheempiricalbasisoftheformer.Additionally,thestudyidentifiedasimilarproportionofitemswithminimalornocorrespondencetoTeachPrimarybehaviors,highlightingthePLAYtool'suniqueabilitytocaptureotheraspectsofteachingconducivetoqualityengagement–basedinstructionandlearning.Quantitativefindingsincludedatotalscorecorrelationof0.15betweentheTeachPrimaryandPLAYinstruments,supportingconcurrentvalidityandreinforcingthetools’measurementofseparatebutrelatedconstructsofqualityteacherpracticeandengagement–basedteachingandlearning,respectively.Takentogether,thesefindingsrevealedthatcombiningqualitativelearningsfromtheDelphimethodwithrobustquantitativevalidationresultscreatesacomprehensivevalidationframework,strengtheningconfidenceintheseclassroomobservationtoolstomeasurequalityengagement–basedandsocio–emotionallearning.

Inadditiontotheanalyticalfindings,valuablelessonswerelearnedfromtheapplicationoftheDelphitechnique,whichcouldimproveitseffectivenessforconsensus–buildinganddecision–makinginfutureresearch.Amongtheseweretheprovisionofsufficienttimefortaskstofacilitatedataaccuracyandtheimportanceofconsistentcommunicationwithexperts.Finally,clearincentivesalongwithstreamlinedprocessesforlabor–intensiveexerciseswererecommendedashelpfulstrategiesforenhancingparticipants’engagement.

1ThenamePLAYisusedthroughoutthispaperasitcorrespondstothenameusedinpubliclyavailablematerialsatthetimeofwriting.TheauthorsacknowledgethatthetoolisnowcalledEngage.

1.Studyobjective

TheLEGOFoundationandtheWorldBankhavecollaboratedtosupporteducationsystemsinadoptingamoreevidence-informedapproachtoimproveholisticteachingandlearning,particularlythroughengagement-basedandsocio-emotionalteachingpractices.Acrucialaspectofthisobjectiveisenhancingthemeasurementofthesepractices,especiallywithinlow-andmiddle-incomecountries(LMICs).OneactivityincludedinthiscollaborationinvolvedadetailedcontentcomparisonoftheLEGOFoundation'sPlayfulLearningAcrosstheYears(PLAY)andtheWorldBank'sTeachPrimaryclassroomobservationtools.Thiscomparisonaimedtounderstandtheextentofcontentandscoreoverlap,aswellastheirconnectionwithstudents'learningexperiencesandoutcomes.Additionally,theexercisefocusedonidentifyingimportantcontentdistinctionsbetweentheinstrumentsandhowtheymayprovidecomplementaryinformationtousers.

Thiscomparisonhasalsoservedasaformofcontentvalidityevidenceandfacilitatessubsequentquantitativeanalysisaimedatunderstandingandcomparingthetechnicalpropertiesofthetools.Forinstance,thiscontentmatchinganalysishighlightsthedegreeofconcordancebetweenitems,subscales,andunderlyinglatentconstructsmeasuredbyPLAYandTeachPrimary.Italsoenhancesourunderstandingoftheassociationbetweenconstructsindicatingthequalityofengagement-basedteachingpracticesandthosere?ectinggeneralteachingpracticesandteacher-studentinteractions.ThiscanbeachievedbyimplementingasimilarmethodologytothatusedbytheWorldBank’sTeachteaminrelationtoinclusiveteachingclassroompractices(see

Molinaetal.,2024)

,examiningtherelationshipbetweentheunderlyingconstruct—engagingteachingpractices—andTeachPrimarybehaviors.ThisapproachenablesmutuallearningandincreasesawarenessofthesynergiesbetweenthePLAYandTeachPrimarymeasurementapproaches,facilitatingmeasurementscale-upandpromotingengagement-basedandsocio-emotionallearningintheclassroom.

Thispaperisstructuredasfollows:Section1beginswithadescriptionoftheLEGOFoundation’sPLAYandtheWorldBank’sTeachPrimaryclassroomobservationtools.Section2detailsthestudy’smethodologicalapproach,includingthestudy’sdesign,pro?lesofexpertsinvolved,andanalyticalmethods.Section3providesanoverviewoftheresultsfromthetwokeyphasesofthisstudy.Finally,Section4presentskeyconclusionsanddiscussesthelimitationsrelatedtothemethodused.

2.Descriptionofobservationtools

PLAYClassroomObservationTool

TheLEGOFoundationhascollaboratedwithRTIInternationalandNewYorkUniversity(NYU)GlobalTIESforChildrentodevelopaculturallyresponsivetoolkitthatincludesclassroomobservationandsurveymeasurementtoolsfocusedoncapturingthequalityofengagingteachingandlearninginclassrooms.ThePLAYtoolkitaimstomeasurehowadultssupportchildren'sengagementinlearning,whichleadstoabroadrangeoflearningoutcomes.

PLAYincludestoolsformultipleagegroupsacrossdi?erentsettings.Forstudentsaged6to12,themeasurementtoolsaredesignedspeci?callyforclassroomuse.Thesetoolsincludeaclassroominventory,teacher-childobservation,ateachersurvey,andachildsurvey.Theteacher-childclassroomobservation

tool,whichisthefocusofthecomparisontaskdescribedinthenextsectionofthisdocument,encompassesfourkeydimensionsessentialforsupportingchildren'sengagementinlearning.Thesedimensions,alsoreferredtoasconstructs,aresummarizedbelowanddetailedfurtherinAnnexA.

?Supportforexploration:Adultsupportforchildren'sinteractionwiththingsandideastoexpandthought.

?Supportforagency:Adultsupportforchildren'sabilitytoin?uencehowandwhattheylearn.

?Supportforpersonalandsocialconnection:Adultsupportfor(1)relatinglearningtochildren'spersonalexperiences,(2)childrenlearningthroughsocialinteraction,and/or(3)childrenfeelingasenseofbelongingandbeingsociallyconnected.

?Supportforemotionalclimate:Adultsfosteranenvironmentwhereinteractionsbetweenadult(s),child(ren),andpeersarewarm,respectful,andpositive.

Thesefourconstructsarecapturedin25itemswithinthetool.Eachitemisscoredona3-pointscale,measuringthefrequencyandqualityofteacherandstudentbehaviors,aswellasparticipationintheclassroom.

TheWorldBank’sTeachPrimary(2ndEdition)ClassroomObservationTool

TheTeachPrimarytoolcapturesteachingpracticesthatsupportqualitylearningandnurturechildren'scognitiveandsocio-emotionalskills.Initiallylaunchedin2019afterarigorousdevelopmentandvalidationprocess,TeachPrimaryunderwentarevisionin2020-2021toenhanceitsmeasurementofinclusiveteachingpracticesthatareresponsiveandfacilitatewhole-childdevelopment.SimilartoPLAY,theTeachPrimarytoolisdesignedforclassroomswithstudentsaged6to12.

TheTeachPrimaryobservationtoolassessestwomainaspects:(i)thetimeteachersspendonlearningandtheextenttowhichstudentsremainontask,and(ii)thequalityofteachingpracticesthatdevelopstudents'socio-emotionalandcognitiveskills.TheTimeonTaskcomponentusesthree"snapshots"of1–10secondstorecordtheteacher'sactionsandthenumberofstudentswhoareontask.Observersnotewhethertheteacherprovidesalearningactivityformoststudents.Ifso,theyscantheclassroomtodetermineifstudentsareontask.O?-taskbehaviorisratedasfollows:0-1studentsbeingo?-task(High);2-5studentsbeingo?-task(Medium);6ormorestudentsbeingo?-task(Low).

TheQualityofTeachingPracticescomponentisorganizedintothreeprimaryareas:

1.ClassroomCulture:Theteachercreatesacultureconducivetolearning.Thisfocusesontheextenttowhichtheteacher:

(i)createsasupportivelearningenvironmentbytreatingallstudentsrespectfully,consistentlyusingpositivelanguage,respondingtostudents'needs,andbothchallengingstereotypesandnotexhibitingbiasintheclassroom.

(ii)establishespositivebehavioralexpectationsbysettingclearbehavioralexpectations,acknowledgingpositivestudentbehavior,ande?ectivelyredirectingmisbehavior.

2.Instruction:Theteacherfacilitateslessonsthatdeepenunderstandingandencouragecriticalthinkingandanalysis.Thisfocusesontheextenttowhichtheteacher:

(i)facilitatesthelessonbyexplicitlyarticulatinglessonobjectivesthatarealignedtothelearningactivity,explainingcontentusingmultipleformsofrepresentation,andconnectingthelearningactivitytoothercontentknowledgeorstudents'dailylives,andbymodelingthelearningactivitythroughenactingorthinkingaloud.

(ii)doesnotsimplymovefromonetopictothenextbutchecksforunderstandingbyusingquestions,prompts,orotherstrategiestodeterminestudents'levelofunderstanding,bymonitoringstudentsduringgroupandindependentwork,andbyadjustinghis/herteachingtothelevelofstudents.

(iii)givesfeedbackbyprovidingspeci?ccommentsorpromptstohelpclarifystudents'misunderstandingsoridentifytheirsuccesses.

(iv)encouragesstudentstothinkcriticallybyaskingopen-endedquestionsandprovidingthemwiththinkingtasksthatrequirethemtoanalyzecontentactively.Studentsexhibitcriticalthinkingabilitybyaskingopen-endedquestionsorperformingthinkingtasks.

3.SocioemotionalSkills:Theteacherfosterssocio-emotionalskillstohelpstudentssucceedinsideandoutsidetheclassroom.Thisfocusesontheextenttowhichtheteacher:

(i)instillsautonomybyprovidingstudentswithopportunitiestomakechoicesandtakeonmeaningfulrolesintheclassroom.Studentsexhibittheirautonomybyvolunteeringtoparticipateinclassroomactivities.

(ii)promotesperseverancebyacknowledgingstudents'e?ortsratherthanfocusingsolelyontheirintelligenceornaturalabilities,havingapositiveattitudetowardstudents'challenges,framingfailureandfrustrationsaspartofthelearningprocess,andencouragingstudentstosetshort-andlong-termgoals.

(iii)fosterssocialandcollaborativeskillsbyencouragingcollaborationthroughpeerinteractionandbypromotinginterpersonalskills,suchasperspective-taking,empathizing,emotionregulation,andsocialproblem-solving.Studentsexhibitsocialandcollaborativeskillsbycollaboratingwithoneanotherthroughpeerinteraction.

Thesethreeareasencompassnineelements,summarizedabove,capturedin28itemslinkedtospeci?cteachingbehaviors.Basedonthequalityofobservedteachingpracticescapturedinobservedbehaviors,theseitemsareratedonarubricaslow,medium,orhigh.Thescoresarethentranslatedintoa5-pointscalederivedfromtwo15-minutelessonobservationsofthesameteacher.FurtherdetailsabouteachbehaviorandelementcanbefoundintheTeachPrimary

manual

(Molinaetal.,2021).

Tooladaptation

TeachPrimaryObservationTool

TheTeachPrimarytoolwasoriginallydevelopedinEnglishbuthasbeenadaptedandimplemented

acrossvariouslinguisticandculturalcontextstoensureitsapplicability.ForSpanish-speakingcountries,thetoolhasbeentranslatedintoSpanishandusedextensively,requiringminimalcontentadaptation

duetoitsbroadalignmentwithteachingpracticesintheseregions.InAfricancountries,wheremultipleo?ciallanguagesofinstructioncoexist,Teachhasbeentranslatedintoseverallocallanguagestoensureaccessibilityforenumeratorsandrelevancefordiverseclassroomsettings,butoverallthe

implementationreliedonEnglishinSierraLeoneandEthiopia.TheseadaptationsofTeachPrimaryhavefocusedprimarilyonlinguistictranslationwhilemaintaining?delitytotheoriginalconstructs,ensuringthetool'svalidityacrossdi?erentcontextsandisunderstoodbyenumerators.

PLAYClassroomObservationTool

ThePLAYtool,developedinEnglish,requiredtranslationintoSpanishforitsimplementationinPeru.ThistranslationfacilitatedthescoringofteachersinSpanish-speakingclassroomswhilemaintainingthe

theoreticalconstructsandscoringframeworkoftheoriginaltool.Whilenoadditionaladaptationsweremadeforothercountriesincludedinthestudy,the?exibilityofthePLAYtoolallowsforfurther

customizationshouldfuturecontextsrequireit.

Enumeratortraining

Trainingenumeratorsinclassroomobservationtoolsisessentialforensuringthatdatacollectedis

accurate,consistent,andreliable.Ithelpsenumeratorsunderstandhowtousethetoolcorrectly,

observeandrecordbehaviorsobjectively,andavoidpersonalbiases.Propertrainingalsoensuresethicalobservations,protectingtherightsofstudentsandteachers,andhelpsenumeratorsadapttodi?erent

classroomsettings.Byaligningallenumeratorsonhowtoapplythetool,thequalityofthedataimproves,makingitmoreusefulfordecision-makingandcreatingbetterinsightsintoclassroompractices.

TeachPrimaryObservationTool

EnumeratortrainingfortheTeachPrimarytoolfollowsastructuredandrigorousprocessfacilitatedbycerti?edmastertrainers.Publiclyavailableresources,includingimplementationguidelinesandtrainingmaterials,provideacomprehensiveframeworkforpreparingenumeratorsintheTeachPrimarytool

administrationandscoring.Trainingtypicallyspansseveraldaysandinvolvesmultiplecomponents:

?ConceptualUnderstanding:TraineesareguidedthroughthetheoreticalunderpinningsoftheTeachtool,includingitsconstructsandscoringframework.

?PracticalExposure:Enumeratorsengagewithcontext-relevantvideosofteachersinclassrooms,gainingpracticalexperienceinscoringclassroombehaviors.

?SkillPractice:TraineespracticeadministeringandscoringTeachPrimarytobuildcon?denceandaccuracy.

?Certi?cation:Enumeratorsmustpassacerti?cationexamtodemonstratemasteryofthetool.

Thisincludesachievingahighlevelofagreementintheirratingsofstandardizedvideos,ensuringreliabilityandreadinessfor?elddatacollection.

Thisrigorousapproachensuresconsistencyandreliabilityindatacollectionacrossdiverseimplementationcontexts.

PLAYClassroomObservationTool

TrainingforthePLAYtoolwasdesignedtobuildcapacityamongenumeratorsthroughacascading

trainingmodel.Acerti?edmastertrainerconductedacomprehensivetrainingsessionforaselectgroupoftrainers,whothenreplicatedthetrainingwithenumerators.KeycomponentsofthePLAYtraining

included:

?Orientation:Enumeratorswereintroducedtothetool’sconstructsandthemethodsforobservingandscoringengagingteachingandlearning.

?PracticeOpportunities:Sessionsincludedopportunitiestopracticescoringbasedonobservationscenariosandexamples.

UnlikeTeach,thePLAYtrainingdidnotrequireenumeratorstoreachapredeterminedlevelofinter-rateragreementforcerti?cation.However,thetrainingensuredthatenumeratorswerefamiliarwiththetoolandcon?dentintheirabilitytoapplyitduringdatacollection.

3.Method

Studydescription

ThisstudycomparedtheitemcontentofthePLAYandTeachPrimaryclassroomobservationtools.Speci?cally,itinvolvedmappingeachofthe25itemsfromPLAYtooneormoreofthe30itemsfromTeachPrimarytodeterminetheextentofoverlapanddi?erencebetweentheinstruments.AtableoutliningthemainfeaturesofthesetoolsisprovidedasAnnexB.

Participantsinthisstudycomprisedagroupofexpertswithbackgroundsinthedevelopmentand/oruseofclassroomobservationmeasures,particularlywithinlow-andmiddle-incomecountrycontextsandacrossmultipleworldregions.Initially,15expertswereidenti?edforpotentialinvolvement.Afterbeingcontactedviaemailandprovidedwithabriefoverviewofthestudy,nineconsentedtoparticipate.Followingthiscon?rmation,participantsreceivedamoredetailedoutlineofthetaskandTermsofReferenceconcerningtheirconsultancyrole.Participantsengagedincontentreviewandcomparisonofthetwotools,item-to-itemmatchingtasks,andparticipatedinanyfollow-updiscussionsortasks.AnoverviewoftheexpertsisprovidedasAnnexC.

?TheDelphimethodforcontentvalidity

ThisstudystronglydrewupontheprinciplesoftheDelphimethod.TheDelphimethod,utilizedinmultiple?elds,includingeducation,health,andpsychology,allowsexpertstoprovidejudgmentsonatopic,whichcanthenbemodi?edorre?nedbasedonreactionstothecollectiveviewsofthegroup

(Green,2014;

Hyatt

etal.,2024;

ManizadeandMason,2010)

.Unlikeface-to-facetechniques,theDelphimethodpromotesindependenceofthoughtandallowsresearcherstoleveragetheexpertiseofnumerousexpertsfromdi?erentlocationswhilefosteringanenvironmentthatencouragesgroupacceptanceandsharedresponsibilityfortheoutcome.TheDelphimethodtypicallyinvolvesseveralkeycharacteristicsforobtaininggroupconsensusonatopic,including(1)participantanonymity,(2)anonymityofresponses,(3)multipleandstructurediterationsofconsultations,(4)groupconsensus,(5)controlledfeedbackofresponsestoparticipants,and(6)statisticalanalysisofdata

(GrantaandKinney,1992,pp.13-14)

.StudiesutilizingtheDelphimethodhavebeenconductedwithasfewassevenandasmanyas1,000participants,thoughsamplesbetween10and50aretypicallyadvisedduetotheextensivedatageneratedandtheanalysesrequiredforeachindividualresponse

(IqbalandPipon-Young,2009)

.

TheDelphimethodhasemergedasapopularapproachfordeterminingcontentvalidity

(Green,2014;

ManizadeandMason,2010)

.Contentvaliditydenotestheextenttowhichtheitemsofameasurementinstrumentarerelevanttoandrepresentativeofthetargetedconstructitisintendedtomeasure.Itconsiderstheimportance,relevance,andclarityofinstrumentitems,domains,andde?nitions,aswellastheappropriatenessandadequacyofitemresponsescales

(Roebiantoetal.,2023)

.Contentvalidityis

particularlyrelevantasitincreasestheaccuracyofmeasuringconstructs.Forexample,inclassroomobs