MachineLearning:

findingpatternsOutlineMachinelearningandClassificationExamples*LearningasSearchBiasWeka2FindingpatternsGoal:programsthatdetectpatternsandregularitiesinthedataStrongpatternsgoodpredictionsProblem1:mostpatternsarenotinterestingProblem2:patternsmaybeinexact(or spurious)Problem3:datamaybegarbledormissing3MachinelearningtechniquesAlgorithmsforacquiringstructuraldescriptionsfromexamplesStructuraldescriptionsrepresentpatternsexplicitlyCanbeusedtopredictoutcomeinnewsituationCanbeusedtounderstandandexplainhowpredictionisderived

(maybeevenmoreimportant)Methodsoriginatefromartificialintelligence,statistics,andresearchondatabaseswitten&eibe4Canmachinesreallylearn?Definitionsof“learning”fromdictionary:Togetknowledgeofbystudy,

experience,orbeingtaughtTobecomeawarebyinformationor

fromobservationTocommittomemoryTobeinformedof,ascertain;toreceiveinstructionDifficulttomeasureTrivialforcomputersThingslearnwhentheychangetheirbehaviorinawaythatmakesthemperformbetterinthefuture.Operationaldefinition:Doesaslipperlearn?Doeslearningimplyintention?witten&eibe5ClassificationLearnamethodforpredictingtheinstanceclassfrompre-labeled(classified)instancesManyapproaches:Regression,DecisionTrees,Bayesian,NeuralNetworks,...Givenasetofpointsfromclasseswhatistheclassofnewpoint?6Classification:LinearRegressionLinearRegressionw0+w1x+w2y>=0Regressioncomputeswifromdatatominimizesquarederrorto‘fit’thedataNotflexibleenough7Classification:DecisionTreesXYifX>5thenblueelseifY>3thenblueelseifX>2thengreenelseblue5238Classification:NeuralNetsCanselectmorecomplexregionsCanbemoreaccurateAlsocanoverfitthedata–findpatternsinrandomnoise9OutlineMachinelearningandClassificationExamples*LearningasSearchBiasWeka10TheweatherproblemOutlookTemperatureHumidityWindyPlaysunnyhothighfalsenosunnyhothightruenoovercasthothighfalseyesrainymildhighfalseyesrainymildnormalfalseyesrainymildnormaltruenoovercastmildnormaltrueyessunnymildhighfalsenosunnymildnormalfalseyesrainymildnormalfalseyessunnymildnormaltrueyesovercastmildhightrueyesovercasthotnormalfalseyesrainymildhightruenoGivenpastdata,CanyoucomeupwiththerulesforPlay/NotPlay?Whatisthegame?11The

weatherproblemGiventhisdata,whataretherulesforplay/notplay?OutlookTemperatureHumidityWindyPlaySunnyHotHighFalseNoSunnyHotHighTrueNoOvercastHotHighFalseYesRainyMildNormalFalseYes……………12The

weatherproblemConditionsforplayingOutlookTemperatureHumidityWindyPlaySunnyHotHighFalseNoSunnyHotHighTrueNoOvercastHotHighFalseYesRainyMildNormalFalseYes……………Ifoutlook=sunnyandhumidity=highthenplay=noIfoutlook=rainyandwindy=truethenplay=noIfoutlook=overcastthenplay=yesIfhumidity=normalthenplay=yesIfnoneoftheabovethenplay=yeswitten&eibe13WeatherdatawithmixedattributesOutlookTemperatureHumidityWindyPlaysunny8585falsenosunny8090truenoovercast8386falseyesrainy7096falseyesrainy6880falseyesrainy6570truenoovercast6465trueyessunny7295falsenosunny6970falseyesrainy7580falseyessunny7570trueyesovercast7290trueyesovercast8175falseyesrainy7191trueno14WeatherdatawithmixedattributesHowwilltheruleschangewhensomeattributeshavenumericvalues?OutlookTemperatureHumidityWindyPlaySunny8585FalseNoSunny8090TrueNoOvercast8386FalseYesRainy7580FalseYes……………15WeatherdatawithmixedattributesRuleswithmixedattributesOutlookTemperatureHumidityWindyPlaySunny8585FalseNoSunny8090TrueNoOvercast8386FalseYesRainy7580FalseYes……………Ifoutlook=sunnyandhumidity>83thenplay=noIfoutlook=rainyandwindy=truethenplay=noIfoutlook=overcastthenplay=yesIfhumidity<85thenplay=yesIfnoneoftheabovethenplay=yeswitten&eibe16ThecontactlensesdataAgeSpectacleprescriptionAstigmatismTearproductionrateRecommendedlensesYoungMyopeNoReducedNoneYoungMyopeNoNormalSoftYoungMyopeYesReducedNoneYoungMyopeYesNormalHardYoungHypermetropeNoReducedNoneYoungHypermetropeNoNormalSoftYoungHypermetropeYesReducedNoneYoungHypermetropeYesNormalhardPre-presbyopicMyopeNoReducedNonePre-presbyopicMyopeNoNormalSoftPre-presbyopicMyopeYesReducedNonePre-presbyopicMyopeYesNormalHardPre-presbyopicHypermetropeNoReducedNonePre-presbyopicHypermetropeNoNormalSoftPre-presbyopicHypermetropeYesReducedNonePre-presbyopicHypermetropeYesNormalNonePresbyopicMyopeNoReducedNonePresbyopicMyopeNoNormalNonePresbyopicMyopeYesReducedNonePresbyopicMyopeYesNormalHardPresbyopicHypermetropeNoReducedNonePresbyopicHypermetropeNoNormalSoftPresbyopicHypermetropeYesReducedNonePresbyopicHypermetropeYesNormalNonewitten&eibe17AcompleteandcorrectrulesetIftearproductionrate=reducedthenrecommendation=noneIfage=youngandastigmatic=no

andtearproductionrate=normalthenrecommendation=softIfage=pre-presbyopicandastigmatic=no

andtearproductionrate=normalthenrecommendation=softIfage=presbyopicandspectacleprescription=myope

andastigmatic=nothenrecommendation=noneIfspectacleprescription=hypermetropeandastigmatic=no

andtearproductionrate=normalthenrecommendation=softIfspectacleprescription=myopeandastigmatic=yes

andtearproductionrate=normalthenrecommendation=hardIfageyoungandastigmatic=yes

andtearproductionrate=normalthenrecommendation=hardIfage=pre-presbyopic

andspectacleprescription=hypermetrope

andastigmatic=yesthenrecommendation=noneIfage=presbyopicandspectacleprescription=hypermetrope

andastigmatic=yesthenrecommendation=nonewitten&eibe18Adecisiontreeforthisproblemwitten&eibe19ClassifyingirisflowersSepallengthSepalwidthPetallengthPetalwidthType0.2Irissetosa24.93.01.40.2Irissetosa…517.0Irisversicolor51.5Irisversicolor…102.5Irisvirginica101.9Irisvirginica…Ifpetallength<2.45thenIrissetosaIfsepalwidth<2.10thenIrisversicolor...witten&eibe20Example:209differentcomputerconfigurationsLinearregressionfunctionPredictingCPUperformanceCycletime(ns)Mainmemory(Kb)Cache(Kb)ChannelsPerformanceMYCTMMINMMAXCACHCHMINCHMAXPRP112525660002561612819822980003200032832269…20848051280003200672094801000400000045PRP= -55.9+0.0489MYCT+0.0153MMIN+0.0056MMAX

+0.6410CACH-0.2700CHMIN+1.480CHMAXwitten&eibe21SoybeanclassificationAttributeNumberofvaluesSamplevalueEnvironmentTimeofoccurrence7JulyPrecipitation3Abovenormal…SeedCondition2NormalMoldgrowth2Absent…FruitConditionoffruitpods4NormalFruitspots5?LeavesCondition2AbnormalLeafspotsize3?…StemCondition2AbnormalStemlodging2Yes…RootsCondition3NormalDiagnosis19Diaporthestemcankerwitten&eibe22TheroleofdomainknowledgeIfleafconditionisnormal

andstemconditionisabnormal

andstemcankersisbelowsoilline

andcankerlesioncolorisbrownthen

diagnosisisrhizoctoniarootrotIfleafmalformationisabsent

andstemconditionisabnormal

andstemcankersisbelowsoilline

andcankerlesioncolorisbrownthen

diagnosisisrhizoctoniarootrotButinthisdomain,“leafconditionisnormal”implies

“leafmalformationisabsent”!witten&eibe23OutlineMachinelearningandClassificationExamples*LearningasSearch

BiasWeka24LearningassearchInductivelearning:findaconceptdescriptionthatfitsthedataExample:rulesetsasdescriptionlanguageEnormous,butfinite,searchspaceSimplesolution:enumeratetheconceptspaceeliminatedescriptionsthatdonotfitexamplessurvivingdescriptionscontaintargetconceptwitten&eibe25EnumeratingtheconceptspaceSearchspaceforweatherproblem4x4x3x3x2=288possiblecombinationsWith14rules2.7x1034possiblerulesetsSolution:candidate-eliminationalgorithmOtherpracticalproblems:MorethanonedescriptionmaysurviveNodescriptionmaysurviveLanguageisunabletodescribetargetconceptordatacontainsnoisewitten&eibe26TheversionspaceSpaceofconsistentconceptdescriptionsCompletelydeterminedbytwosetsL:mostspecificdescriptionsthatcoverallpositiveexamplesandnonegativeonesG:mostgeneraldescriptionsthatdonotcoveranynegativeexamplesandallpositiveonesOnlyLandGneedbemaintainedandupdatedBut:stillcomputationallyveryexpensiveAnd:doesnotsolveotherpracticalproblemswitten&eibe27*Versionspaceexample,1Given:redorgreencowsorchicken

Startwith: L={} G={<*,*>}Firstexample:<green,cow>:positive

HowdoesthischangeLandG?witten&eibe28*Versionspaceexample,2Given:redorgreencowsorchicken

Result: L={<green,cow>} G={<*,*>}Secondexample:<red,chicken>:negativewitten&eibe29*Versionspaceexample,3Given:redorgreencowsorchicken

Result: L={<green,cow>} G={<green,*>,<*,cow>}Finalexample:<green,chicken>:positive

witten&eibe30*Versionspaceexample,4Given:redorgreencowsorchicken

Resultantversionspace: L={<green,*>} G={<green,*>}witten&eibe31*Versionspaceexample,5Given:redorgreencowsorchicken

L={} G={<*,*>}<green,cow>:positive L={<green,cow>} G={<*,*>}<red,chicken>:negative L={<green,cow>} G={<green,*>,<*,cow>}<green,chicken>:positive L={<green,*>} G={<green,*>}witten&eibe32*Candidate-eliminationalgorithmInitializeLandGForeachexamplee: Ifeispositive: DeleteallelementsfromGthatdonotcovere

ForeachelementrinLthatdoesnotcovere: Replacerbyallofitsmostspecificgeneralizations

that 1.covereand 2.aremorespecificthansomeelementinG RemoveelementsfromLthat

aremoregeneralthansomeotherelementinL Ifeis

negative: DeleteallelementsfromLthatcovere

ForeachelementrinGthatcoverse:

Replacerbyallofitsmostgeneralspecializations

that 1.donotcovereand

2.aremoregeneralthansomeelementinL

RemoveelementsfromGthat

aremorespecificthansomeotherelementinGwitten&eibe33OutlineMachinelearningandClassificationExamples*LearningasSearchBiasWeka34BiasImportantdecisionsinlearningsystems:ConceptdescriptionlanguageOrderinwhicht