OverviewOverviewPart1:ReviewofObjectTracking?SingleObjectTracking(SOT)?VideoObjectSegmentation(VOS)?MultipleObjectTracking(MOT)?Multi-ObjectTrackingandSegmentation(MOTS)?SummaryPart2:TowardsGrandUnificationofObjectTracking?GeneralVisionModels(GVM)?UnificationofObjectTracking?Unicorn?Experiments?FurtherAnalysis PartPart1:ReviewofObjectTracking?SingleObjectTracking(SOT)?VideoObjectSegmentation(VOS)?MultipleObjectTracking(MOT)?Multi-ObjectTrackingandSegmentation(MOTS) SingleSingleObjectTracking(SOT)TrackanarbitraryobjectinavideogivenitsinitiallocationSingle-object,Any-classOcclusion,LightChange,BackgroundClutter,etc. zCorrHead !Online !Head !TransformerzHeadxfffSingleObjectTrackingzCorrHead !Online !Head !TransformerzHeadxfffSingleObjectTracking(SOT)SiameseRPNf !fx?SiamRPN(CVPR18)?DaSiamRPN(ECCV18)?SiameRPN++(CVPR19)?Ocean(ECCV20)zDCFx?ATOM(CVPR19)?DiMP(ICCV19)?PrDiMP(CVPR20)?KYS(ECCV20)f !fTransf !f?TransT(CVPR21)?STARK(ICCV21)MostSOTmethodsarebasedonthesearchregion.Pros:Cons:?SavingcomputationV.S?Sensitivetotemporarytrackingfailure?Filteringoutdistractors?Time-consumingwhennumofobjectsislarge UnsupervisedVOSReferringUnsupervisedVOSReferringVOSVideoObjectSegmentation(VOS)nGoalnSegmentspecificobjectspreciselyinavideo.SegmentsalientmovingobjectSemi-supervisedVOSSegmentobjectsgiveninthe1stframebymasksSegmentobjectsgiveninthe1stframebylanguageSTM(ICCVSTM(ICCV19)CFBI(ECCV20)STCN(NeurIPS21)VideoObjectSegmentation(VOS)Semi-supervisedVOSisdominatedbySpace-TimeMemoryNetworkAlthoughachievinggreatperformance,STM-basedmethodssufferfromthefollowingdisadvantages:?Hugetimeandspacecomplexity,especiallyforhighspatialresolutionandthelongsequence.?Highlyrelyingonhigh-qualitymaskannotationsonthefirstframe.MultipleObjectMultipleObjectTracking(MOT)nGoalnTrackallobjectsofspecificclassesinavideo.MOTChallengeBDD100KVisdrone(1class:Person)(8classes:Car,pedestrian,etc)(10classes:Car,pedestrian,etc)ParadigmParadigmMultipleObjectTracking(MOT)RepresentativeMethodsuTrackingbyDetectionuTrackingbyDetection(SORT,DeepSORT,StrongSORT)uJointDetectionandTrackinguJointDetectionandTracking(JDE,FairMOT,CenterTrack,QDTrack)(TrackFormer,GTR)MOTmethodstakesthehigh-resolutionwholeimageastheinputtodetectobjectsascompletelyaspossible.Multi-ObjectTrackingandSegmentation(MOTS)nGoalnSegmentallobjectsofspecificclassesinavideo.MOTSChallengeBDD100KMOTS(1class:Person)(8classes:Car,pedestrian,etc)MOTScanbeseenasavariantofMOTbyreplacingboxeswithmasks.SummarSummaryReferenceOutputsClassTrackspervideoRepresentativeMethodsTypicalInputsSOTInitialboxBoxesagnosticOneOne-ShotDetectionSmallsearchregionVOSInitialmaskMasksagnosticSeveralSTMMedium-resolutionWholeImageMOTNOBoxesspecificTensorhundredsDetection+AssociationHigh-resolutionWholeImageMOTSNOMasksspecificTensorhundredsDetection+AssociationHigh-resolutionWholeImagettherearelargegapsbetweenthefourtrackingtasks?GeneralVisionModels(GVM)?UnificationofObjectTracking?Unicorn?Experiments?FurtherAnalysis entAIvsAGI–CurrentweakAIisdesignedforsolvingonespecifictask.–Artificialgeneralintelligence(AGI)isexpectedtounderstandorlearnanyintellectualtaskthatahumanbeingcan. ?Pioneeringworksinthepastyear2021.082021.112021.112022.01ies Threeobstacleshinderingtheunification:(1)Thecharacteristicsoftrackedobjectsvary(onetargetofanyclassgiveninthereferenceframev.stensevenhundredsofinstancesofspecificcategories)(2)SOTandMOTrequiredifferenttypesofcorrespondence.(pixel-levelcorrespondencedistinguishingthetargetfromthebackgroundv.sinstance-levelcorrespondencematchingthecurrentlydetectedobjectswithprevioustrajectories)(3)DifferentInputs.(smallsearchregiontosavecomputationandfilterpotentialdistractorsv.shigh-resolutionfullimagefordetectinginstancesascompleteaspossible) ?WeproposeUnicorn,aunifiedsolutionforSOT,MOT,VOSandMOTS.?Unicornaccomplishesthegreatunificationofthenetworkarchitectureandthelearningparadigmforfourtrackingtasks.?Unicornputsforwardsnewstate-of-the-artperformanceonmultiplechallengingtrackingbenchmarkswiththesamemodelparameters. Unifiedinputsandbackbone?Takingthefullimagesasinputsforalltasks.?Referenceframeisthe1stframeforSOT&VOSandthe(t-1)thframeforMOT&MOTS?Oneunifiedbackbone(ConvNeXtbydefault)ErefeRhwxcEcureRhwxcCpixeRhwxhwForMOT&MOTS,TheinstanceembeddingeisextractedfromtheframeembeddingE,wherethecenteroftheinstanceislocatederefeRMxc,ecureRNxcCinsteRNxMCinstisthesub-matrixofCpixLearninghighlydiscriminativeembedding{Eref,Ecur}isthekeytobuildingprecisecorrespondenceforalltrackingtasks.Aninteractionmoduleisusedtoenhancedtheoriginalimagefeature.Bydefaultweusethedeformableattentionblockforinteraction.LearningCorrespondencebyPropagation&LearningCorrespondencebyPropagation&Association.?ForSOT&VOS,Correspondencehelpstopropagatethetargetmapfromthereferenceframetothecurrentframe.?ForMOT&MOTS,Correspondencehelpstomatchthedetectionsonthecurrentframewiththetrajectoriesonthereferenceframe.Weintroducethetargetpriorastheswitchamongfourtrackingtasks.?ForSOT&VOS,thetargetpriorcanenhancetheoriginalFPNfeatureandmakesthenetworkfocusonthetrackedtarget.?ForMOT&MOTS,thefusedfeatureF′degeneratesbacktotheoriginalFPNfeatureFtodetectobjectsofspecificclasses.ObjectObjectdetectionheadbasedonYOLOXandCondInst?One-stage,anchor-free?NoRoIoperationssuchasRoI-AlignYOLOXHeadforobjectdetectionCondInstHeadforinstancesegmentationAddthemaskbranchandfreezeotherparametersStage1Target:Correspondence+DetectionLoss:Lstage1=Lcorr+LdetData:1:1fromSOT&MOTSOT:weuseCOCO,LaSOT,GOT-10KandTrackingNetMOT:?ForMOT17,weuseCrowdhuman,ETHZ,CityPerson,MOT17?ForBDD100K,weuseBDD100KStage2Target:MaskLoss:Lstage2=LmaskData:1:1fromVOS&MOTSVOS:weuseCOCO,DAVIS,Youtube-VOSMOTS:?ForMOTS,weuseCOCOandMOTS?ForBDD100K,weuseBDD100K?TrainingofVOS&MOTSwouldnotimpacttheperformanceofSOT&MOT.ForuserswhoareonlyinterestedintheSOT&MOT,runningStage1isenough.?Ineachstage,wetrainthemodel