HighdynamicrangeimagingDigitalVisualEffectsYung-YuChuangwithslidesbyFredoDurand,BrianCurless,SteveSeitz,PaulDebevecandAlexeiEfrosCameraisanimperfectdeviceCameraisanimperfectdeviceformeasuringtheradiancedistributionofascenebecauseitcannotcapturethefullspectralcontentanddynamicrange.Limitationsinsensordesignpreventcamerasfromcapturingallinformationpassedbylens.CamerapipelinelenssensorshutterAssumeastaticscene,Thus,Lisnotafunctionoftime.Camerapipeline12bits8bitsReal-worldresponsefunctionsIngeneral,theresponsefunctionisnotprovidedbycameramakerswhoconsideritpartoftheirproprietaryproductdifferentiation.Inaddition,theyarebeyondthestandardgammacurves.Theworldishighdynamicrange11,50025,000400,0002,000,000,000TheworldishighdynamicrangeRealworlddynamicrangeEyecanadaptfrom~10-6to106cd/m2Often1:100,000inasceneTypical1:50,max1:500forpictures10-6106RealworldHighdynamicrangespotmeterShortexposure10-610610-6106RealworldradiancePictureintensitydynamicrangePixelvalue0to255Longexposure10-610610-6106RealworldradiancePictureintensitydynamicrangePixelvalue0to255CameraisnotaphotometerLimiteddynamicrangePerhapsusemultipleexposures?Unknown,nonlinearresponse

NotpossibletoconvertpixelvaluestoradianceSolution:Recoverresponsecurvefrommultipleexposures,thenreconstructtheradiancemapVaryingexposureWaystochangeexposureShutterspeedApertureNeutraldensityfiltersShutterspeedNote:shuttertimesusuallyobeyapowerseries–each“stop”isafactorof2?,1/8,1/15,1/30,1/60,1/125,1/250,1/500,1/1000secUsuallyreallyis:?,1/8,1/16,1/32,1/64,1/128,1/256,1/512,1/1024secVaryingshutterspeedsHDRIcapturingfrommultipleexposuresCaptureimageswithmultipleexposuresImagealignment(evenifyouusetripod,itissuggestedtorunalignment)ResponsecurverecoveryGhost/flareremovalImagealignmentWewillintroduceafastandeasy-to-implementmethodforthistask,calledMedianThresholdBitmap(MTB)alignmenttechnique.Consideronlyintegraltranslations.Itisenoughempirically.TheinputsareNgrayscaleimages.(YoucaneitherusethegreenchannelorconvertintograyscalebyY=(54R+183G+19B)/256)MTBisabinaryimageformedbythresholdingtheinputimageusingthemedianofintensities.WhyisMTBbetterthangradient?Edge-detectionfiltersaredependentonimageexposuresTakingthedifferenceoftwoedgebitmapswouldnotgiveagoodindicationofwheretheedgesaremisaligned.SearchfortheoptimaloffsetTryallpossibleoffsets.GradientdescentMultiscaletechniquelog(max_offset)levelsTry9possibilitiesforthetoplevelScaleby2whenpassingdown;tryits9neighborsThresholdnoiseignorepixelsthatareclosetothethresholdexclusionbitmapEfficiencyconsiderationsXORfortakingdifferenceANDwithexclusionmapsBitcountingbytablelookupResultsSuccessrate=84%.10%failureduetorotation.3%forexcessivemotionand3%fortoomuchhigh-frequencycontent.Recoveringresponsecurve12bits8bitsRecoveringresponsecurveWewanttoobtaintheinverseoftheresponsecurve0255Dt=

1/4secDt=

1secDt=

1/8secDt=

2secImageseriesDt=

1/2secRecoveringresponsecurve?1?1?1?1?1?3?3?3?3?3?2?2?2?2?20255Dt=

1/4secDt=

1secDt=

1/8secDt=

2secImageseriesDt=

1/2secRecoveringresponsecurve?1?1?1?1?1?3?3?3?3?3?2?2?2?2?2Xij=lnXijIdeabehindthemathln2IdeabehindthemathEachlineforascenepoint.TheoffsetisessentiallydeterminedbytheunknownEiIdeabehindthemathNotethatthereisashiftthatwecan’trecoverBasicideaDesignanobjectivefunctionOptimizeitMathforrecoveringresponsecurveRecoveringresponsecurveThesolutioncanbeonlyuptoascale,addaconstraintAddahatweightingfunctionRecoveringresponsecurveWewantIfP=11,N~25(typically50isused)Wepreferthatselectedpixelsarewelldistributedandsampledfromconstantregions.Theypickedpointsbyhand.ItisanoverdeterminedsystemoflinearequationsandcanbesolvedusingSVDHowtooptimize?1.SetpartialderivativestozeroHowtooptimize?Setpartialderivativestozero

SparselinearsystemAx=b256nn×p1254g(0)g(255)lnE1lnEn:::QuestionsWillg(127)=0alwaysbesatisfied?Whyorwhynot?Howtofindtheleast-squaresolutionforanover-determinedsystem?Least-squaresolutionforalinearsystemTheyareoftenmutuallyincompatible.Weinsteadfindxtominimizethenormoftheresidualvector.Iftherearemultiplesolutions,weprefertheonewiththeminimallength.Least-squaresolutionforalinearsystemIfweperformSVDonAandrewriteitasthenistheleast-squaresolution.pseudoinverseProofProofLibrariesforSVDMatlabGSLBoostLAPACKATLASMatlabcodeMatlabcodefunction[g,lE]=gsolve(Z,B,l,w)n=256;A=zeros(size(Z,1)*size(Z,2)+n+1,n+size(Z,1));b=zeros(size(A,1),1);k=1;%%Includethedata-fittingequationsfori=1:size(Z,1)forj=1:size(Z,2)wij=w(Z(i,j)+1);A(k,Z(i,j)+1)=wij;A(k,n+i)=-wij;b(k,1)=wij*B(i,j);k=k+1;endendA(k,129)=1;%%Fixthecurvebysettingitsmiddlevalueto0k=k+1;fori=1:n-2%%IncludethesmoothnessequationsA(k,i)=l*w(i+1);A(k,i+1)=-2*l*w(i+1);A(k,i+2)=l*w(i+1);k=k+1;endx=A\b;%%SolvethesystemusingSVDg=x(1:n);lE=x(n+1:size(x,1));RecoveredresponsefunctionConstructingHDRradiancemapcombinepixelstoreducenoiseandobtainamorereliableestimationReconstructedradiancemapWhatisthisfor?HumanperceptionVision/graphicsapplications

AutomaticghostremovalbeforeafterWeightedvarianceMovingobjectsandhigh-contrastedgesrenderhighvariance.RegionmaskingThresholding;dilation;identifyregions;BestexposureineachregionLensflareremovalbeforeafterEasierHDRreconstructionrawimage=12-bitCCDsnapshot

EasierHDRreconstructionXij=Ei*Δtj

Exposure(X)Δt12bytesperpixel,4foreachchannelsignexponentmantissaPF7685121<binaryimagedata>FloatingPointTIFFsimilarTextheadersimilartoJeffPoskanzer’s.ppm

imageformat:PortablefloatMap(.pfm)(145,215,87,149)=(145,215,87)*2^(149-128)=(1190000,1760000,713000)(145,215,87,103)=(145,215,87)*2^(103-128)=(0.00000432,0.00000641,0.00000259)Ward,Greg."RealPixels,"inGraphicsGemsIV,editedbyJamesArvo,AcademicPress,1994Radianceformat(.pic,.hdr,.rad)RedGreenBlueExponent32bits/pixelILM’sOpenEXR(.exr)6bytesperpixel,2foreachchannel,compressedsignexponentmantissa

Severallosslesscompressionoptions,2:1typicalCompatiblewiththe“half”datatypeinNVidia'sCgSupportednativelyonGeForceFXandQuadroFX

Availableat/RadiometricselfcalibrationAssumethatanyresponsefunctioncanbemodeledasahigh-orderpolynomialNoneedtoknowexposuretimeinadvance.UsefulforcheapcamerasZXMitsunagaandNayarTofindthecoefficientscmtominimizethefollowingAguessfortheratioof

MitsunagaandNayarAgain,wecanonlysolveuptoascale.Thus,addaconstraintf(1)=1.ItreducestoM-1variables.Howtosolveit?MitsunagaandNayarWesolvetheaboveiterativelyandupdatetheexposureratioaccordinglyHowtodetermineM?SolveuptoM=10andpickuptheonewiththeminimalerror.Noticethatyouprefertohavethesameorderforallchannels.Usethecombinederror.Robertsonet.al.Givenand,thegoalistofindbothandMaximumlikelihoodRobertsonet.al.repeatassumingisknown,optimizeforassumingisknown,optimizeforuntilconvergeRobertsonet.al.repeatassumingisknown,optimizeforassumingisknown,optimizeforuntilconvergeRobertsonet.al.repeatassumingisknown,optimizeforassumingisknown,optimizeforuntilconvergeRobertsonet.al.repeatassumingisknown,optimizeforassumingisknown,optimizeforuntilconvergenormalizesothatSpaceofresponsecurvesSpaceofresponsecurvesPatch-BasedHDRHDRVideoHighDynamicRangeVideo

SingBingKang,MatthewUyttendaele,SimonWinder,RichardSzeliskiSIGGRAPH2003

videoAssortedpixelAssortedpixelAssortedpixelAVersatileHDRVideoSystem

videoAVersatileHDRVideoSystemHDRbecomescommonpracticeManycamerashasbracketexposuremodesiPhone4hasHDRoption,butitismoreexposure