Complimentsof

Accelerating

AIwith

SyntheticData

GeneratingDataforAIProjects

KhaledElEmam

THELEADERINAICOMPUTING.

Signuptogetthe

latestAInewsstraight

toyourinbox.

SUBSCRIBE

AcceleratingAIwith SyntheticDataGeneratingDataforAIProjects

KhaledElEmam

Beijing·Boston·Farnham·Sebastopol·Tokyo

AcceleratingAIwithSyntheticData

byKhaledElEmam

Copyright?2020O’ReillyMedia,Inc.Allrightsreserved.

PrintedintheUnitedStatesofAmerica.

PublishedbyO’ReillyMedia,Inc.,1005GravensteinHighwayNorth,Sebastopol,CA95472.

O’Reillybooksmaybepurchasedforeducational,business,orsalespromotionaluse.Onlineeditionsarealsoavailableformosttitles

(

).Formoreinfor‐mation,contactourcorporate/institutionalsalesdepartment:800-998-9938or

corporate@

.

AcquisitionsEditor:JonathanHassell

DevelopmentEditor:MelissaPotter

ProductionEditor:DanielElfanbaum

Copyeditor:SharonWilkey

Proofreader:ShannonTurlington

InteriorDesigner:DavidFutato

CoverDesigner:KarenMontgomery

Illustrator:RebeccaDemarest

June2020:FirstEdition

RevisionHistoryfortheFirstEdition

2020-06-03:FirstRelease

TheO’ReillylogoisaregisteredtrademarkofO’ReillyMedia,Inc.AcceleratingAIwithSyntheticData,thecoverimage,andrelatedtradedressaretrademarksofO’ReillyMedia,Inc.

Theviewsexpressedinthisworkarethoseoftheauthor,anddonotrepresentthepublisher’sviews.Whilethepublisherandtheauthorhaveusedgoodfaitheffortstoensurethattheinformationandinstructionscontainedinthisworkareaccurate,thepublisherandtheauthordisclaimallresponsibilityforerrorsoromissions,includ‐ingwithoutlimitationresponsibilityfordamagesresultingfromtheuseoforreli‐anceonthiswork.Useoftheinformationandinstructionscontainedinthisworkisatyourownrisk.Ifanycodesamplesorothertechnologythisworkcontainsordescribesissubjecttoopensourcelicensesortheintellectualpropertyrightsofoth‐ers,itisyourresponsibilitytoensurethatyourusethereofcomplieswithsuchlicen‐sesand/orrights.

ThisworkispartofacollaborationbetweenO’ReillyandNVIDIA.Seeour

state‐

mentofeditorialindependence

.

978-1-492-04596-0

[LSI]

v

TableofContents

1.

DefiningSyntheticData 1

WhatIsSyntheticData?

2

TheBenefitsofSyntheticData

5

LearningtoTrustSyntheticData

9

OtherApproachestoAccessingData

11

GeneratingSyntheticDatafromRealData

12

Conclusions

15

2.

TheSynthesisProcess 17

DataSynthesisProjects

17

TheDataSynthesisPipeline

21

SynthesisProgramManagement

27

BestPracticesforImplementingDataSynthesis

28

Conclusions

30

3.

SyntheticDataCaseStudies 33

ManufacturingandDistribution

34

HealthCare

36

FinancialServices

43

Transportation

46

Conclusions

50

4.

TheFutureofDataSynthesis 51

CreatingaDataUtilityFramework

51

RemovingInformationfromSyntheticData

52

vi|TableofContents

UsingDataWatermarking

53

GeneratingSynthesisfromSimulators

54

Conclusions

55

CHAPTER1

1

DefiningSyntheticData

Interestinsyntheticdatahasbeengrowingquiterapidlyoverthelastfewyears.Thishasbeendrivenbytwosimultaneoustrends.Thefirstisthedemandforlargeamountsofdatatotrainandbuildarti-ficialintelligenceandmachinelearning(AIML)models.Thesecondisrecentworkthathasdemonstratedeffectivemethodstogeneratehigh-qualitysyntheticdata.Bothhaveresultedintherecognitionthatsyntheticdatacansolvesomedifficultproblemsquiteeffec-tively,especiallywithintheAIMLcommunity.Groupsandbusi-nesseswithincompanieslikeNVIDIA,IBM,andAlphabet,aswellasagenciessuchastheUSCensusBureau,haveadopteddifferenttypesofdatasynthesistosupportmodelbuilding,applicationdevel-opment,anddatadissemination.

Thisreportprovidesageneraloverviewofsyntheticdatageneration,withafocusonthebusinessvalueandusecases,andhigh-levelcov-erageoftechniquesandimplementationpractices.Weaimtoanswerthequestionsthatabusinessreaderwouldtypicallyask(andhastypicallyasked),butatthesametimeprovidesomedirectiontoanalyticsleadershipseekingtounderstandtheoptionsavailableandwheretolooktogetstarted.

WeshowhowsyntheticdatacanaccelerateAIMLprojects.Someproblemsthatcanbetackledbyusingsyntheticdatawouldbetoocostlyordangerous(e.g.,inthecaseoftrainingmodelscontrollingautonomousvehicles)tosolveusingmoretraditionalmethods,orsimplycannotbedoneotherwise.

2|Chapter1:DefiningSyntheticData

AIMLprojectsrunindifferentindustries,andthemultipleindustryusecasesthatweincludeinthisreportareintendedtogiveyouaflavorofthebroadapplicationsofdatasynthesis.WedefineanAIMLprojectquitebroadlyaswell,toinclude,forexample,thedevelopmentofsoftwareapplicationsthathaveAIMLcomponents.Thereportisdividedintofourchapters.Thisintroductorychaptercoversbasicconceptsandpresentsthecaseforsyntheticdata.

Chap‐

ter2

presentsthedatasynthesisprocessandpipelines,scalingimplementationintheenterprise,andbestpractices.Aseriesofindustry-specificcasestudiesfollowin

Chapter3

.

Chapter4

isforward-lookingandconsiderswherethistechnologyisheaded.

Inthischapter,westartbydefiningthetypesofsyntheticdata.Thisisfollowedbyadescriptionofthebenefitsofusingsyntheticdata—thetypesofproblemsthatdatasynthesiscansolve.Giventherecentadoptionofthisapproachintopractice,buildingtrustinanalysisresultsfromsyntheticdataisimportant.Wethereforealsopresentexamplessupportingtheutilityofsyntheticdataanddiscussmeth‐odstobuildtrust.

Alternativestodatasynthesisexist,andwepresentthesenextwithanassessmentofstrengthsandweaknesses.Thischapterthencloseswithanoverviewofmethodsforsyntheticdatageneration.

WhatIsSyntheticData?

Ataconceptuallevel,syntheticdataisnotrealdatabutisdatathathasbeengeneratedfromrealdataandthathasthesamestatisticalpropertiesastherealdata.Thismeansthatananalystwhoworkswithasyntheticdatasetshouldgetanalysisresultsthataresimilartothosetheywouldgetwithrealdata.Thedegreetowhichasyn‐theticdatasetisanaccurateproxyforrealdataisameasureofutil-ity.Furthermore,werefertotheprocessofgeneratingsyntheticdataassynthesis.

Datainthiscontextcanmeandifferentthings.Forexample,datacanbestructureddata(i.e.,rowsandcolumns),asonewouldseeinarelationaldatabase.Datacanalsobeunstructuredtext,suchasdoc‐tors’notes,transcriptsofconversationsamongpeopleorwithdigitalassistants,oronlineinteractionsbyemailorchat.Furthermore,images,videos,audio,andvirtualenvironmentsarealsotypesofdatathatcanbesynthesized.Wehaveseenexamplesoffakeimages

WhatIsSyntheticData?|3

inthemachinelearningliterature;forinstance,realisticfacesofpeoplewhodonotexistintherealworldcanbecreated,andyoucan

viewtheresults

online.

Syntheticdataisdividedintotwotypes,basedonwhetheritisgen‐eratedfromactualdatasetsornot.

Thefirsttypeissynthesizedfromrealdatasets.Theanalystwillhavesomerealdatasetsandthenbuildamodeltocapturethedistribu‐tionsandstructureofthatrealdata.Here,structuremeansthemul‐tivariaterelationshipsandinteractionsinthedata.Thenthesyntheticdataissampledorgeneratedfromthatmodel.Ifthemodelisagoodrepresentationoftherealdata,thesyntheticdatawillhavesimilarstatisticalpropertiesastherealdata.

Forexample,adatasciencegroupspecializinginunderstandingcustomerbehaviorswouldneedlargeamountsofdatatobuilditsmodels.Butbecauseofprivacyorotherconcerns,theprocessforgettingaccesstothatcustomerdataisslowanddoesnotprovidegoodenoughdatawhenitdoesarrivebecauseofextensivemaskingandredactionofinformation.Instead,asyntheticversionoftheproductiondatasetscanbeprovidedtotheanalystsforbuildingtheirmodels.Thesynthesizeddatawillhavefewerconstraintsputonitsuseandwouldallowthemtoprogressmorerapidly.

Thesecondtypeofsyntheticdataisnotgeneratedfromrealdata.Itiscreatedbyusingexistingmodelsorbyusingbackgroundknowl‐edgeoftheanalyst.Theseexistingmodelscanbestatisticalmodelsofaprocess(forexample,developedthroughsurveysorotherdatacollectionmechanisms)ortheycanbesimulations.Simulationscanbecreated,forinstance,bygamingenginesthatcreatesimulated(andsynthetic)imagesofscenesorobjects,orbysimulationenginesthatgenerateshopperdatawithparticularcharacteristics(say,ageandgender)ofpeoplewhowalkpastthesiteofaprospectivestoreatdifferenttimesoftheday.

Backgroundknowledgecanbe,forexample,amodelofhowafinancialmarketbehavesbasedontextbookdescriptionsorbasedonthebehaviorsofstockpricesundervarioushistoricalconditions,oritcanbeknowledgeofthestatisticaldistributionofhumantrafficinastorebasedonyearsofexperience.Insuchacase,itisrelativelystraightforwardtocreateamodelandsamplefromittogeneratesyntheticdata.Iftheanalyst’sknowledgeoftheprocessisaccurate,thesyntheticdatawillbehaveinamannerthatisconsistentwith

4|Chapter1:DefiningSyntheticData

real-worlddata.Ofcourse,thisworksonlywhenthephenomenonofinterestistrulywellunderstood.

Asafinalexample,whenaprocessisnewornotwellunderstoodbytheanalystandthereisnorealhistoricaldatatouse,ananalystcanmakesomesimpleassumptionsaboutthedistributionsandcorrela-tionsamongthevariablesinvolvedintheprocess.Forexample,theanalystcanmakeasimplifyingassumptionthatthevariableshavenormaldistributionsand“medium”correlationsamongthem,andcreatedatathatway.Thistypeofdatawilllikelynothavethesamepropertiesasrealdatabutcanstillbeusefulforsomepurposes,suchasdebugginganRdataanalysisprogramorforsometypesofper-formancetestingofsoftwareapplications.

Forsomeusecases,havinghighutilitywillmatterquiteabit.Inothercases,mediumorevenlowutilitymaybeacceptable.Forexample,iftheobjectiveistobuildAIMLmodelstopredictcus-tomerbehaviorandmakemarketingdecisionsbasedonthat,highutilitywillbeimportant.Ontheotherhand,iftheobjectiveistoseeifyoursoftwarecanhandlealargevolumeoftransactions,thedatautilityexpectationswillbeconsiderablyless.Therefore,understand-ingwhatdata,models,simulators,andknowledgeexistaswellastherequirementsfordatautilitywilldrivethespecificapproachtouseforgeneratingthesyntheticdata.

Table1-1

providesasummaryofthesyntheticdatatypes.

Table1-1.Typesofdatasynthesiswiththeirutilityandprivacyimplications

Typeofsyntheticdata

Utility

Generatedfromreal(nonpublic)datasetsGeneratedfromrealpublicdata

Canbequitehigh

Canbehigh,althoughlimitationsexistbecause

publicdatatendstobede-identifiedoraggregated

Generatedfromanexistingmodelofa

process,whichcanalsoberepresentedinasimulationengine

Basedonanalystknowledge

Willdependonthefidelityoftheexistinggeneratingmodel

Willdependonhowwelltheanalystknowsthedomainandthecomplexityofthephenomenon

Generatedfromgenericassumptionsnotspecifictothephenomenon

Willlikelybelow

TheBenefitsofSyntheticData|5

Nowthatyouhaveanunderstandingofthetypesofsyntheticdata,wewilllookatthebenefitsofdatasynthesisoverallandforsomeofthesedatatypesspecifically.

TheBenefitsofSyntheticData

Inthissection,wepresentseveralwaysthatdatasynthesiscansolvepracticalproblemswithAIMLprojects.Thebenefitsofsyntheticdatacanbedramatic.Itcanmakeimpossibleprojectsdoable,signif‐icantlyaccelerateAIMLinitiatives,orresultinmaterialimprove‐mentintheoutcomesofAIMLprojects.

ImprovingDataAccess

DataaccessiscriticaltoAIMLprojects.Thedataisneededtotrainandvalidatemodels.Morebroadly,dataisalsoneededforevaluat‐ingAIMLtechnologiesthathavebeendevelopedbyothers,aswellasfortestingAIMLsoftwareapplicationsorapplicationsthatincor‐porateAIMLmodels.

Typically,dataiscollectedforaparticularpurposewiththeconsentoftheindividual;forexample,forparticipatinginawebinarorforparticipatinginaclinicalresearchstudy.Ifyouwanttousethatsamedataforadifferentpurpose,suchasforbuildingamodeltopredictwhatkindofpersonislikelytosignupforawebinarorwhowouldparticipateinastudy,thenthatisconsideredasecondarypurpose.

Accesstodataforsecondaryanalysisisbecomingproblematic.TheUSGovernmentAccountabilityOffice

1

andtheMcKinseyGlobalInstitute

2

bothnotethataccessingdataforbuildingandtestingAIMLmodelsisachallengefortheiradoptionmorebroadly.ADeloitteanalysisconcludedthatdataaccessissuesarerankedinthetopthreechallengesfacedbycompanieswhenimplementingAI.

3

ArecentsurveyfromMITTechnologyReviewreportedthatalmost

1GovernmentAccountabilityOffice,“ArtificialIntelligence:EmergingOpportunities,Challenges,andImplications,”GAO-18-142SP(March2018).

https://oreil.ly/Cpyli

.

2McKinseyGlobalInstitute,“ArtificialIntelligence:TheNextDigitalFrontier?”(June2017).

https://oreil.ly/zJ8oZ

.

3DeloitteInsights,“StateofAIintheEnterprise,2ndEdition”(2018).

https://oreil.ly/

l07tJ

.

6|Chapter1:DefiningSyntheticData

halfoftherespondentsidentifieddataavailabilityasaconstrainttotheuseofAIwiththeircompany.

4

Atthesametime,thepublicisgettinguneasyabouthowtheirdataisusedandshared,andprivacylawsarebecomingmorestrict.ArecentsurveybyO’Reillyhighligh‐tedtheprivacyconcernsofcompaniesadoptingmachinelearningmodels,withmorethanhalfofcompaniesexperiencedwithAIMLcheckingforprivacyissues.

5

InthesameMITsurveymentionedpreviously,64%ofrespondentsnotethat“changesinregulationorgreaterregulatoryclarityondatasharing”isadevelopmentthatwouldbemostlikelytoleadtomoredatasharing.

Contemporaryprivacyregulations,suchastheUSHealthInsurancePortabilityandAccountabilityAct(HIPAA)andtheGeneralDataProtectionRegulation(GDPR)inEurope,imposeconstraintsorrequirementstousingpersonaldataforasecondarypurpose.Anexampleisarequirementtogetanadditionalconsentorauthoriza‐tionfromindividuals.Inmanycases,thisisnotpracticalandcanintroducebiasintothedatabecauseconsentersandnonconsentersdifferinimportantcharacteristics.

6

Datasynthesiscangivetheanalyst,ratherefficientlyandatscale,realisticdatatoworkwith.Giventhatsyntheticdatawouldnotbeconsideredidentifiablepersonaldata,privacyregulationswouldnotapply,andobligationsofadditionalconsenttousethedataforsec‐ondarypurposeswouldnotberequired.

7

ImprovingDataQuality

Giventhedifficultyingettingaccesstodata,manyanalyststrytojustuseopensourceorpublicdatasets.Thesecanbeagoodstartingpoint,buttheylackdiversityandareoftennotwellmatchedtotheproblemsthatthemodelsareintendedtosolve.Furthermore,open

4MITTechnologyReviewInsights,“TheGlobalAIAgenda:Promise,Reality,andaFutureofDataSharing”(March2020).

https://oreil.ly/FHg87

5BenLoricaandPacoNathan,TheStateofMachineLearningAdoptionintheEnterprise(O’Reilly).

6KhaledElEmam,etal.,“AReviewofEvidenceonConsentBiasinResearch,”AmericanJournalofBioethics13,no.4(2013):42–44.

https://oreil.ly/SiG2N.

7However,oneshouldfollowgoodpractices,suchasprovidingnoticetoindividualsabouthowthedataisusedanddisclosed,andhavingethicsoversightontheusesofdataandAIMLmodels.

TheBenefitsofSyntheticData|7

datamaylacksufficientheterogeneityforrobusttrainingofmodels.Forexample,theymaynotcapturerarecaseswellenough.

Sometimestherealdatathatexistsisnotlabeled.Labelingalargenumberofexamplesforsupervisedlearningtaskscanbetime-consuming,andmanuallabelingiserrorprone.Again,syntheticlabeleddatacanbegeneratedtoacceleratemodeldevelopment.Thesynthesisprocesscanensurehighaccuracyinthelabeling.

UsingSyntheticDataforExploratoryAnalysis

Analystscanusesyntheticdatamodelstovalidatetheirassumptionsanddemonstratethekindofresultsthatcanbeobtainedwiththeirmodels.Inthisway,thesyntheticdatacanbeusedinanexploratorymanner.Knowingthattheyhaveinterestingandusefulresults,theanalystscanthengothroughthemorecomplexprocessofgettingtherealdata(eitherraworde-identified)tobuildthefinalversionsoftheirmodels.

Forexample,ananalystwhoisaresearchercouldusetheirexplora-torymodelsonsyntheticdatatothenapplyforfundingtogetaccesstotherealdata,whichmayrequireafullprotocolandmultiplelev-elsofapprovals.Insuchaninstance,workwithsyntheticdatathatdoesnotproducegoodmodelsoractionableresultswouldstillbebeneficialbecauseanalystswouldhaveavoidedtheextraeffortrequiredtogetaccesstotherealdataforapotentiallyfutileanalysis.Anothervaluableuseofsyntheticdataisfortraininganinitialmodelbeforetherealdataisaccessible.Thenwhentheanalystgetstherealdata,theycanusethetrainedmodelasastartingpointfortrainingwiththerealdata.Thiscansignificantlyexpeditethecon-vergenceoftherealdatamodel(hencereducingcomputetime),andcanpotentiallyresultinamoreaccuratemodel.Thisisanexampleofusingsyntheticdatafortransferlearning.

UsingSyntheticDataforFullAnalysis

Avalidationservercanbedeployedtoruntheanalysiscodethatworkedonthesyntheticdataontherealdata.Ananalystwouldper-formalloftheiranalysisonthesyntheticdata,andthensubmitthecodethatworkedonthesyntheticdatatoasecurevalidationserverthathasaccesstotherealdata,asillustratedin

Figure1-1

.Becausethesyntheticdatawouldbestructuredinthesamewayastheorigi-naldata,thecodethatworkedonthesyntheticdatashouldwork

8|Chapter1:DefiningSyntheticData

directlyontherealdata.Theresultsarethensentbacktotheanalysttoconfirmtheirmodels.

Thisisnotintendedtobeaninteractivesystem.Theoutputfromthevalidationserverneedstobecheckedtoensurethatnorevealinginformationisbeingsentoutbythecodeoutput.Therefore,itisintendedtobeusedonceortwicebytheanalystattheveryendoftheiranalysis.Itdoesprovideawaytoprovideassurancetotheana-lyststhatthesynthesisresultsarereplicableontherealdata.

Figure1-1.Thesetupforavalidationserverusedtoexecutefinalcodethatproducedresultsonthesyntheticdata(adaptedfromReplica

AnalyticsLtd.,withpermission)

Whentheutilityofthesyntheticdataishighenough,theanalystscangetsimilarresultswiththesyntheticdataastheywouldhavewiththerealdata,andnovalidationserverisrequired.Insuchacase,thesyntheticdataplaystheroleofaproxyfortherealdata.Thisscenarioisplayingoutinmoreandmoreusecases:assynthesismethodsimproveovertime,thisproxyoutcomeisgoingtobecomemorecommon.

ReplacingRealDataThatDoesNotExist

Insomesituations,realdatamaynotexist.Theanalystmaybetry-ingtomodelsomethingcompletelynew,orthecreationorcollec-tionofarealdatasetfromscratchmaybecostprohibitiveorimpractical.Synthesizeddatacancoveredgeorrarecasesthataredifficult,impractical,orunethicaltocollectintherealworld.

Syntheticdatacanalsobeusedtoincreasetheheterogeneityofatrainingdataset,whichcanresultinamorerobustAIMLmodel.Forexample,unusualcasesinwhichdatadoesnotexistorisdifficulttocollectcanbesynthesizedandincludedinthetrainingdataset.In

LearningtoTrustSyntheticData|9

thatcase,theutilityofthesyntheticdataismeasuredintherobust‐nessincrementitgivestotheAIMLmodels.

Wehaveseenthatsyntheticdatacanplayakeyroleinsolvingaser‐iesofpracticalproblems.Onecriticalfactorfortheadoptionofdatasynthesis,however,istrustinthegenerateddata.Ithaslongbeenrecognizedthathighdatautilitywillbeneededforthebroadadop‐tionofdatasynthesismethods.

8

Thisisthetopicweturntonext.

LearningtoTrustSyntheticData

Initialinterestinsyntheticdatastartedintheearly’90swithpropos‐alstousemultipleimputationmethodstogeneratesyntheticdata.Imputationingeneralistheprocessofreplacingmissingdatavalueswithestimates.Missingdatacanoccur,forexample,inasurveyifsomerespondentsdonotcompleteaquestionnaire.

Accurateimputeddatarequirestheanalysttobuildamodelofthephenomenonofinterestbyusingtheavailabledataandthenusethatmodeltoestimatewhattheimputedvalueshouldbe.Tobuildavalidimputationmodel,theanalystneedstoknowhowthedatawillbeeventuallyused.Withmultipleimputation,youcreatemultipleimputedvaluestocapturetheuncertaintyintheseestimatedvalues.Thisprocesscanworkreasonablywellifyouknowhowthedatawillbeused.

Inthecontextofusingimputationfordatasynthesis,therealdataisaugmentedwithsyntheticdatabyusingthesametypeofimputationtechniques.Insuchacase,therealdataisusedtobuildanimputa‐tionmodelthatisthenusedtosynthesizenewdata.

Thechallengeisthatifyourimputationmodelsaredifferentfromtheeventualusesofthedata,theimputedvaluesmaynotbeveryreflectiveoftherealvalues,andthiswillintroduceerrorsinthedata.Thisriskofbuildingthewrongsynthesismodelhasledtohistoriccautionintheapplicationofsyntheticdata.

Morerecently,statisticalmachinelearningmodelshavebeenusedfordatasynthesis.Theadvantageofthesemodelsisthattheycancapturethedistributionsandcomplexrelationshipsamongthe

8JeromeP.Reiter,“NewApproachestoDataDissemination:AGlimpseintotheFuture(?),”CHANCE17,no.3(June2004):11–15.

https://oreil.ly/x89Vd

.

10|Chapter1:DefiningSyntheticData

variablesquitewell.Ineffect,theydiscovertheunderlyingmodelinthedataratherthanhavingthatmodelprespecifiedbytheanalyst.Andnowwithdeeplearningdatasynthesis,thesemodelscanbequiteaccurateinthattheycancapturemuchofthesignalinthedata—evensubtlesignals.

Therefore,wearegettingclosertothepointwherethegenerativemodelsavailabletodayareproducingdatasetsthatarebecomingquitegoodproxiesforrealdata.Therearealsowaystoassesstheutilityofsyntheticdatamoreobjectively.

Forexample,wecancomparetheanalysisresultsfromsyntheticdatawiththeanalysisresultsfromtherealdata.Ifwedonotknowwhatanalysiswillbeperformedonthesyntheticdata,arangeofpossibleanalysiscanbetriedbasedonknownexamplesofusesofthatdata.Oran“allmodels”evaluationcanbeperformedinwhichallpossiblemodelsarebuiltfromtherealandsyntheticdatasetsandcompared.

9

TheUSCensusBureauhas,atthetimeofwriting,decidedtolever‐agesyntheticdataforsomeofitsmostheavilyusedpublicdatasets,the2020decennialcensusdata.Foritstabulardatadisseminations,theagencywillcreateasyntheticdatasetfromthecollectedindividual-levelcensusdataandthenproducethepublictabulationsfromthatsyntheticdataset.Amixtureofformalandnonformalmethodswillbeusedinthesynthesisprocess.

10

Weprovideanover‐viewofthesynthesisprocessin

Chapter2

.This,arguably,demon‐stratesthelarge-scaleadoptionofdatasynthesisforoneofthemostcriticalandheavilyuseddatasetsavailabletoday.

Asorganizationsbuildtrustinsyntheticdata,theywillmovefromexploratoryanalysisusecases,totheuseofavalidationserver,andthentousingsyntheticdataasaproxyforrealdata.

Alegitimatequestioniswhataretheotherapproachesthatareavail‐abletodaytoaccessdataforAIMLpurposes,inadditiontodata

9AreviewofutilityassessmentapproachescanbefoundinKhaledElEmam,“Seven

WaystoEvaluat