Large

LanguageModels

IntroductiontoLargeLanguageModels

Languagemodels

?Rememberthesimplen-gramlanguagemodel

?Assignsprobabilitiestosequencesofwords

?Generatetextbysamplingpossiblenextwords

?Istrainedoncountscomputedfromlotsoftext

?Largelanguagemodelsaresimilaranddifferent:

?Assignsprobabilitiestosequencesofwords

?Generatetextbysamplingpossiblenextwords

?Aretrainedbylearningtoguessthenextword

Largelanguagemodels

?Eventhroughpretrainedonlytopredictwords

?Learnalotofusefullanguageknowledge

?Sincetrainingonalotoftext

Threearchitecturesforlargelanguagemodels

Decoders

GPT,Claude,Llama

Mixtral

Encoders

BERTfamily,HuBERT

Encoder-decoders

Flan-T5,Whisper

Encoders

Manyvarieties!

?Popular:MaskedLanguageModels(MLMs)

?BERTfamily

?Trainedbypredictingwordsfromsurroundingwordsonbothsides

?Areusuallyfinetuned(trainedonsuperviseddata)forclassificationtasks.

Encoder-Decoders

?Trainedtomapfromonesequencetoanother

?Verypopularfor:

?machinetranslation(mapfromonelanguagetoanother)

?speechrecognition(mapfromacousticstowords)

Large

LanguageModels

IntroductiontoLargeLanguageModels

Large

LanguageModels

LargeLanguageModels:Whattaskscantheydo?

Bigidea

Manytaskscanbeturnedintotasksofpredictingwords!

Thislecture:decoder-onlymodels

Alsocalled:

?CausalLLMs

?AutoregressiveLLMs

?Left-to-rightLLMs

?Predictwordslefttoright

ConditionalGeneration:Generatingtextconditionedonprevioustext!

CompletionText

TransformerBlocks

Encoder

Solongandthanksfor

all

the

the

Language

Modeling

Head

…

…

+i

E

+i

E

+i

E

+i

E

+i

E

Softmax

Unencoderlayer\U/

+i

E

+i

E

U

logits

all

Pre?xText

ManypracticalNLPtaskscanbecastaswordprediction!

Sentimentanalysis:“IlikeJackieChan”

1.Wegivethelanguagemodelthisstring:

Thesentimentofthesentence"I

likeJackieChan"is:

2.Andseewhatworditthinkscomesnext:

P(positive|Thesentimentofthesentence‘‘IlikeJackieChan"is:)

P(negative|Thesentimentofthesentence‘‘IlikeJackieChan"is:)

FraminglotsoftasksasconditionalgenerationQA:“WhowroteTheOriginofSpecies”

1.Wegivethelanguagemodelthisstring:

Q:Whowrotethebook‘‘TheOriginofSpecies"?A:

2.Andseewhatworditthinkscomesnext:

P(w|Q:Whowrotethebook‘‘TheOriginofSpecies"?A:)

3.Anditerate:

P(w|Q:Whowrotethebook‘‘TheOriginofSpecies"?A:Charles)

Summarization

TheonlythingcrazierthanaguyinsnowboundMassachusettsboxingupthepowderywhitestuffandofferingitforsaleonline?Peopleareactuallybuyingit.For$89,self-styledentrepreneurKyleWaringwillshipyou6poundsofBoston-areasnowinaninsulatedStyrofoambox–enoughfor10to15snowballs,hesays.

Original

ButnotifyouliveinNewEnglandorsurroundingstates.“Wewillnotshipsnowtoanystatesinthenortheast!”saysWaring’swebsite,ShipSnowY.“We’reinthebusinessofexpungingsnow!”

Hiswebsiteandsocialmediaaccountsclaimtohave?lledmorethan133ordersforsnow–morethan30onTuesdayalone,hisbusiestdayyet.Withmorethan45totalinches,Bostonhassetarecordthiswinterforthesnowiestmonthinitshistory.Mostresidentsseethehugepilesofsnowchokingtheiryardsandsidewalksasanuisance,butWaringsawanopportunity.

AccordingtoB,itallstartedafewweeksago,whenWaringandhiswifewereshov-elingdeepsnowfromtheiryardinManchester-by-the-Sea,acoastalsuburbnorthofBoston.Hejokedaboutshippingthestufftofriendsandfamilyinwarmerstates,andanideawasborn.[...]

Summary

KyleWaringwillshipyou6poundsofBoston-areasnowinaninsulatedStyrofoambox–enoughfor10to15snowballs,hesays.ButnotifyouliveinNewEnglandorsurroundingstates.

LLMsforsummarization(usingtl;dr)

GeneratedSummary

KyleWaringwill…

U

LMHead

U

U

E

E

E

E

E

E

E

…

E

idea

Theonly

will

tl;dr

Waring

…

wasborn.Kyle

OriginalStoryDelimiter

Large

LanguageModels

LargeLanguageModels:Whattaskscantheydo?

Large

LanguageModels

SamplingforLLMGeneration

DecodingandSampling

Thistaskofchoosingawordtogeneratebasedonthemodel’sprobabilitiesiscalleddecoding.

ThemostcommonmethodfordecodinginLLMs:sampling.Samplingfromamodel’sdistributionoverwords:

?chooserandomwordsaccordingtotheirprobabilityassignedbythemodel.

Aftereachtokenwe’llsamplewordstogenerateaccordingtotheirprobabilityconditionedonourpreviouschoices,

?Atransformerlanguagemodelwillgivetheprobability

Randomsampling

i←1

wi~p(w)

whilewi!=EOSi←i+1

wi~p(wi|w<i)

Randomsamplingdoesn'tworkverywell

Eventhoughrandomsamplingmostlygeneratesensible,high-probablewords,

Therearemanyodd,low-probabilitywordsinthetailofthedistribution

Eachoneislow-probabilitybutaddeduptheyconstitutealargeportionofthedistribution

Sotheygetpickedenoughtogenerateweirdsentences

Factorsinwordsampling:qualityanddiversity

Emphasizehigh-probabilitywords

+quality:moreaccurate,coherent,andfactual,-diversity:boring,repetitive.

Emphasizemiddle-probabilitywords+diversity:morecreative,diverse,-quality:lessfactual,incoherent

Top-ksampling:

1.Choose#ofwordsk

2.ForeachwordinthevocabularyV,usethelanguagemodeltocomputethelikelihoodofthiswordgiventhecontextp(wt|w<t)

3.Sortthewordsbylikelihood,keeponlythetopkmostprobablewords.

4.Renormalizethescoresofthekwordstobealegitimateprobabilitydistribution.

5.Randomlysampleawordfromwithintheseremainingkmost-probablewordsaccordingtoitsprobability.

Top-psampling

Holtzmanetal.,2020

Problemwithtop-k:kisfixedsomaycoververydifferentamountsofprobabilitymassindifferentsituations

Idea:Instead,keepthetopppercentoftheprobabilitymass

ΣP(w|w<t)≥p

w∈V(p)

Temperaturesampling

ReshapethedistributioninsteadoftruncatingitIntuitionfromthermodynamics,

?asystemathightemperatureisflexibleandcanexploremanypossiblestates,

?asystematlowertemperatureislikelytoexploreasubsetoflowerenergy(better)states.

Inlow-temperaturesampling,(τ≤1)wesmoothly

?increasetheprobabilityofthemostprobablewords

?decreasetheprobabilityoftherarewords.

Temperaturesampling

Dividethelogitbyatemperatureparameterτbeforepassingitthroughthesoftmax.

Insteadofy=softmax(u)

Wedo

y=softmax(u/t)

0≤τ≤1

Temperaturesampling

y=softmax(u/t)

Whydoesthiswork?

?Whenτiscloseto1thedistributiondoesn’tchangemuch.

?Thelowerτis,thelargerthescoresbeingpassedtothesoftmax

?Softmaxpusheshighvaluestoward1andlowvaluestoward0.

?Largeinputspusheshigh-probabilitywordshigherandlowprobabilitywordlower,makingthedistributionmoregreedy.

?Asτapproaches0,theprobabilityofmostlikelywordapproaches1

Large

LanguageModels

SamplingforLLMGeneration

Large

LanguageModels

PretrainingLargeLanguageModels:Algorithm

Pretraining

Thebigideathatunderliesalltheamazingperformanceoflanguagemodels

Firstpretrainatransformermodelonenormousamountsoftext

Thenapplyittonewtasks.

Self-supervisedtrainingalgorithm

Wejusttrainthemtopredictthenextword!

1.Takeacorpusoftext

2.Ateachtimestept

i.askthemodeltopredictthenextword

ii.trainthemodelusinggradientdescenttominimizetheerrorinthisprediction

"Self-supervised"becauseitjustusesthenextwordasthelabel!

Intuitionoflanguagemodeltraining:loss

?Samelossfunction:cross-entropyloss

?Wewantthemodeltoassignahighprobabilitytotruewordw

?=wantlosstobehighifthemodelassignstoolowaprobabilitytow

?CELoss:Thenegativelogprobabilitythatthemodelassignstothetruenextwordw

?Ifthemodelassignstoolowaprobabilitytow

?Wemovethemodelweightsinthedirectionthatassignsahigherprobabilitytow

Cross-entropylossforlanguagemodeling

:terenceb

Thecorrectdistributionytknowsthenextword,sois1fortheactualnextwordand0fortheothers.

Sointhissum,alltermsgetmultipliedbyzeroexceptone:thelogpthemodelassignstothecorrectnextword,so:

Teacherforcing

?Ateachtokenpositiont,modelseescorrecttokensw1:t,?Computesloss(–logprobability)forthenexttokenwt+1

?Atnexttokenpositiont+1weignorewhatmodelpredictedforwt+1

?Insteadwetakethecorrectwordwt+1,addittocontext,moveon

Trainingatransformerlanguagemodel

Nexttokenlongandthanksforall

Loss-logyand-logythanks

Language

Modeling

Head

logitslogitslogitslogitslogits

UUUUU

Stacked

Transformer

Blocks

…

x1

x2

x4

x3

x5

3

2

+1

+

+

+

4+5

Input

E

E

E

E

Encoding

E

InputtokensSolongandthanksfor

…

=

…

…

…

…

…

Large

LanguageModels

PretrainingLargeLanguageModels:Algorithm

Large

LanguageModels

PretrainingdataforLLMs

LLMsaremainlytrainedontheweb

Commoncrawl,snapshotsoftheentirewebproducedbythenon-profitCommonCrawlwithbillionsofpages

ColossalCleanCrawledCorpus(C4;Raffeletal.2020),156billiontokensofEnglish,filtered

What'sinit?Mostlypatenttextdocuments,Wikipedia,andnewssites

ThePile:apretrainingcorpus

academicswebbooks

dialog

Filteringforqualityandsafety

Qualityissubjective

?ManyLLMsattempttomatchWikipedia,books,particularwebsites

?Needtoremoveboilerplate,adultcontent

?Deduplicationatmanylevels(URLs,documents,evenlines)Safetyalsosubjective

?Toxicitydetectionisimportant,althoughthathasmixedresults

?CanmistakenlyflagdatawrittenindialectslikeAfricanAmericanEnglish

Whatdoesamodellearnfrompretraining?

?Therearecanineseverywhere!Onedoginthefrontroom,andtwodogs

?Itwasn'tjustbigitwasenormous

?Theauthorof"ARoomofOne'sOwn"isVirginiaWoolf

?Thedoctortoldmethathe

?Thesquarerootof4is2

Bigidea

TextcontainsenormousamountsofknowledgePretrainingonlotsoftextwithallthat

knowledgeiswhatgiveslanguagemodelstheirabilitytodosomuch

Butthereareproblemswithscrapingfromtheweb

Copyright:muchofthetextinthesedatasetsiscopyrighted

?NotcleariffairusedoctrineinUSallowsforthisuse

?Thisremainsanopenlegalquestion

Dataconsent

?Websiteownerscanindicatetheydon'twanttheirsitecrawledPrivacy:

?WebsitescancontainprivateIPaddressesandphonenumbers

Large

LanguageModels

PretrainingdataforLLMs

Finetuning

Large

LanguageModels

Finetuningfordaptationtonewdomains

WhathappensifweneedourLLMtoworkwellonadomainitdidn'tseeinpretraining?

Perhapssomespecificmedicalorlegaldomain?

OrmaybeamultilingualLMneedstoseemoredataonsomelanguagethatwasrareinpretraining?

Finetuning

PretrainingData

Pretraining

PretrainedLM

…

…

…

Fine-

tuning

Data

Fine-tuning

Fine-tunedLM

…

…

…

"Finetuning"means4differentthings

We'lldiscuss1here,and3inlaterlectures

Inallfourcases,finetuningmeans:

takingapretrainedmodelandfurtheradaptingsomeorallofitsparameterstosomenewdata

1.Finetuningas"continuedpretraining"onnewdata

?Furthertrainalltheparametersofmodelonnewdata

?usingthesamemethod(wordprediction)andlossfunction(cross-entropyloss)asforpretraining.

?asifthenewdatawereatthetailendofthepretrainingdata

?Hencesometimescalledcontinuedpretraining

Finetuning

Large

LanguageModels

Large

LanguageModels

EvaluatingLargeLanguageModels

Perplexity

Justasforn-gramgrammars,weuseperplexitytomeasurehowwelltheLMpredictsunseentext

Theperplexityofamodelθonanunseentestsetistheinverseprobabilitythatθassignstothetestset,normalizedbythetestsetlength.

Foratestsetofntokensw1:ntheperplexityis:

Whyperplexityinsteadofrawprobabilityofthetestset?

?Probabilitydependsonsizeoftestset

?Probabilitygetssmallerthelongerthetext

?Better:ametricthatisper-word,normalizedbylength

?Perplexityistheinverseprobabilityofthetestset,normalizedbythenumberofwords

(Theinversecomesfromtheoriginaldefinitionofperplexityfromcross-entropyrateininformationtheory)

Probabilityrangeis[0,1],perplexityrangeis[1,∞]

Perplexity

?Thehighertheprobabilityofthewordsequence,thelowertheperplexity.

?Thusthelowertheperplexityofamodelonthedata,thebetterthemodel.

?Minimizingperplexityisthesameasmaximizingprobability

Also:perplexityissensitivetolength/tokenizationsobestusedwhencomparingLMsthatusethesametokenizer.

Manyotherfactorsthatweevaluate,like:

Size

BigmodelstakelotsofGPUsandtimetotrain,memorytostore

Energyusage

CanmeasurekWhorkilogramsofCO2emitted

Fairness

Benchmarksmeasuregenderedandracialstereotypes,ordecreasedperformanceforlanguagefromoraboutsomegroups.

Large

LanguageModels

DealingwithScale

ScalingLaws

LLMperformancedependson

?Modelsize:thenumberofparametersnotcountingembeddings

?Datasetsize:theamountoftrainingdata

?Compute:Amountofcompute(inFLOPSoretc

Canimproveamodelbyaddingparameters(morelayers,

widercontexts),moredata,ortrainingformoreiterationsTheperformanceofalargelanguagemodel(theloss)scalesasapower-lawwitheachofthesethree

ScalingLaws

LossLasafunctionof#parametersN,datasetsizeD,computebudgetC(ifothertwoareheldconstant)

Scalinglawscanbeusedearlyintrainingtopredictwhatthelosswouldbeifweweretoaddmoredataorincreasemodelsize.

Numberofnon-embeddingparametersN

≈12nlayerd2

ThusGPT-3,withn=96layersanddimensionalityd=12288,has12×96×122882≈175billionparameters.

KVCache

Intraining,wecancomputeattentionveryefficientlyinparallel:

Butnotatinference!Wegeneratethenexttokensoneatatime!

Foranewtokenx,needtomultiplybyWQ,WK,andWVtogetquery,key,values

Butdon'twanttorecomputethekeyandvaluevectorsforallthepriortokensx<i

Instead,storekeyandvaluevectorsinmemoryintheKVcache,andthenwecanjustgrabthemfromthecache

KVCache

KT

A

Q

V

k1

k2

k3

k4

a1

a2

a3

a4

q1

q2

q3

q4

v1

v2

v3

v4

x

=

mask

x

=

dkxN

QKT

q1?k1

q1?k2

q1?k3

q1?k4

q2?k1

q2?k2

q2?k3

q2?k4

q3?k1

q3?k2

q3?k3

q3?k4

q4?k1

q4?k2

q4?k3

q4?k4

Nxdv

Nxdv

NxN

Nxdk

Q

x