1、Boundary Detection in Tokenizing Network Application Payload for Anomaly DetectionRachna Vargiya and Philip ChanDepartment of Computer SciencesFlorida Institute of TechnologyMotivationExisting anomaly detection techniques rely on informatio n derived only from the packet headersMore sophisticated at

2、tacks involve the application payloadExample : Code Red II wormGET /default ida?NNNNNNNNNParsing the payload is required!Problems in hand-coded parsing:Large number of application protocolsFrequent introduction of new protocolsProblem StatementTo parse application payload into tokens without explici

3、t knowledge of the application protocolsThese tokens are later used as features for anomaly detectionRelated workPattern Detection Important TokensFixed Length:Forrest et al. (2019)Variable Length:Wespi et al. (2000)Jiang et al.(2019)Boundary Detection - All Tokenso VOTING EXPERTS by Cohen et al. (2

4、019)Boundary EntropyFrequencyBinary VotesApproachBoundary Finding Algorithms:o Boundary Entropyo FrequencyAugmented Expected Mutual InformationMinimum Description LengthApproach is domain independent (no prior domain knowledge)Combining Boundary FindingAlgorithmsCombination of all or a subset (E.g.

5、Frequency + Minimum Description Length) of techniquesEach algorithm can cast multiple votes, depending on confidence measureBoundary Entropy (Cohen et al)Entropy at the end of each possible window is calculatedHigh Entropy means more variationw XItis 1 rai ny dayis the byte following the current win

6、dowVoting using Boundary Entropy change graph to discrete bars )Itiseirainydayn n n n n n n n n o n n n nEntropy in meaningful tokens starts with a high value, drops, and peaks at the end Vote for positions with the peak entropy Threshold suppresses votes for low entropy valuesThreshold = Average BE

7、Frequency (Cohen et al)Most frequent set of tokens are assumed to be meaningful tokensFrequencies of tokens with length =1,2, 3., 6 Shorter tokens are inherently more frequent than longer toke nsNormalize frequencies for tokens of the same length using standard deviationBoundaries are assigned at th

8、e end of most frequent token in the windowrainy dayFrequency in window:(1 )T = 3(2),lf, = 5(3)=2(4)”lt is” = 3Mutual Information (Ml)Mutual Information given by:Gives us the reduction of uncertainty in presence of event b given event &Ml does not incorporate the counter evidence when & occurs withou

9、t b and vice versaAugmented Expected Mutual Information(AEMI)AEMI sums the supporti ng evide nee and subtracts the counter evidence For each window, the location with the minimum AEMI value suggests a boundaryItisRrainydayabMinimum Description Length(MDL)Shorter code assigned to frequent tokens to m

10、inimize the overall coding lengthBoundary yielding shortest coding length is assigned votesCoding Length per byte:Lg P(tj): no of bits to encode t(o |tj|=lengthoftjI疋渥淪京n藝IJE|rainy ay / t|efttjghtNormalize scores of each algorithm Each algorithm produces list of scores Since the number of votes is p

11、roportional to the score, the scores must be normalized Each score is replaced by the number of standard deviations that the score is away from the mean valueNormalize votes of each algorithmAlgorithms produce list of votes depending on the scoresMake sure each algorithm votes with the same weightNu

12、mber of votes is replaced by the number of standard deviations from the mean valueNormalizing Scores and VotesCombined Normalized VotesCombined Approach with WeightedVotingA list of votes from all the experts is gatheredFor each boundary, the final votes are summedA boundary is placed at a position

13、if the votes at the position exceed threshold. Threshold = Average number of VotesAnomaly Detection Algorithm 一 LERAD(Mahoney and Chan)LERAD forms rules based on 23 attributes o First 15 attributes: from packet headerNext 8 attributes: from the payloado Example Rule:If port = 80 then wordl 二 “GET”Or

14、iginal Payload attributes: space separated tokensOur Payload attributes: Boundary separated tokensExperimental Data2019 DARPA Intrusion Detection Evaluation Data Set Week 3 :attack free (training) dataWeeks 4, 5: attack containing (test) dataEvaluations A, B, C (Known boundaries) : Week 3o trained:

15、days 1 - 4o tested: days 5-7Prevent gaining knowledge from Weeks 4 and 5Evaluation D (Detected attacks)Trained: Week 3Tested :Weeks 4 and 5Evaluation A: % of Space-SeparatedTokens RecoveredMethodPort#25Port#80Port#21Port#79AvgFreq+MDL5226218145.0Freque ncy1516139936.0BE +AEMI + MDL+ Freq211451213.0A

16、EMI5943212.5MDL6732510.3BE33194.0Evaluation B: % of Keywords in RFCsRecoveredMethodPort#25Port#80Port#21AvgFreq+MDL40365945.0Freque ncy31284033.0BE+AEMI+MDL+Freq12132115.3AEMI9525.3MDL7614.7BE3222.3Evaluation C: Entropy of Output (Lower is Better) average across 6 portsMethodAverage ValueFreque ncy5

17、.0MDL5.03Freq+MDL5.06BE5.25BE +AEMI + Freq + MDL5.56AEMI6.38Ranking of AlgorithmsMethodEvaluation AEvaluation BEvaluation CFreq+MDL3Freque ncy221BE+AEMI+ MDL+ Freq335AEMI446MDL552BE664Detection Rate for Space Separated VsBoundary Separated (Freq + MDL)Port #10 FP/day100 FP/daySpaceBoundarySpaceBound

18、ary20224521141614172233332313141314251516161679333380101011131132222Overall59626368% Improvement58Summary of ContributionsUsed payload in formation, while most IDS concentrate on header information.Proposed AEMI + MDL for boundary detection Combined all and subset of algorithmsUsed weighted voting to indicate confidence Proposed techniques find boundaries better than spacesAchieved higher detection rates in an anomaly detection systemFuture WorkFurther evaluation on other portsPick more useful toke ns in stead of first 8DARPA data set is partially synthetic, further evaluation on real traffi