1、海量系統日志采集解析實踐技術創新，變革未來01背景介紹02挑戰及應對03采集手段04案例定制05現狀簡述靈活快速適應日志格式變化量大難以橫向擴容昂貴按EPS 計費難以承受度量日志有效性監測02.挑戰02.應對虛擬化集群支持橫向擴容開源產品自我可控靈活按需解析、規范化格式富化增加監測點03.采集-Linuximuxsockimfileimpstatsomrelp03.采集-WindowsWinlogBeatPacketBeat FileBeatLogstash03.采集-DB/APIJDBCRestKAFKA03. 采集-SF TPSFTPShellPython Bat/Powershell03

2、.采集-過濾/轉換/富化原始日志負載均衡過濾/轉換/富化初級原始日志消息隊列04. 案例定制需求分析解析運營04. 案例定制V 0需求：G t ab中代碼拉取情況監測分析：G t ab中代碼拉取途徑HTTP/SSH,G t ab有多種類型日志文件,ap _son og、app cat on_son og、g t_son og、 g t ab-she og中包含相應拉取代碼的日志信息解析：解析ap _son og、app cat on_son og、g t_son og、g t ab-she og,至少將相關倉庫、地址、賬號、SSHKey- d解 析,富化地址相關的網段信息運營：源網段設備拉取非

3、相關業務倉庫的行為 V 2 0需求：離職人員離職前拉取代碼情況 分析：從人力系統獲得離職人員信息解析：人員拉取數據的事件及關鍵字段抽取入報表引擎數據庫 運營：按需提供長期報表功能method: GET,path: /xxxx,format: */*,controller: Projects:GitHttpController, time: 2020-07-22T04:38:19.394Z,params: key: service,value: git-upload-pack,key: namespace_id, value: xxxxx,key: project_id,value: xxxxx

4、.git,remote_ip: xx.xx.xx.xx, user_id: xxxx, username: xxxx,ua: git/2.21.0 (Apple Git-122.2)”04. 案例定制原始日志初級富化日志檢索timestamp: 2020-07-22 12:38:19 +0800,timegenerated: 2020-07-22 12:38:19 +0800, fromhost-ip: ”xxx. xxx. xxx. xxx , myhostname: xxx. xxx. xxx. xxx. xxx ,syslog-tag: duction_json, inputname:

5、imrelpxxxx, message: xxxxxxxxx04. 案例定制富化數據xx: xx網段name: GET,sourceAddress: xxx.xxx.xxx.xxx,sourceZoneURI: xx-xx.xx.xx.xx_xx.xx.xx.xx_xx產品部, requestMethod: GET,requestClientApplication: git/2.21.0 (Apple Git-122.2), deviceReceiptTime: 1595392699394, destinationUserName: xxxx,deviceSeverity: 200, requ

6、estUrl: /xxxx,deviceCustomString1: git-upload-pack, deviceCustomString2: xxxxx, deviceCustomString3: xxxxx,deviceEventCategory: duction_json, deviceFacility: Projects:GitHttpController, fileType: */*,關聯分析method: GET,path: /xxxx,format: */*,controller: Projects:GitHttpController, time: 2020-07-22T04:

7、38:19.394Z,params: key: service,value: git-upload-pack,key: namespace_id, value: xxxxx,key: project_id,value: xxxxx.git,remote_ip: xx.xx.xx.xx, user_id: xxxx, username: xxxx,ua: git/2.21.0 (Apple Git-122.2)”原始日志關聯數據倉庫/部門映射表04. 案例定制name: GET,sourceAddress: xxx.xxx.xxx.xxx, sourceZoneURI: xxxx 產 品 部 ,

8、 requestMethod: GET, requestClientApplication: git deviceReceiptTime: 1595392699394, destinationUserName: xxxx, deviceSeverity: 200,requestUrl: /xxxx,deviceCustomString1: git-upload-pack, deviceCustomString2: xxxxx, deviceCustomString3: xxxxx, deviceEventCategory: duction_json, deviceFacility: Proje

9、cts:GitHttpController, fileType: */*,規范日志sourceAddress: xxx.xxx.xxx.xxx, sourceZoneURI: xxxx 產 品 部 , destinationUserName: xxxx, deviceCustomString1: git-upload-pack, deviceCustomString2: xxxxx按需定時 匯總報表05. 現狀日志處理架構原始日志過濾轉換富化初級富化日志初級富化日志隊列歸檔解析富化數據規范日志隊列抽取關聯數據實時報警長期報表關聯HDFS模式發現列式數據庫事件運營設備類型設備信息網絡設施華為/華

10、三/銳捷/NTO/安全SD-WAN基礎設施ESXi/VDI/vCSA/TACACS/FreeIPA CMD/BIND/MS DHCP/MS DNS操作系統Linux操作系統日志/Linux Auditd日志Windows 安全/系統/Sysmon/Powershell安全設備網神防火墻/天眼/天擎EDR/天擎DLP/天空衛士DLP/天擎郵件沙箱/椒圖/堡壘機/零信任/行為感 知分析系統/安全DNS/奇安信ID/安域/奇安信VPN/Suricata/Opencanary/OSQuery基礎應用Apache Access Log/Nginx Access Log/IIS/Confluence/Jira/Jenkins/Gitlab/SVN/Sonar/artifactory/Exchange Protocol Logging/Exchange TraceL