從企業數據向大數據的擴展TraditionalApproachStructured,analytical,logicalSystemsofRecordNewApproach

Creative,holisticthought,intuitionSystemsOfEngagementMultimediaSystemsofInsightEnterpriseIntegration

andContextAccumulationStructured

Repeatable

LinearUnstructured

Exploratory

DynamicDataWarehouseWebLogsSocialDataTextData:

emailsSensordata:

imagesRFIDInternalAppDataTransactionDataMainframeDataOLTPSystemDataHadoopand

StreamsTraditionalSourcesNewSourcesERP

data具備洞悉才干的系統SystemsofInsight對新式根底架構的需求在可靠和平安的環境中處置關鍵業務運用存取和處置海量數據——包括構造化和非構造化數據速度及時呼應隨時能夠出現的商業時機，這就需求靈敏、實時性的根底架構ThedynamicsofSoRandSoE:經過負載及資源部署的優化，來加強靈敏性和效益經過采用包括基于開放規范的技術等新技術來改善ITeconomicsSystemofRecord〔SoR〕SystemsofEngagement〔SoE〕對的決策對的地方對的時間點BigData&Analytics大數據分析的新型架構處理方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZone4SmartMeteringGridOperations電網管理FieldService外勤現場效力ResourcePlanning資源規劃CustomerService/CustomerOperations實現真正的有效的法規服從及時發現能源損耗問題、以及偷電和欺詐行為提高客戶稱心度電量運用預測更為準確電網運維優化減少停電次數和時間案例:SmartMetering智慧電力計費大數據分析運用可以帶來真正的業務價值法規服從案例:用大數據分析來加強SmartMetering數據分析的高可用性，以確保隨時了解用戶喜好跨運用的TB級的數據需求–通用虛擬化存儲平臺實時搜集、存儲并分析數據，最快可達50,000datapoints/sec歷史用電形狀數據的復雜查詢處置數據在加載到數據倉庫前的清洗、驗證，這些數據能夠來自很多的用戶、收費系統或斷電維護系統關系掌控

構建和維護電網的獨一試圖對整個企業的構造化和非構造化數據t做全局導覽Navigation，從中發現Discover價值分析用戶用電情況，偵測偷電、改表等行為預測哪些用戶適宜于哪些分時時段電價或需求/呼應效力分時時段電價的實時定價或

提供及時的需求/呼應效力IBMBigData&AnalyticsReferenceArchitectureBigDataPlatformCapabilitiesInformationIngestReal-timeAnalyticsWarehouse&DataMartsAnalyticAppliancesAllDataSourcesAdvancedAnalytics/

NewInsightsNew/

EnhancedApplicationsCognitive認知LearnDynamically?Prescriptive規范BestOutcomes?Predictive預測WhatCouldHappen?Descriptive

描畫WhatHasHappened?ExplorationandDiscoveryWhatDoYouHave?StreamingDataTextDataApplicationsDataTimeSeriesGeoSpatialRelationalSocialNetworkVideo&ImageAutomatedProcessCaseManagementAnalyticApplicationsWatsonCloudServicesISVSolutionsAlertsNewInfrastructureLeveragesDataTypesDatain

MotionDataat

RestDatain

ManyFormsInformationIngestionandOperationalInformationDecision

ManagementBIandPredictiveAnalyticsNavigation

andDiscoveryIntelligence

AnalysisRawDataStructuredDataTextAnalyticsDataMiningEntityAnalyticsMachineLearningLandingArea,AnalyticsZoneandArchiveVideo/AudioNetwork/SensorEntityAnalyticsPredictiveReal-timeAnalyticsExploration,IntegratedWarehouse,andMartZonesDiscoveryDeepReflectionOperationalPredictiveStreamProcessingDataIntegrationMasterDataStreamsInformationGovernance,SecurityandBusinessContinuityBigInsightsStreamsWarehouseInfoSphereBigInsightsHadoop-based低延遲分析，針對多樣化的、海量靜態數據Data-At-RestNetezzaHighCapacityAppliance基于構造化數據的可查詢歸檔Netezza1000基于構造化數據的

BI+定制化分析DataSmartAnalyticsSystem基于構造化數據的運營分析InformixTimeseriesTime-structuredanalyticsInfoSphereWarehouse基于構造化數據的大容量數據分析InfoSphereStreams低延遲流數據分析Velocity,Variety&VolumeData-In-MotionMPPDataWarehouseStreamComputingInformationIntegrationHadoopInfoSphereInformationServer海量數據集成和轉化ApacheHadoop:跨效力器集群的大數據集分布式處置開放系統框架，采用的是一種簡單化編程模型IBMBigDataPlatform大數據平臺What:一種開源軟件，將數據計算分布到整個集群的常見商用效力器和存儲上Why:傳統的計算架構是一種沿縱向擴展方式，經過更快的SAN、大容量內存和多級緩存將數據加載到CPU上，本錢比較高。What:Hadoop把大數據集合拆分區劃為小數據集合，再把小數據集合分發到多臺普通效力器上，是一種橫向擴展方式。Why:Scalable,Flexible,CostEffective,FaultTolerentComponents:MapReduce,HDFSWhatisHadoop?NameNode(Metadatastore)NodesHDFSClusterOperatingSystemNodesElasticStorage-SNCClusterKernelLevelIBMValueforHadoop!HDFS把數據分散存儲在多個存儲節點Node上HDFS設計時就假設存儲節點有失效的能夠，所以HDFS會把一份數據復制3份以上，分散存儲在多個節點上，從而實現系統整體上的可靠性HDFS文件系統是由效力器節點集群組成的，每臺效力器按照HDFS的特有block協議支持網絡化block數據HDFSNameNode有發生單點缺點的危險IBM在改善文件系統的性能同時消除了單點缺點——ElasticStorage-SNC(availableasbetacode)Hadoop闡明,MapReduce,HDFSHadoopStackWhatdoesitlooklike?典型Hadoop存儲的PainPoints在選擇HDFS的組件〔如軟件、效力器、網絡和存儲等〕時很難選對在從測試環境遷移到消費環境時，需求做的調優和調整任務太繁復了長期繼續不斷的運維保證過于繁重，比如老要改換失效組件〔尤其是硬盤〕，這使得保證期望的SLA非常難CPU和存儲去耦本來用戶的CPU和內存曾經滿足計算需求，但為了存儲容量需求安裝更多的硬盤不得不買更多的、不用要的CPU和內存Storageoptionsavailablehavecleargaps本地存儲的利用率低(~25%)，每次需求擴容的時候就要添加更多的效力器，而一旦硬盤失效后需求重建，效力器越多，失效的幾率越高，性能也就越差IBMStorageforHadoop傳統的Hadoop集群運用的是效力器內置硬盤存儲。假設用作測試或科學研討還好，可作為業務運轉的存儲就要采用企業存儲Hadoop集群要擔任數據維護和復制重建〔就是copy〕失效的數據集到不同節點上——嚴重影響CPU性能，無法實現企業級的RASReplicatedata–問題同上擴展的時候同時添加處置器/網絡/存儲，無法做到物盡其用〔nowaytoseparatethese3evenifexcesscapacityexistinginone(e.g.NeededmorestoragebuthadtoaddComputeandNetwork)〕運用外部存儲可以將存儲負載和Hadoop計算節點分別，同時還獲得了企業存儲的益處。SellthevalueofXIV,V7000,SVC,etc.用戶普通會隨HadoopFileSystem部署；采用ElasticStorage可以有很多益處14數據加速ExperiencetheinstantresultsthatcomefromIBMFlashSystemDriveasmuchas45Xfasteranalyticsresultsoncertainworkloads數據負載的多樣性和靈敏性XIVdeliverspredictableperformancethatscaleslinearlywithouthotspotsdeliveringinsightsfromanalyticsfasterwithtuning-freedatadistributionScale-out,parallelprocessingofElasticStoragesoftwareandintegrationwithFlashSystemdramaticallyacceleratesperformanceofAnalyticsclustersVirtualStorageCenterwithSVCautomaticallyoptimizesdatawarehouseperformanceandcostacrossFlashandDiskMainframeDataEnvironmentsIntegrationwithDB2&specialtyanalytics“engines〞leveragingDS8870delivers4xreductioninbatchtimeswithnewHighPerformanceFlashEnclosuresHighspeedencryptiononeverydrivetypesecuresdata數據維護和保管LTFSEEw/tapeprovidesreducedTCObyupto90%overdiskforlongtermretentionofdataatrestwithalargeopenformattaperepositoryReducetheamountofdatatobestoredbyupto25timeswithProtecTIERde-duplication12x更快IBMFlashSystemincreasedSPLUNK&SASapplicationefficiencytoperformbusinessanalytics20x改善inactionablesupplychainanalytics,4xreductioninbatchtimes,virtualizationforplug&play6x時間節省“GPFSallowsustomovethemetadatafromthedisktotheFlashSystemonline.Oncewedidthat,thebackupswerereduceddowntoaboutanhour.〞2hrsbecomes2minutes失效切換時間大幅縮短MappingCharacteristicstoIBMStorageProductsStorageInfrastructure需求適用于一切的5種運用場景OptimizedMulti-TemperatureWarehouse優化的多級存儲庫AllFlashFlashSystemHybridDS8000EasyTierXIV+SSDCachingStorwizeEasyTierFlashSystemSolution(VSC+FlashSystem)PureSystemsPureFlex(XIVorStorwizew/EasyTier)PureDataforTransactions(Storwize)PureDataforAnalytics(Netezza)Midrange&EntryTier0AccelerationSmarterStorageIntegratedSystemsEnterpriseOfferingsXIVzEnterpriseSolutionsforAnalyticswithDS8000PureDataSystemforOperationalAnalyticswithStorwizePureFlexSystemwithStorwizeDS8000SmartAnalyticsSystemswithDS3xxxOpen&ExtensibleStorwizefamilyFlashSystemfamilyIBMSmarterStorage的設計就是支持大數據分析

高效和優化數據根底架構IBMFlashSystem：為大數據分析運用設計的，讓運用和數據實現極速IBMFlashSystem的極速性能讓實時業務決策成為能夠適宜于模塊化數據存儲構造的Hadoop系統。某些或一切數據可以保管到Flash閃存上，其他可以保管到XIVIBMXIV:OptimizeddataworkloaddiversityforBigData&AnalyticsIBMXIV的高性能無須人工干涉配置，且適用于各種各樣的存儲負載IBMXIV的效率高的異乎尋常，而且簡單性業內最高，內置友好界面IBMXIV的彈性是企業級的，完全保證了數據的可用性和業務延續性XIV:為Analytics而生無與倫比的性能可擴展的網格存儲架構恣意時間支持恣意讀寫負載板上的閃存Flash

無與倫比的可靠性精致的數據分布無雙的磁盤重建時間企業級的可用性

無與倫比的簡易性簡單的規劃、供應和靈敏性上線后零維護零調優“XIV最吸引我們的地方就是其超強的性能…we正是由于XIV為我們的精細復雜的分析運用提供了一致的高性能，使得我們可以為我們的用戶帶來更多的價值。〞SAS和XIV網格架構——完美的結合大規模并行計算堅持繼續地最正確性能BalancedPerformance性能平衡年年零調整UnprecedentedScalability史無前例的擴展性配合添加SAS節點和XIV模塊即可IBMSVC:OptimizeddataworkloadflexibilityforBigData&AnalyticsIBMSVC經過如下功能在IBM大數據產品線上添加了靈敏性：完好和數據虛擬化和數據挪動性高級集群和復制多路鏡像，readpreferredoptionRealTimeCompression實時緊縮EasyTierHotExtentcachingStorwizeV7000/UIBMSVC設計原那么Real-TimeCompression實時緊縮是設計來做：作用于ActivePrimaryData公用的緊縮平臺PlatformhandlesALLheavyliftingassociatedwithcompression不會影響性能Wemodifyacompressedfilein-placeefficiently不會改動用戶運用Usersnoradminsneedtochangeanything處置流程不變緊縮是在線完成，不是事后緊縮業界規范緊縮算法所采用的緊縮算法曾經運用了幾十年StorwizeV7000/UIBMSVC24流處置計算&IBMFlashSystemsData:是擁有還是保管?或是是分析和開場行動!DatainDataat25InfoSphereStreams:大數據流分析為分析動態數據而建多并發輸入數據流大規?？蓴U展Massivescalability分析和處置的數據多樣化Structured,unstructured,video,audioAdvancedanalyticoperators自順應實時分析WithDataWarehousesWithHadoopSystemsCurrentfactfinding當前數據查詢分許流動中的數據——在數據落盤前低延遲方式,pushmodel數據驅動——真正的數據分析Historicalfactfinding歷史數據查詢查找和分析存儲在磁盤上的數據信息批處置方式,pullmodel查詢驅動:submitsqueriestostaticdataTraditionalComputingStreamComputing流數據計算代表著計算方式的變化Real-timeAnalyticsRealTimeAnalytics實時分析

想象一下他如何用防火栓喝水來自多個多樣輸入源的大量數據直接處置和過濾數據，而不用存儲僅保管有價值的數據僅關聯對數據最感興趣的用戶隨著數據信息的產生采取行動AdaptiveAnalytics自順應分析

DatainMotionandDataatRest的集成1.DataIngest數據集成，數據發掘，機器學習，統計建模實時和歷史數據洞察力的可視化3.AdaptiveAnalyticsModel數據收取，

在線分析預備，方式校驗Data2.Bootstrap/EnrichControlflowInfoSphereBigInsights,Database&WarehouseInfoSphereStreams

AdaptiveReal-TimeAnalytics自順應實時分析來自多個多樣輸入源的大量數據過去、如今和未來全方位綜合性視圖實時分析，低延時結果Fullcontextfordeepanalysis深度分析的完好的上下文跨datainmotionanddataatrest的常用數據分析自順應-隨機而變當發現非預期行為時，自順應當識別出新數據意義時深度分析之開場沒有認識到的數據意義，隨后才能夠認識到自順應——在開場沒有認識到的，隨后可以找出數據方式StockmarketImpactofweatheronsecuritiespricesAnalyzemarketdataatultra-lowlatenciesMomentumCalculatorFraudpreventionDetectingmulti-partyfraudRealtimefraudpreventione-ScienceSpaceweatherpredictionDetectionoftransienteventsSynchrotronatomicresearchGenomicResearchTransportationIntelligenttrafficmanagementAutomotiveTelematicsEnergy&UtilitiesTransactivecontrolPhasorMonitoringUnitDownholesensormonitoringNaturalSystemsWildfiremanagementWatermanagementOtherManufacturingTextAnalysisERPforCommoditiesReal-timemultimodalsurveillanceSituationalawarenessCybersecuritydetectionLawEnforcement,

Defense&CyberSecurityHealth&LifeSciencesICUmonitoringEpidemicearlywarningsystemRemotehealthcaremonitoringTelephonyCDRprocessingSocialanalysisChurnpredictionGeomapping如何運用InfoSphereStreams?加快數據流入分析系統的速度向買賣方向加速。。。一個高效和靈敏的根底架構顯然可以加快流速，并平衡不同數據分析的需求CoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetwork++預測分析

數據倉庫文本分析HadoopWorkloads優化敏感性分析加快流速價值時間“觸發事件〞數據完備買賣Insight預見獲取數據時間分析數據時間行動時間大數據分析的新式根底架構處理方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZoneExperiencereal-timeanalyticalinsightswithupto50xbetterperformancethanenterprisedisksystemsusingIBMFlashCore?technologyPreserveandprotectinfrastructurecontinuitywhilescalingtoover2petabyteofeffectiveall-flashcapacityunderasingleintegrateinterfaceDeliveragilityanddataeconomicswith4xgreatercapacityinlessrackspacethancompetitiveall-flashproductsSynchronizedandComplimentarytoOverarchingStorageMessaging-Acceleratetimetoinsightsthrough"datawithoutborders."IBMinnovationfreesdatawithagileandsimpletousestoragesolutionsdeliveringsuperiordataeconomicsIBMFlashSystemCoreLaunchMessagingDriveacompleteparadigmshiftinEnterpriseStoragewiththeallnewIBMFlashSystemFamilyIBMFlashSystemFamily

2021ThemeTimetoinsight.Timetovalue.Timetomarket.IBMFlashSystem,it’sabouttime.FlashRealized!IBMFlashSystemV9000

FoundationalPillarsIBMFlashCore?TechnologyistheDNAoftheFlashSystemFamilyScalablePerformanceEnduringEconomicsAgileIntegrationIntroducingtheNewIBMFlashSystemFamilyOfferingsIBMFlashSystem900ExtremePerformance:Delivers100microsecondresponsetimesMacroEfficiency:Lowestlatencyofferingwith>40%greatercapacityatalowercostpercapacityEnterpriseReliability:IBMenhancedMicronMLCflashtechnologywithFlashWearGuaranteePoweredbyIBMFlashCore?TechnologyIBMFlashSystemV9000ScalablePerformance:Growcapacityandperformancewithupto2.2PBscalingcapabilityEnduringEconomics:NextgenerationflashmediawithlowercostpercapacityAgileIntegration:FullyintegratedsystemmanagementtosimplifymanagementandimproveworkforceproductivityunderasinglenamespaceFlashSystem900IntroducingIBMFlashSystem900,thenextgenerationinourlowestlatencyofferingIBMMicroLatency?withupto1.1millionIOPS40%greatercapacityata10%lowercostpercapacityIBMFlashCore?technology,oursecretsauceTechnicalcollaborationwith