OPENINGASFMember,ApacheCeleborn/Flink/HBase/PaimonPMCMember阿里云智能EMR負責人DataTrendsDataVolume:AIfurtherdrivesmassivedataexplosion,farexceedingthedatagrowthofthepreviouseracomputation,andmanagementAIModelsVedioAIModelsAnalyticDataOthersPicturesTheEvolutionofDataArchitectureApplicationsDatabaseReportsETLDataDataExploreETLDatawarehousesstructured,semistructuredandunstructuredDataRealtimeAnalyticsMachineLearningDatascienceReportsstructured,structured,semistructuredandunstructuredDatastreamingAnalyticsMachineLearningDatascienceDataWarehouseReportsDatawarehousesETLApplicationsDatabaseDataWarehouseReportsDatawarehousesETLApplicationsDatabaseTheDatawarehouseArchitecturewillbewillbediscardedApplication··out-of-box,EasytouseStrengthsWeaknessesDataTheDataLakeArchitectureDataLakeApplicationstoreAllReportsMachineLearningDatascienceReportsMachineLearningDatascienceDataExploreETLDatawarehousesStrengthsWeaknessesStrengthsWeaknessesstructured,semistructuredandunstructuredDataDataWarehouseReportsDatawarehousesETLApplicationsDatabaseDataWarehouseReportsDatawarehousesETLApplicationsDatabaseDataLake+Datawarehouse=DataLakehousestreamingAnalyticsMachineLearningDatasciencestructured,semistructuredstreamingAnalyticsMachineLearningDatasciencestructured,semistructuredandunstructuredDataReportsMachineLearningRealtimeAnalyticsDatascienceDataExploreETLDatawarehousesstructured,ReportsMachineLearningRealtimeAnalyticsDatascienceDataExploreETLDatawarehousesstructured,semistructuredandunstructuredDataDevOpsComputingEnginesGovernanceServicesManagementServicesDataFormatsApachepaimonDataStorageAlibabaCloudopenLakeTheLakehousesolutiononAlibabacloudDataworksIDECopilotE-MapReduceApplicationApplicationApachePaimon(LakeFormat)OSS-HDFS（LakeStorage）TieredStorageTieredStorageCompactionAuthenticationAuthorizationBUILDOPENSOURCECOMPATIBLELAKEHOUSEONALIBABACLOUDASFMember,ApacheCeleborn/Flink/HBase/PaimonPMCMember阿里云智能EMR負責人RDBMRDBMSLogsODSDWDDWSLogsRecapTheLakehousesolutiononAlibabacloudE-MapReduceApplicationApplicationApachePaimon(LakeFormat)OSS-HDFS（LakeStorage）TieredStorageCompactionAuthenticationAuthorizationServerlessSparkTransformsDataManagementwithOne-Stop,FullyManagedServicesforSeamlessDevelopment,Scheduling,andMaintenance.?One-stopdataengine?VisualizedjobandworkflowthroughputforIO-intensiveAppScenarioAppScenarioVersionControlAccountingAccountingObjectStorageObjectStorageServiceSessionManagement(ResourceforInteractiveSessionManagement(ResourceforInteractiveQuery)QueueManagement(ResourceforETL)CatalogViewArtifactsManagementcontrolplaneDataCatalogViewArtifactsManagementIntelligentDiagnosecontrolplaneJobMonitorandIntelligentDiagnoseCanvasEditorWorkflowInstanceMonitor-SingleExecutionCanvasEditorWorkflowInstanceMonitor-SingleExecutionViewWorkflowInstanceMonitor-GlobalViewTestingEnvironmentTestingEnvironment?AlibabaCloudLinux3?NativeOperatorHardwareawarenessoptimizationacceleration?zstd-ptgcompressionacceleration?Datashufflereducedupto40%NativeC++Integrationintegration?ApacheTopLevelProject,donatedby?ApacheTopLevelProject,donatedbyAlibabaCloud?Enterprisesecurityassurancewithdataencryption?EnhancedIOscheduling,flowcontrolandquotamanagementMulti-?Widelyadoptedin?Successfullysupportsjobwith600TB+shuffledataScalability69%PerformanceboostthanYARNexternalshufflePerformancegainincreaseswithshuffledatascale?SupportsSparkAQEFunctionality??Spark-submitCompatibleJobSubmission?Gitintegration(PlaOSS-HDFS?Workspace?WorkflowsFunction-wiseEMRServerlessSparkYESYESYESYESWorkflowManagementYESYESYESYESYESCatalogandAuthenticationYESYESYESYESAuditingYESYESYESYESYESAssistant/CopilotYESYESTheLakehousesolutiononAlibabacloudE-MapReduceApplicationApplicationApachePaimon(LakeFormat)OSS-HDFS（LakeStorage）TieredStorageCompactionAuthenticationAuthorizationEMRserverlessstarServerlessStarRocksOffersaHigh-Performance,All-Scenario,Blazing-FastandUnifiedDataLakehouseAnalyticsService.100%CompatiblewithOpen-sourceStarRocks，3XFasterth?SIMD-Optimizedqueryengine?Fullstackvectorizedte?DisAggandVirproductArchitectApplicationApplicationScenarioAd-hocdashboardOperationanalyticsUserprofileReal-timeanalyticsSelf-servicereporting…ProductLayerStarRocks-instanceLayerStoragelayerVirtualWarehouseLakehouseAnalyticsShared-NothingArchitectureDataLakeTableFormatStarRocksTableFormatDataLakeSQLEditorDataLoadingSecuritySQLprofManagementalertVirtualWarehouseVirtualWarehouseHealthanalysisUpgrading…InstanceManagementAuto-ScalingFastandunified?Acomprehensivevectorizedexecutionengine,modernizedcost-basedoptimizer(CBO),withconcurrencyreachingtensofthousandsofqueriespersecond(QPS).?Fullycompatiblewithdatalakeformats,offeringmorethana3XperformanceimprovementrelativetoTrino.?SupportsmaterializedviewELTscenarios,enablingone-stepdatatierprocessing.Separationofstorageandcompute?Optimizedcomputationalelasticityforon-demandusage,withthepotentialtoreducestoragecostsbyupto60%.?Offersmulti-computingclustercapabilities,ensuringresourceisolationbetweendifferentbusinessunitswithoutinterference.?Variouscachingstrategiesavailable,allowingcustomerstoflexiblyconfigureaccordingtotheirbusinessneeds.?Outofbox,theStarRocksManageroffersawiderangeofenterprise-levelfeatures.?Intelligentdiagnosticsandanalysis,providingcomprehensiveanalysisinconjunctionwithcustomerbusinessoperations.controlplanestarManagerHighlightsExtremeHighlightsExtremeElasticityDis-aggregationSupportSlowSQLProfileandDiagnoseInstanceDiagnoseLakeQueryAccelerationLakeQueryAccelerationHive/Paimon/Iceberg/HudiADSADSDWSHive/Hive/Paimon/Iceberg/HudiOn-demandSecond-levelElasticitywithLowCostComprehensiveloadanalysisanddiagnostic3x-5xfasterthanTrinoSignificantlyfasterthanClickHouseandApacheDorisSupportexternalMVandLakehouseHierSophisticatedcachingandtieredstoragecapabilityRecapTheLakehousesolutiononAlibabacloudE-MapReduceApplicationApplicationApachePaimon(LakeFormat)OSS-HDFS（LakeStorage）AuthenticationAuthorizationTieredStorageCompactionAuthenticationAuthorizationTieredStorageCompaction?DatabricksDeltaFunctionality?MetaRetrieval?MetaStatsforCBOFullyManaged?Serverless,Elastic?HighAvailable?HighThroughputs?Import/Exportfrom/toHMSAuthenticationAuthorizationAuditing?AuditLogforAuthorization?AuditLogforMetaOperation?AuditLogforDataOperation(WIP)CompactionManagerITieredStorageManagerThanksYuLiliyu@李勁松ApachePaimonPMCChairCONTENTS1.OpenLake：一套存儲對接全生態2.ApachePaimon與開源計算引擎3.ApachePaimon與自研計算引擎4.ApachePaimon實踐場景CONTENTS數據湖到湖倉一體數據交換0101010101010101010101010100101湖格式SDK讀寫湖倉一體元數據湖格式0101010101010101010101010100101數據架構的選擇批式數倉實時湖倉實時數倉openLakeTheLakehousesolutiononAlibabacloudDataworksIDECopilotE-MapReduceApplicationApplicationApachePaimon(LakeFormat)OSS-HDFS（LakeStorage）TieredStorageTieredStorageCompactionAuthenticationAuthorizationpaimonCOMPUTINGPLATF○RMPaimon+開源大數據?共享存儲，計算平權?流批一體，實時升級?實時離線，極速查詢ApplicationIngestionApplicationIngestionstreamingIngestion實時OLAPOLAP實時OLAPOLAPongoingBatchLeftBatchLeftJoinAggregatestreamingstreamingpartiaupdateAggregate001011101010101010101010101001001阿里云Flink+paimon:streamingLakehouse45流寫更新入湖45流寫更新入湖多表數據打寬45流讀變更日志流讀變更日志spark+paimon:(Higherisbetter)(Higherisbetter)20阿里云離線數據極速阿里云阿里云starPaimon:Deletionvectors模式paimonDataLakeInformation:BridgetoMC&HoloDLF打通自研計算引擎?MaxCompute：ExternalSchema$ApachePaimon(LakeFormat)OSS-HDFS（LakeStorage）即將發布?ALIORC格式?批寫支持ApachePaimon(LakeFormat)OSS-HDFS（LakeStorage）即將發布ApachePaimon(LakeFormat)OSS-HDFS（LakeStorage）paimon某新能源汽車公司在阿里云上的實踐 ApplicationStreamingIngestion streaming異步compactionAppend表changelog=lookup某游戲公司在阿里云上的實踐 ApplicationStreamingIngestion實時OLAP01011101011110101010101010101010010010streamingBatch異步compaction主鍵表主鍵表Append表changelog=inputdeletion-vectorsApacheApachePaimon某本地生活公司在阿里云上的實踐 ApplicationStreamingIngestion高性能OLAPAppend表Cluster:Z-orderThanks李勁松ApachePaimonPMCChair阿里云實時湖倉及Flink產品技術介紹阿里云計算平臺1大數據實時湖倉發展趨勢洞察2基于阿里云實時計算Flink構建實時湖倉3阿里云實時計算Flink產品能力解讀CONTENTS4典型落地架構及案例分享大數據進入實時化湖倉時代!3.02.02.0數倉2023~2020-2022數據湖數據倉庫數據湖結構化，半結構化及非結構化數據02基于阿里云實時計算Flink構建實時湖倉(streaminglakeStreamingLakehouse分鐘級新鮮度分鐘級新鮮度低成本低成本全鏈路實時全鏈路實時秒級查詢響應秒級查詢響應Streaming:T+1s流/批流/批流/批流/批流/批流/批?低成本OSS存儲構建Paimon?深度集成Flink全鏈路實時化?低成本全鏈路實時化?流批存儲計算統一?一套平臺具備數據管理、調度?開放支持多引擎?離線全鏈路實時加速?實時鏈路降本?流批存儲計算統一實時湖倉全鏈路實時加速端到端，全鏈路實時流動，實時更新，分鐘級新鮮度，全鏈路可查，秒級查詢響應！ 數據攝取數據存儲數據計算數據查詢?Streaming?StreamingETL?開放支持多種計算引擎?開放支持多種Olap引擎?外表方式查詢秒級響應?也可直接upload到引擎?基于內存優化查詢性能?Upsert/Partial-Update?TimeTravel?BatchOverwrite/Query實時入湖入倉-簡化操作CTAS分庫分表合并同步CDAS整庫同步實時入湖入倉兼容表變更(schemaEvolution)?支持通過Catalog來實現元數據的自動發現和管理?配合CTAS語法，實現數據的同步和表結構變更自動同步?支持讀取數據變更和表結構變更并同步到下游，數據和表結構變更都可以保證順序?同步到Paimontable時Partitionby可自動兼容有無分區字段實時入湖入倉-多種過程操作SELECTWHEREGROUPBYSELECTWHEREGROUPBYTop-NINSERTSQLAPImapkeyByaggregateDataStreamAPIMoresourcesareontheway實時湖倉低成本存儲 DistributedFileSystem(HDFS/OSS/S3)低延時低延時低成本低成本流批存儲流批存儲支持數據流批計算?通過兩階段提交保證數據ExactlyOnceBatchBatchStreaStream03阿里云實時計算Flink產品能力解讀阿里云實時計算Flink產品豐富的企業級能力數據攝取數據攝取?Yaml模版統一元數據（catalog）上下游SSL支持升級企業級安全能力基礎設施、平臺系統安全多維度，提供全面的安全加固功能來保障數據安全！云上大數據服務如何保障企業數據和服務安全構建全面、多層次的安全管理能力，持續保護云上數據及服務安全通AccessKey帶來的安全風險復復自定義connector管理catalog管理deployment動態更新lineage數據血緣deploymentTarget改造UDF注冊作業資源自動調優基于業務處理復雜度與數據流量，資源動基于業務處理復雜度與數據流量，資源動態調整資源利用率低資源利用率低采集指標采集指標成本高作業吞吐低，延遲高過低作業吞吐低，延遲高過低(易發生FailOver啟動速度慢作業管理平臺更新作業過高(易發生FailOver啟動速度慢作業管理平臺更新作業過高指標分析指標分析綜合各指標生成調優執行計劃重啟作業動態更新作業部署集群04典型落地架構及案例分享 簡單SQL 駕FlinkBinlogDashboardskBinlogDashboardskk?Hologres、Paimon都具備流式訪問能力，故數倉各層可以根據存儲成本、業務時效性進行選擇?數據直接入Hologres：提供秒級時效性+?OLAP引擎可選，支持StarRocks、Trino等典型客戶落地案例過離線數據處理;理離線數據；過程中，兩條技術棧開發、維護成本高，存儲成開發效率提升進一倍，每年節省存儲成本KW，查詢效率提升3倍；?從兩條鏈路簡化到一條鏈路，簡化了系統的復雜度；運維工?一套SQL/Table、一套schema，大幅提升開發效率；?大量縮減Kafka集群，每年節省KW成本；kafkakafkakafkakafka增量databasedatabase增量databasedatabase加工StarRocks二加工Paimon二加工PaimonThanks釘釘信：tute2014阿里巴巴智能引擎事業部技術專家CONTENTS1、產品背景簡介2、解決方案舉例---搜索離線平臺3、生產作業調優及社區合作4、FutureTransactionsAlgorithmdataEventsLogsTransactionsAlgorithmdataEventsLogs業務場景及產品定義BinlogMessageQueueDatabaseMysqlODPSPaimonOfflineSystemStreamStreamProcessingBatchProcessingBatchProcessingMessageQueueODPSPaimonHologresFileSystemSearchEngineAdvertisingEngineRecommendationEngineSampleEngine2、業務多且邏輯復雜3、性能調優難、運維門檻高基于該業務場景我們做了一個提供AI領域e2e的ETL數據處理解決方案的產品產品技術架構 淘寶天貓本地生活菜鳥高德AE飛豬LazadaOpenSearch…產品端 淘寶天貓本地生活菜鳥高德AE飛豬LazadaOpenSearch…產品端據搜推平臺離線推理大模型視覺平臺評測特征樣本平臺UI&&WebIDE（開發、配置、運維、監控、報警）Embedding計算數據集成樣本處理數據集成樣本處理SQLAdHocOLAP流計算批計算流批一體用戶插件調度編排AirflowAirflow調度VVP提作業、開發、運維Catalog（Meta、版本、血緣、Dataset）Restune作業彈性資源Celeborn統一Shuffle服務Paimon湖格式PaimonTDDLSwiftVVP提作業、開發、運維Catalog（Meta、版本、血緣、Dataset）Restune作業彈性資源Celeborn統一Shuffle服務Paimon湖格式PaimonTDDLSwift消息隊列湖表存儲優化服務VVRDRCHA3SparkHologres分布式kv存儲ODPSUDxFCDCPangu（分布式文件系統）ConnectorASI（支持K8S協議的統一調度、統一資源池）計算存儲計算搜索離線平臺定義如何將來源于不同維度數據源的數據匯集到同一個頁面？QueryQuery在線搜索引擎在線搜索引擎user_iduser_name1vivo“103”:“vivo-x50-pro”}3搜索在線集群1131vivo31131vivo3131vivo“103”:“vivo-x50-pro”}3商品維度Hologres寬表DimJoin商品維度Hologres寬表賣家維度Hologres寬表發送到搜索引擎joinKey:賣家維度Hologres寬表發送到搜索引擎user_iduser_name1vivo“103”:“vivo-x50-pro”}3業務邏輯翻譯為VVR作業、流批一體業務邏輯圖VVR批作業VVR流作業nn:11:n1:11:n1:1商品擴展表賣家表商品擴展表賣家表同步層同步層…………ssScanDimJoinDimJoin DimJoinJoin層ScanDimJoinDimJoin DimJoinJoin層業務開發模式為了方便業務開發，提供“托拉拽”開發圖，方便業務方描述業務邏輯業務“托拉拽”開發圖啟動全量(生成調度圖)Airflow任務調度圖分配存儲資源注冊產出信號同步層全量優化---數據集成寫數據Supportclonelatestsnap/apache/paimon/pull/3159/apache/paimon/pull/3287/apache/paimon/pull/3426SupportclonelatestsnapSupportremovedborpModifythedefaultSupportremovedborpModifythedefaultvalueof"target-file-size”/apache/paimon/pull/3721/apache/paimon/pull/3779一、原鏈路缺點1、并發有上限限制，吞吐受限，加并發有拉掛庫的風險。2、核心庫拉取時間只能晚上。二、新鏈路預期收益寫數據HologresSinkHologresSinkOperatorODPSSourceOperatorODPSSourceOperatorHologresSinkOperatorRPCRPCODPSSourceOperatorHologresSinkOperatorRPCRPCRPCODPSSourceOperatorHologresSinkOperatorRPCRPCODPSSourceOperatorHologresSinkOperatorRPCRPCTaskManager嘗試：所有節點都Chain在一起。缺點：由于多個應用共享一個Hologres集群，所以容易造成HologresWorker網絡繁忙，各應用之間相互影響吞吐。Job1JobnJob2NetworkBusy!!!寫數據HologresHologresSinkOperatorhologres_hash_function(key)ODPSSourceOperatorrRPCODPSSourceOperatorrRPCWorkerHologresHologresSinkOperatorODPSSourceOperatorRPCODPSSourceOperatorRPCWorkerHologreHologresSinkOperatorODPSSourceOperatorrODPSSourceOperatorrRPCWorkerHologreHologresSinkOperatorRPCTaskManagerRPCTaskManagerWorkerTaskManager按照Hologres對主鍵Key做Hash分Shard的規則在VVR中自定義Partitioner，加層Shuffle，以此減少HologresWorker的網絡連接。VVRBatchJob資源消耗大。VVRVVRVVRVVRVVRJob2Worker寫數據VVRVVRWriteJobWriteFlushWorkerPanguHologresSQLServerlessServerlessTaskWorkerReadReadWrite1、嘗試Hologres起ServerlessTask,直讀ODPS盤古上的ORC，數據不再經過WAL和MemoryTable。保證資源隔離，不再對HologresWorker上正在跑的Scan、點查等造成影響。2、缺點ODPS與Hologres之間不能有別的算子，比如UDTF。同步層全量優化---結論優點所有節點Chain在一起各應用吞吐易相互影響應用吞吐不會相互影響所以目前生產環境中，是根據不同業務場景選用不同的數據同步方法。ScanDimJoiScanHolo表ShardHolo表Shard數并發DimJoinHologresHologresSourceDimJoinHologresHologresSourceOperator……SinkSinkOperatorHologresHologresSourceHologresHologresSourceOperator……SinkSinkOperatorHologresHologresSourceHologresHologresSourceOperator……SinkSinkOperatorTaskManager缺點：1、Failover代價大，全量產出易延遲。2、混部環境惡劣，作業長尾嚴重ScansScans 近百個OperatorsSourceOperatorHologresSourceOperatorsSourceOperatorCelebornHolosSourceOperatorHologresSourceOperatorsSourceOperatorCelebornHolo表Shard數并發DimJoinHologres…SinkOperatorDimJoinHologreDimJoinHologres…SinkOperatoSinkOperatorDimJoinHologreDimJoinHologres…OperatoOperatorTaskManagerTaskTaskManager缺點：1、JobManagerOOM2、TM與JobManager心跳超時3、JobManager與Zookeeper斷連并發上限HologresSourceOperatorJobManager缺點：穩定性雖高，但耗時不算優秀Holo表ShardHologresSourceOperatorJobManager缺點：穩定性雖高，但耗時不算優秀Holo表Shard數并發并發上限TaskManager起備份Task解長尾問題資源復用Load低的機器跑更多的Task用限制資源的方式實現分批調度，完成一個Task起一個TaskSpeculativeSchedulerDimJoinDimJoinHologresSinkSinkOperatorScanSScan/apache/paimon/pull/3474Supportcustomcommituserprefix/apache/paimon/pull/3507Introducepartitionmarkdonewithend-inputVVRVVR有限流耗時Max(T1,T2)https:///apache/paimon/pull/3584批作業耗時T2…批作業耗時T2…DeltaDeltaFullDeltaFullDelta耗時(T1耗時(T1+T2)優點++理效率相對不高所以目前生產環境中，是根據不同業務場景選用不同的數據處理方法。慢節點慢節點SourceSinkSourceSinRound-RobinSourceOperatorRecSourceOperatorRecordWriterConnectio下游TaskConnectio下游TaskManagerMap<ResultSubMap<ResultSubPartitionId,Weight>ConnectionRecordSerializerResultSubPartition1ResultSubPartitionResultSubPartition2上游TaskManager每當有新消息下發要選擇哪個SubPartition時，DynamicRebalanceChannalSelector會選擇一個權重最低的下游并發發送Buffer。App2App1App2App1App2App1App2App1App3App3App3App3App4App4App4App4VVRBatchJob+PanguVVR有限流Job+PaimonFuture---Open-Lake、湖倉一體阿里云客戶阿里云客戶阿里巴巴集團數據業務SparkSparkMaxComputeRealtimeComputeE-MapReduceOpen-LakeApachePaimon01010101010101010101010101010101E-mail:hongli.wwj@MC湖倉一體實踐及開源開放生態融合王燁(萌豆)-高級技術專家?王燁（花名：萌豆）、高級技術專家?阿里云MaxCompute團隊，湖倉一體化方向產品技術負責人，SQL引擎、優化器方向核心研發Email:ye.wangy@alibaba-inc1.從聯邦計算出發，構建基礎湖倉能力2.以數據湖為重心，全棧優化外表性能CONTENTS3.高效分析非結構化數據，強化AI應用4.多引擎間平權，開源開放融合新架構產品介紹“云原生大數據計算服務(MaxCompute)是一種快速、完全托管的TB/PB級數據倉庫解決方案。MaxCompute向用戶提供了完善的數據導入方案以及多種經典的分布式計算模型，能夠更快速的解決用戶海量數據計算問題，有