1、現代數據中心解決方案提綱思科數據中心產品介紹典型組網方案成功案例LAN/SANLANSAN思科數據中心產品系列CISCO NX-OS: From Hypervisor to Core CISCO DCNM: Single Pane of ManagementDELIVERING TO YOUR DATA CENTER NEEDSResilient, High Performance, Scalable Fabric Operational Efficiency/ ConsistencyP-V-CWorkload Mobility Within/ Across DCsSecure Separat

2、ion/ MultitenancyLAN+SAN ConvergenceCisco MDS 9200 Series Cisco MDS 9500 Series Cisco Nexus 1000VCisco Nexus 1010CiscoNexus 4000Cisco MDS 9100 SeriesCiscoNexus 5000CiscoNexus 3000CiscoNexus 2000CiscoNexus 7000CiscoNexus 6000Nexus 7000系列Nexus 7004Nexus 7009Nexus 7010Nexus 7018Target DeploymentsSmall

3、and Medium DCIdeal migration from Catalyst 6509Ideal for cold aisle - hot aisleHighest 10G DensityBandwidth Per Solt550Gbps550Gbps550Gbps550GbpsMax 10GE Density(Non-Blocking)96 ports336 ports384 ports768 portsFootprint7RU14RU21RU25RU8,800+Customers5 Million+Ports Shipped43,000 Chassis Shipped Nexus

4、7000常用接口板卡F2e 48 ports 1 and 10GE (SFP+) F2e 48 ports 1 and 10GBaseT (RJ45) N7K-F306CF-25N7K-F312FQ-25Nexus 7700系列Nexus 7718Nexus 771018-slot chassis1.3Tbps/slot16 x payload modules2 x supervisor modules6 x fabric modules3 x fan traysFront-to-back airflow16 x 3000W PSUs10-slot chassis1.3Tbps/slot8 x

5、 payload modules2 x supervisor modules6 x fabric modules3 x fan traysFront-to-back airflow8 x 3000W PSUsNexus 7710 ChassisRedundant Supervisor Engines8 payload slots (1.3T/slot)8 3000W AC/DC power supplies14RU3 fan trays6 fabric modules (behind fan trays)RearFrontFront-to-back airflow32”Nexus 7700 主

6、要業務板卡Nexus 7700 F2e 48p 10GNexus 7700 F3 24p 40GNexus 7700 F3 12p 100GNexus 7000 / Nexus 7700 ComparisonNexus 7000Nexus 7700Max System Switching Capacity 17.6+ Tbps42+ Tbps (Fab2)84+ Tbps (future)Per-Slot Bandwidth 550 Gbps1.32 Tbps (Fab2)2.64 Tbps (future)Bandwidth per fabric46Gbps/payload slot (Fa

7、b1)110Gbps/payload slot (Fab2)220Gbps/payload slot (Fab2)440Gbps/payload slot (future)Max Port Density (10G/40G/100G)768/192/96768*/384/192 (F2E/F3)Air FlowFront-to-backSide-to-sideSide-to-backFront-to-backModule typesM1/M2F1/F2/F2E/F3Nexus 7700 F2E/F3FabricsUp to 5 X Fab1Up to 5 X Fab2Up to 6 X Nex

8、us 7700 Fab2* Higher density 10G possible via 40G/100G modules using breakout cables in future48/96 port 4 slot (Q2 CY14)8 slot16 slot (Q2 CY14)高度2/3 RU6-7 RU13 RU21 RU業務板卡槽位1 GEM4816單系統交換容量 (Tbps)NA15 Tbps30 Tbps60 Tbps最大線速的10G端口數量485761152Future最大線速的40G端口數量12144288576可升級到ACI矩陣模式N9K 產品系列Nexus 9396P

9、XNexus 93128TXNexus 9504 Nexus 9508 Nexus 9516 Nexus M12PQNexus 6004PerformanceHighest density line rate, low power per port 10G/40G Ethernet/FCoE switchin the industry!Integrated Layer 2 and Layer 3 feature-setFirst 40G FCoE Switch VersatilityWide range of deployment options High density FEX aggreg

10、atorLarge Scale FabricsLeaf-Spine ArchitectureLatencyLow Latency at scale 1 microsecond latency for all ports, cut through switchingFlexibilityConfigurable port speeds with true 40GE flows or 4 x 10GE flows Modularity provides future expansion capabilitiesEnables High Performance Networking through

11、Cisco Custom SiliconForm-factor / Power4RU; 17.5” W x 30” D96 Line rate 40G ports or 384 Line rate 10G portsPower per 40G port: Max: 31W & Typical: 25WF2B and B2F airflow SKUsUnique InnovationsECN for handling in-cast trafficOptimized multicast scheduling/replicationSupports flexible resource carvin

12、g (TCAM, Buffers)Latency Analytics with conditional SPAN, ERSPAN, IPFIX, Sampled Netflow26MB shared buffer per 12x 10GE ports or 3x 40GE portsUnmatched scale & featuresFCoE, NAT, Line rate SPAN/ERSPAN1024 way ECMP, 16K VLANs, 4K ACLLayer 2 Multi-Path (Fabric Path, TRILL)IPv4 & IPv6 Unicast & Multica

13、st Routing256K MAC/IP Host Tables with flexible carving32K LPM & 32K mroutesNexus 6004 Key Features尺寸及電源1RU; 48 個10G SFP+和4個40G QSFP+接口前進風或后進風交流或直流電源1+1 冗余電源創新特性采用基于條件的SPAN, ERSPAN進行延遲分析精準擁塞通告技術 (ECN) ，可以實現端到端的網絡擁塞控制 優化組播規模及組播復制彈性的資源分配(TCAM, Buffers)大緩存設計：26MB shared buffer per 12x 10GE ports or 3x

14、40GE ports主要技術規格交換容量1.28Tbps線速三層轉發，1us轉發延遲256K MAC/ARP表，支持靈活分配Line rate SPAN/ERSPAN16K VLANs, 4K ACL32K LPM & 32K mroutesIPv4 & IPv6 Unicast & Multicast RoutingLayer 2 Multi-Path (Fabric Path, TRILL)FCoE, NAT, 1024 way ECMP, 支持24個FEX L2/L3模式Nexus 6001Cisco Nexus 5500Offering Choice and Flexibility f

15、or Varied DeploymentsNexus 5500 PlatformNexus 5548UP 1RU Switch (Unified Ports)Nexus 5596UP 2RU Switch (Unified Ports)Nexus 5596T 2RU Switch (10G BaseT)Nexus 5548 Layer 3 Daughter CardNexus 5596 Layer 3 Expansion ModuleNexus 5500 Unified Ports Expansion Module20 Million+2k/5k Ports Shipped125K+ Chas

16、sis Shipped 19,000+2k/5k CustomersNexus 5500 QSFP+ GEM1G10G40G201320122011Nexus 3048TP48-Port 1GT 4-Port SFP+Nexus 3016Q16-Port QSFPNexus 3132Q32-Port QSFP+Nexus 3064-X48-Port SFP 4-Port QSFP+ Integrated PHYNexus 3064-T48-Port 10G-BaseT 4-Port QSFP+ Nexus 3172PQ72-Port SFP+ Nexus 3172TQ72-Port 10GbT

17、Nexus 3064-32T32-Port 10GNexus 3000 SeriesN2224TP24 Port 100/1000M Host Interfaces2 x 10G UplinksN2232PP32 Port 1/10G FCoE Host Interfaces8 x 10G UplinksFET-10GCost Effective Fabric Extender Transceiver (5k/7k)N2248TP-E48 Port 100/1000M Host Interfaces4 x 10G Uplinks32MB Shared BufferB22: HP, Dell,

18、Fujitsu16 x 1/10G Host Interfaces 8 x 10G Uplinks (5k)N2232TM-E32 Port 1/10GBASE-T Host Interfaces8 x 10G Uplinks (Module)Lower Power and BER PHYN2232TM32 Port 1/10GBASE-T Host Interfaces8 x 10G Uplinks (Module)N2248TP48 Port 100/1000M Host Interfaces4 x 10G UplinksNexus 2200系列矩陣擴展器N2248PQ48 1/10G F

19、CoE Host Interfaces4 x QSFP 40G Uplinks思科數據中心接入層交換機定位Nexus 5000 & 6000Nexus 3000 & 3100Nexus 3500Switch-on-Chip (SoC)Multi-Stage FabricHigh Performance TradingHigh Performance ComputeWeb Services DCVirtualized DCUnified FabricUnified Fabric ReadyBuilt for VirtualizationHigh Performance ScaleLowest T

20、CO with Fabric Extender (FEX)High Density, High Performance, Low LatencyLow PowerOpen APIsProgrammableLowest LatencyDeepest VisibilityRich FeaturesProgrammableProgrammable DCNexus 2248TP-E大緩存機制Cisco Unified FabricMAPReduceNIF 1-4x10GEHIF 1-48dedicatedshared32MBIncastN2248TP-E Optimizations32MB 共享緩存避

21、免流量突發時出現丟包。增強的計數器跟蹤Buffer的使用情況。N2248-TP-EModular SwitchFixed backplane=Fabric Extenders(Remote Line Cards)Cross-bar & Supervisor分布式模塊化交換機10Gb Ethernet for the Backplane Physical Chassis Model Virtual Chassis ModelCisco FEX架構 簡化管理，降低成本Nexus 7000/6000/5000 PlatformNexus 2000 Platform虛擬接入交換機（構成一個計算域）L3

22、L2接入層的虛擬化利用思科Nexus6000或Nexus7000作為“父交換機”，Nexus2000作為Fabric Extender，組成一個大型的虛擬接入層交換機整個虛擬接入層交換機本身“無環路”、“無阻塞”，很好的滿足了部署服務器虛擬化的技術要求虛擬接入交換機. . .No Loop No STP利用vPC技術將兩臺設備虛擬為一臺傳統模式與vPC模式對比L2Non-vPCvPCVirtual Port ChannelPhysical TopologyLogical Topology利用vPC技術可以將兩臺設備虛擬化為一臺設備，接入層設備可使用PortChannel方式上聯，不必使用STP

23、協議可為虛擬機遷移、服務器集群提供可靠的二層網絡環境提高了網絡的可用帶寬雙上聯的服務器可工作在Active-Active模式FabricPathVPC+FEXNexus5500Nexus7000VDC利用Fabric Path構建大二層網絡Nexus5500支持16條鏈路的負載均衡（ DC1-5500-2(config)# fabricpath load-balance unicast ）利用ISIS協議構建Fabric Path路由表Conversational MAC地址學習，減小MAC地址表硬件設備已經具備支持TriLL的能力，通過軟件升級即可支持Trill標準思科將推動把FbricPa

24、th的一些增強特性添加到Trill標準中WAN/Core邊緣終結所有的主機二層協議：包括MAC學習, ARP, IGMP, LLDP, DHCP內部嚴格的故障隔離：Fabric內沒有生成樹、ARP廣播、未知單播洪泛等等，象傳統的路由網絡一樣可擴展VLAN Anywhere：任何服務器可在任意VLAN，VLAN可延展到全網Gateway Anywhere：任意位置都可做任意VLAN路由，同一VLAN網關一致Routing Anywhere：無論VLAN內還是VLAN之間都以類似三層路由的方式通信易于擴展、簡化管理、自動部署DFA架構象三層路由網絡一樣的擴展和隔離性，實現統一的二層/三層轉發GW

25、IP: 10.1.1.1GW MAC: 0011:2222:3333IS-IS：維護拓撲iBGP：通告主機路由FabricPath：交換機間轉發新協議來自成熟技術靈活拓撲橫向擴展高可用性動態接入帶來優勢提綱思科數據中心產品介紹典型組網方案成功案例Name NodeRuns a scheduler Job TrackerManages all data nodes, in memorySecondary Name Node Snapshot of meta data of HDFS clusterTypically all three JVM can run on single nodeData

26、 Node Task Tracker Receives Job Info from Job Tracker(Name Node)Map & Reducer Task Managed by Task TrackerConfigurable Ratio of Map & Reduce Task for various workload per Node/CPU/CoreData Locality - IF data not available where the map task is assigned, a missing block be copied over the networkHado

27、op Components and Operations25ToR FEX/switchData node 1Data node 2Data node 3Data node 4Data node 5ToR FEX/switchData node 6Data node 7Data node 8Data node 9Data node 10ToR FEX/switchData node 11Data node 12Data node 13Data node 14Data node 15Name NodeToR FEX/switchData node 1Data node 2Data node 3D

28、ata node 4Data node 5ToR FEX/switchData node 6Data node 7Data node 8Data node 9Data node 10ToR FEX/switchData node 11Data node 12Data node 13Data node 14Data node 15HDFS Architecture261SwitchName NodeFile metadata stored in fsimage.txt and in-memory only (!) map of blocks to data nodes/usr/sean/foo.

29、txt:blk_1,blk_2/usr/jacob/bar.txt:blk_3,blk_4Data node 1:blk_1Data node 2:blk_2, blk_3Data node 3:blk_3112223334444MapMapMapMapMapThe Data Ingest & ReplicationExternal ConnectivityEast West Traffic (Replication of data blocks)Map Phase Raw data Analyzed and converted to name/value pair.Workload tran

30、slate to multiple batches of Map task Reducer can start the reduce phase ONLY after the entire Map set is completeMostly a IO/compute functionHadoop Components and Operations27Hadoop Distributed File SystemUnstructured DataMapMapMapMapMapMapMapMapMapMapKey 1Key 1Key 1Key 1Key 1Key 1Key 1Key 2Key 1Ke

31、y 1Key 1Key 3Key 1Key 1Key 1Key 4ReduceShuffle PhaseReduceReduceResult/OutputReduceMapMapMapMapMapMapMapMapMapMapShuffle Phase - All name/value pair are sorted and grouped by their keys.Reducer is PULLING the data from the Mapper NodesHigh Network ActivityReduce Phase All values associates with a ke

32、y are process for results, three phasesCopy - Get intermediate result from each data node local diskMerge - To reduce the number of filesReduce MethodOutput Replication Phase - Reducer replicating result to multiple nodesHighest Network ActivityNetwork activities dependent on workload behaviorHadoop

33、 Components and Operations28Hadoop Distributed File SystemUnstructured DataMapMapMapMapMapMapMapMapMapMapKey 1Key 1Key 1Key 1Key 1Key 1Key 1Key 2Key 1Key 1Key 1Key 3Key 1Key 1Key 1Key 4ReduceShuffle PhaseReduceReduceResult/OutputReduceMapMapMapMapMapMap and Reduce Traffic29Many-to-Many Traffic Patte

34、rnMap 1Map 2Map NMap 3Reducer 1Reducer 2Reducer 3Reducer NHDFSOutput ReplicationNameNodeJobTrackerZooKeeperSmall Flows/Messaging(Admin Related, Heart-beats, Keep-alive, delay sensitive application messaging)Small Medium Incast(Hadoop Shuffle)Large Incast(Hadoop Replication)Network CharacteristicsThe

35、 relative impact of various network characteristics on Hadoop clusters* Not a scaled or measured data 接入層設計每個機柜的電力消耗不能超過13A，在每個機柜柜頂部署一臺Nexus2248TP-E，服務器雙網卡分別連接本機柜柜頂交換機及相鄰機柜的柜頂交換機按照每個機柜13臺服務器，加上帶外管理接口，將占用39個端口Nexus 2248TP-ENexus 2248TP-E每個機柜13臺服務器，每臺服務器雙網卡上聯，也可以每3個機柜共用2臺Nexus 2248TP-ENexus 2248TP-E千兆

36、接入方案一雙掛方式，一個POD最多可支持24臺N2248TP-EVPCNexus 7000VPCNexus 2248TP-EvPC4*10GNexus 6001POD1810GPOD2810GVPCInternet單掛方式，一個POD最多可支持24臺N2248TP-E,PeerLink建議采用40G接口4*10G采用基于POD的設計思路，每個POD由2臺Nexus6001+20臺Nexus2248TP-E組成（最大支持24臺N2000，阿里為16臺），如果服務器多為單網卡接入，N2000可采用雙掛方式提供更高的冗余能力，如果服務器采用雙網卡聚合的方式接入，N2000可以采用單掛的方式，管理維護

37、更簡單接入層采用4個萬兆上行，可確保40個千兆接口線速轉發（通?？蛻粲貌坏?0個千兆）在POD內是全線速轉發的(一個POD可以支持200臺服務器（按每機柜10臺計算），絕大部分客戶的Hadoop集群規模不會超過200臺服務器）FEX雖然不具備本地轉發能力，但從FEX-N6K-FEX的延遲4us小型客戶，一個POD即可滿足使用要求，大型客戶，可以通過增加N7000作為核心交換機，基于POD進行橫向擴展POD上聯時可以考慮一定的收斂比，最大可支持16個10G捆綁或8個40G捆綁上聯方案優勢：POD內全線速；接入層大緩存；每POD只有2個管理節點；高可靠；便于橫向擴展POD內端到端延遲All tes

38、ting are done in a port-pair configurationAll FEXs are connected to a single N5548 parent switchSpirent Test Center 3.6 and NX-OS 5.0(3)N1(1)CT: Cut-Through, SF: Store-ForwardNexus5548Nexus2000Sprient TEST Center千兆接入方案二VPCNexus 7000VPCNexus 3048/Catalyst 3850410G410GVPCInternet2臺Nexus3048配置VPC可以支持服務

39、器雙網卡聚合也可以采用2臺Catalyst3850堆疊，同樣支持服務器雙網卡聚合Nexus3048更適合服務器單網卡上聯的應用場景，整個DC架構扁平，沒有POD限制，任意服務器之間均為全線速連接服務器雙網卡連接的場景210GCatalyst2960XRNexus3048萬兆接入方案一雙掛方式，一個POD最多可支持24臺N2248PQ-EVPCNexus 7700Nexus9500VPCNexus 2248PQvPC4*40GNexus 6004POD1840G/16*40GPOD2VPCInternet單掛方式，一個POD最多可支持12臺N2232PP,PeerLink建議采用40G接口8*1

40、0G440GNexus 6001采用基于POD的設計思路，每個POD由2臺Nexus6004+24臺Nexus2248PQ組成。也可采用2臺Nexus6001+12臺Nexus2232構成一個POD。如果服務器多為單網卡接入，N2000可采用雙掛方式提供更高的冗余能力，如果服務器采用雙網卡聚合的方式接入，N2000可以采用單掛的方式，管理維護更簡單強調N2248PQ的大緩存機制（32M共享緩存）FEX雖然不具備本地轉發能力，但從FEX-N6K-FEX的延遲1us小型客戶，一個POD即可滿足使用要求，大型客戶，可以通過增加N7000作為核心交換機，基于POD進行橫向擴展POD上聯時可以考慮一定的

41、收斂比，最大可支持16個40G捆綁上聯Nexus 2232采用N2248PQ還是N2232按照覆蓋36個機柜設計POD1包含24臺N2248PQ，每三個機柜共用2臺N2248PQ，可覆蓋36個機柜POD2包含12臺N2232，每個機柜柜頂部署1臺N2232，需要3個POD可覆蓋36個機柜方案一的收斂比為3:1，方案二的收斂比為4:1方案一的擴展性好于方案二方案一的布線成本高于方案二方案二有萬兆電口的選擇Nexus 2248PQvPC4*40GNexus 6004POD1POD28*10GNexus 6001Nexus 2232一個POD包含24臺N2248PQ，按照三個機柜共用2臺N2248P

42、Q設計，可覆蓋36個機柜一個POD包含12臺N2232，每個機柜柜頂部署1臺N2232，覆蓋36個機柜需要3個POD方案一：采用6004+N2248PQ方案二：采用6001+N2232萬兆接入方案二VPCNexus 7700Nexus9500VPCNexus 3064Nexus3172840G440GVPCInternetTOR交換機具備本地轉發能力整個DC架構扁平，沒有POD限制，任意服務器之間均為全線速連接服務器雙網卡連接的場景10GNexus 3064服務器單網卡連接的場景10GFeatureN3064N3172ArchitectureSwitch on ChipSwitch on Ch

43、ipRack Unit11Latency823nsec480nsecL2 ThroughputLine rateLine rateL3 ThroughputLine rateLine rate40GE support4610 GE support6472MAC Table128K288K (shared)IPv4 Unicast routes8K/16K (URPF disabled)16K(shared)IPv4 multicast routes8K32KACL support2K Ingress, 1K Egress4K Ingress, 1K EgressBuffer9 MB share

44、d 12MB shared 利用Spine-Leaf組網模型構建大型DCConfiguration Options:Layer 3 IP fabricLayer 3 all the way to accessLayer 2 extensibility can be provided by VXLANNexus 9500 can be EoRNexus 9300 can be ToR and MoRSpineLeafDC EdgeFabric Extenders2-Tier FabricRouting LeafsN9500/N7000 N9500/N7000 N9500/N7000 N9500/

45、N7000 N9300 N9300/N6000 N9300/N6000 N9300/N6000 N9300/N6000 N9300/N6000 N9300/N6000 N9500/N7000 N9500/N7000 提綱思科數據中心產品介紹典型組網方案成功案例6臺Nexus2248Nexus5548POD 146臺Nexus2248Nexus5548POD 1POD 內：Nexus5548 + TOR，用VPC+ 技術，消除STP，提供多路徑均衡負載。每臺N5K上行8個10GE，下行12個10GE，（考慮還有本地交互）每個POD6臺TOR，上行4個10GE，下行48個GE，提供 288臺服務器

46、的接入阿里巴巴Nexus7K * 2 VPCABTN支持4000臺服務器配置14個POD，共支持4000臺VPCOSPF ECMP(NG Fabric Ready)N6KN6KN6KN6KN6KN6KN6KN6KN6KN6KN6KN6KN2KN2KN2KN2KN2KN2KN2KN2KL3 GatwayLACPHost VPCDistri-SW(Spine)ACC-SW(Leaf)TOR(FEX)40GE面向NG Fabric的架構 (每個POD 768臺服務器 1.28T上行)TOR層: - Nexus 2248TP-E, 采用4個10GE上行.（將來2248PQ，4個40G上行，全萬兆接入）

47、POD SW層: 每臺N6K帶16個FEX, 上行16條40GE 鏈路做OSPF ECMP(將來NG Fabric）, 上下行收斂比1:1每個POD支持768 臺服務器雙網卡 A/A, 共組建8個POD, 支持 6144臺服務器Core SW層:提供4臺Nexus6004, 每臺至少需要64個40GE端口新一代Alibaba模塊內架構: N6k+N6k+N2kFEXCSWCSW騰訊數據中心網絡設計原則As LARGE as it can be massively scalableAs FLAT as it can be unified low-subscriptionAs stupid as

48、is can be easily copiable & fast constructable & operableGeneral Guideline (1G Server)500M East-West traffic anywhere, which is 2:1 oversubscription rate5M South-North traffic per Server which has internet access140ms for 3-way-handshake is the bottom line for apps0.25kpps of ARP per Server, more than 2K Server within single VLAN could be dangerous45% average utilization rate for WAN/MAN LinksDC3.5.2DC3.5vDC3.5.3服務器容量 5,000 5,000 5,0004000+ * 10GE收斂比2:12:1 3:1虛擬化率NY, 4:1NY, 10:1 節點數量 5,000