1、HACMP5.1 concept and planning2022/8/13 HACMP 5.X 概念和實施規劃Advanced Technical SupportIBM Technical Support Center2HACMP5.1 concept and planning2022/8/13 目錄什么是HACMP配置HACMP的規劃和考慮HACMP的安裝和配置HACMP的客戶化和管理3HACMP5.1 concept and planning2022/8/13 目錄什么是HACMP?配置HACMP的規劃和考慮HACMP的安裝和配置HACMP的客戶化和管理4HACMP5.1 concept

2、 and planning2022/8/13 HACMP(High Availability Cluster Multi Processing)為什么需要高可用性？越來越多的應用成為關鍵應用越來越多的業務提供7*24小時服務越來越多的客戶意識到自己是上帝- IBM 公司的解決方案: HACMP什么是HACMP？ 1) High Availability：系統可用性或運行時間最大化系統宕機時間最小化 2) multi-processing？一個cluster里的各個節點上可以運行多個應用共享數據或并發訪問數據.不是容錯機（ fault tolerance )5HACMP5.1 concept a

3、nd planning2022/8/13HACMP for AIX Supplies Clustering ComponentsCluster Manager (clstrmgrES)Event ScriptsLock Manager (cllockd) removed from HACMP 5.2SNMP Agent(clsmuxpd)clinfo d (HACMP 5.1 and later)6HACMP5.1 concept and planning2022/8/13 Cluster ManagerCluster Manager runs on all nodes in the clus

4、ter (clstrmgrES)- Brains of HACMP that finds symptoms, diagnoses problems, and applies remedies - Monitors and responds to changes in cluster state - Synchronizes and coordinates actions within the cluster- Executes Event Scripts in response to changes in cluster state7HACMP5.1 concept and planning2

5、022/8/13 Event ScriptsEvent Scripts determine action to be taken in response to cluster eventsHACMP includes event scripts that provide the behavior that is appropriate for most environmentsThe specific response to an event is easily customizedEvent scripts are written in the default shell and use s

6、tandard AIX commands and HACMP utilities8HACMP5.1 concept and planning2022/8/13 Cluster Lock Manager (removed from HACMP 5.2)In Concurrent Access environments, multiple nodes can simultaneously access the same dataApplications must be written using the Cluster Lock Manager API to register,request an

7、d release locksThe Lock Manager coordinates and serializes access to shared data to prevent data inconsistencies and corruption9HACMP5.1 concept and planning2022/8/13 clinfo & clsmuxpdclinfo is the Cluster Information daemonA clinfo API is provided that allows application programs to determine the c

8、urrent state of the cluster and to take appropriate action in response to changesclinfo also executes a whenever an event occurs within the cluster; this script may be customizedclinfo communications is SNMP-based, relying on a SNMP sub-agent(clsmuxpd) to receive updates on cluster statusThe clinfo

9、daemon is optional on both cluster nodes and client systems10HACMP5.1 concept and planning2022/8/13HACMP Detects and Response to the Following Types of FailuresNode Failures -Processor hardware or operationg system failures -One or more surviving nodes can acquire resourcesNetwork Adapter Failures M

10、ove IP address to standby network adapter in same nodeNetwork Failure -Message displayed on console and event is logged -As every sites network configurations are unique , no other default action is taken -Action to be taken in response to network failures is customizableApplication Failure - Applic

11、ation monitoring 11HACMP5.1 concept and planning2022/8/13What about other types of failures?Disk Drive Failures -LVM Mirroring -RAID Disk DevicesOther Hardware FailureApplication Failure(Customization needed , SRC)HACMP Failure -Promoted to node failurePower Failure -Avoid common power supplies acro

12、ss replicated devices -Use a UPS12HACMP5.1 concept and planning2022/8/13Clusterup to 32 server nodes working together,using TCP/IP for communication and each running the Cluster Manager daemon(program)NodeIndividual machine that is a member of a cluster and running the Cluster Manager daemon . This

13、is an arbitrary string of 30 characters (alphanumeric and underscore, but cannot start with a number).ResourcesEntities that are controlled by a node and are taken over by another node in the event of a node failure-Disks, Volume Groups , file systems-NFS exported file systems-IP Addresses-Applicati

14、on ProgramsThese entities are configured into Resource GroupsCritical ApplicationAn application that is started and stopped under the control of HACMP, For example , a database backend.Definitions13HACMP5.1 concept and planning2022/8/13EventSomething that happens in a cluster -a change in cluster st

15、ate: node down/up, network down/up, network adapter failure etc.BehaviorWhat you would like the cluster to do in response to an EVENT. This behavior is controlled by shell scriptsReintegration The process of bringing a previously failed node , network, or network adapter back into the cluster.Defini

16、tions(Cont.)14HACMP5.1 concept and planning2022/8/13 簡化了HACMP Cluster的配置與管理 通過磁盤傳輸心跳信號(Heartbeat over Disk) IPAT通過Alias實現(IPAT via Alias)(缺省方式) 用戶自定義的資源組(Custom RG) 快遞磁盤接管(Fast Disk Takeover) HACMP-XD HACMP V5.x的新特點ALL IN ONE:在HACMP v5之前，每個版本的HACMP軟件都有四個模塊，HAS，CRM，ES和ESCRM，用戶在定購的時候根據需要選擇其中一個模塊，從HACM

17、P v5開始IBM只提供一個模塊，包括以前四個模塊的所有功能。高可用性15HACMP5.1 concept and planning2022/8/13 目錄什么是HACMP?配置HACMP的規劃和考慮HACMP的安裝和配置HACMP的客戶化和管理16HACMP5.1 concept and planning2022/8/13配置HACMP的規劃和考慮Cluster / Node 規劃Recourse規劃IP網絡及地址（Service IP、Non-Service IP、Persistent IP）Heartbeat網絡（IP、串口、共享硬盤）Volume Group and File Syst

18、emApplication ServerRecourse Group規劃Software的規劃操作系統的版本和補丁要求HACMP的版本和補丁要求應用程序的兼容性17HACMP5.1 concept and planning2022/8/13Cluster/ Node規劃: Node配置要求 (1)一個精心設計的cluster應該盡量避免單點故障（SPOF),為了達到這個目的，典型的cluster一般對硬件有如下要求:至少2個 pseries 服務器作為cluster的節點（安裝并運行HACMP,每個節點的AIX 版本和HACMP的版本要完全一致)內置硬盤：預留一定的硬盤空間滿足HACMP的安裝

19、要求, MIRRORI/O插槽：要滿足網卡、HBA卡（陣列和帶庫要分開）、串口卡至少一種non-IP 網絡，通常采用RS232網絡，如果某些節點集成串口無法作心跳，要配#2943 8口異步卡，節點間要配串口線一個共享磁盤陣列（7133/FastT/ESS等)Cluster / Node18HACMP5.1 concept and planning2022/8/13 Node配置要求(2)：cluster性能考慮為了保證cluster的性能，一個cluster支持的最大組件數目如下：組件 最大數目node 32Resource Groups 64Networks 48Network interf

20、ace,devices, and labels 256Cluster / Node19HACMP5.1 concept and planning2022/8/13Resource: IP網絡規劃用戶的IP網絡類型要求Support: Ethernet, Token ring, FDDI , ATM Not support: SOCC,SLIP, FCS用戶的IP網絡資源規劃SwitchVLANIPAT 方式IP地址規劃Resource20HACMP5.1 concept and planning2022/8/13 IP網絡規劃：網絡連接示意圖test1test2test1_boot2test1

21、_boot1test2_boot1test2_boot2ResourceConceptHow HANetwork ConnectionTwo IPAT method (From version 4.5)21HACMP5.1 concept and planning2022/8/13Concept:Boot IP: 網卡的初始IP地址, AIX啟動后使用的地址Service IP: 對外服務的地址, 會從一塊網卡切換到另一塊網卡Standby IP: 不再需要.22HACMP5.1 concept and planning2022/8/13Non-service labels 在 AIX 啟動時

22、分配在 HACMP 啟動后 ,service IP label 作為一個alias 添加到一塊帶有non-service label 的network interface上non-service label 從來都不會從一個 NIC上移走如果節點失敗 , 一個 takeover 節點會獲得失敗節點的 service IP label ，并作為一個 alias 添加到它的位于同一個HACMP network 里的一塊帶有non-service label 的network interface上 IP網絡規劃： IPAT via IP aliases（一）Resource23HACMP5.1 co

23、ncept and planning2022/8/13設置 IPAT via IP aliases 功能#smitty hacmpExtended Configuration Extended Topology Configuration Configure HACMP Networks Change/Show a Network in the HACMP Cluster Change/Show an IP-Based Network in the HACMP Cluster Change/Show an IP-Based Network in the HACMP ClusterType or

24、 select values in entry fields.Press Enter AFTER making all desired changes.Entry Fields* Network Name net_ether_01 New Network Name * Network Type ether +* Netmask 255.255.255.0 +* Enable IP Address Takeover via IP Aliases Yes + IP Address Offset for Heartbeating over IP Aliases * Network Attribute

25、 public + IP網絡規劃： IPAT via IP aliases（二）Resource24HACMP5.1 concept and planning2022/8/13Non-service labels 在 AIX 啟動時分配HACMP 啟動后 ,service IP label 替換（replaces） non-service IP label 如果節點失敗 , 一個 takeover 節點使用它的位于同一個HACMP network里的一個 non-service interface 來獲得（acquire)失敗節點的service IP label IP網絡規劃： IPAT v

26、ia IP replacementResource25HACMP5.1 concept and planning2022/8/13Persistent Node IP label 是一個 IP alias ，它可以分配給cluster里的一個特定節點How config - extend configuration - 總是位于同一個節點可以位于一塊已經擁有 service 或 non-service IP label 的網卡上 不需在節點上安裝額外的物理網卡不屬于任何資源組能被用于對指定的節點進行管理每個節點可配置多個.在節點啟動后即可用，當HACMP服務停止后也始終保持可用 如果網卡失敗，

27、它只會遷移到相同網絡的同一個節點上的其它網卡如果節點失敗，該IP標識不會遷移到群集中的其它節點 IP網絡規劃： Persistent Node IP labelResource26HACMP5.1 concept and planning2022/8/13Heartbeat網絡規劃確定用戶的Non-IP網絡（心跳）串口方式IP網絡方式硬盤方式Resource27HACMP5.1 concept and planning2022/8/13HACMP5.1的新功能能夠使用下列任何一種共享磁盤陣列 (Fibre Channel, SCSI, 或 SSA)使用的磁盤是一個 enhanced concu

28、rrent volume group 的一部分, 唯一的要求是這個 VG必須在兩個節點都有定義磁盤的這塊特殊區域不能用作任何的 LVM 操作, 而只能用來在兩個節點間傳遞心跳消息 Heartbeat網絡規劃：磁盤心跳（Heartbeat via disk)Resource28HACMP5.1 concept and planning2022/8/13 Volume Group and File System規劃確定存儲設備的連接類型SCSISSAFC了解用戶需求劃分存儲資源劃分RAID創建邏輯盤確定VG的類型Non-concurrentConcurrentEnhanced concurrent

29、確定文件系統類型jfs2JfsResource29HACMP5.1 concept and planning2022/8/13使用RSCT 進行通信支持 ACTIVE 和 PASSIVE 模式ACTIVE Varyon: 類似通常的 varyonPASSIVE Varyon: 不允許 高級操作（high level operations ）HACMP 協調節點間的活動，保證只有一個節點可以 vary on volume group activelyenhanced concurrent volume groups 能夠被包含在一個 non-concurrent resource groups里

30、Enhanced concurrent volume groups in AIX 5.1 and 5.2 Volume Group and File System規劃： Enhanced concurrent Volume GroupResource30HACMP5.1 concept and planning2022/8/13通過提供快速資源組fallover，減少宕機時間對于2塊硬盤的VG，接管時間一般不超過10秒。對于帶有大數量磁盤的VG的資源組有顯著的性能提高。需要enhanced concurrent volume groups 定義在 non-concurrent resource

31、 groups里對于位于共享盤陣上enhanced concurrent volume groups，如果是共享資源組的資源， HACMP 會自動實現快速磁盤接管.Our suggestion. Volume Group and File System規劃： 快速磁盤接管（fast disk takeover)（一）Resource31HACMP5.1 concept and planning2022/8/13你可以查看vg的狀態通過 lsvg 命令：VG STATE 無論是actively 或者passively方式的varyon，都將顯示active狀態。VG PERMISSION 如果是

32、actively方式的varyon ，將顯示read/write，如果是passive方式的varyon ，將顯示passive-only。CONCURRENT 將顯示Capable或者Enhanced-Capable (對于concurrent vg). Volume Group and File System規劃： 快速磁盤接管（fast disk takeover)（二）Resource32HACMP5.1 concept and planning2022/8/13Resource Group規劃 一些可管理的資源集合在一起，和特定的應用相關聯，并可一同執行 failover等行為的組.

33、 包含四種類型：Cascading resource groupRotating resource groupConcurrent resource groupCustom resource group（HA 5.X新增的資源組類型） Resource Group33HACMP5.1 concept and planning2022/8/13Cascading Resource Group:缺省，資源會在優先級最高的節點上啟動低優先級的節點作為standby節點，當高優先級節點fail的時候接管其資源當發生失敗節點重新加入（reintegration）時，資源會返回到優先級更高的節點每個節點都

34、可以配置自己的資源成為 multi-takeover的方式資源（Resources）: Service IP Labels/Addresses Volume Groups Filesystems Application Servers 優先級Resource Group34HACMP5.1 concept and planning2022/8/13Cascading resource group的接管過程Resource Group35HACMP5.1 concept and planning2022/8/13Cascading resource group的屬性Cascading witho

35、ut fallback Inactive takeover Dynamic node priorityCl_highest_free_mem select the node with the highest percentage of free memoryCl_highest_idle_cpu select the node with the most avialable processor timeCl_lowest_disk_busy select the disk that is least busy Cascading 資源組提供下面三種屬性，這些屬性需要在hacmp的Extende

36、d configuration菜單中設置。Resource Group36HACMP5.1 concept and planning2022/8/13Rotating resource group:第一個加入到資源組的節點會獲得資源 當節點失敗 ,第一個備(standby)節點會獲得失敗節點的資源組 一個失敗節點重新加入cluster，將會作為備(standby)節點，不會重新獲得原來的資源組 資源（Resources）: Service IP Labels/Addresses Volume Groups Filesystems Application Servers Resource Gro

37、up37HACMP5.1 concept and planning2022/8/13Rotating resource group的接管過程Resource Group38HACMP5.1 concept and planning2022/8/13Concurrent resource group:所有的節點共享且可并發訪問數據應用在所有節點上并行運行有Lock Manager 協調各節點對數據的訪問一個節點失敗，會降低cluster 性能，但不會發生接管。資源（Resources）: Concurrent Volume Groups Application Servers 共享盤陣Resou

38、rce Group39HACMP5.1 concept and planning2022/8/13Custom resource groupHACMP5.1 增加的新類型 HACMP 5.2 CUSTOM ONLY通過參數設置 ,可以精確定義有關資源組的 startup,failover 和fallback行為的策略資源（Resources）: Service IP Labels/Addresses Volume Groups Use forced varyon of volume groups, if necessary false Filesystems Application Serve

39、rs Resource Group40HACMP5.1 concept and planning2022/8/13Custom resource group 添加一個custom resource group :#smitty hacmp Initialization and Standard Configuration Configure HACMP Resource Groups Add a Resource Group Cascading Rotating Concurrent Custom Add a Custom Resource GroupType or select values

40、 in entry fields.Press Enter AFTER making all desired changes. Entry Fields* Resource Group Name * Participating Node Names (Default Node Priority) + Startup Policy Online On Home Node O + Fallover Policy Fallover To Next Prio + Fallback Policy Fallback To Higher Pr +Resource Group41HACMP5.1 concept

41、 and planning2022/8/13Startup可設置的策略有： Online on Home Node Only. The resource group should be brought online only on its home (highest priority) node during the resource group startup. This requires the highest priority node to be available. Online on First Available Node. The resource group activate

42、s on the first participating node that es available. Online on All Available Nodes. The resource group is brought online on all nodes. Online Using Distribution Policy. Only one resource group is brought online on a node, or on a node per network, depending on the distribution policy specified (node

43、 or network).Fallover可設置的策略有 Fallover to Next Priority Node in the List. In the case of fallover, the resource group that is online on only one node at a time follows the default node priority order specified in the resource groups nodelist. Fallover Using Dynamic Node Priority. Before selecting thi

44、s option, configure a dynamic node priority policy that you want to use. Or you can select one of the three predefined dynamic node priority policies. Bring Offline (on Error Node Only). Select this option to bring a resource group offline on a node during an error conditionFallback可設置的策略有 Fallback

45、to Higher Priority Node in the List. A resource group falls back when a higher priority node joins the cluster. If you select this option, you can use the delayed fallback timer. If you do not configure a delayed fallback policy, the resource group falls back immediately when a higher priority node

46、joins the cluster. Never Fallback. A resource group does not fall back when a higher priority node joins the clusterCustom resource group Resource Group42HACMP5.1 concept and planning2022/8/13在HACMP5.2 以前資源組的定義在HACMP5.2中資源組的定義 Cascading resource groupInactive Takeover = FalseCascading without Fallba

47、ck (CWOF) = FalseStartup: Online on Home Node OnlyFallover: Fallover to Next Priority Node in the ListFallback: Fallback to Higher Priority Node in the ListCascading resource groupInactive Takeover = TrueCWOF = FalseStartup: Online on First Available Node Fallover: Fallover to Next Priority Node in

48、the ListFallback: Fallback to Higher Priority Node in the ListCascading resource groupInactive Takeover = FalseCWOF = TrueStartup: Online on Home Node OnlyFallover: Fallover to Next Priority Node in the ListFallback: Never FallbackCascading resource groupInactive Takeover = TrueCWOF = TrueStartup: O

49、nline on Fisrt Available NodeFallover: Fallover to Next Priority Node in the ListFallback: Never FallbackRotating resource groupStartup: Online Using Distribution PolicyResource Group Distribution Policy: Network distributionFallover: Next Priority Node in the ListFallback: Never FallbackConcurrent

50、resource groupStartup: Online on All Available NodesFallover: Bring Offline (On Error Node Only)Fallback: Never FallbackCustom resource group Resource Group43HACMP5.1 concept and planning2022/8/13配置要求操作系統上需要打的一些額外的包： bos.adt.lib bos.adt.libm bos.adt.syscalls .tcp.client .tcp.server bos.rte.SRC bos.r

51、te.libc bos.rte.libcfg bos.rte.libcur bos.rte.libpthreads bos.rte.odm bos.data 如果需要安裝并行的資源組，需要打下面的包： bos.rte.lvm. 5.1.0.25 or higher bos.clvm.enh.Software44HACMP5.1 concept and planning2022/8/13 目錄什么是HACMP?配置HACMP的規劃和考慮HACMP的安裝和配置HACMP的客戶化和管理45HACMP5.1 concept and planning2022/8/13需要安裝的組件 操作系統的補丁 HA

52、CMP軟件 HACMP軟件的補丁軟件的安裝方法 NIM 光盤安裝 本地硬盤安裝驗證安裝 HACMP軟件的安裝46HACMP5.1 concept and planning2022/8/13HACMP軟件的配置過程 HACMP配置前的準本工作 配置IP地址 編輯/etc/hosts文件 編輯/usr/es/sbin/cluster/etc/rhosts 文件 編寫應用程序的啟動/停止腳本 創建共享的vg和文件系統 配置串口設備HACMP的Standard配置過程 添加Cluster和節點 配置Cluster資源 創建并配置Cluster資源組 同步HACMP的配置HACMP的Extended配置

53、過程 添加心跳 定制Cluster資源 47HACMP5.1 concept and planning2022/8/13HACMP軟件配置方案拓撲圖test1test2test1_boot2test1_boot1test2_boot1test2_boot248HACMP5.1 concept and planning2022/8/13HACMP配置前的準備工作：配置IP地址test1:/netstat -inName Mtu Network Address Ipkts Ierrs Opkts Oerrs Collen0 1500 link#2 0.4.ac.49.f2.d5 77960 0 4

54、7805 0 0en0 1500 100.1 100.1.0.1 77960 0 47805 0 0en1 1500 link#3 0.6.29.ec.44.d6 33 0 11 0 0en1 1500 192.168.0 192.168.0.1 33 0 11 0 0test2:/netstat -inName Mtu Network Address Ipkts Ierrs Opkts Oerrs Collen0 1500 link#2 0.4.ac.49.60.23 31138 0 82582 0 0en0 1500 100.1 100.1.0.2 31138 0 82582 0 0en1

55、 1500 link#3 0.4.ac.3e.b9.4b 36 0 13 0 0en1 1500 192.168.0 192.168.0.2 36 0 13 0 049HACMP5.1 concept and planning2022/8/13HACMP配置前的準備工作：編輯配置文件編輯/etc/hosts文件100.1.0.2 test2_boot1 test2100.1.0.1 test1_boot1 test1192.168.0.1 test1_boot2192.168.0.2 test2_boot210.1.0.1 test1_svc10.1.0.2 test2_svc10.1.0.5

56、 test1_per10.1.0.6 test1_per編輯/usr/es/sbin/cluster/etc/rhosts 文件100.1.0.2100.1.0.1 192.168.0.1 192.168.0.2 50HACMP5.1 concept and planning2022/8/13HACMP配置前的準備工作：編寫腳本查看節點上的應用服務器的啟動/停止腳本test2:/ha51ls l -rwxr-xr-x 1 root system 65 Apr 13 13:51 start-rw-r-r- 1 root system 31 Apr 13 11:49 start.log-rwxr-

57、xr-x 1 root system 66 Apr 13 14:01 start1-rw-r-r- 1 root system 31 Apr 13 14:01 start1.log-rwxrwxrwx 1 root system 64 Apr 13 11:48 stop-rw-r-r- 1 root system 31 Apr 13 11:48 stop.log-rwxr-xr-x 1 root system 66 Apr 13 14:01 stop1-rw-r-r- 1 root system 31 Apr 13 14:01 stop1.log啟動腳本舉例：vi start date /ha

58、51/start.logbanner start app1 /tmp/hacmp.out51HACMP5.1 concept and planning2022/8/13HACMP配置前的準備工作：創建共享的VG和FStest1vg是共享VG：test1 : hdisk0 b07574 rootvg activehdisk3 000438325e22bca7 test1vghdisk4 e5aa8 Nonehdisk5 000438323d0e4487 Nonetest2 : hdisk0 000d29574085126d rootvg activehdisk5 000438325e22bca7

59、 test1vghdisk6 e5aa8 Nonehdisk7 000438323d0e4487 None52HACMP5.1 concept and planning2022/8/13HACMP配置前的準備工作：創建串口設備通過smitty tty命令來添加串口設備：53HACMP5.1 concept and planning2022/8/13HACMP的Standard配置過程：添加CLUSTER和節點 * Cluster Name ha51tsc New Nodes (via selected communication paths) test2_boot1 test1_boot1 C

60、urrently Configured Node(s)通過smitty hacmp命令，按照下列路徑來添加Cluster和節點：- Initialization and Standard Configuration- Add Nodes to an HACMP Cluster。 注：這一步很重要，一般我們都是把每個節點的boot1 作為communication path . 當回車以后，系統會自己discover hacmp 的資源，并有詳細的信息顯示輸出。54HACMP5.1 concept and planning2022/8/13HACMP的Standard配置過程： 添加高可用資源S