1、PAGE HACMP Oracle Rac安裝設置手冊版本 v1.0二零一二年十月北京華勝天成科技股份有限公司文檔操縱更改記錄版本創建/修改時刻編制/修改者 文件/修改內容審批人V1.02012.10批閱姓名職位公布姓名職位目錄 TOC o h z u HYPERLINK l _Toc225161693 目錄 PAGEREF _Toc225161693 h 3 HYPERLINK l _Toc225161694 第1章 HACMP v5.4安裝配置 PAGEREF _Toc225161694 h 4 HYPERLINK l _Toc225161695 1.1、 IP規劃 PAGEREF _To

2、c225161695 h 4 HYPERLINK l _Toc225161696 1.2、 安裝HACMP軟件 PAGEREF _Toc225161696 h 4 HYPERLINK l _Toc225161697 1.2.1、 操作系統版本 PAGEREF _Toc225161697 h 4 HYPERLINK l _Toc225161698 1.2.2、 安裝依靠的軟件包 PAGEREF _Toc225161698 h 5 HYPERLINK l _Toc225161699 1.2.3、 安裝HACMP v5.4 PAGEREF _Toc225161699 h 6 HYPERLINK l

3、_Toc225161700 1.3、 HACMP配置規劃 PAGEREF _Toc225161700 h 8 HYPERLINK l _Toc225161701 1.3.1、 配置IP和網絡 PAGEREF _Toc225161701 h 8 HYPERLINK l _Toc225161702 1.3.2、 編輯/etc/hosts文件 PAGEREF _Toc225161702 h 9 HYPERLINK l _Toc225161703 1.3.3、 創建共享卷組和文件系統 PAGEREF _Toc225161703 h 9 HYPERLINK l _Toc225161704 1.4、 HA

4、CMP Standard配置 PAGEREF _Toc225161704 h 10 HYPERLINK l _Toc225161705 1.4.1、 添加 cluster和節點 PAGEREF _Toc225161705 h 11 HYPERLINK l _Toc225161706 1.4.2、 創建并配置資源組 PAGEREF _Toc225161706 h 11 HYPERLINK l _Toc225161707 1.4.3、 同步HACMP的配置 PAGEREF _Toc225161707 h 13 HYPERLINK l _Toc225161708 1.5、 HACMP啟動 PAGER

5、EF _Toc225161708 h 13 HYPERLINK l _Toc225161709 1.5.1、 啟動HACMP PAGEREF _Toc225161709 h 13 HYPERLINK l _Toc225161710 第2章 Oracle RAC安裝配置 PAGEREF _Toc225161710 h 13 HYPERLINK l _Toc225161711 2.1、 規劃 PAGEREF _Toc225161711 h 13 HYPERLINK l _Toc225161712 2.2、 安裝檢查 PAGEREF _Toc225161712 h 15 HYPERLINK l _T

6、oc225161713 2.3、 安裝前配置 PAGEREF _Toc225161713 h 17 HYPERLINK l _Toc225161714 2.4、 安裝CRS PAGEREF _Toc225161714 h 20 HYPERLINK l _Toc225161715 2.5、 安裝數據庫 PAGEREF _Toc225161715 h 41 HYPERLINK l _Toc225161716 2.6、 創建數據庫 PAGEREF _Toc225161716 h 47 HYPERLINK l _Toc225161717 2.7、 數據測試 PAGEREF _Toc225161717

7、h 59HACMP v5.4安裝配置IP規劃NodeInterface NameTypeIP AddressRegistered Inibm03ibm03Public8Hosts fileibm03ibm03-vipVirtual10Hosts fileibm03ibm03-privPrivate8Hosts fileibm04ibm04Public9Hosts fileibm04ibm04-vipVirtual11Hosts fileibm04ibm04-privPrivate9Hosts file安裝HACMP軟件完成HACMP安裝配置規劃后，能夠開始安裝HACMP軟件，安裝版本為HACM

8、P v5.4。操作系統版本# oslevel -r5300-06安裝依靠的軟件包檢查系統的軟件安裝包，請參照IBM紅皮書中High Availability Cluster Multi-Processing for AIX 5L Installation Guide。那個地點我們建議安裝軟件束APP-DEV和SERVER，其中包括的紅皮書要求的大部分軟件包。安裝rsct軟件包：pat.clientsrsct.opt.storagerm（可選）安裝其它軟件包：bos.clvm（使用并行資源組時需要，RAC下必須安裝此包）bos.data#smitty install_latestInstall

9、SoftwareType or select values in entry fields.Press Enter AFTER making all desired changes.Entry FieldsINPUT device / directory for software /dev/cd0SOFTWARE to install _all_latestPREVIEW only? (install operation will NOT occur) noCOMMIT software updates? yesSAVE replaced files? noAUTOMATICALLY inst

10、all requisite software? yesEXTEND file systems if space needed? yesOVERWRITE same or newer versions? noVERIFY install and check file sizes? noInclude corresponding LANGUAGE filesets? yesDETAILED output? noProcess multiple volumes? yesACCEPT new license agreements? yesPREVIEW new LICENSE agreements?

11、no在“SOFTWARE to install”一欄中選擇要安裝的軟件包，“ACCEPT new license agreements”中同意許可協議，即完成安裝軟件包。或參考以下腳本命令：#安裝軟件包/usr/lib/instl/sm_inst installp_cmd -a -Q -d /dev/cd0 -f bos.clvm ALL I:bos.clvm _all_filesets,bos.data ALL I:bos.data _all_filesets,rsct.basic ALL I:rsct.basic _all_filesets,pat.basic ALL I:pat.basi

12、c _all_filesets,pat.clients ALL I:pat.clients _all_filesets,rsct.opt.storagerm ALL I:rsct.opt.storagerm _all_filesets -c -N -g -X -G -Y #安裝HACMP v5.4除了haview ，netwiew ( Tivoli),的 包以外，所有的HACMP的軟件包都要安裝，rsct.opt.storagerm為可選安裝包，關于差不多包在安裝光盤或Expansion光盤上能夠查找到。#smitty install_latestInstall SoftwareType or

13、 select values in entry fields.Press Enter AFTER making all desired changes.Entry FieldsINPUT device / directory for software /dev/cd0SOFTWARE to install _all_latestPREVIEW only? (install operation will NOT occur) noCOMMIT software updates? yesSAVE replaced files? noAUTOMATICALLY install requisite s

14、oftware? yesEXTEND file systems if space needed? yesOVERWRITE same or newer versions? noVERIFY install and check file sizes? noInclude corresponding LANGUAGE filesets? yesDETAILED output? noProcess multiple volumes? yesACCEPT new license agreements? yesPREVIEW new LICENSE agreements? no在“SOFTWARE to

15、 install”一欄中選擇要安裝的軟件包，“ACCEPT new license agreements”中同意許可協議，即完成安裝軟件包，安裝過程大約在1030分鐘完成。或參考以下腳本命令：#安裝HACMP/usr/lib/instl/sm_inst installp_cmd -a -Q -d /dev/cd0 -f cluster.adt.es ALL I:cluster.adt.es _all_filesets,cluster.doc.en_US.es ALL I:cluster.doc.en_US.es _all_filesets,cluster.es ALL I:cluster.es

16、 _all_filesets,cluster.es.cfs ALL I:cluster.es.cfs _all_filesets,cluster.es.cspoc ALL I:cluster.es.cspoc _all_filesets,cluster.es.nfs ALL I:cluster.es.nfs _all_filesets,cluster.es.plugins ALL I:cluster.es.plugins _all_filesets,cluster.es.worksheets ALL I:cluster.es.worksheets _all_filesets,cluster.l

17、icense ALL I:cluster.license _all_filesets,cluster.man.en_US.es ALL I:cluster.man.en_US.es _all_filesets -c -N -g -X -G -Y系統提示成功后，完成HACMP補丁升級，將HACMP補丁上傳至系統目錄下，我們假設上傳目錄為/tmp/hav54，本地目錄為d:hav54，上傳命令如下：Windows：“開始”-“運行”，輸入“cmd”，依照以下提示完成補丁上傳：Microsoft Windows XP 版本 5.1.2600(C) 版權所有 1985-2001 Microsoft C

18、orp.C:Documents and SettingsAdministratorftp remote-host-ip-address輸入用戶名和密碼（此處需要給用戶設置密碼，假如為空，AIX拒絕登錄），成功登錄后：ftpcd /tmp/hav54ftplcd d:hav54ftppromptftpbinftpmput *.bff上傳完成后退出ftp程序ftpquit升級HACMP v5.4補丁#smitty update_all Update Installed Software to Latest Level (Update All)Type or select values in ent

19、ry fields.Press Enter AFTER making all desired changes.Entry FieldsINPUT device / directory for software /tmp/hav54SOFTWARE to update _update_allPREVIEW only? (update operation will NOT occur) noCOMMIT software updates? yesSAVE replaced files? noAUTOMATICALLY install requisite software? yesEXTEND fi

20、le systems if space needed? yesVERIFY install and check file sizes? noDETAILED output? noProcess multiple volumes? yesACCEPT new license agreements? yesPREVIEW new LICENSE agreements? no回車確認以上命令，即可完成升級工作。注意：假如安裝程序提示“沒有找到相關的fileset”等類似的提示，可在/tmp/hav54目錄下執行命令“#inutoc .”，重新生成軟件包安裝列表。重新啟動AIX安裝完成HACMP后，為

21、幸免各種問題，要求將AIX系統重新啟動，至此我們完成了所有的安裝工作，能夠開始配置HACMP雙機集群了。#shutdown FrHACMP配置規劃配置IP和網絡AIX網卡IP地址配置：#smiitty mktcpip Available Network InterfacesMove cursor to desired item and press Enter.en0 08-08 Standard Ethernet Network Interfaceen1 08-09 Standard Ethernet Network Interface en2 0A-08 Standard Ethernet

22、Network Interface en3 0J-08 Standard Ethernet Network Interface選擇想要配置IP地址的網卡，選擇“en0”，回車確認，進入下一步配置：Minimum Configuration & StartupTo Delete existing configuration data, please use Further Configuration menusType or select values in entry fields.Press Enter AFTER making all desired changes.Entry Field

23、sHOSTNAME ora01Internet ADDRESS (dotted decimal) 2Network MASK (dotted decimal) Network INTERFACE en0NAMESERVERInternet ADDRESS (dotted decimal) DOMAIN Name Default GatewayAddress (dotted decimal or symbolic name) Cost 0Do Active Dead Gateway Detection? noYour CABLE Type N/ASTART TCP/IP daemons Now

24、no使用同樣的方法配置完成網卡的IP地址。編輯/etc/hosts文件關于兩個節點ibm03和ibm04要配置相同的/etc/hosts文件，使用vi配置hosts文件，建議首先對原有的hosts文件進行備份，配置如下：#cp /etc/hosts /etc/hosts.bak#vi /etc/hosts8 ibm039 ibm048 ibm03-priv9 ibm04-priv10 ibm03-vip11 ibm04-vip創建共享卷組和文件系統創建卷組#smitty _mkbvgAdd a Big Volume GroupType or select values in entry fie

25、lds.Press Enter AFTER making all desired changes.Entry FieldsVOLUME GROUP name vgdataPhysical partition SIZE in megabytesPHYSICAL VOLUME names hdisk2FORCE the creation of volume group? yesActivate volume group AUTOMATICALLY noat system restart?Volume group MAJOR NUMBER 100Create VG Concurrent Capabl

26、e? no我們在建立HACMP共享卷組時，要注意的參數 “Activate volume group AUTOMATICALLY at system restart? ”該項，設置為“no”，是使用HACMP的方式將卷組varyon，而不是系統啟動時由操作系統varyon。Major NUMBER是指該設備（orvg）的主設備號，此處設置為系統唯一即可，可通過命令查詢可用的major number：# lvlstmajor43.59,62.或可通過ls l /dev/命令查看當前所有設備的主設備號，第五列即是。創建邏輯卷和文件系統在ibm03節點上按照以下操作：#varyoffvg vgdat

27、a需將vgdata varyoff下來，在ora02節點上才能正確的導入。在ibm04節點上按照一下操作：#lspv hdisk0 00c2ef1fa38df28a rootvg activehdisk1 00c2ef1f38b81fac rootvg activehdisk2 00c2ecef3846c20e None 查看hdisk2是否為物理卷，否則需要使用命令將其屬性更改：#chdev l hdisk2 a pv=yes導入卷組和文件系統的定義#smitty importvgImport a Volume GroupType or select values in entry fiel

28、ds.Press Enter AFTER making all desired changes.Entry FieldsVOLUME GROUP name vgdata* PHYSICAL VOLUME name hdisk2Volume group MAJOR NUMBER 100#varyonvg vgdata#varyoffvg vgdataHACMP Standard配置HACMP配置可采納Standard和Extended兩種方式，其中Standard配置能夠配置Cluter名稱，節點，差不多網絡，應用服務器，各種資源和資源組等差不多的內容，假如要配置永久IP，串口心跳，對資源組進行

29、詳細的配置等工作則需要在Extended中完成，能夠講Standard菜單是Extended菜單的一個子集，能夠在Extended菜單中完成HACMP的所有配置工作。下面我們以Standard菜單完成HACMP的差不多配置工作：添加cluster和節點配置cluster資源創建并配置資源組同步HACMP的配置HACMP配置菜單如下：#smitty hacmp HACMP for AIXMove cursor to desired item and press Enter. Initialization and Standard Configuration Extended Configurat

30、ion System Management (C-SPOC) Problem Determination Tools選擇“Initialization and Standard Configuration”，如下菜單：Initialization and Standard ConfigurationMove cursor to desired item and press Enter.Configuration AssistantsConfigure an HACMP Cluster and NodesConfigure Resources to Make Highly AvailableCo

31、nfigure HACMP Resource GroupsVerify and Synchronize HACMP ConfigurationDisplay HACMP ConfigurationHACMP Cluster Test Tool添加 cluster和節點#smitty hacmp-Initialization and Standard ConfigurationConfigure an HACMP Cluster and NodesConfigure Nodes to an HACMP Cluster (standard)Type or select values in entr

32、y fields.Press Enter AFTER making all desired changes.Entry Fields* Cluster Name ora_clusterNew Nodes (via selected communication paths) ibm03 ibm04Currently Configured Node(s)創建并配置資源組#smitty hacmp- Initialization and Standard ConfigurationConfigure HACMP Resource Groups Configure HACMP Resource Gro

33、upsMove cursor to desired item and press Enter. Add a Resource Group Change/Show a Resource Group Remove a Resource Group Change/Show Resources for a Resource Group (standard)Add a Resource GroupType or select values in entry fields.Press Enter AFTER making all desired changes.Entry Fields* Resource

34、 Group Name ora_rg* Participating Nodes (Default Node Priority) ibm03 ibm04Startup Policy Online On All AvailabFallover Policy Bring Offline (On ErrFallback Policy Never Fallback添加資源至資源組中#smitty hacmp- Initialization and Standard ConfigurationConfigure HACMP Resource Groups Configure HACMP Resource

35、GroupsMove cursor to desired item and press Enter. Add a Resource Group Change/Show a Resource Group Remove a Resource Group Change/Show Resources for a Resource Group (standard)Change/Show All Resources and Attributes for a Resource GroupType or select values in entry fields.Press Enter AFTER makin

36、g all desired changes.Entry FieldsResource Group Name ora_rgParticipating Nodes (Default Node Priority) ibm03 ibm04Startup Policy Online On Home Node OnlyFallover Policy Fallover To Next Priority Node In The ListFallback Policy Fallback To Higher Priority Node In The ListConcurrent Volume Groups vgd

37、ata +Use forced varyon of volume groups, if necessary false +Automatically Import Volume Groups false +Application Servers +Tape Resources + 至此，我們完成了資源和資源組的關聯。同步HACMP的配置#smitty hacmp- Initialization and Standard ConfigurationVerify and Synchronize HACMP Configuration回車確認該操作，如此HACMP的配置差不多同步到兩個節點中了，也確

38、實是ibm03和ibm04擁有同樣的配置了。HACMP啟動啟動HACMP在節點ibm03上執行smitty clstart，回車確認即可啟動HACMP* Start now, on system restart or both now + Start Cluster Services on these nodes ibm03 + BROADCAST message at startup? true + Startup Cluster Information Daemon? false + Reacquire resources after forced down ? false +Oracle

39、 RAC安裝配置規劃數據庫配置規劃ORACLE_BASE/dataCRS_HOME/data/product /10.2.0/crsORACLE_HOME /data/product /10.2.0/dbDB_NAMEorclORACLE_SIDorcl1，orcl2TNS_ADMIN$ORACLE_HOME/network/adminOracle治理帳戶的口令 sys Systemoracleoracle數據庫存放位置裸設備是否使用歸檔方式運行數據庫Yes備份方式講明RMAN/TSM裸設備規劃設備名稱數據庫設施存儲空間/dev/rorcl_ocrOcr信息存放500Mb/dev/rorcl_

40、vdiskvdisk信息存放500Mb/dev/rorcl_spfilespfile信息存放100Mb/dev/rorcl_srvconfigsrvconfig信息存放100Mb/dev/rorcl_systemSystem表空間10G/dev/rorcl_sysauxSysaux表空間10G/dev/rorcl_temp1系統臨時表空間20G/dev/rorcl_temp2系統臨時表空間20G/dev/rorcl_undo1撤銷表空間20G/dev/rorcl_undo2撤銷表空間20G/dev/rorcl_undo3撤銷表空間20G/dev/rorcl_undo4撤銷表空間20G/dev/

41、rorcl_usersUser表空間10G/dev/rorcl_toolsTools表空間10G/dev/rorcl_ctl1操縱文件1G/dev/rorcl_ctl2操縱文件1G/dev/rorcl_ctl3操縱文件1G/dev/rorcl_log1a日志文件500Mb/dev/rorcl_log1b日志文件500Mb/dev/rorcl_log2a日志文件500Mb/dev/rorcl_log2b日志文件500Mb/dev/rorcl_log3a日志文件500Mb/dev/rorcl_log3b日志文件500Mb/dev/rorcl_log4a日志文件500Mb/dev/rorcl_log

42、4b日志文件500Mb/dev/rorcl_data1用戶數據表空間20G/dev/rorcl_data2用戶數據表空間20G/dev/rorcl_data3用戶數據表空間20G/dev/rorcl_data4用戶數據表空間20G安裝檢查操作系統版本：# oslevel -r5300-06物理內存：# lsattr -El mem0goodsize 7744 Amount of usable physical memory in Mbytes Falsesize 7744 Total amount of physical memory in Mbytes False交換區：# lsps -aP

43、age Space Physical Volume Volume Group Size %Used Active Auto Typehd6 hdisk0 rootvg 512MB 3 yes yes lv系統空間：# df -gFilesystem GB blocks Free %Used Iused %Iused Mounted on/dev/hd4 0.50 0.44 13% 4788 5% /dev/hd2 5.00 3.10 38% 41240 6% /usr/dev/hd9var 1.00 0.95 6% 747 1% /var/dev/hd3 1.00 0.65 36% 1082

44、1% /tmp/dev/fwdump 0.38 0.37 1% 4 1% /var/adm/ras/platform/dev/hd1 0.50 0.50 1% 71 1% /home/proc - - - - - /proc/dev/hd10opt 2.00 1.86 7% 3461 1% /opt/dev/fslv00 15.00 4.42 71% 29243 3% /data/dev/lv00 0.12 0.12 4% 18 1% /var/adm/csd文件集：# lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.libper

45、fstat bos.perf.perfstat ctools rsct.basic.rte pat.clients.rte xlC.aix50.rte xlC.rte Fileset Level Path: /usr/lib/objrepos bos.adt.base COMMITTED Base Application Development Toolkit bos.adt.lib COMMITTED Base Application Development Libraries bos.adt.libm COMMITTED Base Application Development Math

46、Library bos.perf.libperfstat COMMITTED Performance Statistics Library Interface bos.perf.perfstat COMMITTED Performance Statistics Interface ctools COMMITTED Proc Filesystem Tools rsct.basic.rte COMMITTED RSCT Basic Function pat.clients.rte COMMITTED RSCT Event Management Client Function xlC.aix50.r

47、te COMMITTED XL C/C+ Runtime for AIX 5.2 xlC.rte COMMITTED XL C/C+ Runtime Path: /etc/objrepos bos.perf.libperfstat COMMITTED Performance Statistics Library Interface bos.perf.perfstat COMMITTED Performance Statistics Interface rsct.basic.rte COMMITTED RSCT Basic Function安裝前配置為確保oracle rac安裝成功，系統治理員

48、root用戶需創建相應的系統用戶、系統組，以便安裝治理oracle軟件。Root用戶還要修改相關系統參數，打破默認值對實際應用的限制。調試rac節點之間的高級功能：rcp/rsh/rlogin，以確定兩節點之間能夠通信正常，能夠在rac環境下進行相關同步操作和心跳監控。創建dba組，id為500# smitty mkgroup創建oracle用戶，id為601# smitty mkuser修改oracle用戶密碼# passwd oracle在oracle用戶下創建.rhosts文件，內容為“+”。# su oracle$ vi .rhosts+修改oracle用戶限制# smitty chu

49、serSoft FILE size -1Soft CPU time -1Soft DATA segment -1Soft STACK size -1Soft CORE file size -1修改系統配置參數，增加系統進程數上限# smitty chgsysMaximum number of PROCESSES allowed per user 2048修改網絡參數no -p -o udp_sendspace=65536no -p -o udp_recvspace=65536no -p -o tcp_sendspace=65536no -p -o tcp_recvspace=65536no -

50、p -o rfc1323=1no -p -o sb_max=1310720no -r -o ipqmaxlen=512創建裸設備mklv -y rac_system_raw_500m -T O -w n -s n -r n vgdata 8依次創建以下裸設備：/dev/rorcl_ocr/dev/rorcl_vdisk/dev/rorcl_spfile/dev/rorcl_srvconfig/dev/rorcl_system/dev/rorcl_sysaux/dev/rorcl_temp1/dev/rorcl_temp2/dev/rorcl_undo1/dev/rorcl_undo2/dev/

51、rorcl_undo3/dev/rorcl_undo4/dev/rorcl_users/dev/rorcl_tools/dev/rorcl_ctl1/dev/rorcl_ctl2/dev/rorcl_ctl3/dev/rorcl_log1a/dev/rorcl_log1b/dev/rorcl_log2a/dev/rorcl_log2b/dev/rorcl_log3a/dev/rorcl_log3b/dev/rorcl_log4a/dev/rorcl_log4b/dev/rorcl_data1/dev/rorcl_data2/dev/rorcl_data3/dev/rorcl_data4更改裸設

52、備權限# cd /dev# chown oracle:dba *orcl*修改oracle用戶環境變量# vi /home/oracle/.profileumask 022export ORACLE_BASE=/dataexport ORACLE_HOME=$ORACLE_BASE/product/10.2.0/dbexport ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crsexport LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/lib32export ORACLE_SID=orclexport PAT

53、H=$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:$PATH安裝CRS在ibm03節點上進入CRS安裝盤，執行runInstaller啟動CRS的安裝；按提示在每個安裝節點上以root用戶執行rootpre.sh，該腳本在CRS安裝盤上的rootpre目錄下；# ./rootpre.sh./rootpre.sh output will be logged in /tmp/rootpre.out_06-04-13.12:33:04Saving the original files in /etc/ora_save_06-04-13.12:33:04Copying new

54、kernel extension to /etcLoading the kernel extension from /etc Oracle Kernel Extension Loader for AIX Copyright (c) 1998,1999 Oracle Corporation Successfully loaded /etc/pw-syscall.64bit_kernel with kmid: 0 x441e300 Successfully configured /etc/pw-syscall.64bit_kernel with kmid: 0 x441e300The kernel

55、 extension was successfuly loaded.Configuring Asynchronous I/OAsynchronous I/O is already definedConfiguring POSIX Asynchronous I/OChecking if group services should be configuredGroup hagsuser does not exist.Creating required group for group services: hagsuserPlease add your Oracle userid to the gro

56、up: hagsuserConfiguring HACMP group services socket for possible use by Oracle.The group or permissions of the group services socket have changed.Please stop and restart HACMP before trying to use Oracle.在兩個節點都執行root.sh腳本：ibm03# ./root.shWARNING: directory /oracle/product/10.2.0 is not owned by root

57、WARNING: directory /oracle/product is not owned by rootChecking to see if Oracle CRS stack is already configuredChecking to see if any 9i GSD is upSetting the permissions on OCR backup directorySetting up NS directoriesOracle Cluster Registry configuration upgraded successfullyWARNING: directory /or

58、acle/product/10.2.0 is not owned by rootWARNING: directory /oracle/product is not owned by rootSuccessfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node : node 1: ibm03 ibm03-priv ibm03node 2: ibm04 ibm04-priv ibm04Creating OCR keys for user root, pr

59、ivgrp system.Operation successful.Now formatting voting device: /dev/rora_voteFormat of 1 voting devices complete.Startup will be queued to init within 30 seconds.Adding daemons to inittabAdding daemons to inittabExpecting the CRS daemons to be up within 600 seconds.CSS is active on these nodes. ibm03CSS is inactive on these nodes. ibm04Local node checking complete.Run root.sh on remaining nodes to start CRS daemons.ibm04 # ./root.shWARNING: directory /oracle/product/10.2.0 is not owned by roo