1、項目 SF故障后鏡像重新同步日期：2012 年 3 月 10 日項目經理：主機工程師：、摘要本文檔描述了復的過程。項目，在兩套鏡像后，有出現故障后，SF鏡像重新同步數據庫并恢1場景目前臺州工商數據庫是存放在電信機房HDS AMS 2300和臺州工商IBM DS4700兩套組成的邏輯鏡像上，這樣數據庫空間有冗余的功能，當其中一套出現故障后，RAC 數據庫還能正常工作，但是恢復后，需要對鏡像盤做底層的重新同步。2處理步驟主要步驟1)1、 當磁盤陣列已恢復完畢后，先做重新掃盤操作：# cfgmgr v 2、 查看磁盤狀態：#lsdev -Cc disk 或 vxdisk -e list2、SF 掃描

2、設備：# vxdctl enable3、將磁盤重新加入到 dg：# vxreattach4、執行完畢后，使用 vxdisk list 命令查看不在有 failed 的磁盤（應該為 online） 5、修復卷信息并做增量同步：# vxrecover -sb6、可以通過命令查看同步進度：# vxtask list2)執行過程roottzgs1:/#vxdctl enable roottzgs1:/#vxreattach roottzgs1:/#vxdisk listDEVICETYPEDISKGROUPSUSams_23000_0 auto:cdsdisk ams_23000_0 datadg o

3、nline shared ams_23000_1 auto:cdsdisk ams_23000_1 datadgonline shared ams_23000_2 auto:cdsdisk ams_23000_2 datadgonline shared ams_23000_3 auto:cdsdisk ams_23000_3 datadgonline shared ams_23000_4 auto:cdsdisk -onlineams_23000_5 auto:cdsdisk -onlineams_23000_6auto:cdsdisk-online ams_23000_7auto:cdsdi

4、sk ams_23000_7ocrvotedgonline shared disk_0auto:LVM- LVMdisk_1auto:LVM-LVM ds4700-0_0auto:cdsdiskds4700-0_0datadgonline shared ds4700-0_1auto:cdsdiskds4700-0_1datadgonline shared ds4700-0_4auto:cdsdiskds4700-0_4datadgonline shared ds4700-0_5auto:cdsdiskds4700-0_5datadgonline shared ds4700-0_8auto:cd

5、sdiskds4700-0_8ocrvotedgonline sharedroottzgs1:/#vxrecover -sb當 vxtask list 執行后沒有輸出內容就表示同步已經完成，否則可以看到同步進度。同步完成后，再查看磁盤狀態，看到 sus 為 online 表示磁盤狀態都正常。32012 年 3 月 21 日 故障故障現象1)小機維保公司 3.15 巡檢的時候發現工商大樓機房的寫入出錯，列出磁盤狀態如下：roottzgs1:/#vxdisk listDEVICETYPEDISKGROUPS USams_23000_0 auto:cdsdisk ams_23000_0 datadg

6、online shared ams_23000_1 auto:cdsdisk ams_23000_1 datadgonline shared ams_23000_2 auto:cdsdisk ams_23000_2 datadgonline shared ams_23000_3 auto:cdsdisk ams_23000_3 datadg online shared ams_23000_4 auto:cdsdisk -onlineams_23000_5 auto:cdsdisk -onlineams_23000_6auto:cdsdisk-online ams_23000_7auto:cds

7、disk ams_23000_7ocrvotedgonline shared disk_0auto:LVM- LVMdisk_1auto:LVM-LVMds4700-0_0 auto-error sharedroottzgs1:/#vxdisk listDEVICETYPEDISKGROUPS USams_23000_0 auto:cdsdisk ams_23000_0 datadgonline shared ams_23000_1 auto:cdsdisk ams_23000_1 datadg online shared ams_23000_2 auto:cdsdisk ams_23000_

8、2 datadgonline shared ams_23000_3 auto:cdsdisk ams_23000_3 datadgonline shared ams_23000_4 auto:cdsdisk -onlineams_23000_5 auto:cdsdisk -onlineams_23000_6auto:cdsdisk-online ams_23000_7auto:cdsdisk ams_23000_7ocrvotedgonline shared disk_0auto:LVM- LVMdisk_1auto:LVM-LVM ds4700-0_0auto:cdsdiskds4700-0

9、_0datadgonline shared ds4700-0_1auto:cdsdiskds4700-0_1datadgonline shared ds4700-0_4auto:cdsdiskds4700-0_4datadgonline shared ds4700-0_5auto:cdsdisk ds4700-0_5datadgonline sharedds4700-0_8auto:cdsdiskds4700-0_8ocrvotedgonline sharedroottzgs1:/#vxtask listTASKIDPTID TYPE/SEPCTPROGRESS168PARENT/R 50.0

10、0% 4/2(1) VXRECOVER171168ATCOPY/R 27.79% 0/798720000/221925376 PLXATT datavol datavol-02datadg smartmove2)故障原因分析經過分析嘗試重新同步磁盤故障依舊，查看磁盤狀態似乎都是正常的。確認出錯的磁盤，測試內容狀態：roottzgs1:/#vxdisk -e listDEVICETYPEDISKGROUPSUS OS_NATIVE_NAMEATTRams_23000_0auto:cdsdiskams_23000_0datadgonline sharedhdisk2 std fcams_23000

11、_1auto:cdsdiskams_23000_1datadgonline sharedhdisk3roottzgs1:/#lsdev -Cc diskhdisk0Available 03-08-00-5,0 16 Bit LVD SCSI Disk Drive hdisk1Available 03-08-00-8,0 16 Bit LVD SCSI Disk Drive hdisk2Available 00-08-01Hitachi Disk Array (Fibre) hdisk3Available 00-08-01Hitachi Disk Array (Fibre) hdisk4Avai

12、lable 00-08-01Hitachi Disk Array (Fibre) hdisk5Available 00-08-01Hitachi Disk Array (Fibre) hdisk6Available 00-08-01Hitachi Disk Array (Fibre) hdisk7Available 00-08-01Hitachi Disk Array (Fibre) hdisk8Available 00-08-01Hitachi Disk Array (Fibre) hdisk9Available 00-08-01Hitachi Disk Array (Fibre) hdis

13、k10 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk11 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk12 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk13 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk14 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk15 Available 0B-08-01 Hitachi

14、 Disk Array (Fibre) hdisk16 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk17 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk18 Available 00-08-01MPIO DS4700 Disk hdisk19 Available 00-08-01MPIO DS4700 Disk hdisk20 Available 00-08-01MPIO DS4700 Disk hdisk21 Available 00-08-01MPIO DS4700 Diskh

15、disk22 Available 00-08-01MPIO DS4700 Diskds4700-0_1 auto-error shared ds4700-0_4 auto-error shared ds4700-0_5 auto-error shared ds4700-0_8 auto-error shared-ds4700-0_0datadgfailed was:ds4700-0_0-ds4700-0_1datadgfailed was:ds4700-0_1-ds4700-0_4datadgfailed was:ds4700-0_4-ds4700-0_5datadgfailed was:ds

16、4700-0_5-ds4700-0_8ocrvotedgfailed was:ds4700-0_8可以看到 hdisk19-hdisk22 出現 error，使用 dd 命令錯誤磁盤內容，發現無法正常內容：初步判斷，是電信大樓服務器無法向工商大樓 DS4700 中寫入數據。3)故障處理結果現象處理時，發現級聯 光纖不通，指示燈不亮，和電信溝通后，經過處理到 2012年 3 月 21 日下午 17:00 點左右， 光纖通過正常。roottzgs1:/#dd if=/dev/hdisk19 of=/hdisk19 bs=128K dd read error: No such device0+0 r

17、ecords in0+0 records outstd fcams_23000_2auto:cdsdiskams_23000_2datadgonline sharedhdisk4 std fcams_23000_3auto:cdsdiskams_23000_3datadgonline sharedhdisk5 std fcams_23000_4auto:cdsdisk-onlinehdisk6 std fcams_23000_5auto:cdsdisk-onlinehdisk7 std fcams_23000_6auto:cdsdisk-onlinehdisk8 std fcams_23000

18、_7auto:cdsdiskams_23000_7ocrvotedgonline sharedhdisk9 std fcdisk_0auto:LVM-LVMhdisk1-disk_1auto:LVM-LVMhdisk0-ds4700-0_0auto-error sharedhdisk19-ds4700-0_1auto-error sharedhdisk21-ds4700-0_4auto-error sharedhdisk18-ds4700-0_5auto-error sharedhdisk20-ds4700-0_8auto-error sharedhdisk22-ds4700-0_0datad

19、gfailed was:ds4700-0_0-ds4700-0_1datadgfailed was:ds4700-0_1-ds4700-0_4datadgfailed was:ds4700-0_4-ds4700-0_5datadgfailed was:ds4700-0_5-ds4700-0_8ocrvotedgfailed was:ds4700-0_8接下來重新掃描磁盤后，查看磁盤狀態正常，重新開始同步數據，到 19:00 后再查看，數據同步已經完成，磁盤狀態正常。roottzgs1:/#cfgmgr -v roottzgs1:/#vxdctl enable roottzgs1:/#vxrea

20、ttach roottzgs1:/#vxdisk listDEVICETYPEDISKGROUPSUSams_23000_0 auto:cdsdisk ams_23000_0 datadgonline shared ams_23000_1 auto:cdsdisk ams_23000_1 datadgonline shared ams_23000_2 auto:cdsdisk ams_23000_2 datadgonline shared ams_23000_3 auto:cdsdisk ams_23000_3 datadg online shared ams_23000_4 auto:cds

21、disk -onlineams_23000_5 auto:cdsdisk -onlineams_23000_6auto:cdsdisk-online ams_23000_7 auto:cdsdisk ams_23000_7ocrvotedgonline shared disk_0auto:LVM- LVMdisk_1auto:LVM-LVM ds4700-0_0auto:cdsdiskds4700-0_0datadgonline shared ds4700-0_1 auto:cdsdisk ds4700-0_1datadgonline shared ds4700-0_4auto:cdsdisk

22、ds4700-0_4datadgonline shared ds4700-0_5auto:cdsdiskds4700-0_5datadgonline shared ds4700-0_8auto:cdsdiskds4700-0_8ocrvotedgonline shared roottzgs1:/#vxrecover -sbroottzgs1:/#vxtask listTASKIDPTID TYPE/SEPCTPROGRESS /數據同步任務正在進行178PARENT/R 25.00% 4/1(2) VXRECOVER181178ATCOPY/R 00.07% 0/798720000/559104 PLXATT datavol datavol-02datadg smartmove roottzgs1:/ roottzgs1:/#vxtask listTASKIDPTID TYPE/SEPCTPROGRESS/到這里同步任務已經結束，同步完成