最近連續有客戶問我,如果修改SSH,會對oracle RAC有什么影響。這個問題,我也看過資料,對oracle RAC的運行是沒有影響的,但是“說”是沒有力度的。 今天正好相對比較空閑,全程針對SSH進行測試,并將測試過程記錄下來,與大家分享一下。 Part I. 測試前狀
最近連續有客戶問我,如果修改SSH,會對oracle RAC有什么影響。這個問題,我也看過資料,對oracle RAC的運行是沒有影響的,但是“說”是沒有力度的。
今天正好相對比較空閑,全程針對SSH進行測試,并將測試過程記錄下來,與大家分享一下。
Part I. 測試前狀態收集
1. 數據庫版本為11.2.0.4的兩節點的RAC,分別是node111g 和node211g
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options
2. 集群狀態正常
[grid@node111g ~]$ crsctl status res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE node111g ONLINE ONLINE node211g ora.DGROUP_01.dg ONLINE ONLINE node111g OFFLINE OFFLINE node211g ora.FRA.dg ONLINE ONLINE node111g ONLINE ONLINE node211g ora.LISTENER.lsnr ONLINE ONLINE node111g ONLINE ONLINE node211g ora.LISTENER_TEST.lsnr ONLINE ONLINE node111g ONLINE ONLINE node211g ora.OCR.dg ONLINE ONLINE node111g ONLINE ONLINE node211g ora.VOTE3D.dg ONLINE ONLINE node111g OFFLINE OFFLINE node211g ora.asm ONLINE ONLINE node111g Started ONLINE ONLINE node211g Started ora.gsd OFFLINE OFFLINE node111g OFFLINE OFFLINE node211g ora.net1.network ONLINE ONLINE node111g ONLINE ONLINE node211g ora.ons ONLINE ONLINE node111g ONLINE ONLINE node211g -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.TEST_SCAN1.lsnr 1 ONLINE ONLINE node111g ora.cvu 1 ONLINE ONLINE node211g ora.node111g.vip 1 ONLINE ONLINE node111g ora.node211g.vip 1 ONLINE ONLINE node211g ora.oc4j 1 ONLINE ONLINE node211g ora.orcl.db 1 ONLINE ONLINE node111g Open 2 ONLINE ONLINE node211g Open ora.orcl.romi.svc 1 ONLINE ONLINE node111g ora.orcl.test.svc 1 ONLINE ONLINE node211g 2 ONLINE ONLINE node111g ora.scan1.vip 1 ONLINE ONLINE node111g
3. SSH配置文件,默認SSH端口是22,文件有提及,不建議修改默認值,但是!!!我就想改,怎么辦呢??
[root@node111g ~]# more /etc/ssh/sshd_config # $OpenBSD: sshd_config,v 1.73 2005/12/06 22:38:28 reyk Exp $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/local/bin:/bin:/usr/bin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options change a # default value. #Port 22 #Protocol 2,1 Protocol 2 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: ...... [root@node111g ~]# [root@node211g ~]# more /etc/ssh/sshd_config # $OpenBSD: sshd_config,v 1.73 2005/12/06 22:38:28 reyk Exp $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/local/bin:/bin:/usr/bin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options change a # default value. #Port 22 #Protocol 2,1 Protocol 2 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: # HostKey for protocol version 1 #HostKey /etc/ssh/ssh_host_key # HostKeys for protocol version 2 #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key # Lifetime and size of ephemeral version 1 server key #KeyRegenerationInterval 1h
Part II 實際修改部分
1. 我們計劃將端口修改為6001,首先我們需要確認這個端口是否被使用
[root@node211g ~]# lsof -i:22 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME sshd 4543 root 3u IPv4 16317 TCP *:ssh (LISTEN) sshd 16000 root 3r IPv4 57216374 TCP node211g:ssh->node111g:56437 (ESTABLISHED) <<<<<<<22端口唄ssh使用 sshd 16004 oracle 3u IPv4 57216374 TCP node211g:ssh->node111g:56437 (ESTABLISHED) sshd 17907 root 3r IPv4 57462432 TCP node211g:ssh->node111g:59861 (ESTABLISHED) sshd 17911 oracle 3u IPv4 57462432 TCP node211g:ssh->node111g:59861 (ESTABLISHED) [root@node211g ~]# lsof -i:1521 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME oracle 1437 oracle 14u IPv4 68052787 TCP node211g:20651->node-cluster-scan:ncube-lm (ESTABLISHED) <<<<<<1521端口被scan使用 oracle 14412 grid 15u IPv4 1313392 TCP node211g:34873->node-cluster-scan:ncube-lm (ESTABLISHED) tnslsnr 14882 grid 16u IPv4 1332718 TCP node211g:ncube-lm (LISTEN) tnslsnr 14882 grid 17u IPv4 1332719 TCP node211g-vip:ncube-lm (LISTEN) [root@node211g ~]# lsof -i:6001 6001端口沒有被使用
2. 同時在兩個節點增加Port 6001 到SSH配置文件中
[root@node111g ~]# vi /etc/ssh/sshd_config #Port 22 Port 6001 #Protocol 2,1 Protocol 2 3. 重啟ssh服務,讓端口生效 [root@node111g ~]# service sshd restart Stopping sshd: [ OK ] Starting sshd: [ OK ] [root@node211g ~]# service sshd restart Stopping sshd: [ OK ] Starting sshd: [ OK ] 4.查看端口,已經被SSH使用 [root@node111g ~]# lsof -i:6001 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME sshd 28964 root 3u IPv4 111172702 TCP *:6001 (LISTEN) [root@node211g ~]# lsof -i:6001 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME sshd 18371 root 3u IPv4 70993328 TCP *:6001 (LISTEN) [root@node211g ~]# 5. 檢查和測試端口,端口22已經無法連接 [oracle@node111g archive_log]$ ssh node111g ssh: connect to host node111g port 22: Connection refused [oracle@node111g archive_log]$ ssh node211g ssh: connect to host node211g port 22: Connection refused
Part III 測試對RAC的影響
1. 通過端口6001連接,一切正常,RAC等價性也沒有受到影響,這里說明SSH等價性是不會受端口的影響的。
這里可以簡單理解為,node1和node2彼此有對方的訪問秘鑰的,只要對方有秘鑰就行,不會關心從什么端口訪問過來的
[oracle@node111g archive_log]$ ssh -p 6001 node211g Last login: Fri Dec 19 10:19:05 2014 from node111g [oracle@node211g ~]$ hostname node211g [oracle@node211g ~]$ exit logout Connection to node211g closed.
2. 端口已經修改成功,檢查crs狀態,一切正常
[grid@node111g ~]$ crsctl status res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE node111g ONLINE ONLINE node211g ora.DGROUP_01.dg ONLINE ONLINE node111g OFFLINE OFFLINE node211g ora.FRA.dg ONLINE ONLINE node111g ONLINE ONLINE node211g ora.LISTENER.lsnr ONLINE ONLINE node111g ONLINE ONLINE node211g ora.LISTENER_TEST.lsnr ONLINE ONLINE node111g ONLINE ONLINE node211g ora.OCR.dg ONLINE ONLINE node111g ONLINE ONLINE node211g ora.VOTE3D.dg ONLINE ONLINE node111g OFFLINE OFFLINE node211g ora.asm ONLINE ONLINE node111g Started ONLINE ONLINE node211g Started ora.gsd OFFLINE OFFLINE node111g OFFLINE OFFLINE node211g ora.net1.network ONLINE ONLINE node111g ONLINE ONLINE node211g ora.ons ONLINE ONLINE node111g ONLINE ONLINE node211g -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.TEST_SCAN1.lsnr 1 ONLINE ONLINE node111g ora.cvu 1 ONLINE ONLINE node211g ora.node111g.vip 1 ONLINE ONLINE node111g ora.node211g.vip 1 ONLINE ONLINE node211g ora.oc4j 1 ONLINE ONLINE node211g ora.orcl.db 1 ONLINE ONLINE node111g Open 2 ONLINE ONLINE node211g Open ora.orcl.romi.svc 1 ONLINE ONLINE node111g ora.orcl.test.svc 1 ONLINE ONLINE node211g 2 ONLINE ONLINE node111g ora.scan1.vip 1 ONLINE ONLINE node111g
3. 開始重啟CRS,看是否有影響,關閉
[root@node211g ~]# /u01/app/11.2.0/grid/bin/crsctl stop crs CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node211g' has completed CRS-4133: Oracle High Availability Services has been stopped. 4. 啟動CRS... [root@node111g ~]# /u01/app/11.2.0/grid/bin/crsctl start crs
5. 狀態依舊正常,完全可以啟動
[grid@node111g ~]$ crsctl status res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE node111g ONLINE ONLINE node211g ora.DGROUP_01.dg ONLINE ONLINE node111g OFFLINE OFFLINE node211g ora.FRA.dg ONLINE ONLINE node111g ONLINE ONLINE node211g ora.LISTENER.lsnr ONLINE ONLINE node111g ONLINE ONLINE node211g ora.LISTENER_TEST.lsnr ONLINE ONLINE node111g ONLINE ONLINE node211g ora.OCR.dg ONLINE ONLINE node111g ONLINE ONLINE node211g ora.VOTE3D.dg ONLINE ONLINE node111g OFFLINE OFFLINE node211g ora.asm ONLINE ONLINE node111g Started ONLINE ONLINE node211g Started ora.gsd OFFLINE OFFLINE node111g OFFLINE OFFLINE node211g ora.net1.network ONLINE ONLINE node111g ONLINE ONLINE node211g ora.ons ONLINE ONLINE node111g ONLINE ONLINE node211g -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.TEST_SCAN1.lsnr 1 ONLINE ONLINE node111g ora.cvu 1 ONLINE ONLINE node111g ora.liyou.db 1 OFFLINE OFFLINE Instance Shutdown 2 OFFLINE OFFLINE Instance Shutdown ora.node111g.vip 1 ONLINE ONLINE node111g ora.node211g.vip 1 ONLINE ONLINE node211g ora.oc4j 1 ONLINE ONLINE node111g ora.orcl.db 1 ONLINE ONLINE node111g Open 2 ONLINE ONLINE node211g Open ora.orcl.romi.svc 1 ONLINE ONLINE node111g ora.orcl.test.svc 1 ONLINE ONLINE node211g 2 ONLINE ONLINE node111g ora.scan1.vip 1 ONLINE ONLINE node111g
6. 再查個數據,就可以放心了吧!!
$ sqlplus / as sysdba Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options SQL> select username from dba_users; select username from dba_users; USERNAME ------------------------------------------------------------------------------------------ MGMT_VIEW SYS SYSTEM DBSNMP SYSMAN SCOTT ...... 38 rows selected.
階段總結,SSH在數據庫和GI安裝完成之后,正常運行階段,SSH是沒有任何使用的,oracle節點間通信都是通過oracle自己的process,通過TCP和UDP協議通信,使用這里更改SSH對于RAC數據庫的運行沒有任何影響。
7. 那!!!我們要是打patch呢?嘿嘿!!
不巧,這個測試數據庫已經應用最新的PSU 11.2.0.4.4,那么我們就先卸載 ......
cd $ORACLE_HOME/OPatch [oracle@node111g OPatch]$ ./opatch lsinventory Oracle Interim Patch Installer 11.2.0.3.6 Copyright (c) 2013, Oracle Corporation. All rights reserved。 Oracle Home : /u01/app/oracle/product/11.2.0/dbhome_1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc OPatch version : 11.2.0.3.6 OUI version : 11.2.0.4.0 Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2014-12-19_15-59-03午後_1.log Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2014-12-19_15-59-03午後.txt -------------------------------------------------------------------------------- Oracle Database 11g 11.2.0.4.0 Patch 19121551 : applied on Thu Nov 27 17:50:58 CST 2014 Unique Patch ID: 17949166 Patch description: "Database Patch Set Update : 11.2.0.4.4 (19121551)" <<<<<<<< Created on 6 Oct 2014, 10:07:57 hrs PST8PDT Sub-patch 18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)" Sub-patch 18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)" Sub-patch 17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)" Bugs fixed: ...... Patch 18031740 : applied on Thu Nov 27 14:29:51 CST 2014 Unique Patch ID: 17253722 Patch description: "OCW Patch Set Update : 11.2.0.4.2 (18031740)" Created on 19 Mar 2014, 09:06:31 hrs PST8PDT Bugs fixed: ...... Local node = node111g Remote node = node211g -------------------------------------------------------------------------------- OPatch succeeded.
8. Rollback失敗了,說明在rollback過程中,如果需要到對方節點執行操作的情況下,就需要通過SSH訪問,但是由于端口更改,會導致連接失敗。
同理應用patch,和節點添加刪除的時候,也需要copy文件到對方節點,同樣也會遇到這個問題。
(借用同事的測試環境,是日文的,但是通過LANG=en_US,一直沒有修改成英文輸出,這部分就伴有日文了,在同事的幫助下,理解其中的含義了,不過,日文很好猜的,“消除” ==“刪除”, “失敗”就不用翻譯了吧,嘿嘿!!)
All-node error message = リストされたファイル'/u01/app/oracle/product/11.2.0/dbhome_1/.patch_storage/NRollback/2014-12-19_16-16-49午後/rac/remove_files.txt.instantiated'に基づいて、ノード'node211g'でファイルを削除中にエラーが発生しました。 OPatch remote node node211g, delete the file fail. 続行しますか。[y|n] y User Responded with: Y Instantiating the file "/u01/app/oracle/product/11.2.0/dbhome_1/.patch_storage/NRollback/2014-12-19_16-16-49午後/rac/remove_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11.2.0/dbhome_1/.patch_storage/NRollback/2014-12-19_16-16-49午後/rac/remove_dirs.txt" with actual path. Removing directories on remote nodes... OPatchはリモートノード'node211g' でディレクトリの削除に失敗しました。 詳細: '/u01/app/oracle/product/11.2.0/dbhome_1/.patch_storage/NRollback/2014-12-19_16-16-49午後/rac/remove_dirs.txt.instantiated'に基づいてリストされたディレクトリをノード'node211g'から削除できませんでした。[PRKC-1083 : 指定したノード"node211g "のいずれかに、"/u01/app/oracle/product/11.2.0/dbhome_1/.patch_storage/NRollback/2014-12-19_16-16-49午後/rac/remove_dirs.txt.instantiated"に示されているディレクトリを削除することに失敗しました。 ノードnode211g:PRKC-1044 : シェル/usr/bin/sshおよび/usr/bin/rshを使用したノードnode211gのリモートコマンド実行設定のチェックに失敗しました <<<<<使用/usr/bin/sshおよび/usr/bin/rsh執行遠端node211g命令檢查失敗。 node211g: Connection refused fail.
Part IV 測試修改IP對SSH等價性的影響
1. 下面我們在測試一下,修改IP是不是會影響SSH等價性(注意,修改private IP,一定是要同步調整GI中記錄的inter-connect網絡信息的,不然GI是不能正常通信的,oracle有相關文檔介紹如何修改Public和Pricate IP的)
2. 從Private IP下手,只要修改node2節點的就可以了
# ifconfig eth2 eth2 Link encap:Ethernet HWaddr 00:0C:29:4D:2D:D0 inet addr:10.0.0.2 Bcast:10.0.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8739422 errors:0 dropped:0 overruns:0 frame:0 TX packets:11094020 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5201547204 (4.8 GiB) TX bytes:8400210712 (7.8 GiB) [root@node211g tmp]# more /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 node211g localhost 192.168.1.1 node111g 192.168.1.2 node211g 192.168.1.3 node111g-vip 192.168.1.4 node211g-vip 192.168.1.5 node-cluster-scan 10.0.0.1 node111g-priv 10.0.0.2 node211g-priv <<<<<<<<<<<<<<<<<<<
3. 修改網卡Private IP為10.0.0.12
[root@node211g tmp]# ifconfig eth2 10.0.0.12 netmask 255.255.255.0
4. 查看,已經修改成功
[root@node211g tmp]# ifconfig eth2 eth2 Link encap:Ethernet HWaddr 00:0C:29:4D:2D:D0 inet addr:10.0.0.12 Bcast:10.0.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8744646 errors:0 dropped:0 overruns:0 frame:0 TX packets:11100718 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5204200949 (4.8 GiB) TX bytes:8404710975 (7.8 GiB)
5. 切換到grid用戶測試, 發現在第一次連接的時候,還是要輸入密碼的,但是第二次就不需要的,說明SSH等價性,在更改IP之后是有影響的,需要在RSA文件中記錄新IP登陸的授權信息,但是不用重新配置等價性。
[root@node211g tmp]# su - grid [grid@node211g ~]$ ssh -p 6001 10.0.0.1 Last login: Mon Dec 22 10:34:07 2014 from node211g-priv [grid@node111g ~]$ ssh -p 6001 10.0.0.12 The authenticity of host '10.0.0.12 (10.0.0.12)' can't be established. RSA key fingerprint is 9b:11:59:5b:0f:0d:85:17:94:0c:e0:76:be:c4:7e:9e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.0.0.12' (RSA) to the list of known hosts. Last login: Thu Dec 18 14:45:30 2014 from dhcp-tokyo-twvpn-1-vpnpool-10-191-12-97.vpn.oracle.com [grid@node211g ~]$ exit logout Connection to 10.0.0.12 closed. [grid@node111g ~]$ ssh -p 6001 10.0.0.12 Last login: Mon Dec 22 10:34:55 2014 from node111g-priv
到此,本次測試結束,分別測試了在SSH端口變更情況下,對RAC的影響。還有修改IP對于SSH等價性的影響。
------------測試,才有說服力!!!------------
聲明:本網頁內容旨在傳播知識,若有侵權等問題請及時與本網聯系,我們將在第一時間刪除處理。TEL:177 7030 7066 E-MAIL:11247931@qq.com