RedHat Enterprice Linux Server 5 X64上安装Oracle 10G R2 RAC安装指南

RedHat Enterprice Linux Server 5 X64上安装Oracle 10G R2 RAC安装指南 Oracle 10G R2 RAC安装指南 For RedHat Enterprice Linux Server 5 X64 1 硬件配置 主机主要配置: HP DL580G5两台，4C/16G/2*146/1*FC 操作系统: RedHat Enterprice Linux Server 5.2 X64 数据库: Oracle 10g RAC R2(10.2.0.4) 磁盘阵列: HP XP24000 2 安装设计 2.1 操作系统安装设计 2.1.1 操作系统选择 Linux操作系统安装版本要求:Redhat Linux AS 5.2 2.1.2 安装组件时选择定制 安装组件时选择定制:(一种方式你只选择你认识的包，操作系装完后，再安装需要的包，另一种方式是所有包都装上) 2.1.3 关闭防火墙 关闭Firewall，关闭selinux 2.1.4 需要安装的Patches/Packages 操作系统必备的基本包如下: binutils-2.17.50.0.6-6.el5.x86_64.rpm compat-db-4.2.52-5.1.x86_64.rpm compat-gcc-34-3.4.6-4.x86_64.rpm compat-gcc-34-c++-3.4.6-4.x86_64.rpm compat-libstdc++-33-3.2.3-61.x86_64.rpm compat-libstdc++-296-2.96-138.i386.rpm control-center-2.16.0-16.el5.x86_64.rpm gcc-4.1.2-42.el5.x86_64.rpm gcc-c++-4.1.2-42.el5.x86_64.rpm glibc-2.5-24.x86_64.rpm glibc-common-2.5-24.x86_64.rpm glibc-devel-2.5-24.x86_64.rpm glibc-headers-2.5-24.x86_64.rpm gnome-libs-1.4.1.2.90-44.1 ibaio-0.3.106-3.2.x86_64.rpm libgomp-4.1.2-42.el5.x86_64.rpm libstdc++-4.1.2-42.el5.x86_64.rpm libstdc++-devel-4.1.2-42.el5.x86_64.rpm libXp-1.0.0-8.1.el5.x86_64.rpm make-3.81-3.el5.x86_64.rpm pdksh-5.2.14-30 sysstat-7.0.2-1.el5.x86_64.rpm xscreensaver-5.04-1.0.el5.src.rpm 需要特别说明的是如果你的基本包高于以上版本的话不要紧，没有任何关系，事实上我安装的时候就有许多基本包是高于以上版本的。其中标为经包的3个包在RedHat AS5里已经不提供了(估计是被其它包所代替，个人意见，示经官方证实)，但是不装这三个包不影响Oracle 10g在AS5下的安装。 查询方法，例如: rpm -q binutils compat-db compat-gcc-34 compat-gcc-34-c++ compat-libstdc++-33 control-center gcc gcc-c++ glibc glibc-common glibc-devel glibc-headers gnome-libs libstdc++ libstdc++-devel libgomp libXp make pdksh sysstat xscreensaver libaio 安装包的方法如下: #rpm -Uvh libaio-0.3.105-2.x86_64.rpm 卸载包的方法如下: #rpm –e libaio-0.3.105-2(卸载包) 2.1.5 安装特别包 需要特注意的是libaio包，这个包在AS4下默认安装的是32包的，我们需要把它装成64位的。 2 查看libaio的版本用，以下命令 #rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' libaio 如果出来的结果中有:libaio-0.3.105-2 (x86_64)就可以了，如果没有就要安装libaio-0.3.105-2.x86_64.rpm这个包了。 安装该包的命令如下: #rpm -Uvh libaio-0.3.105-2.x86_64.rpm 如果已经安装了32位libaio包则需要先卸载掉32位包,然后再安装64位的包。 卸载该包的命令如下: #rpm –e libaio-0.3.105-2(卸载包) 2.1.6 调整操作系统参数 要达到调整操作系统参数的目录，只需修改/etc/sysctl.conf文件即可，在本环境中，只要在sysctl.conf文件里添加如下内容: kernel.shmall = 2097152 kernel.shmmax = 8589934592(其中这个数值为内存的一半，以byte为单位，我的机器内存 为16384MB，那这这个参数值即为8192*1024*1024) kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536 net.ipv4.ip_local_port_range = 1024 65000 net.ipv4.tcp_wmem = 262144 262144 262144 net.ipv4.tcp_rmem = 4194304 4194304 4194304 net.core.rmem_default = 1048576 net.core.rmem_max = 1048576 net.core.wmem_default = 262144 net.core.wmem_max = 262144 2.1.7 网卡聚合设计 安装网卡聚合软件或驱动，将两块网卡配置为链路一主一备模式运行，文档服务器和银电联网服务器除外。 3 2.1.8 主机网络标识设计 主机名IP标签 网卡 IP地址 掩码 网关 功能 称 接口 (位) hbyxdnl1 hbyxdnl11 Bind0 10.122.10.162/25 25 10.122.10.254 固定IP Hbyxdnl1-vip Bind0:1 10.122.10.163/25 25 10.122.10.254 vip Hbyxdnl1-prv Bind1 192.168.16.1/25 - priv Hbyxdnl2 Hbyxdnl2 Bind0 10.122.10.164/25 25 10.122.10.254 固定IP Hbyxdnl2-vip Bind0:1 10.122.10.165/25 25 10.122.10.254 vip Hbyxdnl2-prv Bind1 192.168.16.2/25 25 - priv 注:两台机器上都安装oracle 10g RAC电能采集中间库、95598中间库和weblogic92; 2.1.9 内置磁盘安装设计 本项目中，两台电能量采集前置机将采用HP DL580G5，配置8块146G磁盘，服务器磁盘安装设计如下: 磁盘分配 用途 RAID需求 2块 系统盘镜像，卷组vg00。 1 6块 数据存储，卷组vg01。RAID 5 5 2.1.10 内置磁盘定制(CUSTOM)文件系统设计(1号机) 卷组名称 文件系统 Mount点 大小要求 类型 用途 - /dev/sda1 /boot 120M ext3 引导区 vg00 /dev/vg00/lvol3 / 20G Ext3 根区 /dev/vg00/oralv /oracle 30G ext3 oracle /dev/vg00/app /app 20G ext3 weblogic /dev/vg00/lvol2 16G swap 交换分区 vg01 /dev/vg01/arch01lv /arch01 175G ext3 Oracle归档 /dev/vg01/explv /exp 500G ext3 Oracle备份 - hbyxdnl2:/dev/vg01/a/arch02 175G NFS Oracle归档 rch02lv 说明:主机共包含8块146G内置磁盘，两块以镜像方式创建卷组vg00，;剩余6块以RAID5方式创建卷组vg01，建立默认swap区，16G。 4 2.1.11 内置磁盘定制(CUSTOM)文件系统设计(2号机) 卷组名称 文件系统 Mount点 大小要求 类型 用途 - /dev/sda1 /boot 120M ext3 引导区 vg00 /dev/vg00/lvol3 / 20G Ext3 根区 /dev/vg00/oralv /oracle 30G ext3 oracle /dev/vg00/app /app 20G ext3 weblogic /dev/vg00/lvol2 16G swap 交换分区 vg01 /dev/vg01/arch02lv /arch02 175G ext3 Oracle归档 /dev/vg01/explv /exp 500G ext3 Oracle备份 - hbyxdnl1:/dev/vg01/a/arch01 175G NFS Oracle归档 rch01lv 说明:主机共包含8块146G内置磁盘，2块以镜像方式创建卷组vg00，;剩余6块以RAID5方式创建卷组vg01，建立默认swap区，16G。 2.1.12 阵列磁盘定制分区设计 集群锁盘分区 磁盘 分区 裸设备 大小要求 类型 用途 sdg /dev/sdg /dev/raw/raw1 300M raw ocr盘 sdh /dev/sdh /dev/raw/raw2 300M raw vote盘 sdi /dev/sdi /dev/raw/raw3 300M raw spfile文件盘 集群数据盘分区 磁盘 分区 裸设备 ASM盘 大小要求 类型 用途 sdc /dev/sdc /dev/raw/raw4 ORA1 286G ASM Oracle数据 sdd /dev/sdd /dev/raw/raw5 ORA2 286G ASM Oracle数据 2.2 Oracle安装设计 2.2.1 Oracle电能量库(RAC数据库) Oracle的RAC数据库安装在本项目的两台HP DL580G5 PC服务器上。 2.2.2 安装oracle clusterware 安装项 配置说明 5 crs安装路径 /oracle/product/crs/10.2 cluster name crs_dnl cluster节点名称 hbyxdnl1,hbyxdnl2 指定网卡使用(VIP,PRIV) 见“7.5.2.2章节” Ocr盘指定 /dev/raw/rocrlv vote盘指定 /dev/raw/rvotelv 注:此处的ocr和vote盘指定应与卷组裸设备划分章节中创建的裸设备保持一致～ 2.2.3 安装DATABASE软件 安装项 配置说明 安装方式 定制custom 安装路径(ORACLE_HOME) /oracle/product/db/10.2 cluster安装模式 cluster installation: 两个节点:hbyxdnl1, hbyxdnl2 创建库选项 只安装软件，不建库 2.2.4 Oracle数据库升级 在上节安装完后数据库软件后，升级oracle数据库到10.2.0.4，过程同上。 注意安装路径要和原来的目录保存一致。 2.2.5 创建RAC数据库 安装项 配置说明 创建类型 oracle real application cluster database 节点选择 hbyxdnl1,hbyxdnl2 数据库类型 custom 数据库名称及SID前缀 dnl 6 数据库服务 dnl 两个节点上的实例:dnl1,dnl2 TAF策略:basic 用户口令 oracle 存储模式 裸设备方式 是否启用归档 启用归档运行模式; 两个实例的归档路径:/arch01和/arch02 内存设置 选择typical percentage:44% Block大小 8k processes 2500 语言 AMERICAN_AMERICA 数据库字符集 UTF8 国家区域字符集 UTF8 Connection mode dedicated server mode Spfile路径 /dev/vgdata/rspfile 数据存储 该部分数据数据文件采用ASM磁盘，磁盘名为(控制文件、系统空间、RAC_DISK，名为RAC_DISK的ASM盘所采用的物理盘 为,/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf(注:建redolog、用户表空间等) 立二个undo表空间undotbs1和undotbs2，每个实列1 个undotbs表空间，建12组redo log，每个实例6组) 2.2.6 安装Oracle Workflow Server2.6.3 1 只在1号节点hbyxdnl1上发起安装过程 2 安装路径与此前的ORACLEHOME保持一致。 3 只安装oracle workflow 2.6.3组件。 7 3 安装配置操作系统 3.1 安装操作系统 3.2 创建oracle用户和dba组 首先创建dba用户，组ID为300 再创建oracle用户，密码为oracle，用户ID为300 8 修改oracle用户，将oracle加到dba组，主组为dba 9 3.3 修改oracle用户限制 1(将下列内容添加到/etc/security/limits.conf文件里: oracle soft nproc 4096 oracle hard nproc 16384 oracle soft nofile 4096 oracle hard nofile 65536 2(将下列内容添加到/etc/pam.d/login文件里: session required pam_limits.so 3. 将下列内容添加到/etc/profile文件里: if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi 4(将先前创建的几个文件系统授权给oracle #chown –R oracle:dba /orainstall /orasys /oradata #chmod –R 775 /orainstall /orasys /oradata 5(修改/etc/hosts 10.122.10.162 hbyxdnl1 10.122.10.163 hbyxdnl1-vip 192.168.10.162 hbyxdnl1-priv 10.122.10.164 hbyxdnl2 10.122.10.165 hbyxdnl2-vip 192.168.10.164 hbyxdnl2-priv 3.4 安装额外系统包 操作系统必备的基本包如下: binutils-2.17.50.0.6-6.el5.x86_64.rpm compat-db-4.2.52-5.1.x86_64.rpm compat-gcc-34-3.4.6-4.x86_64.rpm compat-gcc-34-c++-3.4.6-4.x86_64.rpm compat-libstdc++-33-3.2.3-61.x86_64.rpm compat-libstdc++-296-2.96-138.i386.rpm 10 control-center-2.16.0-16.el5.x86_64.rpm gcc-4.1.2-42.el5.x86_64.rpm gcc-c++-4.1.2-42.el5.x86_64.rpm glibc-2.5-24.x86_64.rpm glibc-common-2.5-24.x86_64.rpm glibc-devel-2.5-24.x86_64.rpm glibc-headers-2.5-24.x86_64.rpm gnome-libs-1.4.1.2.90-44.1 ibaio-0.3.106-3.2.x86_64.rpm libgomp-4.1.2-42.el5.x86_64.rpm libstdc++-4.1.2-42.el5.x86_64.rpm libstdc++-devel-4.1.2-42.el5.x86_64.rpm libXp-1.0.0-8.1.el5.x86_64.rpm make-3.81-3.el5.x86_64.rpm pdksh-5.2.14-30 sysstat-7.0.2-1.el5.x86_64.rpm xscreensaver-5.04-1.0.el5.src.rpm 需要特别说明的是如果你的基本包高于以上版本的话不要紧，没有任何关系，事实上我安装的时候就有许多基本包是高于以上版本的。其中标为经包的3个包在RedHat AS5里已经不提供了(估计是被其它包所代替，个人意见，示经官方证实)，但是不装这三个包不影响Oracle 10g在AS5下的安装。 查询方法，例如: #rpm -q binutils compat-db compat-gcc-34 compat-gcc-34-c++ compat-libstdc++-33 control-center gcc gcc-c++ glibc glibc-common glibc-devel glibc-headers gnome-libs libstdc++ libstdc++-devel libgomp libXp make pdksh sysstat xscreensaver libaio 安装包的方法如下: #rpm -Uvh libaio-0.3.105-2.x86_64.rpm 卸载包的方法如下: #rpm –e libaio-0.3.105-2(卸载包) 11 需要特注意的是libaio包，这个包在AS4下默认安装的是32包的，我们需要把它装成64位的。 查看libaio的版本用，以下命令 #rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' libaio 如果出来的结果中有:libaio-0.3.105-2 (x86_64)就可以了，如果没有就要安装libaio-0.3.105-2.x86_64.rpm这个包了。 安装该包的命令如下: #rpm -Uvh libaio-0.3.105-2.x86_64.rpm 如果已经安装了32位libaio包则需要先卸载掉32位包,然后再安装64位的包。 卸载该包的命令如下: #rpm –e libaio-0.3.105-2(卸载包) 3.5 配置Hangcheck 计时器 [root@node1 ~]# vi /etc/rc.local 增加: modprobe hangcheck-timer hangcheck-tick=30 hangcheck_margin=180 然后得启机器 3.6 修改/etc/redhat-release文件 由于10g发行时AS5.2还没有发布，因此安装程序的操作系检查不支持RedHat AS 5.2，这就需要手工修改/etc/redhat-release文件。将Red Hat Enterprise Linux Server release 5.2 (Tikanga)替换为Red Hat Enterprise Linux Server release 4。 另一种方法就是在执行Oracle安装方件是将参数，忽略操作系统版本检查，具体方法如下: $./runInstaller -ignoreSysPrereqs IGNORE_PREREQ=true 3.7 设置ssh 在clusterware (CRS) 和Rac Database 安装过程中，Oracle Universal Installer (OUI) 必须能够以oracle 的身份自动将软件复制到所有RAC 节点。这里我们通过配置ssh 让oracle 用户拥有不输入密码即可访问 各节点的能力。 12 首先两个节点间互ping，要确保都是通的 [root@node1 ~]# ping 192.168.100.102 [root@node1 ~]# ping 10.10.17.222 要建立用户等效性，需要在两个节点上以oracle 用户身份生成用户的公钥和私钥，首先在 node1 执行: [root@node1 opt]# su - oracle [oracle@node1 ~]$ mkdir ~/.ssh [oracle@node1 ~]$ chmod 700 ~/.ssh [oracle@node1 ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_rsa. Your public key has been saved in /home/oracle/.ssh/id_rsa.pub. The key fingerprint is: d2:69:eb:ac:86:62:27:50:99:ff:e8:1e:a2:e6:5d:7f oracle@node1 [oracle@node1 ~]$ ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_dsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_dsa. Your public key has been saved in /home/oracle/.ssh/id_dsa.pub. The key fingerprint is: 0a:9a:20:46:a2:28:ec:72:23:82:f2:9d:f8:62:9b:d1 oracle@node1 然后换node2 执行 [root@node2 ~]# ping 192.168.100.102 [root@node2 ~]# ping 10.10.17.222 [root@node2 opt]# su - oracle [oracle@node2 ~]$ mkdir ~/.ssh [oracle@node2 ~]$ chmod 700 ~/.ssh [oracle@node2 ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_rsa. Your public key has been saved in /home/oracle/.ssh/id_rsa.pub. The key fingerprint is: dd:be:7a:37:e4:b5:f0:b2:24:95:50:61:ea:a1:61:07 oracle@node2 [oracle@node2 ~]$ ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_dsa): 13 Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_dsa. Your public key has been saved in /home/oracle/.ssh/id_dsa.pub. The key fingerprint is: bc:b5:cb:43:c7:19:53:d6:f7:16:69:85:12:7f:aa:be oracle@node2 切换回node1，接着执行: [oracle@node1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys [oracle@node1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys 提示:下列命令会提示你输入node2 的oracle 密码，按照提示输入即可，如果失败可重新 尝试执行命 令。 [oracle@node1 ~]$ ssh node2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys The authenticity of host 'node2 (192.168.100.102)' can't be established. RSA key fingerprint is 92:d1:ce:5b:c8:a1:52:d5:ac:00:5f:48:5d:12:06:e4. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node2,192.168.100.102' (RSA) to the list of known hosts. oracle@node2's password: [oracle@node1 ~]$ ssh node2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys oracle@node2's password: [oracle@node1 ~]$ scp ~/.ssh/authorized_keys node2:~/.ssh/authorized_keys oracle@node2's password: authorized_keys 100% 1992 2.0KB/s 00:00 两机相互执行，看看是否还需要输入密码 [oracle@node1 ~]$ ssh node1 date [oracle@node1 ~]$ ssh node2 date [oracle@node1 ~]$ ssh node1-priv date [oracle@node1 ~]$ ssh node2-priv date 切换至node2 执行 [oracle@node2 ~]$ ssh node1 date [oracle@node2 ~]$ ssh node2 date [oracle@node2 ~]$ ssh node1-priv date [oracle@node2 ~]$ ssh node2-priv date 提示: 注意命令的返回信息，由于rac 对节点间时间较敏感，因此如果此处显示节点单时间不一至， 建议同 步(一般相隔超过20 秒就需要同步时间了)。 !!!!!最后别忘了ssh一下自已!!!!! 14 3.8 升级操作系统内核 3.8.1 升级内核 升级内核到最新版，包括的主要文件有如下: kernel-2.6.18-92.1.22.el5.x86_64.rpm kernel-devel-2.6.18-92.1.22.el5.x86_64.rpm kernel-doc-2.6.18-92.1.22.el5.noarch.rpm kernel-headers-2.6.18-92.1.22.el5.x86_64.rpm 下载地址: 安装方法如下: hbyxdnl1:/tmp/kennel#rpm –ivh * 3.8.2 校验初始 RAM 磁盘映像 如果系统使用 ext3 文件系统或 SCSI 控制器，你就需要初始 RAM 磁盘。初始 RAM 磁盘的目的是允许模块化的内核在它能够进入模块通常驻留的设备之前具备进入内核需要从该设备引导的模块的能力。 初始 RAM 磁盘通过使用 mkinitrd 命令来创建。然而，如果内核及其相关文件是从 Red Hat, Inc. 发行的 RPM 软件包中安装或升级的话，这个步骤会被自动执行;因此，它不必被手工进行。要校验它是否被创建了，使用 ls -l /boot 命令来确定 initrd-2.4.20-2.47.1.img 文件被创建了(版本应该匹配刚刚安装了的内核的版本)。 下一步是校验引导装载程序已被配置来引导新内核。详情请参阅 第 30.6 节 。 3.8.3 校验引导装载程序 如果你安装了 GRUB 或 LILO 引导装载程序， kernel RPM 软件包配置它们来引导刚刚安装的内核，但是它并不配置引导装载程序默认引导新内核。 15 确认一下引导装载程序已被配置成引导新内核总是值得提倡的。这是至关重要的一步。如果引导装载程序被配置得不正确，你将无法引导你的系统。若这种情况发生了，使用你从前创建的引导盘来引导你的系统，然后再试图配置你的引导装载程序。 3.8.3.1 GRUB 如果你选择了 GRUB 作为引导装载程序，请确认 /boot/grub/grub.conf 文件中包含的 title 部分中的版本与你刚刚安装的 kernel 软件包的版本相同(如果你安装了 kernel-smp 和/或 kernel-bigmem ，你也会看到这个部分): # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/hda2 # initrd /initrd-version.img #boot=/dev/hda default=3 timeout=10 splashimage=(hd0,0)/grub/splash.xpm.gz title Red Hat Linux (2.4.20-2.47.1) root (hd0,0) kernel /vmlinuz-2.4.20-2.47.1 ro root=LABEL=/ initrd /initrd-2.4.20-2.47.1.img title Red Hat Linux (2.4.20-2.30) root (hd0,0) kernel /vmlinuz-2.4.20-2.30 ro root=LABEL=/ initrd /initrd-2.4.20-2.30.img 如果你创建了单独的 /boot 分区，到内核与 initrd 映像的路径是相对于 /boot 分区而言的。 注意，默认引导项目没有被设置为新内核。要配置 GRUB 来默认引导新内核，把 default 变量的值改成包含新内核的 title 部分的号码。这个号码从 0 开始。例如，如果新内核是第二个 title 部分，把 default 设置为 1 。 16 你可以重新引导计算机来开始测试这个新内核，观察屏幕上的消息来确保硬件被正确地检测到了。 3.8.3.2 LILO 如果你选择了 LILO 作为引导装载程序，请确认 /etc/lilo.conf 文件中包含的 image 部分中的版本与你刚刚安装的 kernel 软件包的版本相同(如果你安装了 kernel-smp 和/或 kernel-bigmem ，你也会看到这个部分): prompt timeout=50 default=2.4.20-2.30 boot=/dev/hda map=/boot/map install=/boot/boot.b message=/boot/message linear image=/boot/vmlinuz-2.4.20-2.47.1 label=2.4.20-2.47.1 initrd=/boot/initrd-2.4.20-2.47.1.img read-only append="root=LABEL=/" image=/boot/vmlinuz-2.4.20-2.30 label=2.4.20-2.30 initrd=/boot/initrd-2.4.20-2.30.img read-only append="root=LABEL=/" 注意，默认引导项目没有被设置为新内核。要配置 LILO 来默认引导新内核，把 default 变量的值改成包含新内核的 image 部分中的 label 的值。以根用户身份运行 /sbin/lilo 命令来启用改变。运行后，其输出会与如下相似: Added 2.4.20-2.47.1 * Added linux 2.4.20-2.47.1 后面的 * 意味着那部分中的内核是 LILO 会默认引导的内核。 17 你可以重新引导计算机来开始测试这个新内核，观察屏幕上的消息来确保硬件被正确地检测到了。 3.9 配置网卡绑定 所谓bonding，就是将多块网卡绑定同一IP地址对外提供服务，可以实现高可用或者负 载均衡。当然，直接给两块网卡设置同一IP地址是不可能的。通过bonding，虚拟一块网卡对 外提供连接，物理网卡的被修改为相同的MAC地址。Kernels 2.4.12及以后的版本均提供 bonding模块，以前的版本可以通过patch实现。 3.9.1 首先要看linux是否支持bonding RHEL4已经默认支持了.(大部分发行版都支持) # modinfo bonding filename: /lib/modules/2.6.18-8.el5/kernel/drivers/net/bonding/bonding.ko author: Thomas Davis, tadavis@lbl.gov and many others description: Ethernet Channel Bonding Driver, v3.0.3 version: 3.0.3 license: GPL srcversion: 2547D22885C2FDF28EF7D98 如果有类似上面的信息输出,说明已经支持了. 如果没有,说明内核不支持bonding,需要重新编译内核 编译内核的步骤这里就不多写了,在make menuconfig里面 找到Bonding driver support这样的对话框,选中它. 如果是模块编译,还需要挂载模块.挂载bonding 模块的方法; [root@localhost#] modprobe bonding 如果/sbin/ifenslave没有,我们还需要把它也单独编译出来 [root@lin:/home/beinan#] cd /usr/src/linux-2.6.20/Documentation/networking [root@lin:/home/networking#] gcc -Wall -Wstrict-prototypes -O -I/usr/src/linux-2.6.20/include ifenslave.c -o ifenslave [root@lin:/home/networking#] cp ifenslave /sbin/ifenslave 3.9.2 确认你目前使用的网卡 检查/etc/sysconfig/network-scripts目录下以ifcfg-开头的文件，应该 为eth0, eth1... 18 3.9.3 配置虚拟网卡bond0 1、编辑两个虚拟网络接口配置文件,指定网卡IP ，并将第一行改为分别改为 DEVICE=bond0和DEVICE=bond1: [root@hbyxdnl1]# cp /etc/sysconfig/network-scripts/ifcfg-eth0 ifcfg-bond0 [root@hbyxdnl1]# vi ifcfg-bond0 [root@hbyxdnl1]# cat ifcfg-bond0 DEVICE=bond0 BOOTPROTO=none ONBOOT=yes NETWORK=10.122.10.128 NETMASK=255.255.255.128 IPADDR=10.122.10.162 USERCTL=no GATEWAY=10.122.10.254 TYPE=Ethernet [root@hbyxdnl]# cp /etc/sysconfig/network-scripts/ifcfg-eth0 ifcfg-bond1 [root@hbyxdnl1]# vi ifcfg-bond1 [root@hbyxdnl1]# cat ifcfg-bond1 DEVICE=bond1 BOOTPROTO=none ONBOOT=yes NETWORK=192.168.16.0 NETMASK=255.255.255.128 IPADDR=192.168.16.1 USERCTL=no TYPE=Ethernet 2、修改eth0， eth1，eth2，eth3配置文件 [root@hbyxdnl1 network-scripts]#vi ifcfg-eth0 [root@hbyxdnl1 network-scripts]#cat ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none ONBOOT=yes MASTER=bond0 SLAVE=yes USERCTL=yes [root@hbyxdnl1 network-scripts]# ifcfg-eth1 DEVICE=eth1 BOOTPROTO=none 19 ONBOOT=yes MASTER=bond0 SLAVE=yes USERCTL=yes [root@hbyxdnl1 network-scripts]#cat ifcfg-eth2 DEVICE=eth2 BOOTPROTO=none ONBOOT=yes MASTER=bond1 SLAVE=yes USERCTL=yes [root@hbyxdnl1 network-scripts]# ifcfg-eth3 DEVICE=eth3 BOOTPROTO=none ONBOOT=yes MASTER=bond1 SLAVE=yes USERCTL=yes 3 、编辑 /etc/modules.conf 文件 编辑 /etc/modprobe.conf 文件，加入如下一行内容，以使系统在启动时加载bonding模块，对外虚拟网络接口设备为 bond0、bond1加入下列4行: alias bond0 bonding options bond0 miimon=100 mode=1 alias bond1 bonding options bond1 miimon=100 mode=1 说明:miimon是用来进行链路监测的。 比如:miimon=100，那么系统每100ms监测一次链路连接状态，如果有一条线路不通就转入另一条线路;mode的值表示工作模式，他共有0,1,2,3四种模式，常用的为0,1两种。 mode=0表示load balancing (round-robin)为负载均衡方式，两块网卡都工作。 mode=1表示fault-tolerance (active-backup)提供冗余功能，工作方式是主备的工作方式,也就是说默认情况下只有一块网卡工作,另一块做备份。 bonding只能提供链路监测，即从主机到交换机的链路是否接通。如果只是交换机对外的链路down掉了，而交换机本身并没有故障，那么bonding会认为 20 链路没有问题而继续使用。 需要说明的是如果想做成负载均衡,仅仅设置这里options bond0 miimon=100 mode=0是不够的,还需要设置交换机的端口。从原理一下(bond运行在mode 0下): mode 0下bond所绑定的网卡的IP都被修改成一样的mac地址，如果这些网卡都被接在同一个交换机，那么交换机的arp表里这个mac地址对应的端口就有多个，那么交换机接受到发往这个mac地址的包应该往哪个端口转发呢,正常情况下mac地址是全球唯一的，一个mac地址对应多个端口肯定使交换机迷惑了。所以mode0下的bond如果连接到交换机，交换机这几个端口应该采取聚合方式(cisco称为ethernetchannel，foundry称为portgroup)，因为交换机做了聚合后，聚合下的几个端口也被捆绑成一个mac地址。 4、编辑/etc/rc.local 编辑/etc/rc.local，在其后内加入如下4行: ifenslave bond0 eth0 eth1 ifenslave bond1 eth2 eth3 3.9.4 重起机器，确认配置 到这时已经配置完毕重新启动机器。重启会看见以下信息就表示配置成功了 ................ Bringing up interface bond0 OK Bringing up interface eth0 OK Bringing up interface eth1 OK Bringing up interface bond1 OK Bringing up interface eth2 OK Bringing up interface eth3 OK ................ 通过查看bond0的工作状态查询能详细的掌握bonding的工作状态 [root@hbyxdnl1 /]## cat /proc/net/bonding/bond0 bonding.c:v2.4.1 (September 15, 2003) 21 Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 0 Up Delay (ms): 0 Down Delay (ms): 0 Multicast Mode: all slaves Slave Interface: eth1 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:0e:7f:25:d9:8a Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:0e:7f:25:d9:8b 3.10 配置存储多路径 本案例中存储设备是HP XP24000高端存储，主机是DL580G5，HBA卡是HP FC1142SR(qlogic芯片)单端口光纤通道卡(2块)，操作系统是RedHat AS5.2 X86_64，内核为kernel-2.6.18-92.1.22(x86_64)。 3.10.1 安装device-mapper包 首先确认系统中是否已经安装了device-mapper工具 [root@hbyxdnl1]# rpm -qa|grep device device-mapper-1.02.24-1.el5.x86_64.rpm device-mapper-event-1.02.24-1.el5.x86_64.rpm device-mapper-multipath-0.4.7-17.el5.x86_64.rpm 22 如果没有或缺少相关的安装包，则安装包，安装命令如下: #rpm –ivh device-mapper-1.02.24-1.el5.x86_64.rpm device-mapper-event-1.02.24-1.el5.x86_64.rpm device-mapper-multipath-0.4.7-17.el5.x86_64.rpm 3.10.2 安装光纤通道卡最新驱动程序 从HP官方网站上下载最新光纤通道卡驱动程序，下载地址如下: ，驱动 程序文件名类似hp_qla2x00-2007-10-05b.tar.gz。 将驱动程序上传到主机上并解包，解包方法如下: gunzip -c < hp_qla2x00-2007-10-05b.tar.gz | tar -xvpf - 文件解包后里有一个INSTALL文件，直接运行即可: #/tmp/hp_qla2x00-2007-10-05b/INSTALL 安装完成后，安装脚本会自动在/etc/modprobe.conf文件里写入如下两条记录 (注意，我这里装的是Qlogic芯卡的HBA卡，并且操作系统是RedHat的Linux， 如果是Emulex芯卡的卡，或操作系统是SUSE那么又不一样，其它模式的设置 见附件1: DeviceMapper Multipath Enablement Kitfor HP StorageWorks Disk Arrays v4.0.2): 1.options qla2xxx ql2xmaxqdepth=16 qlport_down_retry=10 ql2xloginretrycount=30 ql2xfailover=0 ql2xlbType=1 ql2xautorestore=0xa0 ConfigRequired=0 2. 其中ql2xfailover这一项是选择是否打开failover模式，0是关闭，1是 打开，其它各个参数的具体意义见附件2 :QLogic Fibre Channel Driver for Kernel 2.6.x read me。在本项目中我选的是0，即关闭模式，个人理解的是如 果关闭failover模系，系统将会采了loadbalance模式，因为在实测中如果采 用的是failover模式打开的话，访问磁盘会有短暂中断，如果关闭falover模 式，访问磁盘不会有任何中断。 23 安装完成后仔细检查/etc/modprobe.conf文件，确认failover模式是否满 足需要，如果不满足，如即修改，修改完成后执行如下命令，生产新的初始化 内存盘: # /opt/hp/src/hp_qla2x00src/make_initrd 3.10.3 安装HP多路径软件包 从HP官方网站上下载最新的多路径软件包，Multi-path Device Mapper for Linux Software下载网址: ; cc=us&prodNameId=3559652&taskId=135&prodTypeId=18964&prodSeriesId=3559651&submit.y=1&submit.x=9&lang=en&cc=us&displaylang=en 解包方法见上一个小节。 文件解包后里有一个INSTALL文件，直接运行即可: #/tmp/HPDMmultipath-4.1.0 /INSTALL 安装过程中程序将会生成一个新的/etc/multipath.conf覆盖原有的主机自带 的multipath.conf文件。新生产的/etc/multipath.conf文件内容类似下面的格式: ### The Device Mapper Multipath Template configuration file for RHEL5 ### releases to be used with HP Storageworks Arrays. ### Use this configuration file as your /etc/multipath.conf file. ### If you already have a valid working configuration file, refer here ### for the recommended configuration for HP arrays. ### For a list of configuration options with descriptions, please refer ### to /usr/share/doc/HPDMmultipath-tools-/multipath.conf.annotated # The defaults section defaults { udev_dir /dev polling_interval 10 selector "round-robin 0" path_grouping_policy failover getuid_callout "/sbin/scsi_id -g -u -s /block/%n" prio_callout "/bin/true" path_checker tur rr_min_io 100 rr_weight uniform 24 failback immediate no_path_retry 12 user_friendly_names yes } # The blacklist section - use this to blacklist a multipath device based on # it's wwid ( using wwid ) or device names ( using devnode ) or # vendor and product id ( using device block). blacklist { # wwid 26353900f02796769 devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z][[0-9]*]" # devnode "^cciss!c[0-9]d[0-9]*" # device { # vendor "HP" # product "OPEN-*" # } } # The blacklist_exceptions section - to list device names to be treated # as multipath candidates even if they are on the blacklist. # Note: blacklist exceptions are only valid in the same class. # It is not possible to blacklist devices using the devnode keyword # and to exclude some devices of them using the wwid keyword. #blacklist_exceptions { # wwid 2345234245647657 # devnode "sda" # device { # vendor "HP" # product "OPEN-*" # } #} # The multipaths section - uncomment this section to define a per multipath # device settings. #multipaths { # uncomment the multipath block to modify the settings of a multipath device # based on its wwid 25 #multipath { # wwid 2039485769900000000 # alias red # path_grouping_policy group_by_prio # path_selector "round-robin 0" # failback immediate # rr_weight uniform # no_path_retry 10 # rr_min_io 100 #} #} # The devices section - used to define per storage array model settings devices { # If you are already using a valid configuration file and do not have a # device subsection for HP arrays, please add the appropriate device subsection # for the respective arrays from the entries below. # If you already have a device subsection for HP arrays which has different # parameters from the entries below, modify it appropriately. # For EVA A/A arrays device { vendor "HP|COMPAQ" product "HSV1[01]1 \(C\)COMPAQ|HSV[2][01]0|HSV300" path_grouping_policy group_by_prio getuid_callout "/sbin/scsi_id -g -u -s /block/%n" path_checker tur path_selector "round-robin 0" prio_callout "/sbin/mpath_prio_alua /dev/%n" rr_weight uniform failback immediate hardware_handler "0" no_path_retry 12 rr_min_io 100 } # For XP device { vendor "HP" 26 product "OPEN-.*" path_grouping_policy multibus getuid_callout "/sbin/scsi_id -g -u -s /block/%n" path_selector "round-robin 0" rr_weight uniform path_checker tur hardware_handler "0" failback immediate no_path_retry 12 rr_min_io 1000 } # For MSA A/A arrays device { vendor "HP" product "MSA VOLUME*" path_grouping_policy group_by_prio getuid_callout "/sbin/scsi_id -g -u -s /block/%n" path_checker tur path_selector "round-robin 0" prio_callout "/sbin/mpath_prio_alua /dev/%n" rr_weight uniform failback immediate hardware_handler "0" no_path_retry 12 rr_min_io 100 } device { vendor "HP" product "MSA2012sa" path_grouping_policy group_by_prio getuid_callout "/sbin/hp_scsi_id -g -u -n -s /block/%n" path_checker tur path_selector "round-robin 0" prio_callout "/sbin/mpath_prio_alua %d" rr_weight uniform failback immediate hardware_handler "0" no_path_retry 18 rr_min_io 100 } 27 device { vendor "HP" product "MSA2[02]12fc|MSA2012i" path_grouping_policy multibus getuid_callout "/sbin/scsi_id -g -u -s /block/%n" path_selector "round-robin 0" rr_weight uniform path_checker tur hardware_handler "0" failback immediate no_path_retry 18 rr_min_io 100 } # To blacklist a device by vendor and product say, to blacklist a XP device # uncomment the below block #device { # vendor "HP" # product_blacklist "OPEN-.*" #} } 这个文件就是多路径的配置文件，由于是HP出的驱动，所有device里列的都是HP的磁盘阵列，如果你的主机是连的是IBM或EMC又或是其它厂商的阵更也可以加到里面，只要把相关阵列的信息相入即可。 其配置例子文件为: /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.defaults 配置参数解释文档为: /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.annotated 3.10.4 配置多路径磁盘 首先查看磁盘唯一SCSI编码(unique SCSI identifiers)，执行如下命令: [root@hbyxdnl2 ~]# for i in `cat /proc/partitions | awk '{print $4}' |grep sd`; do echo "### $i: `scsi_id -g -u -s /block/$i`"; done ### sda: 360060e80054932000000493200000043 ### sdb: 360060e80054932000000493200000044 ### sdc: 360060e80054932000000493200000045 ### sdd: 360060e8005493200000049320000002d 28 ### sde: 360060e8005493200000049320000002e ### sdf: 360060e80054932000000493200000043 ### sdg: 360060e80054932000000493200000044 ### sdh: 360060e80054932000000493200000045 ### sdi: 360060e8005493200000049320000002d ### sdj: 360060e8005493200000049320000002e ### sdk: 360060e80054932000000493200000043 ### sdl: 360060e80054932000000493200000044 ### sdm: 360060e80054932000000493200000045 ### sdn: 360060e8005493200000049320000002d ### sdo: 360060e8005493200000049320000002e ### sdp: 360060e80054932000000493200000043 ### sdq: 360060e80054932000000493200000044 ### sdr: 360060e80054932000000493200000045 ### sds: 360060e8005493200000049320000002d ### sdt: 360060e8005493200000049320000002e 其次查看各磁盘的实际容量: #fdisk -l 从上述结果中我们可以看到我们5个磁盘唯一SCSI编码，对应20个逻辑磁盘，也就是说实际有5个阵列映射过来的磁盘，每个磁盘有4条路径，根据这个内容我们编写/etc/multipath.conf文件，格式如下: 1.排除掉主机内置的磁盘: 原内容: blacklist { # wwid 26353900f02796769 devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z][[0-9]*]" # devnode "^cciss!c[0-9]d[0-9]*" # device { # vendor "HP" # product "OPEN-*" # } } 修改后内容: blacklist { # wwid 26353900f02796769 devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" devnode "^hd[a-z][[0-9]*]" devnode "^cciss!c[0-9]d[0-9]*" # device { # vendor "HP" # product "OPEN-*" 29 # } } 主要是将 devnode "^cciss!c[0-9]d[0-9]*"这一条前面的注释符册掉，从 面排除主机内置磁盘，不要让它生成多路径盘(主要是没必要)。 2.定义多路径磁盘别名: 原内容: #multipaths { # uncomment the multipath block to modify the settings of a multipath device # based on its wwid #multipath { # wwid 2039485769900000000 # alias red # path_grouping_policy group_by_prio # path_selector "round-robin 0" # failback immediate # rr_weight uniform # no_path_retry 10 # rr_min_io 100 #} #} 修改后内容: multipaths { multipath { wwid 360060e80054932000000493200000043 alias ocrdisk path_grouping_policy group_by_prio path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 10 rr_min_io 100 } multipath { wwid 360060e80054932000000493200000044 alias votedisk path_grouping_policy group_by_prio path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 10 30 rr_min_io 100 } multipath { wwid 360060e80054932000000493200000045 alias spfiledisk path_grouping_policy group_by_prio path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 10 rr_min_io 100 } multipath { wwid 360060e8005493200000049320000002d alias oradisk1 path_grouping_policy group_by_prio path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 10 rr_min_io 100 } multipath { wwid 360060e8005493200000049320000002e alias oradisk2 path_grouping_policy group_by_prio path_selector "round-robin 0" failback immediate rr_weight uniform no_path_retry 10 rr_min_io 100 } } 3.对磁盘进行分区: 分区时要注意，目标磁盘请选择sd开关的磁盘，在本例选择如下标红的5 个磁盘进行分区: ### sda: 360060e80054932000000493200000043 ### sdb: 360060e80054932000000493200000044 ### sdc: 360060e80054932000000493200000045 ### sdd: 360060e8005493200000049320000002d ### sde: 360060e8005493200000049320000002e 31 ### sdf: 360060e80054932000000493200000043 ### sdg: 360060e80054932000000493200000044 ### sdh: 360060e80054932000000493200000045 ### sdi: 360060e8005493200000049320000002d ### sdj: 360060e8005493200000049320000002e ### sdk: 360060e80054932000000493200000043 ### sdl: 360060e80054932000000493200000044 ### sdm: 360060e80054932000000493200000045 ### sdn: 360060e8005493200000049320000002d ### sdo: 360060e8005493200000049320000002e ### sdp: 360060e80054932000000493200000043 ### sdq: 360060e80054932000000493200000044 ### sdr: 360060e80054932000000493200000045 ### sds: 360060e8005493200000049320000002d ### sdt: 360060e8005493200000049320000002e #fdisk /dev/sda 选对主分区，起始块为1，结束块选择系统默认即可。 3.10.5 启动multipathd服务 1.分好区后，执行multipathd命令，启动multipathd服务: #/etc/init.d/multipathd restart 2.开启multipathd服务自动启动功能: #chkconfig --level 0123456 multipathd on 3.重启操作系统(在两台主机上都完成上述配发后): #reboot 系统重启后，你便会在/dev/mapper目录下看到你刚才定义的五个多路径盘和5个分区，共10个。名称类似下面: ocrdisk,orcdiskp1,votedisk,votediskp1,spfiledisk,spfilediskp1,oradisk 1,oradisk1p1,oradisk2,oradisk2p1 并且在/dev下会新生成10个dm-x这样的盘，这个盘在我们装Oracle RAC环境下用不到，不去管它了。 32 3.11 配置裸设备 由于RHEL5 中取消了rawdevices，如果不想将映射命令放到rc.local 中的话，我们就需要将其配置到/etc/udev/rules.d/60-raw.rules文件中，但是实际测试目前在AS5编辑rc.local文件方法最可靠，因此配置/etc/udev/rules.d/60-raw.rules的方法就不介绍了。 编辑/etc/rc.local文件(，添加如下内容: # cat /etc/rc.local #!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local ##### # Oracle Cluster Registry (OCR) devices ##### chown root:dba /dev/mapper/ocr** chmod 664 /dev/mapper/ocr* raw /dev/raw/raw1 /dev/mapper/ocrdiskp1 sleep 2 chown root:dba /dev/raw/raw1 chmod 664 /dev/raw/raw1 ##### # Oracle Cluster Voting disks ##### chown oracle:dba /dev/mapper/vote* chmod 660 /dev/mapper/vote* raw /dev/raw/raw2 /dev/mapper/votediskp1 sleep 2 chown oracle:dba /dev/raw/raw2 chmod 664 /dev/raw/raw2 由于spfile文件可以放到ASM里，因此就不给它配裸设备了 3.12 配置asm 注意:必须以root 身份进行 首先在node1 执行: [root@node1 ~]# /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library 33 driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: oracle Default group to own the driver interface []: dba Start Oracle ASM library driver on boot (y/n) [n]: y Fix permissions of Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: [ OK ] Loading module "oracleasm": [ OK ] Mounting ASMlib driver filesystem: [ OK ] Scanning system for ASM disks: [ OK ] 然后切换至node2 执行 [root@node2 ~]# /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: oracle Default group to own the driver interface []: dba Start Oracle ASM library driver on boot (y/n) [n]: y Fix permissions of Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: [ OK ] Loading module "oracleasm": [ OK ] Mounting ASMlib driver filesystem: [ OK ] Scanning system for ASM disks: [ OK ] 再切换回node1 执行 [root@node1 ~]# /etc/init.d/oracleasm createdisk VOL1 /dev/mapper/ocrdiskp1 Marking disk "/dev/sdd1" as an ASM disk: [ OK ] [root@node1 ~]# /etc/init.d/oracleasm createdisk VOL2 /dev/mapper/votediskp1 Marking disk "/dev/sde1" as an ASM disk: [ OK ] 在node1搜索当前ASM磁盘组: [root@node1 linux-patch]# /etc/init.d/oracleasm scandisks Scanning system for ASM disks: [ OK ] 在node1查看一下当前的磁盘组: [root@node1 ~]# /etc/init.d/oracleasm listdisks VOL1 VOL2 Node2 上也搜索/查看一下当前磁盘组: 34 [root@node2 ~]# /etc/init.d/oracleasm scandisks Scanning system for ASM disks: [ OK ] [root@node2 ~]# /etc/init.d/oracleasm listdisks VOL1 VOL2 Ok,配置完成，准备进入到crs 的安装阶段 4 安装OracleCRS软件 4.1 安装的准备工作 以二进制方式上传安装文件到/oracle/soft目录 解压缩cpio文件 [oracle@hbyxdnl1 soft]$ gunzip 10201_clusterware_linux_x86_64.cpio.gz [oracle@hbyxdnl1 soft]$ cpio -idmv <10201_clusterware_linux_x86_64.cpio clusterware/stage/prereq/crs/crsprereq.jar clusterware/stage/prereq/crs/refhost.xml clusterware/stage/prereq/crs/crs_prereq.xml clusterware/stage/prereq/crs … clusterware/doc clusterware/welcome.html clusterware 641117 blocks 4.2 图形下安装 在第一个节点上图形界面上执行安装: # xhost + # su - oracle $ cd /oracle/soft/clusterware $ ./runInstaller 35 root用户执行 [root@hbyxdnl1 ~]# /oracle/soft/clusterware/rootpre/rootpre.sh No OraCM running [root@hbyxdnl2 ~]# /oracle/rootpre/rootpre.sh No OraCM running 36 单击Next按钮 37 单击Next按钮 38 单击Next按钮 39 单击Next按钮 40 41 单击Next按钮 单击Add按钮，填入节点2的名称 单击OK按钮 42 单击Next按钮 43 单击Edit按钮，把bond0设置成public 单击OK按钮 44 单击Next按钮 45 填入ocr文件的裸设备文件名，单击Next按钮 46 单击Next按钮 47 单击Install按钮 48 开始安装 49 在两个节点分别执行oraInstRoot.sh和root.sh， 注意:这里先不要点OK按钮: 第一台orainstRoot.sh [root@hbyxdnl1 ~]# /oracle/oraInventory/orainstRoot.sh Changing permissions of /oracle/oraInventory to 770. Changing groupname of /oracle/oraInventory to dba. The execution of the script is complete 第二台orainstRoot.sh [root@hbyxdnl2 rootpre]# /oracle/oraInventory/orainstRoot.sh Changing permissions of /oracle/oraInventory to 770. Changing groupname of /oracle/oraInventory to dba. The execution of the script is complete 第一台root.sh [root@hbyxdnl1 ~]# /oracle/product/10.2.0/crs/root.sh WARNING: directory '/oracle/product/10.2.0' is not owned by root WARNING: directory '/oracle/product' is not owned by root WARNING: directory '/oracle' is not owned by root Checking to see if Oracle CRS stack is already configured /etc/oracle does not exist. Creating it now. Setting the permissions on OCR backup directory Setting up NS directories Oracle Cluster Registry configuration upgraded successfully WARNING: directory '/oracle/product/10.2.0' is not owned by root WARNING: directory '/oracle/product' is not owned by root WARNING: directory '/oracle' is not owned by root Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node : node 1: hbyxdnl1 hbyxdnl1-priv hbyxdnl1 node 2: hbyxdnl2 hbyxdnl2-priv hbyxdnl2 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Now formatting voting device: /dev/raw/raw2 Format of 1 voting devices complete. Startup will be queued to init within 90 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. CSS is active on these nodes. hbyxdnl1 CSS is inactive on these nodes. hbyxdnl2 Local node checking complete. Run root.sh on remaining nodes to start CRS daemons. 50 历史报错: Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up NS directories Failed to upgrade Oracle Cluster Registry configuration 这是oracle软件本身的一个bug,参考metalink,bug number: 4679769 解决办法如下: 先安装软件包4679769 # Patch Installation Instructions: # -------------------------------- # To apply the patch, unzip the PSE container file: # p4679769_10201_LINUX.zip # Set your current directory to the directory where the patch # is located: # % cd 4679769 # Copy the clsfmt.bin binary to the $ORA_CRS_HOME/bin directory where # clsfmt is being run: # % cp $ORA_CRS_HOME/bin/clsfmt.bin $ORA_CRS_HOME/bin/clsfmt.bin.bak # % cp clsfmt.bin $ORA_CRS_HOME/bin/clsfmt.bin # Ensure permissions on the clsfmt.bin binary are correct: # % chmod 755 $ORA_CRS_HOME/bin/clsfmt.bin 就是把clsfmt.bin文件覆盖掉就OK了 第二台root.sh [root@hbyxdnl2 ~]# /oracle/product/10.2.0/crs/root.sh WARNING: directory '/oracle/product/10.2.0' is not owned by root WARNING: directory '/oracle/product' is not owned by root WARNING: directory '/oracle' is not owned by root Checking to see if Oracle CRS stack is already configured /etc/oracle does not exist. Creating it now. Setting the permissions on OCR backup directory Setting up NS directories Oracle Cluster Registry configuration upgraded successfully WARNING: directory '/oracle/product/10.2.0' is not owned by root WARNING: directory '/oracle/product' is not owned by root WARNING: directory '/oracle' is not owned by root clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node : node 1: hbyxdnl1 hbyxdnl1-priv hbyxdnl1 node 2: hbyxdnl2 hbyxdnl2-priv hbyxdnl2 clscfg: Arguments check out successfully. 51 NO KEYS WERE WRITTEN. Supply -force parameter to override. -force is destructive and will destroy any previous cluster configuration. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 90 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. CSS is active on these nodes. hbyxdnl1 hbyxdnl2 CSS is active on all nodes. Waiting for the Oracle CRSD and EVMD to start Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps 历史报错: /oracle/product/10.2.0/crs/jdk/jre//bin/java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory 这是由于LD_ASSUME_KERNEL参数设置不正确引起 在所有节点机器的/oracle/product/10.2.0/crs/bin/vipca文件中 找到如下内容: Remove this workaround when the bug 3937317 is fixed arch=`uname -m` if [ "$arch" = "i686" -o "$arch" = "ia64" ] then LD_ASSUME_KERNEL=2.4.19 export LD_ASSUME_KERNEL fi #End workaround 在fi后新添加一行: unset LD_ASSUME_KERNEL 以及srvctl文件 (需要在所有节点上操作) # vi /oracle/product/10.2.0/crs/bin/srvctl 找到如下内容: LD_ASSUME_KERNEL=2.4.19 export LD_ASSUME_KERNEL 同样在其后新增加一行: unset LD_ASSUME_KERNEL 4.3 root执行vipca 在一号机上运行如果马上运行vipca会

报告

软件系统测试报告下载sgs报告如何下载关于路面塌陷情况报告535n,sgs报告怎么下载竣工报告下载

如下错误 [root@hbyxdnl1 ~]# /oracle/product/10.2.0/crs/bin/vipca Error 0(Native: listNetInterfaces:[3]) 52 [Error 0(Native: listNetInterfaces:[3])] 必须先设置 [root@hbyxdnl1 ~]# oifcfg setif -global bond0/10.122.10.128:public [root@hbyxdnl1 ~]# oifcfg setif -global bond1/192.168.10.128:cluster_interconnect [root@hbyxdnl1 ~]# oifcfg getif bond0 10.122.10.128 global public bond1 192.168.10.128 global cluster_interconnect 然后在图形界面下执行 # /oracle/product/10.2.0/crs/bin/vipca 单击Next按钮，选中public网卡bond0 53 单击Next按钮，填入hbyxdnl1-vip和hbyxdnl2-vip其他根据/etc/hosts文件自动完成，注意 子网掩码的设置与public ip一致。 54 单击Next按钮 单击Finish按钮 完成以后单击OK按钮 55 56 回到crs安装界面继续OK按钮 57 安装检查成功 58 单击Exit完成oracle CRS安装 5 安装Oracle数据库软件 在hbyxdnl1节点上 执行/oracle/soft/database/runInstaller.sh 59 单击下一步 选择安装企业版数据库，单击下一步 60 确定安装路径，单击下一步 选中集群内所有节点，单击下一步 61 单击下一步 选择仅安装数据库软件，单击下一步 62 单击安装 63 此时安装向导将提示你以root用户的身份在所有节点上执行上述脚本。执 行效果如下: Running Oracle10 root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /app/oracle/product/10.2.0/db Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 执行完毕后单击确定 64 单击退出，完成数据库软件的安装。 6 安装Oracle10.2.0.4补丁 6.1 安装oracle CRS补丁 停止相关数据库服务 [oracle@hbyxdnl1 ~]$ emctl stop dbconsole TZ set to PRC OC4J Configuration issue. /oracle/product/10.2.0/db/oc4j/j2ee/OC4J_DBConsole_hbyxdnl1_bdhbyxdnl1 not found. [oracle@hbyxdnl1 ~]$ isqlplusctl stop iSQL*Plus 10.2.0.1.0 Copyright (c) 2003, 2005, Oracle. All rights reserved. getnameinfo failed getnameinfo failed getnameinfo failed iSQL*Plus instance on port 5560 is not running ... [root@hbyxdnl1 ~]# srvctl stop nodeapps -n hbyxdnl1 CRS-0210: Could not find resource 'ora.emshbyxdnl1.LISTENER_HBYXDNL1.lsnr'. [root@hbyxdnl1 ~]# srvctl stop nodeapps -n hbyxdnl2 65 CRS-0210: Could not find resource 'ora.emshbyxdnl2.LISTENER_HBYXDNL2.lsnr'. [root@hbyxdnl1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....is1.gsd application OFFLINE OFFLINE ora....is1.ons application OFFLINE OFFLINE ora....is1.vip application OFFLINE OFFLINE ora....is2.gsd application OFFLINE OFFLINE ora....is2.ons application OFFLINE OFFLINE ora....is2.vip application OFFLINE OFFLINE oracle用户执行/oracle/soft/patch10203/Disk1/runInstaller 单击Next按钮，选择crs的目录 66 单击Next按钮，集群安装 单击Next按钮 67 单击Install按钮 68 完成时需要各节点执行脚本 执行脚本 [root@hbyxdnl1 ~]# /oracle/product/10.2.0/crs/bin/crsctl stop crs Stopping resources. Successfully stopped CRS resources Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@hbyxdnl1 ~]# /oracle/product/10.2.0/crs/install/root102.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /oracle/product/10.2.0/crs Relinking some shared libraries. Relinking of patched files is complete. WARNING: directory '/oracle/product/10.2.0' is not owned by root WARNING: directory '/oracle/product' is not owned by root WARNING: directory '/oracle' is not owned by root Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds. Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . 69 10203 patch successfully applied. clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node : node 1: hbyxdnl1 hbyxdnl1-priv hbyxdnl1 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully 6.2 安装Oracle数据库补丁 停止相关数据库服务 [oracle@hbyxdnl1 ~]$ emctl stop dbconsole TZ set to PRC OC4J Configuration issue. /oracle/product/10.2.0/db/oc4j/j2ee/OC4J_DBConsole_hbyxdnl1_bdhbyxdnl1 not found. [oracle@hbyxdnl1 ~]$ isqlplusctl stop iSQL*Plus 10.2.0.1.0 Copyright (c) 2003, 2005, Oracle. All rights reserved. getnameinfo failed getnameinfo failed getnameinfo failed iSQL*Plus instance on port 5560 is not running ... [root@hbyxdnl1 ~]# srvctl stop nodeapps -n hbyxdnl1 CRS-0210: Could not find resource 'ora.emshbyxdnl1.LISTENER_HBYXDNL1.lsnr'. [root@hbyxdnl1 ~]# srvctl stop nodeapps -n hbyxdnl2 CRS-0210: Could not find resource 'ora.emshbyxdnl2.LISTENER_HBYXDNL2.lsnr'. [root@hbyxdnl1 ~]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....is1.gsd application OFFLINE OFFLINE ora....is1.ons application OFFLINE OFFLINE ora....is1.vip application OFFLINE OFFLINE ora....is2.gsd application OFFLINE OFFLINE ora....is2.ons application OFFLINE OFFLINE ora....is2.vip application OFFLINE OFFLINE oracle用户执行/oracle/soft/patch10203/Disk1/runInstaller 70 单击Next按钮，选择数据库软件的目录 单击Next按钮，集群安装 71 单击Next按钮 单击Install按钮 72 安装完成检查确认 73 完成时需要各节点执行脚本 第一个节点 第二个节点 74 7 创建数据库 创建/oracle/dbca.conf文件，内容如下: 7.1 dbca配置ASM 7.1.1 先配置ASM lib 先安装ASM所需要的软件包: 75 开始配置ASM设备: 输入oracle用户和dba组，其他选项默认 76 开始扫面ASM磁盘 77 执行如下脚本，把设备信息同步到另外一个节点 /etc/init.d/oracleasm scandisks /etc/init.d/oracleasm listdisks 78 在第二个节点执行 /etc/init.d/oracleasm scandisks 至此ASM设备配置完成 79 7.1.2 Dbca配置ASM 单击Next按钮，单击Select All按钮 80 单击Next按钮 单击Next按钮 81 单击Next按钮，输入spfile所对应的裸设备名称:/dev/raw/raw3 82 单击Next按钮 83 点OK，然后next 84 此处报错是因为没有创建监听，点YES，会自动创建监听，然后点击next 85 出现如下图: 86 点finish,出现可供选择的设备名称，选择raw4提前规划好的raw5，并选择external(已经做 好冗余): 87 选择OK，开始配置ASM 88 点finish配置完成，并做下一部操作，开始建库: 89 90 7.1.3 Dbca创建数据库 单击Next按钮，单击Select All按钮 91 单击Next按钮，选择Custom Database 输入SID:dnl 92 93 输入用户名和口令 94 选择ASM建库方式 95 选择RAC_DISK ASM组 96 默认选项 97 默认选项 98 99 默认选项 100 创建 service name:dnlcrs 101 102 103 字符集修改为:UTF8 104 连接模式为默认: 105 Processes数量修改为2500 106 SGA为系统默认，可以后根据需要修改 107 删除已存在的redo log groups,重新创建共12个，分别为两个实例所用，大小为512M 108 点FINISH开始创建数据库 109 110 确认参数设置，点OK，开始建库 111 112 建库完成，不需要修改密码，点EXIT退出即可 113 会自动把CRS所有资源启动 安装完成 8 附件1 Release Notes Device Mapper Multipath Enablement Kit for HP 114 StorageWorks Disk Arrays v4.1.0 Part number: AA-RWF9E-TE First edition: November 2008 Legal and notice information ? Copyright 2008 Hewlett-Packard Development Company, L.P. Overview HPDM Multipath software kit is a HP released multipath enablement kit for HP StorageWorks Disk Arrays, based on Device Mapper Multipath included in Linux operating system distributions. This release notes discusses the recent product information about HP Device Mapper Multipath (HPDM Multipath) Enablement Kit for HP StorageWorks Disk Arrays v4.1.0. Device Mapper Multipath offers the following features: • I/O failover and failback: Provides transparent failover and failback of I/Os by rerouting I/Os automatically to an alternative path when a path failure is sensed and routing them back when the path is restored. • Path grouping policies: Paths are coalesced based on the following path-grouping policies: 1. Priority based path-grouping • Providespriority to group paths based on Asymmetric Logical Unit Access (ALUA) state • Provides static load balancing policy by assigning higher priority to the preferred path 2. Multibus • All paths are grouped under a single path group 3. Group by serial • Paths are grouped together based on controller serial number 4. Failover only • Provides failover without load balancing by grouping the paths into individual path groups • I/O load balancing policies: Provides weighted Round Robin load balancing policy within a path group. • Device name persistence: Device names are persistent across reboots and Storage Area Network (SAN) reconfigurations. Device Mapper also provides configurable device name aliasing feature for easier management. • Persistent device settings: All the device settings such as load balancing policies, path grouping policies are persistent across reboots and SAN reconfigurations. • Device exclusion: Provides device exclusion feature through blacklisting of devices. • Path monitoring: Periodically monitors each path for status and enables faster failover and failback. • Online device addition and deletion: Devices can be added to or deleted from Device Mapper 115 (DM) multipath without rebooting the server or disrupting other devices or applications. • Management Utility: Provides Command Line Interface (CLI) to manage multipath devices. • Boot from SAN: Provides multipathing for operating system installation partitions on SAN device. NOTE: Multipathing for SAN Boot environment is supported on the following operating systems: RHEL 5 U1/RHEL 5 U2/SLES 10 SP1/SLES 10 SP2. For more details on multipathing support for SAN Boot environment, see the Booting Linux x86 and x86_64 systems from a Storage Area Network with Device Mapper Multipath document available at What’s new HPDM Multipath v4.1.0 provides the following additional features: • Provides utilities and device parameter values to enable multipathing for HP BladeSystem external SAS Storage (MSA2012sa) Disk Arrays. • Provides multipath support for Xen enabled virtual machines on RHEL 5 U2 operating system. • Provides multipath support with data replication solutions, Business Copy and Continuous Access for HP StorageWorks Enterprise Virtual Arrays and XP Disk Arrays. Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays v4.1.0 3 • Provides support for multipathing in Oracle 10g environment. NOTE: For operating systems supported with HP StorageWorks Disk Arrays, refer the support matrix section in the respective Disk Array manual. Device Mapper Multipath support matrix Table 1 lists the hardware and software prerequisites for installing HPDM Multipath. Table 1 Hardware and software prerequisites System feature Supported hardware and software Operating system versions RHEL 4 Update 6 RHEL 4 Update 7 RHEL 5 Update 1 RHEL 5 Update 2 SLES 9 - SP3 errata 2.6.5–7.283 or later SLES 9 - SP4 SLES 10 - SP1 SLES 10 - SP2 116 Host Bus Adapters (HBA) SAN Switches See , , and DriverDownload.jsp?lang=en&cc=us&prodNameId=3628653& taskId=135&prodTypeId=332283&prodSeriesId=3628652&submit.y=2& submit.x=5&lang=en&cc=us Servers HP BladeSystem c-Class Server Blades, ProLiant x86, ProLiant AMD64, ProLiant EM64T Servers, Integrity Servers Supported arrays EVA 3000 (HSV101) VCS 4.007 or later EVA 5000 (HSV111) VCS 4.007 or later EVA 4000 (HSV200) XCS 5.110/6.100 or later EVA 6000 (HSV200) XCS 5.110/6.100 or later EVA 8000 (HSV210) XCS 5.110/6.100 or later EVA 4100 (HSV200) XCS 6.110 or later EVA 6100 (HSV200) XCS 6.110 or later EVA 8100 (HSV210) XCS 6.110 or later EVA 4400 (HSV300) XCS 0900 or later EVA iSCSI Connectivity Option XP10000 fw rev 50-07-30-00/00 or later XP12000 fw rev 50-09-34-00/00 or later XP 20000 fw rev 60-02-04-00/00 or later XP 24000 fw rev 60-02-04-00/00 or later MSA 1000/MSA 1500 fw rev 7.0.0 or later MSA 2000 Storage product family (MSA2012fc/MSA2212fc) fw rev J200P19 or later MSA2012i fw rev J210R10 or later MSA2012sa fw rev J300P10–01 or later 4 Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays HBA drivers and Smart Array Controller drivers HP SC08Ge Host Bus Adapter: 4.00.13.04-2 or later (for RHEL 5 U1/SLES 10 SP1), 3.12.14.00-2 or later (for RHEL 4 U6/ SLES 9 SP4) available at: ;cc=us&prodNameId=3759720&taskId=135& prodTypeId=329290&prodSeriesId=3759718&lang=en&cc=us Emulex: 8.0.16.40 or later (for RHEL 4 U7), 8.2.0.22 or later (for SLES 10 SP2/RHEL 5 U2), 8.0.16.32 or later (for SLES 9/RHEL 4 U6), 8.1.10.11 or later (for RHEL 5 U1/SLES 10 SP1) available at: products/storageworks/4gbpciehba/index.html Qlogic: 8.02.11 or later (for RHEL 4 U7), 8.02.11 or later (for SLES 10 117 SP2/RHEL 5 U2), 8.01.07.25 or later (for SLES 9/ SLES 10 SP1/ RHEL 4 U6/ RHEL 5 U1) available at: storageworks/fca2214/index.html HP Smart Array P700m Controller: TechSupport/DriverDownload.jsp?lang=en&cc=us&prodNameId=3628653& taskId=135&prodTypeId=332283&prodSeriesId=3628652&submit.y=2& submit.x=5&lang=en&cc=us NOTE: HPDM Multipath provides support for iSCSI devices on the following operating systems: RHEL 5 U1/RHEL 5 U2/SLES 9 SP3/SLES 9 SP4/SLES 10 SP1/SLES 10 SP2. Installing Device Mapper Multipath tools Ensure the following RPMs bundled with the operating system distributions are installed on the system: • For RHEL 4 Update 7: device-mapper-1.02.25-2.el4 or later, device-mapper-multipath-0.4.5- 31.el4 or later • For RHEL 5 Update 2: device-mapper-1.02.24-1.el5 or later, device-mapper-multipath-0.4.7- 17.el5 or later • For SLES 10 SP2: device-mapper-devel-1.02.13-6.14 or later, device-mapper-1.02.13-6.14 or later NOTE: For RHEL 4 U6/RHEL 5 U1/SLES 9 SP3/SLES 10 SP1 operating systems, install HP Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays v4.0.0 and then install HPDM Multipath v4.1.0. For more information on installing HPDM Multipath v4.0.0, see the Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays 4.0.0 Installation and Reference Guide. Prerequisites for SLES 10 SP2 Ensure the following RPMs are installed on SLES 10 SP2: readline-devel sysfsutils aaa_base acl attr audit-libs autoconf automake bash bind-libs bind-utils binutils bison brp-check-internal bzip2 coreutils cpio cpp cracklib cvs cyrus-sasl db device-mapper device-mapper-devel diffutils e2fsprogs file filesystem fillup findutils flex gawk gcc gdbm gdbm-devel gettext gettext-devel glibc glibc-32bit glibc-devel glibc-locale gpm grep groff gzip info insserv klogd less libacl libattr libcom_err libgcc libgssapi libmudflap libnscd libstdc++ libtool libxcrypt libzio m4 make man mktemp module-init-tools ncurses ncurses-devel net-tools netcfg openldap2-client openssl pam pam-modules patch perl permissions 118 Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays v4.1.0 5 popt post-build-checks procinfo procps psmisc pwdutils rcs readline readline-devel rpm sed strace sysfsutils sysvinit tar tcpd texinfo timezone unzip util-linux vim zlib zlib-devel Prerequisites for SLES 9 SP4 Ensure the following RPMs are installed on SLES 9 SP4: aaa_base acl attr bash bind-utils bison bzip2 coreutils cpio cpp cracklib cvs cyrus-sasl db devs diffutils e2fsprogs file filesystem fillup findutils flex gawk gdbm-devel glibc glibc-devel glibc-locale gpm grep groff gzip info insserv kbd less libacl libattr libgcc libselinux libstdc++ libxcrypt m4 make man mktemp module-init-tools ncurses ncurses-devel net-tools netcfg openldap2-client openssl pam pam-modules patch permissions popt procinfo procps psmisc pwdutils rcs readline sed strace syslogd sysvinit tar tcpd texinfo timezone unzip util-linux vim zlib zlib-devel autoconf automake binutils device-mapper gcc gdbm gettext libtool perl readline-devel rpm termcap udev Installing HPDM Multipath Enablement kit 4.1.0 To install HPDM Multipath 4.1.0, complete the following steps: 1. Download the HPDM Multipath Enablement Kit for HP StorageWorks Disk Arrays v4.1.0 available at 2. Log in as root to the host system. 3. Copy the installation tar package to a temporary directory (for instance, /tmp/HPDMmultipath). 4. Unbundle the package by executing the following commands: #cd /tmp/HPDMmultipath #tar -xvzf HPDMmultipath-4.1.0.tar.gz #cd HPDMmultipath-4.1.0 5. Verify that the directory contains README.txt, COPYING, INSTALL, bin, conf, SRPMS, and docs directories. 6. To install HPDM Multipath v4.1.0, execute the following command: #./INSTALL Configuring Device Mapper Multipath to enable HP arrays This section describes the following: • Recommended device parameter values • Setting up HPDM Multipath • Setting up Device Mapper Multipath daemon Recommended device parameter values To enable the HP arrays, edit /etc/multipath.conf file by adding the following under devices 119 section: 6 Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays For EVA3000 / EVA5000 / EVA4x00/ EVA6x00 / EVA8x00 device { vendor "(COMPAQ|HP)" product "HSV1[01]1|HSV2[01]0|HSV300" getuid_callout "/sbin/scsi_id -g -u -s /block/%n" prio_callout "/sbin/mpath_prio_alua /dev/%n" hardware_handler "0" path_selector "round-robin 0" path_grouping_policy group_by_prio failback immediate rr_weight uniform no_path_retry 12 rr_min_io 100 path_checker tur } For MSA2012fc/MSA2212fc/MSA2012i device { vendor "HP" product "MSA2[02]12fc|MSA2012i" getuid_callout "/sbin/scsi_id -g -u -s /block/%n" hardware_handler "0" path_selector "round-robin 0" path_grouping_policy multibus failback immediate rr_weight uniform no_path_retry 18 rr_min_io 100 path_checker tur } For MSA1000/MSA1500 device { vendor "HP" product "MSA VOLUME*" getuid_callout "/sbin/scsi_id -g -u -s /block/%n" prio_callout "/sbin/mpath_prio_alua /dev/%n" hardware_handler "0" path_selector "round-robin 0" path_grouping_policy group_by_prio failback immediate 120 rr_weight uniform no_path_retry 12 rr_min_io 100 path_checker tur } Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays v4.1.0 7 For MSA2012sa device { vendor "HP" product "MSA2012sa" getuid_callout "/sbin/hp_scsi_id -g -u —n —s /block/%n" prio_callout "/sbin/mpath_prio_alua %d" hardware_handler "0" path_selector "round-robin 0" path_grouping_policy group_by_prio failback immediate rr_weight uniform no_path_retry 18 rr_min_io 100 path_checker tur } For XP device { vendor "HP" product "OPEN-.*" getuid_callout "/sbin/scsi_id -g -u -s /block/%n" hardware_handler "0" path_selector "round-robin 0" path_grouping_policy multibus failback immediate rr_weight uniform no_path_retry 12 rr_min_io 1000 path_checker tur } NOTE: • For SLES 10 SP2, in the device section, replace prio_callout ―/sbin/mpath_prio_alua %d‖ with prio alua • For SLES 10 SP2, in the device section for MSA 2012sa, replace getuid_callout ―/sbin/hp_scsi_id —g —u —n —s /block/%n‖ 121 with getuid_callout ―/sbin/scsi_id —g —u —n —s /block/%n‖ • For more information on editing /etc/multipath.conf, see the Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays 4.0.0 Installation and Reference Guide. You can find this document on the Manuals page of Multi-path Device Mapper for Linux Software, which is accessible at 8 Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays Setting up HPDM Multipath Setting up HPDM Multipath includes configuring HBA parameters for multipathed environment. This involves the following: • Configuring QLogic HBA parameters • Configuring Emulex HBA parameters Configuring QLogic HBA parameters To configure the QLogic HBA parameters for QLogic 2xxx family of HBAs, complete the following steps: 1. Edit the /etc/modprobe.conf file in RHEL hosts and /etc/modprobe.conf.local file in SLES hosts with the following values: options qla2xxx ql2xmaxqdepth=16 qlport_down_retry=10 ql2xloginretrycount=30 ql2xfailover=0 ql2xlbType=1 ql2xautorestore=0xa0 ConfigRequired=0 2. Rebuild the initrd by executing the following script: /opt/hp/src/hp_qla2x00src/make_initrd 3. Reboot the host. Configuring Emulex HBA parameters To configure the Emulex HBA parameters, complete the following steps: 1. For Emulex lpfc family of HBAs: • InRHEL4hosts,edit the/etc/modprobe.conf file with the following values: options lpfc lpfc_nodev_tmo=14 lpfc_lun_queue_depth=16 lpfc_discovery_threads=32 • InSLES9hosts,edit the/etc/modprobe.conf.local file with the following values: options lpfc lpfc_nodev_tmo=14 lpfc_lun_queue_depth=16 lpfc_discovery_threads=32 • InSLES10hosts,edit the/etc/modprobe.conf.local file with the following values: options lpfc lpfc_nodev_tmo=28 lpfc_lun_queue_depth=16 lpfc_discovery_threads=32 122 • InRHEL5hosts,edit the/etc/modprobe.conf file with the following values: options lpfc lpfc_nodev_tmo=28 lpfc_lun_queue_depth=16 lpfc_discovery_threads=32 2. Rebuild the initrd by executing the following script: /opt/hp/hp-lpfc/make_initrd 3. Reboot the host. Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays v4.1.0 9 NOTE: Ensure you have a HBA driver installed in single path mode. For more information, see the installation and reference guides for the respective HBA drivers. Setting up Device Mapper Multipath daemon You must set the Device Mapper Multipath daemon to start at boot time. For RHEL hosts, complete the following steps to start the multipathd daemon at boot time: 1. Run the following command to check if the daemon is configured to start at boot time: # chkconfig --list multipathd 2. Run the following commands to start the Device Mapper Multipath daemon: # chkconfig [--level levels] multipathd on # chkconfig multipathd For SLES hosts, complete the following steps to start the multipathd daemon at boot time: 1. Run the following commands to check if the daemon is configured to start at boot time: # chkconfig --list boot.device-mapper # chkconfig --list boot.multipath # chkconfig --list multipathd 2. Run the following commands to start the Device Mapper Multipath daemons: # chkconfig boot.device-mapper [levels] # chkconfig boot.multipath [levels] Known issues Following are the known issues in the HPDM Multipath 4.1.0 release: • Path failure messages are seen in the log file when encountered with a Unit Attention condition. This behavior could be observed during online LUN addition, deletion, LUN transition across the controller. These paths would be recovered after the polling interval set. • multipath commands may take longer time to execute on heavily loaded servers or under path failure conditions. • User friendly multipath device names may change on reboot if /var is mounted on a separate partition other than the root file system. It is recommended to have /(root) and /var on the same partition. • Blacklisting the multipath device in the file /etc/multipath.conf and restarting the multipath 123 service may not remove the device on RHEL 4 distributions. Execute the following command to remove the blacklisted device: # multipath -f • Using fdisk command to create partitions may fail to create Multipath device for the partition device. It is recommended to use parted command to create partitions for the device. • multipathd daemon crashes on systems configured with device paths more than the system open file limits (default system open file limit=1024). It is recommended to change the system open file limits by using either the ’max_fds’ parameter in /etc/multipath.conf file or by using the ulimit -n command and restart the multipathd demon. • Multipath -l command may not reflect the correct path status for Logical Units presented from MSA2012sa array when paths fail or are restored under heavy load conditions. To refresh the path status, execute the # multipath -v0 command. • Multipath devices may not be created for Logical Units when the system disks or internal controllers are cciss devices. It is recommended to blacklist these devices in the /etc/multipath.conf file and restart the multipathd daemon. 10 Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays Support Telephone numbers for worldwide technical support are listed on the HP support website: Collect the following information before calling the worldwide technical support: • Technical support registration number (if applicable) • Product serial numbers • Product model names and numbers • Applicable error messages • Operating system type and revision level • Detailed, specific questions For continuous quality improvement, calls may be recorded or monitored. HP recommends that customers sign up online using the Subscriber’s choice website: • Subscribing to this service provides you with e-mail updates on the latest product enhancements, newer versions of drivers, and firmware documentation updates as well as instant access to numerous other product resources. • After signing up, you can locate your products by selecting Business support > Storage under Product Category. Device Mapper Multipath Enablement Kit for HP StorageWorks Disk Arrays v4.1.0 11 124 9 附件2 Fibre Channel Driver for Kernel 2.6.x This software license applies only to QLogic customers. QLogic Corporation. All rights reserved. Table of Contents 1. OS Support 2. Supported Features 3. Release History 4. Saving the Driver Source to Diskette 5. Installing the Driver 5.1 Building the Driver from the Source 5.2 Load the Driver Manually Using insmod or modprobe 5.3 Making a RAMDISK Image to Load the Driver 6. Driver Parameters 6.1 NVRAM Parameters 6.2 Driver Command Line Parameters 7. SNIA API Library Package 7.1 Installing SNIA API Library 7.2 Uninstalling SNIA API Library 8. Additional Notes 8.1 Failover Support 8.2 Persistent Binding 8.3 Configuration Data 8.4 How to Force Rescan of Devices 8.5 Booting from SAN 8.6 Compilation Messages 8.7 Dynamically Modifying SCSI Black-list Entries 9. IP Support 125 9.1 Load the Driver Manually Using insmod or modprobe 9.2 Configuring Interfaces 10. Contacting Support 1. OS Support This driver supports 2Gb and 4Gb HBAs. Refer to the release notes for a list of kernel versions and different distributions tested with this release. 2. Supported Features , FCAL - direct attach loop , Point-to-point , Fabric support , Initiator mode only , Fault recovery on down loops , Persistent binding , Extended LUN support up to 255 LUNs , FC tape support , Non Failover and Failover capability 3. Release History Please refer to Release Notes provided in the package (release.txt). 4. Saving the Driver Source to Diskette 1. Download the qla2xxx-vx.yy.zz-dist.tgz file from QLogic's web-site. 2. If prompted "What would you like to do with this file?" select Save this file to disk. 3. Insert a blank diskette and download to the diskette directly. 5. Installing the Driver This section makes extensive use of the build.sh script located in driver source (extras/build.sh). This script currently supports driver compilation (installation and updates) on SLES9 and RHEL4 distributions on four main hardware platforms (x86, x86_64, ia64 and ppc64). The build.sh script supports for following directives: # ./extras/build.sh Build the driver sources based on the standard SLES9/RHEL4 build environment. # ./extras/build.sh clean Clean driver source directory of all build files (i.e. *.ko, *.o, etc). 126 # ./extras/build.sh new Rebuild the driver sources from scratch. This is essentially a shortcut for: # ./build.sh clean # ./build.sh # ./extras/build.sh install Build and install the driver module files. This command performs the following: 1. Builds the driver .ko files. 2. Copies the .ko files to the appropriate /lib/modules/... directory. 3. Adds the appropriate directive in the modprobe.conf[.local] to remove the qla2xxx_conf module when the qla2xxx modules in unloaded. 4. Updates the newly built qla2xxx_conf.ko module with any previously saved data in /etc/qla2xxx.conf. # ./extras/build.sh initrd 5.1 Building a Driver from the Source Code From the source code, you can build a qla2xxx.ko and a qla2200.ko, qla2300.ko, qla2322.ko, qla6312.ko, or qla2400.ko for your host system, and load the driver manually or automatically using a RAMDISK image during system boot time. 1. Ensure the appropriate build environment is present on the system. For example, for SLES9, the kernel-headers and kernel-sources RPM files are required. # cd /mnt/cdrom/SuSE/RPMS (#cd /mnt/RedHat/RPMS for RHEL 4.) # rpm -ivh kernel-source*.rpm # rpm -ivh kernel-syms*.rpm RHEL 4 does not require the kernel source to be installed to build the qla2xxx modules. 2. Using the diskette you created in Section 4, copy the qla2xxx-vx.yy.zz-dist.tgz file to /qla2x00. Follow these steps from the "/" (root) directory: # mkdir qla2x00 # cd qla2x00 # mount /mnt/floppy # cp /mnt/floppy/*.[bz2|gz] . (the period at the end is required) # tar -xvzf *.tgz # cd qlogic # ./drvrsetup (this will extract the source files directory in to the current directory) # cd qla2xxx-x.yy.zz 3. Build the Driver modules from the source code by executing the build.sh script. # ./extras/build.sh 127 4. To load the driver manually, see section 5.2. To make a RAMDISK image to load the driver during system boot time, see section 5.3. 5.2 Load the Driver Manually using insmod or modprobe Before loading the driver manually, first build the driver binary from the driver source files as described in section 5.1. 1. To load the driver directly from the local build directory, load the driver in following in order: # insmod qla2xxx_conf.ko or # insmod qla2xxx.ko or # insmod qla2300.ko 2. To load the driver using modprobe, install the driver modules (*.ko) files to the appropriate kernel module directory: # ./extras/build.sh install 3. Type the following to load the driver: # modprobe -v qla2300 (QLA246X/QLE246X, QLA23XX) or, # modprobe -v qla2322 (QLE236X) or, # modprobe -v qla2400 (QLA24XX/QLE24XX) NOTE: The modprobe -v qla2300 command will automatically load the qla2xxx.ko and qla2xxx_conf components. 4. To unload the driver using modprobe: # modprobe -r qla2300 This will unload qla2300.ko and qla2xxx.ko modules. # modprobe -r qla2xxx_conf This will unload qla2xxx_conf.ko 5.3 Making a RAMDISK Image to Load the Driver To build a RAMDISK image on RHEL4 and SLES9, follow the steps listed in the following subsections: , 5.3.1 RHEL 4 RAMDISK Image , 5.3.2 SLES 9 RAMDISK Image 5.3.1 RHEL 4 RAMDISK Image To build the RAMDISK image perform the following steps: 1. Follow the steps in section 5.1 2. Install the driver modules (*.ko) files to the appropriate kernel module directory: # ./extras/build.sh install 3. Edit the /etc/modprobe.conf file and add the following entries: alias scsi_hostadapter2 qla2xxx_conf (SANsurfer use only) alias scsi_hostadapter3 qla2322 (QLE236X) or qla2300 (QLA234X) or 128 alias scsi_hostadapter4 qla2400 (QLA24XX) 4. Change to the /boot directory. 5. Take backup of current RAMDISK image: # cp -f initrd-2.6.[kernel_version].img initrd-2.6.[kernel_version].img.bak 6. Build the RAMDISK image by executing the following command: # mkinitrd -f initrd-2.6.[kernel_version].img [kernel_version] 7. Reboot the system to load the RAMDISK image with the QLogic driver. 5.3.2 SLES 9 RAMDISK Image To build the RAMDISK image on SLES 9 perform the following steps: 1. Follow the steps in section 5.1 2. Install the driver modules (*.ko) files to the appropriate kernel module directory: # ./extras/build.sh install 3. Edit the /etc/sysconfig/kernel file and modify the INITRD_MODULES directive. The first modules must be qla2xxx_conf (for SANSurfer) followed by qla2xxx. After qla2xxx, modules specific to the HBA installed in the system should be added (see Notes below). ... INITRD_MODULES=".... qla2xxx_conf qla2xxx qla2300 qla2322 qla6312 qla2400" ... Notes: qla2xxx_conf: SANsurfer use only qla2xxx: Common module qla2300: For QLA234X qla2322: For QLE236X qla2400: For QLA24XX qla6312: For QLA2XX 4. Change to the /boot directory. 5. Take backup of current RAMDISK image: # cp -f initrd-2.6.[kernel_version] initrd-2.6.[kernel_version].bak 6. Build the RAMDISK image by executing the following command: # /sbin/mk_initrd 7. Reboot the system to load the RAMDISK image with the QLogic driver. 6. Driver Parameters The Driver parameters are divided into System Parameters and NVRAM Parameters sections. 6.1 NVRAM Parameters 129 The NVRAM features described below are hard-coded in the Driver. The changes made for the particular NVRAM feature in the Fast!Util do not take effect unless otherwise noted. 6.2 Driver Command Line Parameters The driver gets its parameters from the command line itself or from modprobe option directive found in the modprobe.conf[.local] file. The parameters are in simple =value format, i.e. ql2xfailover=1. Where is one of the following option parameters: Usage: insmod qla2xxx.ko =value Parameter Description Values This parameter defines whether the extended_error_logging driver will print verbose logging 0 to disable; 1 to enable information. This parameter defines whether ql2xfailover 0 to disable; 1 to enable; Default: 1 failover mode is enable or disable. . This parameter defines the maximum queue depth reported to ql2xmaxqdepth Scsi Mid-Level per device. The Default: 32 Queue depth specifies the number of outstanding requests per lun. This parameter defines the Login ql2xlogintimeout timeout value in seconds during the Default: 20 seconds initial login. This parameter defines how long to wait for a port that returns a Default: 0 (use value specified in qlport_down_retry PORT-DOWN status before returning NVRAM) I/O back to the OS. This parameter defines the maximum number of Scsi Default: 20 (standard mode value), ql2xretrycount Mid-Level retries allowed per 30 (failover mode value) command. This parameter defines whether to 0 (don't display the configuration) displayConfig display the current configuration. 1(display the configuration) 0 (bind by Portname); 1 (bind by This parameter defines what target Bind PortID); Default: 0 (portname persistent binding method to use. binding) This parameter defines how to bind 0 (present all the devices ConfigRequired the devices. discovered to the OS); 1 (present 130 only the configured devices, such as the device defined in /etc/qla2xxx.conf) to the OS; Default: 0 This parameter defines maximum MaxPathsPerDevice Default: 8 (compile time only) number of paths to a device. This parameter defines how many retries to perform on the current path MaxRetriesPerPath Default: 3 before failing over to the next path in the path list. This parameter defines total retries to do before failing the command Default: MaxRetriesPerPath * MaxRetriesPerIo and returning to the OS with MaxPathsPerDevice ) + 1 selection timeout (DID_NO_CONNECT). This parameter defines type of failover notification mechanism to use when a failover or failback QlFailoverNotifyType Default: 0 (none) occurs. Certain storage systems require special CDBs to be issued to do failover or failback. This parameter defines the delay in seconds before a failback is FailbackTime Default: 5 seconds performed to ensure all paths are available. This parameter defines the recovery time in seconds required before RecoveryTime Default: 10 seconds commands can be sent to the restored path. This parameter enables or disables the logic that restores the previous failed preferred path and/or controller for a given lun. This option toggles the default state. Combine one or more of the ql2xautorestore Default: 0 following model numbers into an inclusion mask: 0x80 - MSA A/A (auto-restore disabled) 0x20 - HSV111, HSV101, HSV200, HSV210 (auto-restore disabled) 131 0x10 - DSXXX (auto-restore disabled) 0x04 - HSV110, HSV100 (auto-restore disabled) 0x02 - MSA1000 (auto-restore disabled) 0x01 - XP (auto-restore enabled) This parameter defines the load Balance Method for the driver to use either static or dynamic: 0 (None) - Expose luns on the first active path and make them the preferred path or the first active optimize path and make them the preferred path (storages: MSA A/A and EVA A/A). ql2xlbType 1 (Static load balancing) - Distribute Default: 0 and expose the LUNs across the active optimize port(s) or active un-optimize port(s) and HBA(s). 2 (Least outstanding I/O) - send command to the path with the lowest I/O count. 3 (Least Service time) - Send request to the path with the shortest execution time. This parameter excludes device models from being a failover capable target. Combine one or more of the following model numbers into an exclusion mask: 0x80 - MSA A/A ql2xexcludemodel Default: 0 0x20 - HSV111, HSV101, HSV200, HSV210 0x10 - DSXXX 0x04 - HSV110, HSV100 0x02 - MSA1000 0x01 - XP Enable/Disable target level grouping 1 - Enable target level grouping emulation. This option is necessary emulation; 0 - Disable target level ql2xtgtemul for the GUI to work correctly if the grouping emulation; driver is set for lun level grouping of Default: Enable paths by lunid. The following 132 storages uses this method of combining paths: HSV210, DSXXX, HSV110, MSA1000, XP. NOTE: For a comprehensive list of parameters enter the following command: # /sbin/modinfo qla2xxx.ko 7. SNIA API Library Package The SNIA API library is distributed (qlapi--rel.tgz) as part of driver combo package qla2xxx-vx.yy.zz-dist.tgz, which also includes the Source RPM package. Using the diskette you created in Section 4, copy the distribution file qla2xxx-vx.yy.zz-dist.tgz to /qla2x00. Follow these steps from the "/" (root) directory: # mkdir qla2x00 # cd qla2x00 # mount /mnt/floppy # cp /mnt/floppy/*.tgz . (the period at the end is required) # tar -xvzf *.tgz # cd qlogic 7.1 Installing SNIA API Library Type the following command in current directory to install/setup API library: # ./libinstall (this installs/sets up HBA API library) 7.2 Uninstalling SNIA API Library Type the following command in current directory to remove API library: # ./libremove (Script file to remove HBA API library) 8. Additional Notes 8.1 Failover Support 8.1.1 How to Disable The Failover Support In The Driver Failover support can be disabled in the qla2xxx driver by using the ql2xfailover module parameter: # insmod qla2xxx.ko ql2xfailover=0 ; insmod qla2300.ko To disable in modprobe.conf add the following after the driver: # options qla2xxx ql2xfailover=0 133 NOTE: Failover is enabled by default when the 8.x driver is built. 8.1.2 Configuration Changes Made via SANsurfer LUN Masking NOTE: The Linux SCSI mid-layer requires communication to a target via LUN 0. Therefore, the driver will not mask Lun 0 if it is so defined to be by the application. For the new LUN masking configuration to take effect, the driver must be reloaded. The following is an example of the sequence of actions to take: 1. Load the driver: # modprobe qla2300 2. Load the qlremote agent. 3. Start the GUI and connect it to the destination system. 4. Make LUN masking changes. 5. Disconnect the host from GUI and stop qlremote agent. 6. Unload the driver: modprobe -r qla2300 ; modprobe -r qla2xx_conf 7. Reload the driver: # modprobe qla2300 8. Load qlremote agent again. 9. Start the GUI and connect it to the destination system. 10. Now you should see the updated LUN masking configuration. 8.2 Persistent Binding The Persistent Binding information consists of some adapter configuration entries along with some target entries. Persistent Binding can be specified in two ways: Manually or using SANsurfer. We recommend using SANsurfer for ease of use. The following is the procedure to manually add persistent binding commands. The driver displays the current configuration when the displayConfig command line option is specified. The persistent binding configuration is found in /var/log/messages file. It prints the configuration information in the format required by the driver. The best way to extract configuration messages is to use grep and direct the output to a file. You need to remove the Linux timestamp at the beginning of each message and combine them together on single line. For example: # insmod qla2300.ko displayConfig=1 # grep "scsi-qla" /var/log/messages > /tmp/info.cfg 134 The format of the persistent binding commands is as follows: Device descriptions: scsi-qla<#>-adapter-port=; The designated by qla<#>, where the <#> is the adapter instance number. The parameter specifies the FC port name to be used for the adapter. where is the FC port name value in hexa-decimal format. If this entry is not specified in the conf file, the default value is the adapter's port name as saved in the NVRAM. Example: scsi-qla0-adapter-port=210000e08b01158d\; host adapter instance 0 has a portname of 210000e08b01158d scsi-qla<#1>-tgt-<#2>-di-<#3>-node=; This parameter associates the specified with the SCSI target ID value specified by <#2> and a device id value specified by <#3> where is the SCSI target ID FC name> to be assigned to the device and <#3> is the device unique id. Where: <#1> Specifies the adapter instance number <#2> Specifies the SCSI ID of Target <#3> Specifies the path/device id scsi-qla<#1>-tgt-<#2>-di-<#3>-port=; This parameter associates the specified with the SCSI target ID value specified by <#2> and a device id value specified by <#3> where Where: <#1> Specifies the adapter instance number <#2> Specifies the SCSI ID of Target <#3> Specifies the path/device id (always 0 for non-failover) scsi-qla<#1>-tgt-<#2>-di-<#3>-disabled=<256 bit mask>; This parameter associates the specified <256 bit mask> with the SCSI target ID value specified by <#2> and a device id value specified by <#3>. Where: <#1> Specifies the adapter instance number <#2> Specifies the SCSI ID of Target 135 <#3> Specifies the path/device id <256 bit mask> msb lsb 000000000000000000000000000000000000000000000000000000000000000F The mask above makes the first four luns, 3, 2, 1, and 0 of a given Target disabled on that target/path. This mask specification is heavily type checked to be a sequence of 64 hex digits. 8.3 Configuration Data Configuration/persistent data is loaded in the driver automatically when the driver is installed and loaded. Normally this information is passed to the driver via the command line, but due to the constraints inherent in using the command line we have provide an alternate method using QLA_OPTS. 8.3.1 QLA_OPTS QLA_OPTS reads the configuration data from qla2xxx.conf and updates the binary module qla2xxx_conf.ko. The driver automatically tries to load the binary module qla2xxx_conf.ko at driver initialization time. Once loaded, the module passes the configuration information directly to the driver. The configuration data and be found in /etc/qla2xxx.conf. NOTE: Approximately 300K of configuration space has been pre-allocated within the qla2200_conf/qla2300_conf module for configuration/persistent data. 8.3.2 Compatibility with SANsurfer FC Manager QLA_OPTS works seamlessly with updated the SANsurfer FC Manager. All appropriate configuration data is written to the _conf.ko module. 8.3.3 Updating Configuration Manually Manually updating the /etc/qla2xxx.conf file is strongly discouraged. Because it can cause the binary module and the configuration file to get out of sync. However, in the event that some persistent/configuration value needs to be changed manually, first change the /etc/qla2xxx.conf file, then run the following command in the driver's build directory to update the configuration module: 136 # ./extras/build.sh install 8.4 How to Force Rescan Of Devices This software driver version allows you to force the driver to re-scan devices, which enables you to add a new device. This triggers the driver to initiate lun discovery process. To force a rescan, follow these steps: 1. Enter the following text from the command line: # echo "scsi-qlascan" > /proc/scsi// (qlogic driver re-scans) where can be either one: qla2100/qla2200/qla2300 is the instance number of the HBA. 2. Force the scsi mid-layer to do its own scan and build the device table entry for the new device: # echo "scsi add-single-device 0 1 2 3" >/proc/scsi/scsi (scsi mid layer re-scans) where 0 1 2 3 is replaced by your Host Channel Id Lun. 8.5 Booting from SAN 8.5.1 Creating a Driver Disk (DD Kit) image The driver disk (DD Kit) image enables to install the Red Hat 4/SLES 9 OS and QLogic driver on the Fibre Channel boot disk connected to the QLogic Host Bus adapters. To build the driver disk image perform the following steps. (Example RHEL4-U1 IA-32): 1. Download 8.x_linux_driver_ddkit_for_redhat_suse_dist.tgz from the QLogic website (). 2. Untar the tgz file: # tar -xvzf *.tgz 3. Type: # dd if=qla2xxx-8.01.00-x86-dd-RHEL4-U1.img of=/dev/fd0 bs=1440k (Requires a 1.44MB Floppy) NOTE: For 2Gb HBAs (QLA23xx) the embedded driver may also be used for the initial installation then updated to the latest driver. 8.5.2 RHEL 4 - Using the Driver Disk (DD Kit) image NOTE: For QLA246X/QLE246X the dd disk must be used for Boot from SAN. 137 1. At the boot prompt boot:, type linux dd. 2. Continue the OS installation as normal. 8.5.3 RHEL 4 - Boot from SAN installation on non-LUN 0 LUN The following procedure applies to the RHEL 4 distribution only. When installing to a LUN other than LUN 0 and LUN 0 is present the following steps must be taken in order to successfully boot from the LUN. 1. At the Boot Loader Configuration screen, select Configure Advance Boot Loader Option and click the Next button. 2. At the Advanced Boot Loader Configuration screen, click the Change Driver Order button. 3. At the Edit Drive Order window move the entry for the boot LUN to the top of the list. 4. Click OK to continue. 5. Continue the OS installation as normal. 8.6 Compilation Messages During driver compilation, you may see a message similar to the following: warning: 'inter_module_*' is deprecated (declared at /usr/src/...) This message is non-fatal and does not interfere with the functionality of the driver. 8.7 Dynamically Modifying SCSI Black-list Entries On 2.6.x kernels you can dynamically change the SCSI blacklist either by writing to a /proc entry or using the scsi_mod moduled parameter (to be persistent across reboot). This requires the SCSI Vendor/Model information for the SCSI device, which you can find in /proc/scsi/scsi. Black list entries are in the form: vendor:model:flags[,v:m:f] where flags can be the following integer values: 0x001 /*?Only scan LUN 0 */ 0x002 /* Known to have LUNs, force scanning, deprecated: Use max_luns=N */ 0x004 /* Flag for broken handshaking */ 0x008 /* unlock by special command */ 0x010 /* Do not use LUNs in parallel */ 0x020 /* Buggy Tagged Command Queuing */ 0x040 /* Non consecutive LUN numbering */ -- -> value need to be passed to "flags" variable for sparse lun 0x080 /* Avoid LUNS >= 5 */ 138 0x100 /* Treat as (removable) CD-ROM */ 0x200 /* LUNs past 7 on a SCSI-2 device */ 0x400 /* override additional length field */ 0x800 /* ... for broken inquiry responses */ 0x1000 /* do not do automatic start on add */ 0x2000 /* do not send ms page 0x08 */ 0x4000 /* do not send ms page 0x3f */ 0x8000 /* use 10 byte ms before 6 byte ms */ 0x10000 /* 192 byte ms page 0x3f request */ 0x20000 /* try REPORT_LUNS even for SCSI-2 devs (if HBA supports more than 8 LUNs) */ 0x40000 /* don't try REPORT_LUNS scan (SCSI-3 devs) */ 00000 /* device is actually for RAID config */ 0x200000 /* select without ATN */ 0x400000 /* retry HARDWARE_ERROR */ For example: # echo ::040 > /proc/scsi/device_info To enable persistence across reboots, you need to add the following line: options scsi_mod dev_flags=:: to the following file (based on distribution): /etc/modprobe.conf for RHEL 4.0 and above and /etc/modprobe.conf.local for SLES 9 and above and rebuild the ramdisk. Please refer to section 5.3 on how to build ramdisk. 9. IP Support 9.1 Load the Driver Manually using insmod or modprobe Before loading the driver manually, first build the driver binary from the driver source files as described in section 5.1. To load the driver directly from the local build directory, type the following in order: # insmod qla2xxx_conf.ko # insmod qla2xxx.ko # insmod qla2300.ko # insmod qla2xip.ko To load the driver using modprobe: 1. Install the driver modules (*.ko) files to the appropriate kernel module directory: # ./extras/build.sh install 2. Type the following to load the driver for qla23xx HBAs: 139 # modprobe -v qla2xxx_conf # modprobe -v qla2300 # modprobe -v qla2xip The modprobe -v qla2300 command will automatically load qla2xxx.ko the component. To unload the driver using modprobe: # modprobe -r qla2xip # modprobe -r qla2300 This will unload qla2300.ko and qla2xxx.ko modules. # modprobe -r qla2xxx_conf This will unload qla2xxx_conf.ko. The qla2xip driver will create a network-interface binding to each IP-capable recognized HBA. Binding entries can be viewed from the messages file after the IP driver has loaded: qla2xip: QLogic IP via Fibre Channel Network Driver qla2xip: Driver Version 1.0b2, Entry point: e08e5060 qla2xip: Mapping interface fc0 to HBA 210100e08b20a15b qla2xip: Mapping interface fc1 to HBA 210200e08b40a25b 9.2 Configuring Interfaces The following is for 2Gb HBAs only (QLA23xx/QLE236x). You must configure the network interfaces to allow TCP/IP applications running on the host to communicate with other IP-capable QLogic HBAs. Compile the following basic host information to allow the IP driver to pass TCP/IP data over QLogic HBAs in your host: , Interface name: fc0 (From above) , IP Address: 192.168.1.x (A non-routable address) Where 'x' is a unique number between (1 and 254) , Netmask: 255.255.255.0 (Standard class C mask) , Interface name: fc1 (From above) , IP Address: 192.168.2.x (A non-routable address) Where 'x' is a unique number between (1 and 254) , Netmask: 255.255.255.0 (Standard class C mask) Use the ifconfig program to configure an interface with the compiled host information: # ifconfig fc0 192.168.1.x up 140 Configure other interfaces (if multiple IP-capable HBAs are present in the host): # ifconfig fc1 192.168.1.x up Verify the configured interfaces: # ifconfig Output similar to the following should be displayed for the newly configured interfaces: fc0 Link encap:Ethernet HWaddr 00:E0:8B:20:A1:5B inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MTU:4096 Metric:1 RX packets:1214577458 errors:0 dropped:0 overruns:0 frame:0 TX packets:1214213174 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:32 RX bytes:3081095492 (2938.3 Mb) TX bytes:2751945609 (2624.4 Mb) fc1 Link encap:Ethernet HWaddr 00:E0:8B:40:A2:5B inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MTU:4096 Metric:1 RX packets:1204464697 errors:0 dropped:0 overruns:0 frame:0 TX packets:1194873236 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:32 RX bytes:1454694706 (1387.3 Mb) TX bytes:991094469 (945.1 Mb) Configuration is now complete. Please verify that basic networking is possible between two hosts connected via two HBAs by using a simple ping command: # ping 192.168.1.2 For additional information concerning Linux networking, please read the Linux Networking HOWTO available from . 10. Contacting Support Please feel free to contact your QLogic approved reseller or QLogic Technical Support at any phase of integration for assistance. QLogic Technical Support can be reached by the following methods: Web: North America Contact Information Email: support@qlogic.com Phone: (952) 932-4040 Support contact information for other regions of the world is available at the 141 QLogic website: Go to Top ? Copyright 2006. All rights reserved worldwide. QLogic, the QLogic logo, and the Powered by QLogic logo are registered trademarks of QLogic Corporation. All other brand and product names are trademarks or registered trademarks of their respective owners. 10 Linux下集群管理方法 10.1 停具体某一个或几个服务: $srvctl stop databse -d racdb $srvctl stop nodeapps -n rac1 10.2 停掉集群管理的所有服务: $$ORA_CRS_HOME/bin/crs_stop -all 10.3 停掉集群: $$ORA_CRS_HOME/bin/crsctl stop crs 10.4 关闭集群自动启动服务 $$ORA_CRS_HOME/bin/crsctl disable crs 142 10.5 Linux IP Bonding 配置笔记 (2009-12-19 15:24:36) 转载 标签: 分类: Linux 10.5.1 杂谈 11 操作系统:CentOS Linux 5(Kernel:2.6.18-53.el5) 网卡说明:双网卡(非同品牌型号)名字分别为eth0、eth1 网络设置:192.168.1.201/24 11.1 2.1 相关文件 /etc/sysconfig/network-scripts/ifcfg-bond0 /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 /etc/modprobe.conf /etc/rc.local /proc/net/bonding/bond0 11.2 2.2 开始配置 1.备份eth0、eth1配置文件 # cd /etc/sysconfig/network-scripts # cp ifcfg-eth0 bak.ifcfg-eth0 143 # cp ifcfg-eth1 bak.ifcfg-eth1 2.建立ifcfg-bond0 # vi ifcfg-bond0 加入如下内容: DEVICE=bond0 ONBOOT=yes BOOTPROTO=none BROADCAST=192.168.1.255 IPADDR=192.168.1.201 NETMASK=255.255.255.0 GATEWAY=192.168.1.1 3.编辑ifcfg-eth0 # vi ifcfg-eth0 加入如下内容覆盖原有内容: DEVICE=eth0 TYPE=Ethernet ONBOOT=yes BOOTPROTO=none MASTER=bond0 SLAVE=yes HWADDR=00:16:EC:AE:07:54 144 4.编辑ifcfg-eth1 # vi ifcfg-eth1 加入如下内容覆盖原有内容: DEVICE=eth1 TYPE=Ethernet ONBOOT=yes BOOTPROTO=none MASTER=bond0 SLAVE=yes HWADDR= 00:50:BA:0C:65:02 5.编辑modprobe.conf # cd /et c # vi modprobe.conf 追加如下内容: alias bond0 bonding options bond0 miimon=100 mode=1 说明: miimon表示链路检查间隔，单位为毫秒;mode的值表示工作模式，他共有0、1、2、3、4、 5、6七种模式， 0模式:负载均衡模式，RR方式，全负载均衡，需要Switch侧做Trunk，可失效一网卡; 1模式:热备模式，不需要Switch侧支持; 2模式:负载均衡模式，XOR方式，根据来源MAC; 3模式:广播模式，所有网卡一起收发数据包，可失效一网卡; 145 4模式:802.3ad模式，需要Switch支持802.3ad Dynamic Link Aggregation; 5模式:负载均衡模式，TLB方式，半负载均衡，发送为负载均衡，接收为动态分配，不需要Switch侧支持，可失效一网卡; 6模式:负载均衡模式，ALB方式，全负载均衡，网卡需支持动态更改MAC，不需要Switch侧支持，可失效一网卡; 常用的为0,1两种。 6.编辑rc.local # vi rc.local 追加如下内容: ifenslave bond0 eth0 eth1 route add -net 192.168.1.0 netmask 255.255.255.0 bond0 12 3 结果测试 重启服务器，在开机加载服务有如下提示输出则说明模块成功加载。 Bringing up interface bond0 [ OK ] Bringing up interface eth0 [ OK ] Bringing up interface eth1 [ OK ] 12.1 3.1 查看IP配置 输出信息应该与下方的输出类似 # ifconfig -a bond0 Link encap:Ethernet HWaddr 00:16:EC:AE:07:54 146 inet addr:192.168.1.201 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::216:ecff:feae:754/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:10452933 errors:0 dropped:0 overruns:0 frame.:0 TX packets:2245803 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2313966546 (2.1 GiB) TX bytes:1240415537 (1.1 GiB) eth0 Link encap:Ethernet HWaddr 00:16:EC:AE:07:54 inet6 addr: fe80::216:ecff:feae:754/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:10353385 errors:0 dropped:0 overruns:0 frame.:0 TX packets:2245519 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2305237371 (2.1 GiB) TX bytes:1240369619 (1.1 GiB) Interrupt:225 Base address:0xc800 eth1 Link encap:Ethernet HWaddr 00:16:EC:AE:07:54 inet6 addr: fe80::216:ecff:feae:754/64 Scope:Link UP BROADCAST SLAVE MULTICAST MTU:1500 Metric:1 RX packets:99554 errors:0 dropped:0 overruns:0 frame.:0 TX packets:291 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:8729599 (8.3 MiB) TX bytes:47044 (45.9 KiB) 147 Interrupt:217 Base address:0xac00 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:12133 errors:0 dropped:0 overruns:0 frame.:0 TX packets:12133 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:18922380 (18.0 MiB) TX bytes:18922380 (18.0 MiB) 12.2 3.2 网关连通状态 # ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. 64 bytes from 192.168.1.1: icmp_seq=1 ttl=255 time=0.608 ms 64 bytes from 192.168.1.1: icmp_seq=2 ttl=255 time=0.593 ms 64 bytes from 192.168.1.1: icmp_seq=3 ttl=255 time=0.800 ms 64 bytes from 192.168.1.1: icmp_seq=4 ttl=255 time=0.515 ms --- 192.168.1.1 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3000ms rtt min/avg/max/mdev = 0.515/0.629/0.800/0.104 ms 148 12.3 3.3 查看bond0状态 # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.1.2 (January 20, 2007) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up Link Failure Count: 1 Permanent HW addr: 00:16:ec:ae:07:54 Slave Interface: eth1 MII Status: down Link Failure Count: 2 Permanent HW addr: 00:50:ba:0c:65:02 149 12.4 3.4 Bond迁移状态 在同一网段用一台机器持续ping Bonding主机，然后拔掉eth0的网线，同时使用其他网络接入服务。 # ping 192.168.1.201 Pinging 192.168.1.201 with 32 bytes of data: Reply from 192.168.1.201: bytes=32 time<1ms TTL=64 Reply from 192.168.1.201: bytes=32 time<1ms TTL=64 Reply from 192.168.1.201: bytes=32 time<1ms TTL=64 Reply from 192.168.1.201: bytes=32 time<1ms TTL=64 Reply from 192.168.1.201: bytes=32 time=3ms TTL=64 Reply from 192.168.1.201: bytes=32 time<1ms TTL=64 Reply from 192.168.1.201: bytes=32 time<1ms TTL=64 Reply from 192.168.1.201: bytes=32 time<1ms TTL=64 Reply from 192.168.1.201: bytes=32 time<1ms TTL=64 Ping statistics for 192.168.1.201: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms # 在ping的同时开一个SSH登录到主机，并输入# top，可以看到中间只是顿了一下就恢复了，并没有断;查看bond0状态，发现Currently Active Slave已经切换到eth1。 150 # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.1.2 (January 20, 2007) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: eth1 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: down Link Failure Count: 1 Permanent HW addr: 00:16:ec:ae:07:54 Slave Interface: eth1 MII Status: up Link Failure Count: 2 Permanent HW addr: 00:50:ba:0c:65:02 eth0网线插回去后，bonding并不会及时的将工作网卡切换到eth0，而是等到eth1链路失 效后才会切换回eth0 再拔掉eth1的网线查看bond0状态会发现Currently Active Slave已经切换回eth0 经过测试，实验成功～ 151 转自: 152