Error in invoking target 'agent nmhs' of makefile

11g 설치시 나오는 에러

Error in invoking target 'agent nmhs' of makefile ....

이라고 에러가 나오는 경우.


Oracle 11g R2를 Install하는 동안 “Error in invoking target 'agent nmhs' of makefile - during 11g Installation”과 같은 오류가 보고되는 경우 그냥 무시를 하고 진행해도 되지만 “$ emctl status dbconsole” 또는 오라클 에이젼트가 오작동을 일으키는 경우가 있습니다.


위와 같이 오류가 보고되는 순간 계속 진행하지 말고,


1. “$ORACLE_HOME/sysman/lib/ins_emagent.mk”을 vi로 Open


2. “$(MK_EMAGENT_NMECTL)”를 찾아서 “$(MK_EMAGENT_NMECTL) -lnnz11” 로 수정


3. 인스톨로 “Retry”를 선택하여 무시하지 말고 계속 진행.

리눅스7 버전에서 11g grid 설치시 root.sh 실패

root.sh 실행시 

아래와 같이 나오며 설치 실패 되는 경우


[root@localhost /]# /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

LOCAL ADD MODE

Creating OCR keys for user ‘oracle’, privgrp ‘oinstall’..

Operation successful.

LOCAL ONLY MODE

Successfully accumulated necessary OCR keys.

Creating OCR keys for user ‘root’, privgrp ‘root’..

Operation successful.

CRS-4664: Node localhost successfully pinned.

Adding Clusterware entries to inittab

ohasd failed to start

Failed to start the Clusterware. Last 20 lines of the alert log follow:

2016-01-01 02:14:46.806:

[client(11401)]CRS-2101:The OLR was formatted using version 3.

2016-01-01 02:14:49.572:

[client(11424)]CRS-1001:The OCR was formatted using version 3.


ohasd failed to start at /u01/app/11.2.0/grid/crs/install/roothas.pl line 377, line 4.




해결 방안


1. deinstall 해서 설치를 삭제


[root@localhost deinstall]# su – oracle

Last login: Fri Jan 1 02:17:02 EST 2016 on pts/1

[oracle@localhost ~]$ cd /u01/app/11.2.0/grid/deinstall

[oracle@localhost deinstall]$ ./deinstall


Checking for required files and bootstrapping …

Please wait …

Location of logs /tmp/deinstall2016-01-01_02-30-16AM/logs/


############ ORACLE DEINSTALL & DECONFIG TOOL START ############

######################### CHECK OPERATION START ###########################

[START] Install check configuration ##


Checking for existence of the Oracle home location /u01/app/11.2.0/grid

Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Standalone Server

Oracle Base selected for deinstall is: /u01/app/oracle

Checking for existence of central inventory location /u01/app/oraInventory

Checking for existence of the Oracle Grid Infrastructure home /u01/app/11.2.0/grid

Checking for sufficient temp space availability on node(s) : ‘localhost.localdomain’


## [END] Install check configuration ##

Traces log file: /tmp/deinstall2016-01-01_02-30-16AM/logs//crsdc.log

Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16A/logs/netdc_check2016-01-01_02-30-40-AM.log

Network Configuration check config END

Asm Check Configuration START


ASM de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16AM/logs/asmcadc_check2016-01-01_02-30-41-AM.log

ASM configuration was not detected in this Oracle home. Was ASM configured in this Oracle home (y|n) [n]: n

ASM was not detected in the Oracle Home


######################### CHECK OPERATION END #########################

####################### CHECK OPERATION SUMMARY #######################

Oracle Grid Infrastructure Home is: /u01/app/11.2.0/grid

The cluster node(s) on which the Oracle home deinstallation will be performed are:null

Oracle Home selected for deinstall is: /u01/app/11.2.0/grid

Inventory Location where the Oracle home registered is: /u01/app/oraInventory

ASM was not detected in the Oracle Home

Do you want to continue (y – yes, n – no)? [n]: y

A log of this session will be written to: ‘/tmp/deinstall2016-01-01_02-30-16AM/logs/deinstall_deconfig2016-01-01_02-30-35-AM.out’

Any error messages from this session will be written to: ‘/tmp/deinstall2016-01-01_02-30-16AM/logs/deinstall_deconfig2016-01-01_02-30-35-AM.err’


######################## CLEAN OPERATION START ########################

ASM de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16AM/logs/asmcadc_clean2016-01-01_02-30-57-AM.log

ASM Clean Configuration END


Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2016-01-01_02-30-16A/logs/netdc_clean2016-01-01_02-30-57-AM.log

De-configuring backup files…

Backup files de-configured successfully.


The network configuration has been cleaned up successfully.

Network Configuration clean config END


—————————————->


Run the following command as the root user or the administrator on node “localhost”.


/tmp/deinstall2016-01-01_02-30-16AM/perl/bin/perl -I/tmp/deinstall2016-01-01_02-30-16AM/perl/lib -I/tmp/deinstall2016-01-01_02-30-16AM/crs/install /tmp/deinstall2016-01-01_02-30-16AM/crs/install/roothas.pl -force -deconfig -paramfile “/tmp/deinstall2016-01-01_02-30-16AM/response/deinstall_Ora11g_gridinfrahome1.rsp”


Press Enter after you finish running the above commands


<————————————-


This is the point where you have to execute the command highlighted and execute it from root user and wait for it to complete in order to deconfigure the Oracle Restart Stack.


[root@localhost ~]# /tmp/deinstall2016-01-01_02-30-16AM/perl/bin/perl -I/tmp/deinstall2016-01-01_02-30-16AM/perl/lib -I/tmp/deinstall2016-01-01_02-30-16AM/crs/install /tmp/deinstall2016-01-01_02-30-16AM/crs/install/roothas.pl -force -deconfig -paramfile “/tmp/deinstall2016-01-01_02-30-16AM/response/deinstall_Ora11g_gridinfrahome1.rsp”

Using configuration parameter file: /tmp/deinstall2016-01-01_02-30-16AM/response/deinstall_Ora11g_gridinfrahome1.rsp

CRS-2613: Could not find resource ‘ora.cssd’.

CRS-4000: Command Stop failed, or completed with errors.

CRS-2613: Could not find resource ‘ora.cssd’.

CRS-4000: Command Delete failed, or completed with errors.

CRS-4133: Oracle High Availability Services has been stopped.

Successfully deconfigured Oracle Restart stack



#########################

Setting the force flag to false

Setting the force flag to cleanup the Oracle Base

Oracle Universal Installer clean START


Detach Oracle home ‘/u01/app/11.2.0/grid’ from the central inventory on the local node : Done

Delete directory ‘/u01/app/11.2.0/grid’ on the local node : Done

Delete directory ‘/u01/app/oraInventory’ on the local node : Done

Delete directory ‘/u01/app/oracle’ on the local node : Done

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory ‘/tmp/deinstall2016-01-01_02-30-16AM’ on node ‘localhost’

## [END] Oracle install clean ##

######################### CLEAN OPERATION END #########################

####################### CLEAN OPERATION SUMMARY #######################

Oracle Restart was already stopped and de-configured on node “localhost”

Oracle Restart is stopped and de-configured successfully.

Successfully detached Oracle home ‘/u01/app/11.2.0/grid’ from the central inventory on the local node.

Successfully deleted directory ‘/u01/app/11.2.0/grid’ on the local node.

Successfully deleted directory ‘/u01/app/oraInventory’ on the local node.

Successfully deleted directory ‘/u01/app/oracle’ on the local node.

Oracle Universal Installer cleanup was successful.


Run ‘rm -rf /etc/oraInst.loc’ as root on node(s) ‘localhost’ at the end of the session.

Run ‘rm -rf /opt/ORCLfmap’ as root on node(s) ‘localhost’ at the end of the session.

Run ‘rm -rf /etc/oratab’ as root on node(s) ‘localhost’ at the end of the session.

Oracle deinstall tool successfully cleaned up temporary directories.

#######################################################################

############# ORACLE DEINSTALL & DECONFIG TOOL END #############


2. Oracle Support 홈페이지 (metalink)에서 패치 18370031 를 받는다.


3. Grid를 재설치 하고 root.sh 실행 바로전에 OPatch 18370031을 패치 한다.


 $ opatch napply -local ./18370031


4. 패치가 끝나면 root.sh 실행

Oracle 11g 11.2.0.4 RAC ASM 설치 (Last Ver.) - 3. DB 구축

ASMCA 로 ASM 디스크 그룹 구성




DB 설치전에 db_env 로딩


$ db_env

$ cd ../database

$ ./runInstaller



[root@rac2 ~]# /app/oracle/product/11.2.0.4/db_1/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /app/oracle/product/11.2.0.4/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.


엔진 설치 완료.



DBCA로 db 구성


RAC 노드 2개 모두 선택


EM을 설치


Datafile DATA 그룹에


FRA 는 RECO 영역에 넣음.


acfs 모듈을 불러오지 않은 상태기 때문에 아카이브는 패치 후에 지정.


필요 없는 옵션은 해제


필요한 프로세스 값과 캐릭터셋을 선택


DB 구성 완료.

Oracle 11g 11.2.0.4 RAC ASM 설치 (Last Ver.) - 2. Grid 설치

설치에 필요한 파일은 총 5개


Database 설치파일

p13390677_112040_Linux-x86-64_1of7.zip

p13390677_112040_Linux-x86-64_2of7.zip


Grid 설치 파일

p13390677_112040_Linux-x86-64_3of7.zip


OPatch 최신파일

p6880880_112000_Linux-x86-64.zip


DBPSU+GRID Patchset

p23615403_112040_Linux-x86-64


※ 11g 부터는 archive log dest를 공유 스토리지에 저장을 해야하는데, AIX나 HP 같은 UNIX자체적인 공유 파일 시스템을 갖추고 있다. 그래서 gpfs나 hacmp를 통해 공유 파일시스템에 파일 형식으로 archive를 저장 하는데 반해, 리눅스는 그것이 되질 않는다.

예전에 오라클에서 제공한 방식인 ocfs2는 장시간 사용 문제가 발생한다.

그래서 ASM 스토리지에 직접 Archive를 저장하거나 아니면 ASM에서 acfs 모듈을 띄워서 공유 파일시스템을 만든후 저장한다.

Oracle Linux 최신 커널에서 11.2.0.4 기본 설치파일에 내장되어 있는 acfs 모듈이 버전이 낮아 로딩이 되지 않는 관계로 acfs를 사용하려면 최신 패치까지 다 씌워야 한다.



GUI 환경에서 작업한다. (콘솔 or X Manager)


1번에 설치 파일을 올리고 오라클 계정으로 접속.


# su - oracle


콘솔 로그인시 프로파일을 못불러오는 경우가 있으니 강제로 로딩


$ cd ~

$ . .bash_profile

$ grid_env


Grid를 설치하기 위해 grid_env를 로딩


설치 파일이 있는 곳으로 가서

$ ./runInstaller


아래 순서대로 진행 하면 된다.


Add 버튼을 눌러 rac2를 추가해주고, SSH COnnectivity 버튼을 눌러 두 노드간의 SSH 통신을 구축한다.


비번만 넣고 setup


이렇게 에러 메세지가 나오지만 그냥 무시하고 설치 하면 된다.

아무런 영향 없다. (오라클 공식 메뉴얼에도 이렇게 나옴)



rac1 /app/oraInventory/orainstRoot.sh

rac2 /app/oraInventory/orainstRoot.sh

rac1 /app/grid/11.2.0.4/grid.root.sh

rac2 /app/grid/11.2.0.4/grid.root.sh


위 순서대로 반드시 Root 계정에서 실행 해준다.


[root@rac1 ~]# /app/oraInventory/orainstRoot.sh
Changing permissions of /app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac1 ~]# /app/grid/11.2.0.4/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /app/grid/11.2.0.4/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /app/grid/11.2.0.4/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

ASM created and started successfully.

Disk Group OCRVOTE created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 278b92ca45434f2ebfe4ba8720932ef9.
Successfully replaced voting disk group with +OCRVOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   278b92ca45434f2ebfe4ba8720932ef9 (/dev/asm-ocrvote01) [OCRVOTE]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.OCRVOTE.dg' on 'rac1'
CRS-2676: Start of 'ora.OCRVOTE.dg' on 'rac1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded


[root@rac2 ~]# /app/oraInventory/orainstRoot.sh
Changing permissions of /app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac2 ~]# /app/grid/11.2.0.4/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /app/grid/11.2.0.4/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /app/grid/11.2.0.4/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded


Grid 설치 완료.


맨 마지막에 100% 되고 무슨무슨 verify 에러 뜨는데 그냥 무시하고 next 누르면 된다.



사족 - 설치시 그냥 ASM 그룹 하나만 만들어서 거기에 OCRVOTE, DATA, FRA, Archive 다 넣어도 되긴한다.

그래도 구분해두는 것이 나중에 교체작업이나 ASM 관리하는데 있어서 좀 더 편하다고 생각한다.



Oracle 11g 11.2.0.4 RAC ASM 설치 (Last Ver.) - 1. 서버 세팅

오라클 리눅스 6 버전 64bit 기준 (영문)

 - 영문으로 설치하는 이유: 리눅스의 한글 시스템은 UTF-8 이기 때문에 EUC-KR로 DB를 세팅할 경우 로그에 나오는 한글이 전부 깨짐는 현상 발생. UTF-8로 세팅할 경우 한글로 설치 해도 무방.


★ 주의 - 오라클 11g 11.2.0.4 RAC 설치시 서버의 HOSTNAME이 대문자 일경우 인스톨러에서 SSH 통신이 되지 않아서 설치가 불가능하다. 


설치 환경 VMWare 12

네트워크는 Bridged 와 Host Only

HDD 120GB (/, swap, /app)



리눅스 설치시 설치 패키지


    Base System > Base
    Base System > Client management tools
    Base System > Compatibility libraries
    Base System > Hardware monitoring utilities
    Base System > Large Systems Performance
    Base System > Network file system client
    Base System > Performance Tools
    Base System > Perl Support
    Servers > Server Platform
    Servers > System administration tools
    Desktops > Desktop
    Desktops > Desktop Platform
    Desktops > Fonts
    Desktops > General Purpose Desktop
    Desktops > Graphical Administration Tools
    Desktops > Input Methods
    Desktops > X Window System
    Applications > Internet Browser
    Development > Additional Development
    Development > Development Tools


리눅스 설치 후 설정


필수 패키지 설치

# yum -y install libaio-devel ksh libstdc++ elfutils-libelf elfutils-libelf-devel libtool-ltdl ncurses readline unixODBC


OS 업데이트

# yum -y update


6.7 버전은 3.8 커널 탑재  업데이트시  커널이 올라가는데 4.1 버전까지 올라간다.

6.8 버전은 처음부터 4.1 버전이 탑재 되어 있다.

ACFS 모듈은 3.8 버전까지만 지원한다.


# vi /etc/sysctl.conf

fs.aio-max-nr = 1048576
fs.file-max = 6815744
#kernel.shmall = 2097152 
#kernel.shmmax = 1054504960
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048586

# sysctl -p


# vi /etc/security/limits.conf

oracle              soft    nproc   2047
oracle              hard    nproc   16384
oracle              soft    nofile  4096
oracle              hard    nofile  65536
oracle              soft    stack   10240

# vi /etc/pam.d/login

session    required     pam_limits.so

# rpm -Uvh cvuqdisk*

 (Grid 설치 디렉토리의 압축을 풀면 RPM 폴더에 있음)


그룹 및 유저 추가

groupadd -g 1000 oinstall
groupadd -g 1200 dba
useradd -u 1100 -g oinstall -G dba oracle
passwd oracle

※ Oracle이 제공하는 11.2.0.3이상 버전의 설치 메뉴얼에 따르면 따로 Grid 계정을 생성하지 않는것으로 변경했다.


# passwd oracle


# vi /etc/hosts

127.0.0.1       localhost.localdomain   localhost
# Public
192.168.0.111   ol6-112-rac1.localdomain        ol6-112-rac1
192.168.0.112   ol6-112-rac2.localdomain        ol6-112-rac2
# Private
192.168.1.111   ol6-112-rac1-priv.localdomain   ol6-112-rac1-priv
192.168.1.112   ol6-112-rac2-priv.localdomain   ol6-112-rac2-priv
# Virtual
192.168.0.113   ol6-112-rac1-vip.localdomain    ol6-112-rac1-vip
192.168.0.114   ol6-112-rac2-vip.localdomain    ol6-112-rac2-vip
# SCAN
192.168.0.115   ol6-112-scan.localdomain ol6-112-scan
192.168.0.116   ol6-112-scan.localdomain ol6-112-scan
192.168.0.117   ol6-112-scan.localdomain ol6-112-scan

※ IP 대역은 각자의 설정에 맞게 넣는다.

※ 11g는 SCAN IP를 통해 로드밸런싱 및 접속를 할 수 있다. SCAN 1개이상 설정 할 수 있다.


# vi /etc/security/limits.d/90-nproc.conf

# 이렇게 되어 있는 부분을 * soft nproc 1024 # 이렇게 고친다. * - nproc 16384

※ 왜 고치는지는 MOS Note [ID 1487773.1] 참조


# vi /etc/selinux/config

SELINUX=permissive


리눅스 방화벽 중지

# service iptables stop
# chkconfig iptables off


# vi /etc/sysconfig/ntpd

OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"


ORACLE 설치 폴더 생성

mkdir -p /app/grid/11.2.0.4/grid mkdir -p /app/oracle/product/11.2.0.4/db_1 chown -R oracle:oinstall /app chmod -R 775 /app/

※ 고객사가 원하는 폴더 혹은 자신이 원하는 (관리하기 쉬운) 폴더로 생성.



ORACLE 계정 Profile 수정


# su - oracle

$ vi .bash_profile

# Oracle Settings TMP=/tmp; export TMP TMPDIR=$TMP; export TMPDIR ORACLE_HOSTNAME=ol6-112-rac1.localdomain; export ORACLE_HOSTNAME ORACLE_UNQNAME=RAC; export ORACLE_UNQNAME ORACLE_BASE=/app/oracle; export ORACLE_BASE GRID_HOME=/app/grid/11.2.0.4/grid; export GRID_HOME DB_HOME=$ORACLE_BASE/product/11.2.0.4/db_1; export DB_HOME ORACLE_HOME=$DB_HOME; export ORACLE_HOME ORACLE_SID=RAC1; export ORACLE_SID ORACLE_TERM=xterm; export ORACLE_TERM BASE_PATH=/usr/sbin:$PATH; export BASE_PATH PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH alias grid_env='. /home/oracle/grid_env' alias db_env='. /home/oracle/db_env'

$ vi grid_env

ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

$ vi db_env

ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

※ Grid 계정을 따로 생성해주지 않고 oracle 계정으로 DB와 ASM 모두 관리한다.

 - Alias를 통해 grid_env, db_env 명령을 통해 DB와 ASM 인스턴스 간의 선택을 할 수 있다..


서버 재구동하여 모든 설정 적용

# shutdown -r now


VMware에 HDD Disk를 공유 볼륨에 생성. (생성 방법은 http://db.necoaki.net/49 포스팅 참조)



ASM Disk 그룹은 크게 4가지로 나뉜다.


OCR과 VOTING DISK 정보가 들어있는 OCRVOTE

DATA가 들어가는 DATA

복구 영역이 들어가는 RECO (FRA)

Archive log가 저장되는 ARCH


여기에 들어갈 Disk를 자신이 선택하여 직접 생성해주면 된다.

OCRVOTE - 3GB

DATA - 50GB (10GB 5개)

RECO - 10GB

ARCH - 5GB


어차피 현업에서는 공유 볼륨을 잡아주는 일은 스토리지 엔지니어와, OS 엔지니어가 알아서 원하는대로 해준다.

리눅스를 잘한다면 대용량 디스크 하나만 붙여서 lv 하면 된다.


[root@rac1 ~]# ls -l /dev/sd*

brw-rw----. 1 root disk 8,  0 Aug  4 21:47 /dev/sda
brw-rw----. 1 root disk 8,  1 Aug  4 21:48 /dev/sda1
brw-rw----. 1 root disk 8,  2 Aug  4 21:47 /dev/sda2
brw-rw----. 1 root disk 8,  3 Aug  4 21:48 /dev/sda3
brw-rw----. 1 root disk 8, 16 Aug  4 21:47 /dev/sdb
brw-rw----. 1 root disk 8, 32 Aug  4 21:47 /dev/sdc
brw-rw----. 1 root disk 8, 48 Aug  4 21:47 /dev/sdd
brw-rw----. 1 root disk 8, 64 Aug  4 21:47 /dev/sde
brw-rw----. 1 root disk 8, 80 Aug  4 21:47 /dev/sdf
brw-rw----. 1 root disk 8, 96 Aug  4 21:47 /dev/sdg


각각의 디스크를 Fdisk 한다.


[root@rac1 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xec049fb2.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-391, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-391, default 391):
Using default value 391

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


fdisk를 다하면 아래와 같이 된다.


[root@rac1 ~]# ls -l /dev/sd*
brw-rw----. 1 root disk 8,  0 Aug  4 21:47 /dev/sda
brw-rw----. 1 root disk 8,  1 Aug  4 21:48 /dev/sda1
brw-rw----. 1 root disk 8,  2 Aug  4 21:47 /dev/sda2
brw-rw----. 1 root disk 8,  3 Aug  4 21:48 /dev/sda3
brw-rw----. 1 root disk 8, 16 Aug  4 21:54 /dev/sdb
brw-rw----. 1 root disk 8, 17 Aug  4 21:54 /dev/sdb1
brw-rw----. 1 root disk 8, 32 Aug  4 21:54 /dev/sdc
brw-rw----. 1 root disk 8, 33 Aug  4 21:54 /dev/sdc1
brw-rw----. 1 root disk 8, 48 Aug  4 21:54 /dev/sdd
brw-rw----. 1 root disk 8, 49 Aug  4 21:54 /dev/sdd1
brw-rw----. 1 root disk 8, 64 Aug  4 21:54 /dev/sde
brw-rw----. 1 root disk 8, 65 Aug  4 21:54 /dev/sde1
brw-rw----. 1 root disk 8, 80 Aug  4 21:54 /dev/sdf
brw-rw----. 1 root disk 8, 81 Aug  4 21:54 /dev/sdf1
brw-rw----. 1 root disk 8, 96 Aug  4 21:56 /dev/sdg
brw-rw----. 1 root disk 8, 97 Aug  4 21:56 /dev/sdg1


기존에는 oracleasm 이라는 패키지를 통해 모듈을 띄워서 ASM 디스크를 적용했었는데 그 과정이 사라졌다.

Udev를 통해 그냥 Direct로 붙인다.


# vi /etc/scsi_id.config (없으니까 만든다.)

options=-g

#[root@rac1 ~]# /sbin/scsi_id -g -u -d /dev/sdb
36000c292d9f969dda718b42875402ba4
[root@rac1 ~]# /sbin/scsi_id -g -u -d /dev/sdc
36000c2966133c5d6de9c4f15e2c2b977
[root@rac1 ~]# /sbin/scsi_id -g -u -d /dev/sdd
36000c29446c31cffc46c1a9073719639
[root@rac1 ~]# /sbin/scsi_id -g -u -d /dev/sde
36000c29597c280fe33b8f0e9831ccfef
[root@rac1 ~]# /sbin/scsi_id -g -u -d /dev/sdf
36000c295fdea390d70fbf564d399a614
[root@rac1 ~]# /sbin/scsi_id -g -u -d /dev/sdg
36000c29571ed9c2b0aa031c228cb4098


# vi /etc/udev/rules.d/99-oracle-asmdevices.rules


KERNEL=="sdb1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c292d9f969dda718b42875402ba4",NAME="asm-ocrvote01", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdc1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c2966133c5d6de9c4f15e2c2b977",NAME="asm-data01", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdd1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29446c31cffc46c1a9073719639",NAME="asm-data02", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sde1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29597c280fe33b8f0e9831ccfef",NAME="asm-data03", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdf1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c295fdea390d70fbf564d399a614",NAME="asm-reco01", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdg1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="36000c29571ed9c2b0aa031c228cb4098",NAME="asm-arch01", OWNER="oracle", GROUP="dba", MODE="0660"


블락 디바이스 파티션 테이블로 로딩 한다.

# /sbin/partprobe /dev/sdb1
# /sbin/partprobe /dev/sdc1
# /sbin/partprobe /dev/sdd1
# /sbin/partprobe /dev/sde1

UDEV 룰을 다시 불러 온다.

# /sbin/udevadm control --reload-rules
# /sbin/start_udev

[root@rac1 ~]# ls -l /dev/asm*
brw-rw----. 1 oracle dba 8, 97 Aug  4 22:53 /dev/asm-arch01
brw-rw----. 1 oracle dba 8, 33 Aug  4 22:53 /dev/asm-data01
brw-rw----. 1 oracle dba 8, 49 Aug  4 22:53 /dev/asm-data02
brw-rw----. 1 oracle dba 8, 65 Aug  4 22:53 /dev/asm-data03
brw-rw----. 1 oracle dba 8, 65 Aug  4 22:51 /dev/asm-disk1
brw-rw----. 1 oracle dba 8, 17 Aug  4 22:53 /dev/asm-ocrvote01
brw-rw----. 1 oracle dba 8, 81 Aug  4 22:53 /dev/asm-reco01


# shutdown -h now


서버를 끄고 2번으로 복사. (복사 방법은 http://db.necoaki.net/49 포스팅 참조)


호스트 네임 수정

# vi /etc/sysconfig/network


NETWORKING=yes
HOSTNAME=rac2
GATEWAY=192.168.219.1
NTPSERVERARGS=iburst


2번으로 복사하면 네트워크가 다른 이름으로 잡힌다.

eth0, eth1 로 바꿔준다.


[root@rac1 ~]# cd /etc/sysconfig/network-scripts/

[root@rac1 network-scripts]# mv ifcfg-Auto_eth2 ifcfg-eth0
[root@rac1 network-scripts]# mv ifcfg-Auto_eth3 ifcfg-eth1


ip 변경

# vi ifcfg-eth0

IPADDR=192.168.xxx.xxx


# vi ifcfg-eth1

IPADDR=192.168.xxx.xxx



[root@rac1 network-scripts]# vi /etc/udev/rules.d/70-persistent-net.rules


# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:55:a4:0f", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:55:a4:05", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"


# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:eb:bc:2e", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:eb:bc:38", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"



이렇게 4개가 나오는데, 위에 두개를 지우고 아래 두개는 수정해준다.


# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:eb:bc:2e", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:eb:bc:38", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"



2번에서도 디스크 마운트가 되어 있는지 확인

[root@rac1 network-scripts]# ls-lh /dev/asm*
brw-rw----. 1 oracle dba 8, 97 Aug  4 22:59 /dev/asm-arch01
brw-rw----. 1 oracle dba 8, 33 Aug  4 22:59 /dev/asm-data01
brw-rw----. 1 oracle dba 8, 49 Aug  4 22:59 /dev/asm-data02
brw-rw----. 1 oracle dba 8, 65 Aug  4 22:59 /dev/asm-data03
brw-rw----. 1 oracle dba 8, 17 Aug  4 22:59 /dev/asm-ocrvote01
brw-rw----. 1 oracle dba 8, 81 Aug  4 22:59 /dev/asm-reco01


# su - oracle

$ vi .bash_profile


# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR

ORACLE_HOSTNAME=rac2; export ORACLE_HOSTNAME
ORACLE_UNQNAME=rac; export ORACLE_UNQNAME
ORACLE_BASE=/app/oracle; export ORACLE_BASE
GRID_HOME=/app/grid/11.2.0.4/grid; export GRID_HOME
DB_HOME=$ORACLE_BASE/product/11.2.0.4/db_1; export DB_HOME
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
ORACLE_SID=rac2; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'
alias ll='ls -lh'


rac2 로 모두 수정


$ vi grid_env


ORACLE_SID=+ASM2; export ORACLE_SID
ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH


$ vi db_env


ORACLE_SID=rac2; export ORACLE_SID
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH



준비가 완료 됐으니 2번을 재부팅하고 1번을 같이 켜준다.


# shutdown -r now