Linux multipath 설정 및 raw device 설정

CentOS 6.7 에서의 스토리지 연동 후 multipath 설정 입니다.

 

스토리지 연결 후 linux 상에 파일시스템이 보이게 됩니다.

 

[root@sam ~]# fdisk -l |grep /dev/sd
Disk /dev/sda: 2000.4 GB, 2000365379584 bytes
/dev/sda1   *           1          26      204800   83  Linux
/dev/sda2              26       13080   104857600   83  Linux
/dev/sda3           13080       26134   104857600   83  Linux
/dev/sda4           26134      243198  1743560704    5  Extended
/dev/sda5           26135       39189   104857600   83  Linux
/dev/sda6           39189       52243   104857600   83  Linux
/dev/sda7           52243       60598    67108864   82  Linux swap / Solaris
/dev/sda8           60598       67125    52428800   83  Linux
/dev/sda9           67125       73652    52428800   83  Linux
/dev/sda10          73652      243198  1361872896   83  Linux
Disk /dev/sdb: 2000.4 GB, 2000365379584 bytes
/dev/sdb1               1      478795  1953480704   83  Linux
Disk /dev/sdc: 2147 MB, 2147483648 bytes
Disk /dev/sdd: 2147 MB, 2147483648 bytes
Disk /dev/sde: 2147 MB, 2147483648 bytes
Disk /dev/sdf: 2147 MB, 2147483648 bytes
Disk /dev/sdg: 2147 MB, 2147483648 bytes
Disk /dev/sdi: 2147 MB, 2147483648 bytes
Disk /dev/sdh: 2147 MB, 2147483648 bytes
Disk /dev/sdj: 2147 MB, 2147483648 bytes
Disk /dev/sdk: 2147 MB, 2147483648 bytes
Disk /dev/sdl: 2147 MB, 2147483648 bytes
Disk /dev/sdm: 2147 MB, 2147483648 bytes
Disk /dev/sdn: 2147 MB, 2147483648 bytes

 

sdc 부터 스토리지에서 할당된 영역이 보이내요.

 

처음엔 multipath 가 실행되지 않아 multipath -l 을 해보아도 아무것도 나오지 않습니다.

 

multipath.conf 파일을 복사 합니다.

 

# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf  /etc/multipath.conf

 

위와 같이 복사 후  devices 부분을 보니다.

 

devices {
        device {
                vendor "IBM"
                product "2145"
                path_grouping_policy    group_by_prio
                getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
                prio_callout            "/sbin/mpath_prio_alua /dev/%n"
                }
}

 

해당서버에 붙인 스토리지는 IBM v9000 장비 입니다. 해당 정보를 입력 합니다.

 

# chkconfig --level 2345 multipathd on

 

# service multipathd start

 

하게 되면 multipath 정보가 나옵니다.

 

multipath.conf 파일에 수정한게 없으므로 하드웨어가 임의로 뒤죽박죽 붙게 되죠. 아래처럼 말이죠...

 

[root@sam ~]# multipath -ll
Aug 31 16:06:01 | /etc/multipath.conf line 97, invalid keyword: device
Aug 31 16:06:01 | /etc/multipath.conf line 98, invalid keyword: device
Aug 31 16:06:01 | /etc/multipath.conf line 99, invalid keyword: vendor
Aug 31 16:06:01 | /etc/multipath.conf line 100, invalid keyword: product
Aug 31 16:06:01 | /etc/multipath.conf line 101, invalid keyword: path_grouping_policy
Aug 31 16:06:01 | /etc/multipath.conf line 102, invalid keyword: getuid_callout
Aug 31 16:06:01 | unmatched '}' at line 104 of /etc/multipath.conf
Aug 31 16:06:01 | /etc/multipath.conf line 104, invalid keyword: }
Aug 31 16:06:01 | unmatched '}' at line 105 of /etc/multipath.conf
Aug 31 16:06:01 | /etc/multipath.conf line 105, invalid keyword: }
mpathe (3600507680c81834b4000000000000048) dm-0 IBM,2145
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 7:0:1:0 sdf 8:80  active ready running
| `- 7:0:2:0 sdi 8:128 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 7:0:0:0 sdc 8:32  active ready running
  `- 7:0:3:0 sdl 8:176 active ready running
mpathg (3600507680c81834b400000000000004a) dm-2 IBM,2145
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 7:0:1:2 sdh 8:112 active ready running
| `- 7:0:2:2 sdk 8:160 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 7:0:0:2 sde 8:64  active ready running
  `- 7:0:3:2 sdn 8:208 active ready running
mpathf (3600507680c81834b4000000000000049) dm-1 IBM,2145
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 7:0:0:1 sdd 8:48  active ready running
| `- 7:0:3:1 sdm 8:192 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 7:0:1:1 sdg 8:96  active ready running
  `- 7:0:2:1 sdj 8:144 active ready running

 

위에처럼 쓰신다고하면 뭐 상관없지만 좀 정리해보겠습니다. 순서도 맞추고 원하는 이름으로 mapper 설정을 해보죠

 

보기좋게 붙여보겠습니다.

 

# vi /etc/multipath.conf

 

아래처럼 추가 하였습니다.

 

 

multipaths {
       multipath {
        wwid    3600507680c81834b4000000000000048
        alias                   mpath1
        }
       multipath {
        wwid    3600507680c81834b4000000000000049
        alias                   mpath2
        }
       multipath {
        wwid    3600507680c81834b400000000000004a
        alias                   mpath3
        }
}

 

# serivce multipath reload

 

[root@sam ~]# multipath -ll
mpath2 (3600507680c81834b4000000000000049) dm-1 IBM,2145
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 7:0:0:1 sdd 8:48  active ready running
  |- 7:0:1:1 sdg 8:96  active ready running
  |- 7:0:2:1 sdj 8:144 active ready running
  `- 7:0:3:1 sdm 8:192 active ready running
mpath1 (3600507680c81834b4000000000000048) dm-0 IBM,2145
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 7:0:0:0 sdc 8:32  active ready running
  |- 7:0:1:0 sdf 8:80  active ready running
  |- 7:0:2:0 sdi 8:128 active ready running
  `- 7:0:3:0 sdl 8:176 active ready running
mpath3 (3600507680c81834b400000000000004a) dm-2 IBM,2145
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 7:0:0:2 sde 8:64  active ready running
  |- 7:0:1:2 sdh 8:112 active ready running
  |- 7:0:2:2 sdk 8:160 active ready running
  `- 7:0:3:2 sdn 8:208 active ready running

 

수정된게 깔끔하게 되었습니다.

 

해당 내용을 raw device 와 binding 하기위해서 다음의 곳에 해당 내용을 추가합니다.

raw device 는 oracle 에서 사용하기 위해서 작업 하는데 해당 권한을 설정합니다.

 

# vi /etc/udev/rules.d/60-raw.rules

 

여기에 다음과 같이 입력합니다.

 

ACTION=="add", KERNEL=="/dev/mapper/mpath1" RUN+="raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="/dev/mapper/mpath2" RUN+="raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="/dev/mapper/mpath3" RUN+="raw /dev/raw/raw3 %N"

 

ACTION=="add", KERNEL=="raw*", OWNER=="oracle", GROUP=="dba", MODE=="0660"
ACTION=="add", KERNEL=="rawctl", OWNER=="root", GROUP=="disk", MODE=="0660"

 

# vi /etc/rawdevices

 

여기에 다음과 같이 입력합니다.

 

/dev/raw/raw1   /dev/mapper/mpath1
/dev/raw/raw2   /dev/mapper/mpath2
/dev/raw/raw3   /dev/mapper/mpath3

 

모든 준비가 완료 되었습니다.

 

# service multipath restart

 

이렇게 한 후 raw 장치가 생성 되어있는지 확인해 봅니다.

 

[root@sam ~]# ll /dev/mapper/
total 0
crw-rw---- 1 root root 10, 58 Aug 31 19:05 control
lrwxrwxrwx 1 root root      7 Aug 31 20:38 mpath1 -> ../dm-0
lrwxrwxrwx 1 root root      7 Aug 31 20:38 mpath2 -> ../dm-1
lrwxrwxrwx 1 root root      7 Aug 31 20:38 mpath3 -> ../dm-2
[root@storage-test-04 ~]# ll /dev/raw
total 0
crw-rw---- 1 root disk 162, 1 Aug 31 19:06 raw1
crw-rw---- 1 root disk 162, 2 Aug 31 19:06 raw2
crw-rw---- 1 root disk 162, 3 Aug 31 19:06 raw3
crw-rw---- 1 root disk 162, 0 Aug 31 19:05 rawctl

 

깔끔하게 리부팅 한번 해줍니다. 혹시 모를 재부팅 시 정상적으로 적용이 되었는지를 위해 확인차 진행해 봅니다.

 

※ internal Disk 가 2개 이상일 경우 정상적인 부팅이 안될 수 있습니다. 일단 OS영역의 /dev/sda* 부분 이외의 또다른 하드디스크가 있다면 fstab 에서 # 으로 막아주고 다시 부팅 시도합니다.

 

이럴 경우 multipath 에서 internal 디스크도 같이 잡게 되는데 하드디스크의 정보가 multipath.conf에 들어있지 않아 부팅이 되지 않으니 참고 하세요. 일단 막고 부팅이 된다면 다음과 같이 내용 추가해 줍니다.

 

[root@storage-test-04 ~]# multipath -ll
mpath2 (3600507680c81834b4000000000000049) dm-2 IBM,2145
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 7:0:0:1 sdd 8:48  active ready running
| `- 7:0:3:1 sdm 8:192 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 7:0:1:1 sdg 8:96  active ready running
  `- 7:0:2:1 sdj 8:144 active ready running
mpath1 (3600507680c81834b4000000000000048) dm-1 IBM,2145
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 7:0:1:0 sdf 8:80  active ready running
| `- 7:0:2:0 sdi 8:128 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 7:0:0:0 sdc 8:32  active ready running
  `- 7:0:3:0 sdl 8:176 active ready running
mpathd (3600508b1001c408a940e2809e4553e25) dm-0 HP,LOGICAL VOLUME
size=1.8T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 6:0:0:1 sdb 8:16  active ready running
mpath3 (3600507680c81834b400000000000004a) dm-3 IBM,2145
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 7:0:1:2 sdh 8:112 active ready running
| `- 7:0:2:2 sdk 8:160 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 7:0:0:2 sde 8:64  active ready running
  `- 7:0:3:2 sdn 8:208 active ready running

 

위와 같이 mpathd (3600508b1001c408a940e2809e4553e25) 부분이 생겼내요.. 이건 OS영역말고 다른 internal disk 인데 같이 잡혔내요.. 이부분을 multipath 에서 예외 시켜줍니다.

 

# vi /etc/multipath.conf

 

blacklist {
        wwid "3600508b1001c408a940e2809e4553e25"
}

 

위와 같이 넣어주면 예외 처리가 됩니다.

 

다시 한번 재부팅 후 정상적인 부팅이라면 설정 끝~

 

이제 오라클 엔지니어에게 토스~!!

 

############

설정파일 내용

############

 

[root@sam ~]# cat /etc/multipath.conf

devices {
        device {
                vendor "IBM"
                product "2145"
                path_grouping_policy    group_by_prio
                getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
        #       prio_callout            "/sbin/mpath_prio_alua /dev/%n"
                }
}

blacklist {
        wwid "3600508b1001c408a940e2809e4553e25"
}

multipaths {
       multipath {
        wwid    3600507680c81834b4000000000000048
        alias                   mpath1
        }
       multipath {
        wwid    3600507680c81834b4000000000000049
        alias                   mpath2
        }
       multipath {
        wwid    3600507680c81834b400000000000004a
        alias                   mpath3
        }
}

 

[root@sam ~]# cat /etc/rawdevices
/dev/raw/raw1   /dev/mapper/mpath1
/dev/raw/raw2   /dev/mapper/mpath2
/dev/raw/raw3   /dev/mapper/mpath3

 

[root@sam ~]# cat /etc/udev/rules.d/60-raw.rules
ACTION=="add", KERNEL=="/dev/mapper/mpath1" RUN+="raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="/dev/mapper/mpath2" RUN+="raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="/dev/mapper/mpath3" RUN+="raw /dev/raw/raw3 %N"

 

ACTION=="add", KERNEL=="raw*", OWNER=="oracle", GROUP=="dba", MODE=="0660"
ACTION=="add", KERNEL=="rawctl", OWNER=="root", GROUP=="disk", MODE=="0660"