View source

Run out of space on one of SMTP nodes in DZM due to inappropriate logrotate settings in PureMessage for UNIX.

<pre>
root@dmz-sea-01:~# df -h /opt
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/datavg-optlv   19G   16G  2.3G  88% /opt

root@dmz-sea-01:~# du -k  /opt|sort -rn | head -20
16052372	/opt
15966600	/opt/pmx6
14901396	/opt/pmx6/postgres-9.0
14840964	/opt/pmx6/postgres-9.0/var
14840960	/opt/pmx6/postgres-9.0/var/data
13756756	/opt/pmx6/postgres-9.0/var/data/base
13741232	/opt/pmx6/postgres-9.0/var/data/base/37894172
1081352	/opt/pmx6/postgres-9.0/var/data/pg_xlog
641452	/opt/pmx6/var
397524	/opt/pmx6/var/digest
396844	/opt/pmx6/var/digest/pending
184544	/opt/pmx6/etc
</pre>

Add new disk in vCenter

Then run the below command in the VM to rescan SCSI bus:
<pre>
root@dmz-sea-01:~# echo "- - -" > /sys/class/scsi_host/host2/scan
</pre>

Watch the new disk added:

<pre>
root@dmz-sea-01:~# dmesg | tail -20
[14381973.690254] [0]: VMCI: Updating context from (ID=0xfa83d017) to (ID=0xfa83d017) on event (type=0).
[14427863.701641] [0]: VMCI: Updating context from (ID=0xfa83d017) to (ID=0xfa83d017) on event (type=0).
[14600494.612139] [15976]: VMCI: Updating context from (ID=0xfa83d017) to (ID=0xfa83d017) on event (type=0).
[15674992.301746] scsi 2:0:2:0: Direct-Access     VMware   Virtual disk     1.0  PQ: 0 ANSI: 2
[15674992.301761] scsi target2:0:2: Beginning Domain Validation
[15674992.302445] scsi target2:0:2: Domain Validation skipping write tests
[15674992.302448] scsi target2:0:2: Ending Domain Validation
[15674992.302495] scsi target2:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
[15674992.303282] sd 2:0:2:0: [sdc] 41943040 512-byte logical blocks: (21.4 GB/20.0 GiB)
[15674992.303306] sd 2:0:2:0: [sdc] Write Protect is off
[15674992.303308] sd 2:0:2:0: [sdc] Mode Sense: 61 00 00 00
[15674992.303340] sd 2:0:2:0: [sdc] Cache data unavailable
[15674992.303342] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[15674992.303768] sd 2:0:2:0: [sdc] Cache data unavailable
[15674992.303770] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[15674992.308148] sd 2:0:2:0: Attached scsi generic sg3 type 0
[15674992.309127]  sdc: unknown partition table
[15674992.309271] sd 2:0:2:0: [sdc] Cache data unavailable
[15674992.309274] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[15674992.309381] sd 2:0:2:0: [sdc] Attached SCSI disk
</pre>

No partition there:

<pre>
root@dmz-sea-01:~# fdisk -l /dev/sdc

Disk /dev/sdc: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table
</pre>

Adding new PV and extend VG:

<pre>
root@dmz-sea-01:~# pvcreate /dev/sdc
  Physical volume "/dev/sdc" successfully created

root@dmz-sea-01:~# vgextend datavg /dev/sdc
  Volume group "datavg" successfully extended

root@dmz-sea-01:~# vgdisplay datavg | grep Free
  Free  PE / Size       5374 / 20.99 GiB
</pre>

Now extend the LV and re-size filesystem:
<pre>
root@dmz-sea-01:~# lvextend -L+10G /dev/datavg/optlv 
  Extending logical volume optlv to 29.00 GiB
  Logical volume optlv successfully resized

root@dmz-sea-01:~# resize2fs /dev/datavg/optlv 
resize2fs 1.42 (29-Nov-2011)
Filesystem at /dev/datavg/optlv is mounted on /opt; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 2
Performing an on-line resize of /dev/datavg/optlv to 7602176 (4k) blocks.
The filesystem on /dev/datavg/optlv is now 7602176 blocks long.

root@dmz-sea-01:~# df -h /opt
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/datavg-optlv   29G   16G   12G  58% /opt
</pre>