How to recover from a I/O error in VxVM

While working as a Unix Admin , many a times i have experienced critical issues with Veritas Volume Manager. Most of the times its a heart stopping moment when you hear something is wrong with your disk Storage subsystem. After all , your data sits on Veritas. I am sure any Unix Admin worth his salt will agree with me.

But the beauty of Veritas is its simplicity. It stores objects in a very structured and systematic fashion and if you work out the problem logically you can solve it.

Here i present a typical issue which is encountered with Veritas Volumes i.e filesystems becoming stale or unmountable after a sudden power-off or a sudden loss of connectivity from SAN or HBA card. Moreover this is a very useful question for interviews as well.

Veritas throws I/O error , dont worry vxmend comes to your rescue ...... just unmount the filesystem forcefully with a '-f' and run below commands in sequence.

Vxmend......
Force the plex to go off ( its already stale)

# vxmend -o force off <plex>

Bring it on again ....

# vxmend on <plex>

Clean the plex of any errors as shown below:-

# vxmend fix clean <plex>

Start the volume.

# vxvol start<volume>

Run a fsck as usual

# fsck -F vxfs/dev/vx/rdsk/<diskgroup>/<volume>

Now mount the filesystem, it should be back up in a sane manner.

# mount -F vxfs /dev/vx/dsk/<diskgroup>/<volume> /mountpoint

I have always found Veritas to be very robust and powerful. Just keep learning without fear and worries. Cheers ............................

VxVM interview questions

1)    Can you reduce a FS in VxVM ? What is the risk involved ?
There are a few risks and issues involved while reducing a Filesystem in VxVM.
1.1)    UFS volumes  cannot be shrunk, only grow operation is permitted.
1.2)    VxFS volumes  must be mounted for grow and shrink operation, if it is unmounted no operation can be performed
1.3)    You cannot resize a volume that contains plexes with different layout types ( e.g concat and stripe). Attempting to do so results in the following error message:
              VxVM vxresize ERROR V-5-1-2536 Volume volume has different
               organization in each mirror 
To resize such a volume successfully, you must first reconfigure it so that each data plex has the same layout type.

2)    What is the role of vxconfigd daemon ?
vxconfigd daemon handles all configuration management tasks for VxVM objects. It maintains disk and disk group configuration details, communicates configuration changes to the kernel, and modifies the persistent configuration information stored on disks
 vxconfigd provides the interface between VxVM commands and the kernel device drivers.
 vxconfigd handles   configuration change requests from VxVM utilities, communicates the change requests  to the VxVM kernel, and modifies configuration information stored on disk.
 vxconfigd also initializes VxVM when the system is booted.
 
So practically any command which changes the configuration of VxVM objects (plexes, volumes) interact with vxconfigd daemon.

3)    Where is the diskgroup configuration information is stored in VxVM ?
The DG configuration info is stored in /etc/vx/cbr/bk directory. Last 5 configuration changes are stored.

4)    What is stored in privlen of a VxVM disk ? Do all the disks of a diskgroup have the same information in their private region ?
The private region of a vxvm disk stores disk header label and configuration information about vxvm objects such as volumes,plexes and sub disks.
 
Yes, each  disk in a diskgroup stores an entire copy of the configuration information.

5)    Can you import a DG with incomplete set of disks ?
Yes, If some of the disks in a Diskgroup have failed , you can import the DG using –f option
# vxdg –f import diskgroup

6) How will you replace a faulty disk in VxVM ?
A disk which has failed can be seen from the output of vxdisk list command. The disk device ctd will be missing from the output and it can be seen as failed was :c#t#d#. This confirms that the disk has indeed failed.
We can use vxdiskadm menu options to replace the faulted disk. Choose option 5 from the men , then list the failed disk using 'list' option. We would have to choose an alternate disk for the replacement. The list of available disks is also shown by vxdiskadm. Press 'y' and complete the replacement activity.