VCS Interview questions answers

1) What is split brain and amnesia prevention in cluster ?

2) Suppose one of the high-priority heartbeat connection between nodes is lost, what will be the condition of the cluster known as ? What action VCS will take in such a scenario ?
When one of the high-priority heartbeat connectivity between nodes is lost and there is only one remaining heartbeat link, VCS will place the node in a special membership category known as jeopardy membership.
In such case, VCS will autodisable the SG amd the servicegroup state will not change i.e offline or online servicegroups continue to be in that state. VCS  prevents any failover from happening to prevent data corruption.
3) During patching if we want to stop a servicegroup from failover, what actions we will take ?
It is a best practice to freeze a servicegroup during server patching activity. When you freeze a Group, VCS will take no action on that Group or its Resources. It will not try to bring the servicegroup online on any other node.  After the maintenance is over, you can bring the resources online and VCS will refresh its view at that time.

4) How do you check logs of a servicegroup and a resource in VCS ? How will you troubleshoot if a resource has faulted ?
The default VCS log directory is /var/VRTSvcs/log
The main event log of VCS is /var/VRTSvcs/log/engine_A.log. This file is the best place to begin troubleshooting for a failed resource.
Individual agent types have their own log files e.g Mount_A.log , Apache_A.log or Weblogic_A.log. These log files contain more  detailed info than the engine_A.log.

Checking for the word 'clean' can provide clues related to the cause of failure.

5) What is the main purpose of llt and had daemons ?
LLT is the transport mechanism of VCS and is responsible for load balancing of cluster communications and maintaining heartbeat. HAD is the main VCS deamon and is responsible for taking operator input and performing the relevant actions. HAD also takes all types of corrective actions required.

6) What are the difference between LLT and GAB ?
LLT and GAB purpose and differences
LLT is the layer 2 protocol developed by Veritas. It takes care of the heartbeat connection and is a carrier.
GAB distributes the cluster config among nodes. GAB uses LLT as its tranport mechanism for distributing cluster configuration changes.
HAD communicates with GAB and maintains / tracks all cluster configuration. Uses main.cf file to build cluster config. HAD also takes all types of corrective actions required.

7) What is the difference between high priority and low priority link in VCS ?
High priority link is used for transmitting cluster communication and configuration information between nodes and to GAB as well as for heartbeat communications.
Low priority link is used only for heartbeat in normal scenario, but in case of failure of high priority link it can take over the task of transmitting cluster communication also.

8) What are the components in VCS i/o fencing setup ?
Following are the components required for IO fencing in VCS
i)    Coordinator diskgroup with 3 disks
ii)    Data diskgroup
iii)    Dynamic multipathing software (VXDMP)

9) What is a Jeopardy condition in VCS ? What happens to the ServiceGroup and Resources running on a system which is under Jeopardy condition ?
Jeopardy membership condition occurs when a node in a cluster is having only one heartbeat connection remaining with the rest of the cluster. At this point, VCS cannot reliably distinguish between a node failure or network failure if the last heartbeat interconnect also fails. Hence under jeopardy condition, VCS prevents the ServiceGroup from failover. The Applications and ServiceGroup running on the node keep on running as usual and will not be failed over in case of a Node failure. But in case of a resource or group fault, the ServiceGroup fails over to available systems in the cluster. This is a safety mechanism to prevent data corruption.

10) During RACE condition for membership arbitration in case of a node or link failure, how VCS will determine the eligible host for aquiring the lock on co-ordinator disks ? Which sub-cluster will win the RACE and based on what logic ?
During a RACE condition, the partitioned nodes will form a sub-cluster and try to acquire the co-ordinatore disks. Among the nodes in the sub-cluster , the node with the lowest LLT ID will run for the RACE on behalf of itself and other nodes in sub-cluster. If it is successful it will eject keys of other systems (i.e nodes which are not part of the newly formed sub-cluster) from the co-ordinator disks and send a WON_RACE communication to nodes in its sub-cluster. Other nodes which fail to acquire the disks will panic.

VxVM interview questions

1)    Can you reduce a FS in VxVM ? What is the risk involved ?
There are a few risks and issues involved while reducing a Filesystem in VxVM.
1.1)    UFS volumes  cannot be shrunk, only grow operation is permitted.
1.2)    VxFS volumes  must be mounted for grow and shrink operation, if it is unmounted no operation can be performed
1.3)    You cannot resize a volume that contains plexes with different layout types ( e.g concat and stripe). Attempting to do so results in the following error message:
              VxVM vxresize ERROR V-5-1-2536 Volume volume has different
               organization in each mirror 
To resize such a volume successfully, you must first reconfigure it so that each data plex has the same layout type.

2)    What is the role of vxconfigd daemon ?
vxconfigd daemon handles all configuration management tasks for VxVM objects. It maintains disk and disk group configuration details, communicates configuration changes to the kernel, and modifies the persistent configuration information stored on disks
 vxconfigd provides the interface between VxVM commands and the kernel device drivers.
 vxconfigd handles   configuration change requests from VxVM utilities, communicates the change requests  to the VxVM kernel, and modifies configuration information stored on disk.
 vxconfigd also initializes VxVM when the system is booted.
 
So practically any command which changes the configuration of VxVM objects (plexes, volumes) interact with vxconfigd daemon.

3)    Where is the diskgroup configuration information is stored in VxVM ?
The DG configuration info is stored in /etc/vx/cbr/bk directory. Last 5 configuration changes are stored.

4)    What is stored in privlen of a VxVM disk ? Do all the disks of a diskgroup have the same information in their private region ?
The private region of a vxvm disk stores disk header label and configuration information about vxvm objects such as volumes,plexes and sub disks.
 
Yes, each  disk in a diskgroup stores an entire copy of the configuration information.

5)    Can you import a DG with incomplete set of disks ?
Yes, If some of the disks in a Diskgroup have failed , you can import the DG using –f option
# vxdg –f import diskgroup

6) How will you replace a faulty disk in VxVM ?
A disk which has failed can be seen from the output of vxdisk list command. The disk device ctd will be missing from the output and it can be seen as failed was :c#t#d#. This confirms that the disk has indeed failed.
We can use vxdiskadm menu options to replace the faulted disk. Choose option 5 from the men , then list the failed disk using 'list' option. We would have to choose an alternate disk for the replacement. The list of available disks is also shown by vxdiskadm. Press 'y' and complete the replacement activity.

Solaris Interview Questions

1) Can you reduce a FS in VxVM ? What is the risk involved ?
2) What are branded zones in solaris ?
3) What is the main enhancement in NFS v4 over v3 ?
4) How can we refresh automount config without affecting connected users ?
5) How many types of automount maps are there ?
6) How can you patch a solaris zone which is a sparse root ?

7) What is the difference between zone and container in Solaris ?
8) How can we assign zpool to NGZ ?
9) How thin provisioning / sparse volume is created in ZFS ?


10) Suppose after a power outage or a mistake by storage admin, one of your LUN in VxVM was unavailable and now it has become available again. But your mountpoint is throwing I/O error. How will you resolve this situation ?
11) What are the advantages of using ZFS instead of UFS ?
12) Which is better hardware  RAID or software RAID ? why ?
13) After a power outage or kernel panic ZFS pool is not getting imported ? What are the steps for troubleshooting this ?
14) Which virtualization solution would you recommend to customer LDOM or zones ? under what scenario and considerations ?
15) Suppose you have a low memory condition on solaris server . How will you troubleshoot further ?
16) What is the change in regards to network configuration in Solaris 10 over Solaris 8 ?

17) How can you restore LDOM configuration ?
18) How can we clone LDOM ?
19) Which field of iostat indicates a IO bottleneck ? What are the important metrics to be looked at in iostat output ?
20) In maintaining Solaris security what are the common steps / procedures followed by System Admins ?

21) In Solaris we have resource pools ( rpool ). We can assign rpools to zones as well to set cpu shares. Then why do we need resource controls in zones ( add rctl ) . What is the advantage of using resource controls when resource pooling is already there ?

22) How to install multiple patches in a single patchadd invocation ?   

VCS VXVM DiskGroup switchover example

Step 1:- Start the desired haagent ( DiskGroup) on both the machines.

client1: /tmp/vcs> haagent -start DiskGroup -sys client1
client1: /tmp/vcs> haagent -start DiskGroup -sys client2

Step 2:- Add the servicegroup and populate the SystemList.

client1: /tmp> haconf -makerw
client1: /tmp> hagrp -add DG_servicegroup
VCS NOTICE V-16-1-10136 Group added; populating SystemList and setting the Parallel attribute recommended before adding resources
client1: /tmp> hagrp -modify VG_servicegroup SystemList client1 1 client2 2

Step 3:- Add the resource, specify the type of the resource and assign it to a servicegroup, enable the resource, specify the Disk Group name etc and bring the resource online.

client1: /tmp> hares -add VXVMDG DiskGroup DG_servicegroup
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before agent monitors
client1: /tmp> hares -modify VXVMDG Enabled 1
client1: /tmp> hares -modify VXVMDG DiskGroup clusterdg
client1: /tmp> hares -modify VXVMDG StartVolumes 1
client1: /tmp> hares -online VXVMDG -sys client1

Step 4:- Add a ‘Mount’ resource type named clustervol ( or whichever name it is easy to identify with ), specify the correct blockdevice name, FS type, mountpoint and fsck option. Enable and bring the resource online.

client1: /tmp> hares -add clustervol Mount DG_servicegroup
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before agent monitors
client1: /tmp> hares -modify clustervol BlockDevice /dev/vx/dsk/clusterdg/clustervol
client1: /tmp> hares -modify clustervol FSType vxfs
client1: /tmp> hares -modify clustervol MountPoint /clustervol
client1: /tmp> hares -modify clustervol FsckOpt %-y
client1: /tmp> hares -modify clustervol Enabled 1
client1: /tmp> hares -online clustervol -sys client1

Step 5 :- Link the two resources i.e “DiskGroup and Mount” and enable resources contained with the servicegroup.

client1: /tmp> hares -link clustervol VXVMDG
client1: /tmp> hagrp -enableresources DG_servicegroup
client1: /tmp> hagrp -online DG_servicegroup -sys client1
client1: /tmp> haconf -dump –makero
client2: /tmp> mkdir /clustervol
client1: /tmp> hagrp -switch DG_servicegroup -to client2

Extended process list in Solaris 10 and HPUX, without truncated lines

Some processes typically have a very long list (more than 1000 characters) with command line arguements  and ps –ef | grep <pid> will show only one line (max 80 characters). Both solaris and hpux have extensions through which we can view the full command line, without truncation.


To retrieve the full process list for a process by running ps

on Solaris :-
/usr/ucb/ps –agxuwwwww PID

On HPUX :-
ps –exx

Unix / Solaris Password Expiration Automated email notification

I have been entrusted with setting up a mail alert system for user password expiration. The user should automatically get intimated through mail a few days before his password expiration date. I wrote a small script by taking help from www.unix.com and other forums.
Below is the script for checking the age of the password and alert the user if password is going to expire in next 15 days.

Script Name  :- /usr/bin/solchage

---script start here----
#!/usr/bin/bash
umask 0022
PATH=/usr/bin:/usr/sbin
SHADOW=/etc/shadow
DSHADOW=/etc/shadow.dummy
USER=$1

# Copy the contents of /etc/shadow to a dummy file and make sure the entries for system  
# users are not there in the dummy file. Also replace the encrypted password field with      
# *LK* to make sure passwords are not visible or cannot be copied by someone else.

cat ${SHADOW} | egrep -v "root|daemon|etc" | awk -F: '{print $1,"*LK*",$3,$4,$5,$6,$7,$8}' | sed 's/ /:/g' > ${DSHADOW}

PASSWDFILE=/etc/passwd

# Specify the mail domain of your company here.
DOMAIN=xyz.com

# The next line extracts the users email id from GECOS field of /etc/passwd file. So as a pre # requisite to running this script, you must enter the email id of the user, without the            # domain name, in GECOS field as i have assumed here. Let me know if you can think of a
# more elegant way of extracting this information.

EMAIL=`grep ^${USER} ${PASSWDFILE} | awk -F: '{print $5}'`

# Save the message in a file.
FILE=/tmp/msg.$$

# Set the password policy here, i.e the number of days after which user must change              # password.
PWPOLICY=90
# Set the warning period here.
WARN=15

# Calculate the number of seconds elapsed since Jan 1 ,1970 i.e Unix epoch.

EPOCH=`perl -e 'print time;'`

# Convert the number of seconds into days.

DAYSEPOCH=`expr ${EPOCH} / 86400`

# Calculate the number of days since password was changed for the last time for a particular # user. This info can be extracted from 3rd field of /etc/passwd file. This is expressed as
# the number of days between January 1,  1970, and  the  date  that  the  password was last
# modified.

LASTCHG=`grep ^${USER} ${DSHADOW} | awk -F: '{print $3}'`


# Subtract the above value from the number of days since epoch to arrive at the number    #  of days since last password change. 

PASSWDCHANGE=`expr ${DAYSEPOCH} - ${LASTCHG}`

EXPIRED=`expr ${PWPOLICY} - ${PASSWDCHANGE}`

if [ "${EXPIRED}" -lt "${WARN}" ]; then

cat > ${FILE} <<EOF
Dear ${USER},
Your password will expire in ${EXPIRED} days. Please change it as soon as possible.
EOF

mailx -s "Password expiring soon." ${EMAIL}@${DOMAIN} < ${FILE}

fi--- script end here---

To run the above main script, you have to run another small script which i produce below.
Copy the above script and place it under /usr/bin and name it solchage. Ofcourse you can give it another name, its upto you but make corresponding changes in below script as well if you do so.

Lets name the second script as /var/pwexpire.sh. So put this script in crontab for execution once everyday. It will run for all users, and send them a mail if their password is going to expire within 15 days.

Script Name:- /var/pwexpire.sh

--- script begin here ---

cat /etc/passwd | egrep -v "root|daemon|etc|sys|adm|lp|uucp|nuucp|smmsp|listen|gdm|webservd|postgres|svctag|nobody|noaccess|nobody4" | awk -F: '{print $1}' | egrep -v "bin" | xargs -I {} /usr/bin/solchage {}

---script end here---


What the above script does ? Let us examine step by step.

1) It reads /etc/passwd file and cuts out system users from the list
2) Then prints the remaining usernames using awk and removes all other entries except first filed from the output.
3) Then xargs executes our script /usr/bin/solchage one by one for every listed user. This is required because the our script takes username as argument ( see USER=$1 above ) and runs for that particular user.

You will have to give execute permissions to both the scripts.

SUNWjet server installation steps ( jumpstart )

SUNWjet is a new enhanced version of jumpstart and is easier to configure than older versions of jumpstart. You can download SUNWjet packge from OTN at this link http://www.oracle.com/technetwork/systems/jet-toolkit/index.html/

The steps to install and configure your JET server are:-

1) # pkgadd -d . SUNWjet   ( install the package)

2) # mount -o ro -F hsfs /dev/dsk/c0t4d0s2 /cdrom                  (mount the solaris DVD)

3) # /opt/jet/bin/copy_solaris_media /cdrom    (by default image will get copied to /export/install/media)

4) # /opt/jet/bin/list_solaris_locations

5) # mkdir /export/install/patches

6) # mkdir /export/install/pkgs

7) # /opt/jet/bin/make_template solclnt01     (create a template file)

8) # vi /opt/jet/Templates/solclnt01           (edit the 3 parameters listed below)

base_config_ClientArch="sun4u"
base_config_ClientEther=0:3:ba:ef:60:39
base_config_ClientOS="10"

9) # /opt/jet/bin/make_client solclnt01

 

From ok prompt of the client machine, type the below command to get started

10) ok   boot net - install -w

 

After this step, rest of the installation is vanilla.

Disabling sendmail daemon (SMTP) on solaris 10

The sendmail daemon runs on port 25 and is enable by default on solaris boxes.
The sendmail daemon is not needed to be running on servers which are meant to be mail clients. To disable sendmail service use below steps:-

1. Edit /etc/default/sendmail . Create the file if its not already there and include the following values:
MODE=Ac
QUEUEINTERVAL=”15m”

2. Stop the sendmail service 
/etc/init.d/sendmail stop

3. Now edit /etc/sendmail/submit.cf

and change the line shown here: D{MTAHost}[127.0.0.1]
to :
D{MTAHost}[<ur-mail-server-ip>]

4.  Start the sendmail service.
    /etc/init.d/sendmail start

Now port 25 on localhost would be disabled and server wont be listening on that port anymore.

svcadm enhancements in Solaris 10 - Bind to localhost

Some services in solaris 10 operate based on local and global properties.

For example, in rpcbind configuration if the value for local_only is set to true, all rpc services are accessible from inside the machine but an outside machine cannot access these services.

bash-3.00# svccfg -s rpc/bind listprop config/local_only

config/local_only  boolean  false

bash-3.00#

bash-3.00# svccfg -s rpc/bind setprop config/local_only=true

bash-3.00#

bash-3.00# svcadm refresh rpc/bind

bash-3.00#

bash-3.00# svccfg -s rpc/bind listprop config/local_only

config/local_only  boolean  true

bash-3.00#

bash-3.00# svccfg -s rpc/bind setprop config/local_only=false

bash-3.00#

bash-3.00# svcadm refresh rpc/bind

bash-3.00# svcadm refresh rpc/bind

bash-3.00#

bash-3.00# svccfg -s rpc/bind listprop config/local_only

config/local_only  boolean  false

This in my opinion is a significant security enhancement, especially in some cases where you want a particular service to be accessible from localhost but disabled for outside machines. Many solaris services have this kind of configurability.

Solaris 10, project files

On Solaris 10, system V  IPC paramters( e.g shmmax,shmseg) are set under user specific projects and these values take effect on a per-project basis only and are not system-wide values. We do not need to set them in /etc/system and even if set, the values are ignored.

All the processes started by users who are member of a project inherit the parameter values from /etc/project file.

hostA: /etc\> projects -l

user.oracle

        projid : 1001

        comment: "Oracle Project"

        users  : oracle

        groups : dba

                 oinstall

        attribs: process.max-sem-nsems=(priv,256,deny)

                 project.max-sem-ids=(priv,100,deny)

                 project.max-shm-ids=(priv,128,deny)

                 project.max-shm-memory=(priv,4294967296,deny)

hostA: /etc\> more /etc/project

system:0::::

user.root:1::::

noproject:2::::

default:3::::

group.staff:10::::

user.oracle:100:Oracle Project:oracle:dba,oinstall:process.max-sem-nsems=(priv,256,deny);project.max-sem-ids=(priv,100,deny);project.max-shm-ids=(priv,128,den

y);project.max-shm-memory=(priv,4294967296,deny)

      After editing the /etc/project file, we also need to give privilege to oracle user to be able to use projects, otherwise even if projects are created properly , it will not take effect.

ggnqccita2: /etc\> more /etc/user_attr

oracle::::project=user.oracle

The values set in /etc/project file are dynamic and do not need reboot to take effect. In previous versions of solaris, the values set in /etc/system did not take effect until reboot.