Moscow time zone in AIX

AIX does not seem to have a pre-defined time zone for Moscow. The closest thing in "smit chtz" is Saudi Arabia, which also corresponds to CUT+3 but isn't exactly indicative. However, nothing is stopping you from naming your own time zone using the TZ variable in /etc/environment. This is my value of TZ for the Moscow time zone:

TZ=MSK-3MSD-4,M3.5.0/02:00:00,M10.5.0/03:00:00

During daylight saving period the "date" command gives:

Thu Oct 4 10:12:55 MSD 2007

And during standard time period:

Sun Oct 28 23:02:32 MSK 2007

Process memory usage summary in AIX

This is from the Performance Management book on the AIX Infocenter. A useful command to list the top 15 memory-using processes on the system.

svmon -Pt15 | perl -e 'while(<>){print if($.==2||$&&&!$s++);$.=0 if(/^-+$/)}'

CPU-bound threads in AIX

Sometimes a DLPAR CPU remove operation fails because one or more threads are bound to a logical CPU ("0930-033 Resource 0x2 is busy and cannot be released"). The command

ps -mefo bnd,pid,args

will show each thread and the logical CPU it is bound to, if applicable (first column of output). For example:

- 1044680 db2fmp (C) 0
- - -
3 - -
- - -
3 - -

If the thread PID is '-', look up the first numeric PID above — this is the parent process. In this example, PID 1044680 has 4 child threads, two of which are bound to logical CPU 3. If a DLPAR operation tried to reduce the number of logical CPUs to less than 3, it would fail with error 0930-033. To unbind these threads, use the 'bindprocessor -u' command on the parent PID:

bindprocessor -u 1044680

Now the DLPAR operation should work.

ctrmc subsystem missing after ML upgrade

After upgrading from AIX 5.3 ML03 to ML04 (and possibly from/to other levels also), for some reason the ctmrc subsystem is no longer present. This is a problem for LPARs because all DLPAR functionality is lost. To re-create and start the subsystem, execute these commands:

/usr/sbin/rsct/bin/rmcctrl -a
/usr/sbin/rsct/bin/rmcctrl -s

'/usr/linux/bin/ls --color' on AIX

I have a habit of using GNU coreutils on non-GNU systems (e.g. AIX) for my shell environment, so I install the coreutils package from the AIX Toolbox for Linux on all my machines. My 'ls' is aliased to '/usr/linux/bin/ls --color' to have ANSI colored output. Unfortunately, this has unwanted side effects when piping 'ls' output to other commands, apparently because of some color-related control characters. Consider:

# alias ls
alias ls='/usr/linux/bin/ls --color'

# ls ./foobar
./foobar

# ls ./foobar | xargs /usr/bin/rm -e
rm: ./foobar: A file or directory in the path name does not exist.

However, using the standard AIX 'ls' works:

# /usr/bin/ls ./foobar | xargs /usr/bin/rm -e
rm: Removing ./foobar

Had to break my head a little before guessing this one.

NIM subnet mismatch and LED 613

Here's something that can occur fairly easily: we changed the IP address of a NIM client to another subnet, but forgot to update the client's subnet on the NIM server, i.e. 'lsnim -l <client>' still showed the old subnet. Then we did a BOS install on the client, and right after loading the kernel the client got stuck in LED 613 ('8 mm 80 GB VXA-2 tape device'). Of course, the LED's meaning has nothing to do with the original problem, so this is one of those "better-write-that-down" things.

Neat lslpp output

My memory must be going, as I can never seem to remember the options order for this:

lslpp -qlcOu [fileset]

This lists the "usr" part of the fileset(s), in colon-separated format, with no headers.

errclear after VIOS reboot

I have LPARs with rootvg mirrored across two VIO servers (client LVM mirroring). If one VIOS is rebooted, several errors will get logged in the errpt on the client LPARs (LVM I/O error, PV declared missing, etc). To clear these errors in one shot while preserving other unrelated errors in the errpt, run this command on each client LPAR:

errclear -j DE3B8540,EAA3D429,F7DDA124,41BF2110,613E5F38,E86653C3,52715FA5,AA8AB241,857033C6 0

PRODUCT_TAPE parameter in image.data file

I found this out the hard way and haven't found it yet in the official documentation.

In a file generated by mkszfile(1), the value of PRODUCT_TAPE is set to "no". If you want to create a NIM image_data resource from this file and use it to install BOS, the value needs to be changed to "yes". Otherwise the BOS installation will bomb out in the initial stages (when copying CuDv to disk after creating logical volumes and filesystems).

To find this out I had to set BOSINST_DEBUG to "yes" in the bosinst.data file and examine the saved console output.

EMC AIX ODM package

Dilemma of the day: to install or not to install the EMC AIX ODM package on AIX 5.3 machines connected to an EMC DMX SAN??

This is from the AIX 5.3 release notes:

A device configured as MPIO other FC disk has the following properties:
[...]
Is supported in a production environment. Device-specific vendor ODM pre-definitions are not required to be installed before using in a production environment.

So, I install my machines with the default AIX configuration and they run happily for weeks with EMC disks configured as "MPIO Other FC SCSI Disk Drive", with no problems whatsoever.

Today I find this in the "EMC Host Connectivity Guide for IBM AIX":

A Symmetrix device configured as an MPIO other FC device is not
supported. The only supported configuration is with the EMC
Symmetrix FCP MPIO ODM predefined attributes which has been
certified to operate with the AIX default PCMs. These predefines get
installed with the EMC.Symmetrix.fcp.MPIO.rte fileset.

My machines haven't gone into production yet, but some will very soon. So now I have to varyoff the VGs, delete the disks, install the goddamn EMC filesets (which does a bosboot), and reboot each machine. Luckily I only have ten machines so far.

According to the same EMC doc, two filesets are necessary, from the following two combinations:

EMC.Symmetrix.aix.rte
EMC.Symmetrix.fcp.rte

or

EMC.Symmetrix.aix.rte
EMC.Symmetrix.fcp.MPIO.rte

The former configures the disks without MPIO (lspath shows nothing for those disks), while the latter configures them as MPIO devices with a path to each HBA they are connected to.

But can EMC.Symmetrix.fcp.rte and EMC.Symmetrix.fcp.MPIO.rte coexist? No, they can't. This is what happens if you try to install the two at the same time:

*************************************************************************
* EMC Symmetrix FCP MPIO Software Support for Symmetrix devices has *
* been found installed or Symmetrix FCP Software Support is being *
* installed simultaneously. It is necessary to remove all previous *
* versions of the EMC.Symmetrix.fcp.MPIO.rte software before Symmetrix *
* FCP Software Support can be installed. *
* Select the correct software fileset to install or run the following *
* command to remove the EMC.Symmetrix.fcp.MPO.rte software. *
* *
* Please run 'installp -u EMC.Symmetrix.fcp.MPIO.rte' to remove the *
* installed fileset. *
* *
*************************************************************************
instal: Failed while executing the EMC.Symmetrix.fcp.rte.pre_i script.

*************************************************************************
* EMC Symmetrix FCP Software Support for Symmetrix devices has been *
* found installed or Symmetrix FCP MPIO Software Support is being *
* installed simultaneously. It is necessary to remove all previous *
* versions of the EMC.Symmetrix.fcp.rte software before Symmetrix FCP *
* MPIO Software Support can be installed. *
* Select the correct software fileset to install or run the following *
* command to remove the EMC.Symmetrix.fcp.rte software *
* *
* Please run 'installp -u EMC.Symmetrix.fcp.rte' to remove the *
* installed fileset. *
* *
*************************************************************************
instal: Failed while executing the EMC.Symmetrix.fcp.MPIO.rte.pre_i script.

So, it's either one or the other. Why the hassle, you may ask? Thing is, right now we do not have multiple HBA attachments for the same disks. In this situation, either case will work. In the case of MPIO, each disk will have one path through the lone HBA. However, management says one day we might do double HBA attachment, and the choice will be between MPIO and EMC Powerpath. Powerpath requires EMC.Symmetrix.fcp.rte, which cannot coexist with EMC.Symmetrix.fcp.MPIO.rte, as we have just seen. So, the possible scenarios are:

1. Configure now with MPIO and schedule down time if Powerpath is chosen
2. Configure now without MPIO, and schedule down time if MPIO is chosen

To me, choice 1 seems clear. Screw EMC!!