Users started complaining about spotty application performance; sometimes it was good, sometimes it was horrible. The first thing came to my mind was some kind of database bottleneck. In my experience, CPU and I/O contention are usually the culprits but one time (not this time) it actually was a network problem. Here's how I broke it down...

The application in question was a J2EE application running in an AIX LPAR on top of Oracle (9.2.0.7) running in AIX on a separate LPAR. In addition, there were a bunch of batch jobs, written in a variety of languages, that run on the database LPAR throughout the day.

The first thing that I did was to collect data; without it, I would have been guessing.

Figure out disk

There are two things I used to investigate I/O: sar and oracle statspack data. I analyzed the sar data at a daily level and the statspack data over a weekly period. It so happened that this client bounces oracle every week (clearing statspack data) so I pulled the data towards the end of the week.

SAR

Since we were running on AIX the sar utility is really helpful for getting disk and cpu stats. We scheduled it to run every 5 minutes 24/7. After a few days we had a nice sample set of data to graph and analyze.

I usually pull the sar data together into an excel spreadsheet (attached at bottom of article). It is easier to explain hot disks by seeing them on a topographical 3D map, and how many times do you get to use a 3D surface area chart?

I found several hdisks were at 100% utilization at certain times of the day and were over 80% average utilization for the entire day. Other disks were hardly being used. This looked like a classic I/O bottleneck. The question on my mind was what are these hdisks and how do they map to mount points?

What really kind of suprised me was what happened when I rotated the graph (nice Excel feature)...

Why were hdisk9-24 mirrors of hdisk27-42 and hdisk50-54 mirros of hdisk55-59? What was causing this mirroring?

DB STATS

The Oracle DBAs told me that they had configured oracle to balance our datafiles over a certain set of mount points. Since Oracle is the main I/O user on the LPAR, I wanted to make sure that this was working properly. If Oracle were balanced across the mount points, what was causing the imbalance at the hdisk level?

Oracle Query for I/O on each mount point

SELECT 
  substr(filename,0,
    instr( filename,'/',1,4)) mount,
  sum(phyrds) reads,
  sum(phywrts) writes,
  sum(phyrds + phywrts) total
FROM 
  perfstat.stats$filestatxs data
WHERE 
  filename like '/oracle/kna/p151i0%'
GROUP BY 
  substr(filename, 0,
    instr( filename, '/', 1, 4))

Note: in this case our mount point is everything to the left of the 4th '/'.

It gives us an output that looks like this (when put into excel):

So Oracle wasn't really all that balanced across the mount points; several mount points are being hit much harder than others. This seems to correlate with the sar data.

Checkpoint

So far I had determined that the I/O picture wasn't all that great. The I/O was not balanced across the hdisks or mount points and sometimes spiked to 100% utilization. At this point, I needed to put the I/O data together and figure out what was going on. More specifically, I needed to know:

1. How does an hdisk map to a mount point?
2. How does hdisk mirroring happen?

Mapping mount points to hdisks

This required a bit of web learning. Here's what I've learned about the AIX I/O stack.

Here's the general structure of an AIX I/O stack:
mount point
logical volume (in a volume group)
physical volume\virtual path
hdisk

So in order to answer How does an hdisk map to a mount point? and to generate the above diagram, I needed to go through each layer and figure out how it mapped to the layer below.

Thankfully there is an AIX configuration tool called smit. Essentially it is a text based UI that runs normal system commands. The key benefit is that you don't have to know the underlying system commands (or slog through a bunch of man pages of commands that you don't even know!). It is pretty easy to navigate the menus but here are the commands that I ended up running:

• Find out logical volume and mount point per volume group
  • lsvg -o|lsvg -i -l
• Find out physical volumes for a logical volume
  • lslv '-l' logical_volume_name
• Find out hdisks in a physical volume
  • lsvpcfg

I'm sure someone has written a handy script to smash all of those together to produce a mapping from volume group down to hdisk but I assembled it manually.

What was very interesting was the fact that two hdisks were mapped to each vpath (physical volume). Was AIX somehow load balancing I/O across the hdisks in a vpath? Yep.. that's exactly what was going on. It turns out that each fiber pathway to a LUN gets its own hdisk in AIX (by default). So if you have two fiber cards connecting to the SAN, you'll get two hdisks/LUN; four fiber cards will make four hdisks/LUN, etc. So that answers my question about mirroring.

Conclusion

In this case, there were three different technologies attempting to evenly distribute and load-balance I/O generated by Oracle: Oracle itself, the AIX I/O stack and the SAN/DASD device. The storage team was telling me (after I asked) that certain LUNs were getting hit a lot harder than others. SAR gave me information about the AIX stack which confirmed that certain mirrored hdisk pairs were getting hit a lot harder than others. Oracle's db stats told me that certain mount points were being used more heavily than others.

We're in the process of fixing this issue.. our main attack strategy is:

• go over our AIX configuration parameters.. make certain they are in line with Oracle best practices.
• re-balance our data across the mount points at the Oracle level.
• change our AIX I/O mappings so that mount points map 1:1 with LUNs. Our current design has some overlap-- two mount points sometimes share the same LUN.
• upgrade the filesystems to JFS2.