hotFilesInTmp.htm
05-05-2002 Bikle
-------------------------------
Hot Files on /tmp
Suppose you are the proud owner of an Oracle 9i database. It serves a
decision support application which means that on Friday night you wipe
it clean and then you load it up with a terabyte of data every
weekend. It is probable that this database would be operated during
the weekend with archiving turned off.
If you place your online redolog files in /tmp, this will boost
performance during each weekend load. The load will run faster
than if the online redolog files reside on a disk device like
a raw partition.
Why is this?
I conceptualize /tmp to be a giant memory structure rather than a file
system (on Solaris 8 anyway). It does not live on disk; it lives in
RAM. If you buy a Sun box with 64 GB of RAM (A fully loaded V880 for
example) then /tmp has the ability to easily hold three online redolog
files which are each 1GB in size.
Be aware, however, that any file placed in /tmp is subject to removal
during machine reboot. So, if the machine reboots during the
dataload, your database is toast. Perhaps that is a risk you are
willing to take (after all, you are not running XP on a Dell). Also,
some DSS databases are designed to be wiped clean before each data load.
So, if the machine does reboot perhaps it will be no big deal to just
wipe the broken database clean and restart the load. During the week
when the database is serving the DSS application, the DBA would probably
want to move the online redologs to a normal disk location and perhaps
even turn on archiving.
Lets run with the idea a little further. During the weekend load,
aside from the redologs, could you put any other "hot" files in /tmp?
Maybe yes. I'd certainly take a look at data files which hold the
UNDO tablespace. A Sun box with 64 GB of RAM would give you a lot
of wiggle room.
You might ask, "Do hot files in /tmp generate any I/O at all?"
I say, "Lets write a simple shell script and find out."
Here is a script I wrote which generates many reads/writes on a file in /tmp:
hotFilesInTmp-sh.txt
Here is a script I wrote which generates many reads/writes on a file in /bikle:
hotFilesInBikle-sh.txt
Next, I worked towards putting these scripts into action and using iostat
to tell me about I/O activity.
First I used the df command to give me some info about /tmp:
bikle22# df -k /tmp
Filesystem kbytes used avail capacity Mounted on
swap 26999136 88 26999048 1% /tmp
It appears that /tmp has a 26GB of free space available.
I used prtdiag to tell me about RAM:
bikle22# prtdiag
System Configuration: Sun Microsystems sun4u Sun Fire 880
System clock frequency: 150 MHz
Memory size: 32768 Megabytes
========================= CPUs ===============================================
Run E$ CPU CPU
Brd CPU MHz MB Impl. Mask
--- --- ---- ---- ------- ----
A 0 750 8.0 US-III 5.4
B 1 750 8.0 US-III 5.4
A 2 750 8.0 US-III 5.4
B 3 750 8.0 US-III 5.4
C 4 750 8.0 US-III 5.4
D 5 750 8.0 US-III 5.4
C 6 750 8.0 US-III 5.4
D 7 750 8.0 US-III 5.4
========================= Memory Configuration ===============================
Logical Logical Logical
MC Bank Bank Bank DIMM Interleave Interleaved
Brd ID num size Status Size Factor with
---- --- ---- ------ ----------- ------ ---------- -----------
A 0 0 1024MB no_status 512MB 8-way 0
A 0 1 1024MB no_status 512MB 8-way 0
A 0 2 1024MB no_status 512MB 8-way 0
A 0 3 1024MB no_status 512MB 8-way 0
B 1 0 1024MB no_status 512MB 8-way 1
B 1 1 1024MB no_status 512MB 8-way 1
B 1 2 1024MB no_status 512MB 8-way 1
B 1 3 1024MB no_status 512MB 8-way 1
A 2 0 1024MB no_status 512MB 8-way 0
A 2 1 1024MB no_status 512MB 8-way 0
A 2 2 1024MB no_status 512MB 8-way 0
A 2 3 1024MB no_status 512MB 8-way 0
B 3 0 1024MB no_status 512MB 8-way 1
B 3 1 1024MB no_status 512MB 8-way 1
B 3 2 1024MB no_status 512MB 8-way 1
B 3 3 1024MB no_status 512MB 8-way 1
C 4 0 1024MB no_status 512MB 8-way 2
C 4 1 1024MB no_status 512MB 8-way 2
C 4 2 1024MB no_status 512MB 8-way 2
C 4 3 1024MB no_status 512MB 8-way 2
D 5 0 1024MB no_status 512MB 8-way 3
D 5 1 1024MB no_status 512MB 8-way 3
D 5 2 1024MB no_status 512MB 8-way 3
D 5 3 1024MB no_status 512MB 8-way 3
C 6 0 1024MB no_status 512MB 8-way 2
C 6 1 1024MB no_status 512MB 8-way 2
C 6 2 1024MB no_status 512MB 8-way 2
C 6 3 1024MB no_status 512MB 8-way 2
D 7 0 1024MB no_status 512MB 8-way 3
D 7 1 1024MB no_status 512MB 8-way 3
D 7 2 1024MB no_status 512MB 8-way 3
D 7 3 1024MB no_status 512MB 8-way 3
========================= IO Cards =========================
Bus Max
IO Port Bus Freq Bus Dev,
Brd Type ID Side Slot MHz Freq Func State Name Model
---- ---- ---- ---- ---- ---- ---- ---- ----- -------------------------------- ----------------------
I/O PCI 8 B 0 33 33 5,0 ok pci-pci1011,26.5/SUNW,qlc (scsi-+ PCI-BRIDGE
I/O PCI 8 B 0 33 33 4,0 ok SUNW,qlc-pci1077,2200.1077.4083.+ device on pci-bridge
I/O PCI 8 B 0 33 33 5,0 ok SUNW,qlc-pci1077,2200.1077.4083.+ device on pci-bridge
I/O PCI 9 B 6 33 33 2,0 ok network-pci108e,2bad SUNW,pci-gem
I/O PCI 9 A 8 33 66 1,0 ok pci108e,1000-pci108e,1000.1
I/O PCI 9 A 8 33 66 1,1 ok SUNW,hme-pci108e,1001 SUNW,qsi-cheerio
I/O PCI 9 A 7 33 66 2,0 ok pci108e,1000-pci108e,1000.1
I/O PCI 9 A 7 33 66 2,1 ok SUNW,hme-pci108e,1001 SUNW,qsi-cheerio
No failures found in System
===========================
It appears that of the 32GB of RAM on this box, only 6GB is being used.
Next, I used the "swap -l" command to tell me about the swap device:
bikle22# swap -l
swapfile dev swaplo blocks free
/dev/dsk/c1t0d0s1 118,65 16 4096576 4096576
I need to know about the swap device because "df" is telling me that /tmp on this box
is mounted on "swap". If any I/O happens in /tmp it will show up on this device.
In this case the swap device is this: /dev/dsk/c1t0d0s1
The corresponding disk is this: /dev/dsk/c1t0d0
Next, I used the "iostat -xn" command to generate some column headings for a
simple iostat report I intended to create:
bikle22# iostat -xn
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t6d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0 0 c1t2d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.6 0 0 c1t4d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0 0 c1t8d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0 0 c1t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.6 0 0 c1t5d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0 0 c1t10d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0 0 c1t11d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0 c1t3d0
0.0 0.9 0.1 11.3 0.2 0.0 193.8 34.8 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0 0 c1t9d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.6 0 0 c1t12d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0 0 c1t13d0
0.0 0.6 0.3 2.7 0.0 0.0 0.0 1.9 0 0 c3t2B0000602200443Bd1
1.1 0.3 8.7 2.4 0.0 0.0 0.0 1.6 0 0 c3t2B0000602200443Bd0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 bikle22:vold(pid463)
bikle22#
These are the column headings I want:
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
They will come in handy later.
Next, I started shell script to generate lots of I/O in /tmp:
bikle22# cd /bikle
bikle22# ./hotFilesInTmp.sh
Then, I ran to another shell and used iostat to tell me about the
activity on the swap disk:
bikle22# iostat -xn 7 | grep c1t0d0
0.0 0.9 0.1 11.3 0.2 0.0 195.6 35.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.6 0.0 3.6 0.0 0.0 0.0 9.1 0 1 c1t0d0
0.0 3.9 0.0 30.9 0.0 0.4 0.0 107.2 0 3 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.1 0.0 0.1 0.0 0.0 0.0 6.8 0 0 c1t0d0
0.0 3.4 0.0 27.4 0.0 0.3 0.0 98.8 0 3 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 2.0 0.0 16.0 0.0 0.1 0.0 62.6 0 2 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.3 0.0 1.3 0.0 0.0 0.0 6.2 0 0 c1t0d0
0.0 1.9 0.0 10.9 0.0 0.0 0.0 6.0 0 1 c1t0d0
0.0 0.7 0.0 3.7 0.0 0.0 0.0 5.9 0 0 c1t0d0
0.0 2.7 0.0 20.7 0.0 0.1 0.0 52.4 0 2 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 1.4 0.0 7.4 0.0 0.0 0.0 6.8 0 1 c1t0d0
0.0 1.6 0.0 11.6 0.0 0.0 0.0 29.9 0 1 c1t0d0
^C
Next, I placed the column headings on the above output:
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.9 0.1 11.3 0.2 0.0 195.6 35.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.6 0.0 3.6 0.0 0.0 0.0 9.1 0 1 c1t0d0
0.0 3.9 0.0 30.9 0.0 0.4 0.0 107.2 0 3 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.1 0.0 0.1 0.0 0.0 0.0 6.8 0 0 c1t0d0
0.0 3.4 0.0 27.4 0.0 0.3 0.0 98.8 0 3 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 2.0 0.0 16.0 0.0 0.1 0.0 62.6 0 2 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.3 0.0 1.3 0.0 0.0 0.0 6.2 0 0 c1t0d0
0.0 1.9 0.0 10.9 0.0 0.0 0.0 6.0 0 1 c1t0d0
0.0 0.7 0.0 3.7 0.0 0.0 0.0 5.9 0 0 c1t0d0
0.0 2.7 0.0 20.7 0.0 0.1 0.0 52.4 0 2 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 1.4 0.0 7.4 0.0 0.0 0.0 6.8 0 1 c1t0d0
0.0 1.6 0.0 11.6 0.0 0.0 0.0 29.9 0 1 c1t0d0
The key column to notice is the "%b" column. I can see that
the most busy the disk got was 3% busy. That is very quiet.
A rule of thumb I use with iostat is that any disk less than
20% busy is not worth looking at when I'm looking for I/O problems.
Next, I wanted to see what iostat reported when I put hot files
in /bikle (which is a true disk location) rather than /tmp.
Here is a copy of the script:
#! /bin/sh
# hotFilesInBikle.sh
# simple script to see if a small hot file in /bikle contributes to any I/O
# cd to the right place
cd /bikle
cp /bin/date file1
while [ 0 ]
do
/bin/dd if=file1 of=file2 bs=123456 > /dev/null 2>&1
done
I used df to tell me which disk holds /bikle:
bikle22# df -k /bikle
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c1t0d0s0 32870474 5870303 26671467 19% /
It happens to be the same disk as the disk which holds the swap partition.
I started the hotFilesInBikle.sh script.
Here is what I found from iostat:
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.9 0.1 11.3 0.2 0.0 195.6 35.0 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
0.0 91.4 0.0 729.4 0.0 0.5 0.0 5.7 0 52 c1t0d0
0.0 167.3 0.0 1338.3 0.0 1.0 0.0 5.7 0 95 c1t0d0
0.0 159.7 0.0 1275.7 0.0 1.2 0.0 7.4 0 92 c1t0d0
0.0 167.3 0.0 1338.3 0.0 1.0 0.0 5.7 0 95 c1t0d0
0.0 167.4 0.0 1339.4 0.0 1.0 0.0 5.7 0 95 c1t0d0
0.0 167.3 0.0 1338.3 0.0 1.0 0.0 5.7 0 95 c1t0d0
0.0 97.1 0.0 777.1 0.0 0.6 0.0 5.7 0 55 c1t0d0
0.0 2.4 0.0 18.4 0.0 0.1 0.0 52.7 0 2 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0
I notice a very different picture here. The script ramped the disk
up to 95% busy.
So, I consider this good evidence that hot files (which are small
compared to available RAM) will generate negligible I/O if they
are placed in /tmp and they will generate significant I/O if they
are placed in a normal file system.
This information might be useful to a team looking at different
ways to deploy a large DSS system.