What is taking up all this space?

We had a filesystem that kept growing in size even though the user was deleting the log files. du and df didn't match up and it was starting to be a problem. We restarted one of the apps and got back a little space so I figured there were still open file handles holding on to the space. 

>>lsof | grep deleted

 That showed me the app that was holding on to the space.

>>lsof | grep deleted | awk '{print $1, $2, $7, $9}'

 Then on the PID of the app I ran:

 >>kill -HUP 8787

 Presto, that freed up all the disk space.

Issue starting a VM on Citrix Xen after an outage

After a recent problem with our Citrix Xen cluster we had one Windows VM that refused to start up:

[root@XENSERVER1 ~]# xe vm-start uuid=4dc9528c-0eb7-38fc-7246-66b387d6aa0e
Error code: SR_BACKEND_FAILURE_46
Error parameters: , The VDI is not available [opterr=VDI aae84c26-4520-45cd-ad66-5e379874f5dd already attached RW],

After a lot of poking and prodding the VM's disks seemed fine.
I tried to export the VM and received this message:

[root@XENSERVER1 ~]# xe vm-export uuid=4dc9528c-0eb7-38fc-7246-66b387d6aa0e filename=/var/run/sr-mount/3184f86f-9743-b819-5cd5-e84ccf7e7c6c/win-server-export.vdi
The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.
message: Failure("The VDI aae84c26-4520-45cd-ad66-5e379874f5dd is already attached in RW mode; it can't be attached in RO mode!")

That got me on the right track and I did this:

[root@XENSERVER1 ~]# xe-toolstack-restart
Stopping xapi: .. [ OK ]
Stopping the v6 licensing daemon: [ OK ]
Stopping the memory ballooning daemon: [ OK ]
Stopping perfmon: [ OK ]
Stopping the fork/exec daemon: [ OK ]
Stopping the multipath alerting daemon: [ OK ]
Starting the multipath alerting daemon: [ OK ]
Starting the fork/exec daemon: [ OK ]
Starting perfmon: [ OK ]
Starting the memory ballooning daemon: [ OK ]
Starting the v6 licensing daemon: [ OK ]
Starting xapi: ....start-of-day complete. [ OK ]
done.

And presto the VM booted:

[root@XENSERVER1 ~]# !250
xe vm-start uuid=4dc9528c-0eb7-38fc-7246-66b387d6aa0e

The problem was that XENSERVER1 was the only one that was not rebooted during the outage since it was the master.

Simple log/file rotation

I wanted a nice way to do an hourly backup of a mysql database and only keep a days worth.

In the past I have written scripts that would check how many files were there or used find to delete
old only.

I wondered if I could use logrotate to handle this since it is installed on most systems and is simple to use.
Unfortunately it doesn't understand the term hourly and only does daily and monthly.

I wrote my backup script and created a simple logrotate.conf file but left out the section that said
how often to run.

[root@server1 bin]# cat backup-www-db.logrotate
/var/database_backup/mysql_dump.sql {
# hourly
rotate 24
compress
delaycompress
missingok
create 640 root adm
}

Then I just told the backup script to run logrotate as the first thing it does giving it that config file:
/usr/sbin/logrotate /usr/adm/bin/backup-www-db.logrotate

It rotates the previous database dump and compresses them.

It only runs when the script runs so I set cron to run every hour and we have a nice simple solution.

Cloning Redhat Servers in Vmware and networking

After cloning a Redhat/Centos box in vmware and booting it, the network refused to come up.

I found that udev had the MAC's for the old server hard coded and did not clean them up on the reboot.

Commenting out the old lines and rebooting forced udev to recreate new entries and away we went.

{ krkardlnxapp2 } [ /etc/udev/rules.d ]$ cat 70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x15ad:0x07b0 (vmxnet3) (custom name provided by external tool)
#SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:b5:00:c9", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x15ad:0x07b0 (vmxnet3)
#SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:b5:08:a0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x15ad:0x07b0 (vmxnet3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:b5:08:a0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

Solaris Ldom's

I received a loaner Sparc T5240 box from oracle the other day to test out their ldom's. Here are some basic commands I used to create a guest. Note, I didn't go into how to install Solaris inside the ldom.

I installed version 1.3 of the Ldom software which I downloaded directly from Oracle. ( LDoms_Manager-1_3.zip )
The software has an installer that sets up many of the things you need.

One thing I found after looking around was that the networking inside a guest couldn't talk to the outside world.
I adjusted the networking including plumbing the virtual switch and giving it the same MAC as the underlying nxge4 ( 10G ) interface.

ifconfig nxge0 down
ifconfig nxge0 unplumb
ifconfig vsw0 plumb
ifconfig vsw0 X.X.X.X netmask 255.255.255.0
ifconfig vsw0 up
mv /etc/hostname.nxge4 /etc/hostname.vsw0
ldm remove-vswitch primary-vsw0
ldm add-vsw mac-addr=0:14:4f:c1:c2:c8 net-dev=nxge4 primary-vsw0 primary

Now on to the guest:

First I created two zfs zvols. One for the OS and one for a zpool.

zfs create -V 12000m pool0/vdomains/ldom2
zfs create -V 100g pool0/vdomains/zpool-ldom2

To install solaris inside the guest I am going to use the 10u9 iso image and have it in /tmp/iso/sol-10-u9-ga-sparc-dvd.iso

I gave it 4G of RAM and 1 virtual CPU:

ldm add-domain ldom2
ldm stop test1
ldm add-vcpu 1 ldom2
ldm add-memory 4096m ldom2
ldm add-vnet vnet1 primary-vsw0 ldom2
ldm add-vdsdev /dev/zvol/dsk/pool0/vdomains/ldom2 zdisk2@primary-vds0
ldm add-vdisk zdisk2 zdisk2@primary-vds0 ldom2
ldm add-vdsdev /dev/zvol/dsk/pool0/vdomains/zpool-ldom2 zdisk3@primary-vds0
ldm add-vdisk zdisk3 zdisk3@primary-vds0 ldom2
ldm add-vdsdev /tmp/iso/sol-10-u9-ga-sparc-dvd.iso iso_vol@primary-vds0
ldm-vdisk vdisk vdisk_iso iso_vol@primary-vds0 ldom2
ldm bind-domain ldom2
ldm set-variable autoboot\?=false ldom2
ldm start ldom2

Now I can telnet to port 5000 on the localhost and I'll will be on the console but the machine will be at the OK> prompt.

Type: show-disks to find the ISO, or CDROM link and then type boot and the long path to boot off the ISO.

I did a typical Solaris install for the guest and actually used a flar.

Once the install was done I created a zpool on the second disk and I had a fully working solaris guest with zfs inside.

To get the guest to book automatically and not stop at the OK> prompt each time I changed the autoboot to true from false.

Look at the latest patch number in solaris

Here is a quick way to print out just the latest patch versions on a solaris box:
showrev -p | perl -lane '($p,$rev) = ($F[1] =~ /(\d+)-(\d+)/); $has{$p}=$rev if $has{$p}<$rev; END { map { print "$_-$has{$_}" } sort keys %has }'

Deleting all the blastwave packages on a Solaris node

I needed to delete all of the Blastwave packages on a Solaris node to clean it up and I didn't want to do them one at a time or keep hitting yes.

Here is a quick way that I found:

>> yes | pkgrm $(pkinfo | awk '/CSW/ { print $2}')