Issue starting a VM on Citrix Xen after an outage

After a recent problem with our Citrix Xen cluster we had one Windows VM that refused to start up:

[root@XENSERVER1 ~]# xe vm-start uuid=4dc9528c-0eb7-38fc-7246-66b387d6aa0e
Error code: SR_BACKEND_FAILURE_46
Error parameters: , The VDI is not available [opterr=VDI aae84c26-4520-45cd-ad66-5e379874f5dd already attached RW],

After a lot of poking and prodding the VM's disks seemed fine.
I tried to export the VM and received this message:

[root@XENSERVER1 ~]# xe vm-export uuid=4dc9528c-0eb7-38fc-7246-66b387d6aa0e filename=/var/run/sr-mount/3184f86f-9743-b819-5cd5-e84ccf7e7c6c/win-server-export.vdi
The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.
message: Failure("The VDI aae84c26-4520-45cd-ad66-5e379874f5dd is already attached in RW mode; it can't be attached in RO mode!")

That got me on the right track and I did this:

[root@XENSERVER1 ~]# xe-toolstack-restart
Stopping xapi: .. [ OK ]
Stopping the v6 licensing daemon: [ OK ]
Stopping the memory ballooning daemon: [ OK ]
Stopping perfmon: [ OK ]
Stopping the fork/exec daemon: [ OK ]
Stopping the multipath alerting daemon: [ OK ]
Starting the multipath alerting daemon: [ OK ]
Starting the fork/exec daemon: [ OK ]
Starting perfmon: [ OK ]
Starting the memory ballooning daemon: [ OK ]
Starting the v6 licensing daemon: [ OK ]
Starting xapi: ....start-of-day complete. [ OK ]
done.

And presto the VM booted:

[root@XENSERVER1 ~]# !250
xe vm-start uuid=4dc9528c-0eb7-38fc-7246-66b387d6aa0e

The problem was that XENSERVER1 was the only one that was not rebooted during the outage since it was the master.

Simple log/file rotation

I wanted a nice way to do an hourly backup of a mysql database and only keep a days worth.

In the past I have written scripts that would check how many files were there or used find to delete
old only.

I wondered if I could use logrotate to handle this since it is installed on most systems and is simple to use.
Unfortunately it doesn't understand the term hourly and only does daily and monthly.

I wrote my backup script and created a simple logrotate.conf file but left out the section that said
how often to run.

[root@server1 bin]# cat backup-www-db.logrotate
/var/database_backup/mysql_dump.sql {
# hourly
rotate 24
compress
delaycompress
missingok
create 640 root adm
}

Then I just told the backup script to run logrotate as the first thing it does giving it that config file:
/usr/sbin/logrotate /usr/adm/bin/backup-www-db.logrotate

It rotates the previous database dump and compresses them.

It only runs when the script runs so I set cron to run every hour and we have a nice simple solution.

Cloning Redhat Servers in Vmware and networking

After cloning a Redhat/Centos box in vmware and booting it, the network refused to come up.

I found that udev had the MAC's for the old server hard coded and did not clean them up on the reboot.

Commenting out the old lines and rebooting forced udev to recreate new entries and away we went.

{ krkardlnxapp2 } [ /etc/udev/rules.d ]$ cat 70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x15ad:0x07b0 (vmxnet3) (custom name provided by external tool)
#SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:b5:00:c9", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x15ad:0x07b0 (vmxnet3)
#SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:b5:08:a0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x15ad:0x07b0 (vmxnet3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:b5:08:a0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"