A few Netapp dedup commands

I found these Netapp commands when I started digging into some dedup performance issues:

• > sis start -s /vol/$volume    
• This fixes very slow normal dedup operations due to this bug http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=657692
• > sis status   
• Shows the status and a few settings of all of your volumes
• > df -s /vol/$volume   
• Shows some space savings
• > sysstat 1
• Shows you some basic performance stats updated once per second

Compare two files and show the additional lines in one that aren't in the other.

Quote often I have two sorted files and I want to "subtract" one from the other.  I've used a few different tools like perl, grep, and awk with success but they can be slow.

In typical linux fashion there is a tool that does exactly what I need very quickly called join.

Here is a quick example: join -1 3 -2 1 -v 1 < file1  < file2  > output

That is saying use column 3 on file1 ( -1 3 ), and column 1 on file2 ( -2 1 ).  It will then show only the lines that are in file1 but not file2 ( -v 1 ).

The files must already be sorted on those columns before you run join.

Way to create an rpm

I ran across a tool that looks very promising to turn a directory of files into an rpm.

It is called fpm and I'm starting to look into it now.

Running Parallel Commands on a group of Linux boxes

I have been using a program called cssh for a while now and really appreciated the features of it.  Every once in a while I wanted to run commands from the command line and not spawn a bunch of new windows.  I recently ran across polysh which works very nicely and does exactly what I wanted.

What is taking up all this space?

We had a filesystem that kept growing in size even though the user was deleting the log files. du and df didn't match up and it was starting to be a problem. We restarted one of the apps and got back a little space so I figured there were still open file handles holding on to the space. 

>>lsof | grep deleted

 That showed me the app that was holding on to the space.

>>lsof | grep deleted | awk '{print $1, $2, $7, $9}'

 Then on the PID of the app I ran:

 >>kill -HUP 8787

 Presto, that freed up all the disk space.

Issue starting a VM on Citrix Xen after an outage

After a recent problem with our Citrix Xen cluster we had one Windows VM that refused to start up:

[root@XENSERVER1 ~]# xe vm-start uuid=4dc9528c-0eb7-38fc-7246-66b387d6aa0e
Error code: SR_BACKEND_FAILURE_46
Error parameters: , The VDI is not available [opterr=VDI aae84c26-4520-45cd-ad66-5e379874f5dd already attached RW],

After a lot of poking and prodding the VM's disks seemed fine.
I tried to export the VM and received this message:

[root@XENSERVER1 ~]# xe vm-export uuid=4dc9528c-0eb7-38fc-7246-66b387d6aa0e filename=/var/run/sr-mount/3184f86f-9743-b819-5cd5-e84ccf7e7c6c/win-server-export.vdi
The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.
message: Failure("The VDI aae84c26-4520-45cd-ad66-5e379874f5dd is already attached in RW mode; it can't be attached in RO mode!")

That got me on the right track and I did this:

[root@XENSERVER1 ~]# xe-toolstack-restart
Stopping xapi: .. [ OK ]
Stopping the v6 licensing daemon: [ OK ]
Stopping the memory ballooning daemon: [ OK ]
Stopping perfmon: [ OK ]
Stopping the fork/exec daemon: [ OK ]
Stopping the multipath alerting daemon: [ OK ]
Starting the multipath alerting daemon: [ OK ]
Starting the fork/exec daemon: [ OK ]
Starting perfmon: [ OK ]
Starting the memory ballooning daemon: [ OK ]
Starting the v6 licensing daemon: [ OK ]
Starting xapi: ....start-of-day complete. [ OK ]
done.

And presto the VM booted:

[root@XENSERVER1 ~]# !250
xe vm-start uuid=4dc9528c-0eb7-38fc-7246-66b387d6aa0e

The problem was that XENSERVER1 was the only one that was not rebooted during the outage since it was the master.

Simple log/file rotation

I wanted a nice way to do an hourly backup of a mysql database and only keep a days worth.

In the past I have written scripts that would check how many files were there or used find to delete
old only.

I wondered if I could use logrotate to handle this since it is installed on most systems and is simple to use.
Unfortunately it doesn't understand the term hourly and only does daily and monthly.

I wrote my backup script and created a simple logrotate.conf file but left out the section that said
how often to run.

[root@server1 bin]# cat backup-www-db.logrotate
/var/database_backup/mysql_dump.sql {
# hourly
rotate 24
compress
delaycompress
missingok
create 640 root adm
}

Then I just told the backup script to run logrotate as the first thing it does giving it that config file:
/usr/sbin/logrotate /usr/adm/bin/backup-www-db.logrotate

It rotates the previous database dump and compresses them.

It only runs when the script runs so I set cron to run every hour and we have a nice simple solution.

Cloning Redhat Servers in Vmware and networking

After cloning a Redhat/Centos box in vmware and booting it, the network refused to come up.

I found that udev had the MAC's for the old server hard coded and did not clean them up on the reboot.

Commenting out the old lines and rebooting forced udev to recreate new entries and away we went.

{ krkardlnxapp2 } [ /etc/udev/rules.d ]$ cat 70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x15ad:0x07b0 (vmxnet3) (custom name provided by external tool)
#SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:b5:00:c9", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x15ad:0x07b0 (vmxnet3)
#SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:b5:08:a0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x15ad:0x07b0 (vmxnet3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:b5:08:a0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

Solaris Ldom's

I received a loaner Sparc T5240 box from oracle the other day to test out their ldom's. Here are some basic commands I used to create a guest. Note, I didn't go into how to install Solaris inside the ldom.

I installed version 1.3 of the Ldom software which I downloaded directly from Oracle. ( LDoms_Manager-1_3.zip )
The software has an installer that sets up many of the things you need.

One thing I found after looking around was that the networking inside a guest couldn't talk to the outside world.
I adjusted the networking including plumbing the virtual switch and giving it the same MAC as the underlying nxge4 ( 10G ) interface.

ifconfig nxge0 down
ifconfig nxge0 unplumb
ifconfig vsw0 plumb
ifconfig vsw0 X.X.X.X netmask 255.255.255.0
ifconfig vsw0 up
mv /etc/hostname.nxge4 /etc/hostname.vsw0
ldm remove-vswitch primary-vsw0
ldm add-vsw mac-addr=0:14:4f:c1:c2:c8 net-dev=nxge4 primary-vsw0 primary

Now on to the guest:

First I created two zfs zvols. One for the OS and one for a zpool.

zfs create -V 12000m pool0/vdomains/ldom2
zfs create -V 100g pool0/vdomains/zpool-ldom2

To install solaris inside the guest I am going to use the 10u9 iso image and have it in /tmp/iso/sol-10-u9-ga-sparc-dvd.iso

I gave it 4G of RAM and 1 virtual CPU:

ldm add-domain ldom2
ldm stop test1
ldm add-vcpu 1 ldom2
ldm add-memory 4096m ldom2
ldm add-vnet vnet1 primary-vsw0 ldom2
ldm add-vdsdev /dev/zvol/dsk/pool0/vdomains/ldom2 zdisk2@primary-vds0
ldm add-vdisk zdisk2 zdisk2@primary-vds0 ldom2
ldm add-vdsdev /dev/zvol/dsk/pool0/vdomains/zpool-ldom2 zdisk3@primary-vds0
ldm add-vdisk zdisk3 zdisk3@primary-vds0 ldom2
ldm add-vdsdev /tmp/iso/sol-10-u9-ga-sparc-dvd.iso iso_vol@primary-vds0
ldm-vdisk vdisk vdisk_iso iso_vol@primary-vds0 ldom2
ldm bind-domain ldom2
ldm set-variable autoboot\?=false ldom2
ldm start ldom2

Now I can telnet to port 5000 on the localhost and I'll will be on the console but the machine will be at the OK> prompt.

Type: show-disks to find the ISO, or CDROM link and then type boot and the long path to boot off the ISO.

I did a typical Solaris install for the guest and actually used a flar.

Once the install was done I created a zpool on the second disk and I had a fully working solaris guest with zfs inside.

To get the guest to book automatically and not stop at the OK> prompt each time I changed the autoboot to true from false.

Look at the latest patch number in solaris

Here is a quick way to print out just the latest patch versions on a solaris box:
showrev -p | perl -lane '($p,$rev) = ($F[1] =~ /(\d+)-(\d+)/); $has{$p}=$rev if $has{$p}<$rev; END { map { print "$_-$has{$_}" } sort keys %has }'

Deleting all the blastwave packages on a Solaris node

I needed to delete all of the Blastwave packages on a Solaris node to clean it up and I didn't want to do them one at a time or keep hitting yes.

Here is a quick way that I found:

>> yes | pkgrm $(pkinfo | awk '/CSW/ { print $2}')

ludelete problem in Live Upgrade

I needed to delete a live upgrade environment to get ready to make a new one for 10u8.

Running the ludelete command gave me this error:

>>ludelete 10u6_disk2

System has findroot enabled GRUB

Checking if last BE on any disk...

ERROR: Last BE on disk

ERROR: This boot environment <10u6_disk2> is the last BE on the above disk.

ERROR: Deleting this BE may make it impossible to boot from this disk.

ERROR: However you may still boot solaris if you have BE(s) on other disks.

ERROR: You *may* have to change boot-device order in the BIOS to accomplish this.

ERROR: If you still want to delete this BE <10u6_disk2>, please use the force option (-f).

Unable to delete boot environment.

So I tried it with the -f option after I checked to make sure that /dev/dsk/c3t0d0 was in fact the correct disk ( booting off /dev/dsk/c3t4d0 ).

The next problem I ran in to was that it couldn't mount some zfs filesystems that it thought should be there but didn't exist anymore.

>>ludelete -f 10u6_disk2

System has findroot enabled GRUB

No entry for BE <10u6_disk2> in GRUB menu

ERROR: mount: /storage/pool0/blah: No such file or directory

ERROR: cannot mount mount point device

ERROR: failed to mount file system on

ERROR: unmounting partially mounted boot environment file systems

ERROR: cannot mount boot environment by icf file

ERROR: Cannot mount BE <10u6_disk2>.

Unable to delete boot environment.

I manually edited /etc/lu/ICF.1 and cleared out all of the zfs mounts and re-ran ludelete which returned even though it was still upset about a few mounts.

>>ludelete -f 10u6_disk2

System has findroot enabled GRUB

No entry for BE <10u6_disk2> in GRUB menu

Determining the devices to be marked free.

ERROR: cannot open 'pool0/blah2/sendsrc': dataset does not exist

ERROR: cannot mount mount point device

ERROR: failed to mount file system on

ERROR: unmounting partially mounted boot environment file systems

ERROR: cannot mount boot environment by icf file

WARNING: Unable to mount ABE <10u6_disk2>: cannot complete lumk_iconf

WARNING: Unable to determine disk partition configuration information for BE <10u6_disk2>.

WARNING: Unable to determine the devices/datasets to be freed for BE <10u6_disk2>.

Updating boot environment configuration database.

Updating boot environment description database on all BEs.

Updating all boot environment configuration databases.

Boot environment <10u6_disk2> deleted.

And to check the status and make sure it is gone:

>>lustatus

Boot Environment Is Active Active Can Copy

Name Complete Now On Reboot Delete Status

-------------------------- -------- ------ --------- ------ ----------

10u6 yes yes yes no -

Solaris PCA patch utility

I needed to patch one of our solaris nodes before taking a flar of it so I pulled out the trusty PCA tool.

When I ran it, with my Sunsolve account username/password, I received a 401 unauthorized error for each patch.

I have an active Sunsolve account linked to our contracts, and I can sign into the web page with no problems, so I didn't understand why it wasn't working.

I then logged into the Sunsolve account, searched for the list of patches that pca -l gave me, and built up their download script.

When I ran it I received the same error, unauthorized.

I took the exact link that the download script was using for one patch and put it in my browser. A page to accept the license agreement came up and after I accepted the license everything started working again.

I sure hope this isn't a foreshadow into what things will be like with "Oracle".

Bash History

A quick way to add a date/time stamp to your bash history:

export HISTTIMEFORMAT="::%h/%d - %H:%M:%S:: "

You can put it in your favorite bash start up file like .bash_profile or .bashrc

How to convert a mac from Cisco/Foundry format to Unix format

I needed to associate servers to ports on our switches but the formats are different so I needed a way to convert them.

Here is a quick perl one line to go from Cisco/Foundry to Unix:

perl -ane '$_=~s/(..)\.?/\1:/g;$_=~s/:$//g; print $_'

An example:

echo 0019.b9f5.93f3 | perl -ane '$_=~s/(..)\.?/\1:/g;$_=~s/:$//g; print $_'

00:19:b9:f5:93:f3

Updating the file extension association in OSX

I wanted to change the association of the .avi extention on my MacBook Pro running Leopard to VLC.

To do it, right click on the file and click Get Info.  Then at the bottom of the drop down menu change the program from the list.  Then click "Change All" button if you want to change it for every file.

Linksys WAP54G and DD-WRT

I am staying in a place that has a nice connection and wireless.  Perfect for my Macbook Pro, not so good for the desktop.  I found a v3 WAP54G and thought I would just hook it up as a wireless client and put my desktop behind that.  Well, it turns out that isn't as easy as one might hope.

All is not lost, welcome DD-WRT.  I went here: http://www.dd-wrt.com/wiki/index.php/Installation
and downloaded this firmware: http://www.dd-wrt.com/dd-wrtv2/downloads/stable/dd-wrt.v24%20SP1/Consumer/Linksys/WAP54G_v3/dd-wrt.v24_micro_generic.bin

I uploaded it on to the Linksys through their firmware upgrade page and when it was done I pulled the plug and waited about 15sec ( I was impatient! ).

After some trial and error ( and wondering ) I found it at http://192.168.1.1

I set wireless to be client, put in the house SSID, and under Wireless Security, I set the Security mode and entered the house WPA Shared Key.

I also made sure my internal LAN network ( 192.168.1.0/24 ) was NOT the same as the house LAN network.  The house is set to be 192.168.0.0/24 so I was okay, however I changed it to a 172.16.1.0/24 network just to make sure there wasn't any confusion.

I set up the DHCP server for the LAN, told it to use the House Router as it's primary DNS and then use OpenDNS http://opendns.com ( 208.67.222.222 and 208.67.220.220 ) as the second and third.  I've been very happy with OpenDNS and highly recommend it.

So now, my workstation is behind the WAP54G which is acting like a firewall/router to the house network and all is happy.

I'll probably set up some port forwarding so I can talk to my workstation from the office but that is trivial under DD-WRT.

I am very pleased with how well it is working and how easy it was to set up.  There is a tone of info in the interface and I'm going to enjoy playing around with it.

Clock setting in Linux

In almost all applications time being in sync is very important. Even simple logging needs accurate time to keep the logs in tune.

Servers have a hardware clock and an OS software clock and both need to be in sync as well as in sync with a certified time source.

If you already use NTP to keep your software clock in sync with a certified time source it is pretty easy to keep your hardware clock up to date with the OS time.

Here are some basic commands:

To set the syste/OSm clock from the hardware clock:
#> sudo hwclock --hctosys

Likewise if your hardware clock is set incorrectly (for example, if you replaced the
CMOS battery on your motherboard), you can set the hardware clock from your system clock as follows:

Likewise, set the hardware clock from the software/OS clock:

#> hwclock --systohc

As always, there are a lot more options available so check out the man pages.

How to add a disk to a vmware linux guest without rebooting

I have a whole series of linux guests running on some ESX hosts off NFS shares.

Today I wanted to add another disk to a running guest and couldn't reboot it.
ESX let me add the disk but the CentOS 5.3 guests didn't see it automatically.

I ran this command:
echo - - - > /sys/class/scsi_host/host0/scan

"That is echo dash space dash space dash /sys/class/scsi_host/host zero /scan"

and then the disk showed up under fdisk -l

Put a new LVM on it, added a filesystem, and mounted it.

Vlan tagging between HP and Cisco switches

I needed to get some vlans between a couple HP and Cisco switches and ran into some problems with it. After some digging I found the solution:

• On the HP set traffic on the trunk port to be tagged for every VLAN you want to propagate
• Allow access to the VLAN on the non-trunk ports but set them to be untagged

Good luck!