8th Annual Memphis Walk


For eight years now (see here or here or here or here or here), Richie, Robert, and I have been doing this thing where we meet at the Mississippi River cobblestones at dawn during a Saturday in the fall and walk until nearly sunset, seeing a good chunk of Memphis at a relaxed pace through out the day. For this edition of the Memphis Walk, we were joined by John Christenson, with the goal of following urban rivers.


This year, the Memphis Walk took us down Riverside Drive, past the finally-completed Beale Street Landing complex. I still think the cost of it was too high and time required for construction was too long, but at least it is here now and it is an impressive structure for visitors. I hope we are still using it 13 years from now, unlike the Pyramid. I’m sure we will still be paying for it then.



We walked through the downtown River Arts Festival before it opened for the day. I saw nearly as many food stands as artist booths, but I’m happy to see a vibrant and active South Main District.




Heading south on Main, you eventually come across one of my favorite companies in Memphis, at least in terms of the name. I love the winking screw lady…


Continuing south, we entered the area around I-55 and Crump Blvd that features a number of abandoned warehouses. I’ve driven past many of them before, and this was our chance to really peek inside and really take in how large, and sometimes beautiful, they are. (I even startled a homeless person that was living in the first floor of one of them.) We also came across some gems, such as an old fire engine next to a diesel repair shop.




There is also a huge recycling center in this area, near Kansas St at Wisconsin Ave, that I had never seen before.



Eventually, we circled around past the French Fort Historic District, near the National Ornamental Metal Museum, and headed south again, bypassing the turn-off to President’s Island. All along the way, we could see cobblestone streets that had been covered up by multiple layers of concrete and then asphalt, which was gradually breaking up and revealing the original roadbed below.



Seeking a way to get away from the industrial area and stick closer to the river, we walked through the Riverside Golf Course, which offered bluff views of McKellar Lake and the Riverside Bayou.



Even while you are still in Martin Luther Riverside Park, you can see the steam? stacks of the Valero Memphis Refinery, just over the tree line.


After passing the scrumptious smells of Jim Neely’s Interstate BBQ on Highway 61 (also South 3rd St), we turned off into an interesting area, bisected by levees and raised railroad bed and bridges that exists just to the south of I-55 and north of Nonconnah Creek. This grassy area is highly maintained and contains several gas and other fuel pipelines, as well as (possibly) the rumored direct Valero fuel pipeline that goes directly to the Memphis Airport and FedEx.


Most intriguing, there are a number of small farm operations in this area which look like something you would find in the most remote parts of the southern US, complete with horses, chickens, and other livestock. One appeared to have power run to it, while another did not.




Here, we are standing at the intersection of Nonconnah Creek and Cane Creek, trying to figure out how to cross either one of them.


Eventually, we found a shallow section of Nonconnah Creek and waded across, watching out for all of the broken glass in the creek bed.




We found what must be the boyfriend of the screw logo lady when we emerged from the woods around Nonconnah Creek in a modern warehouse district. Those two should definitely get together.



After a tasty and very filling meal at Uncle Lou’s Fried Chicken and observing a minor car wreck, we headed toward Winchester Road and points east.



The Airways Blvd at Democrat Rd area of Memphis has seen better days. At the Airways Transit Terminal, we just missed the bus downtown, so we called for a car from Uber, which got us to Bardog for a quick pint in less than 15 minutes.


All in all, a fun and interesting way to spend the day.

Scott and Lia Come to Memphis!

Scott and Lia in the Swan

Scott and Lia spent a couple of days with us in Memphis last week. We explored the Mud Island River Park and Ornamental Metal Museum, ate Top’s beef brisket from a styrofoam cup, chased after Sadie holding a tennis ball, paddled plastic animals around a symbolic Gulf of Mexico, listened to Lia recite pi to about 20 places, chowed on some pizza with Dave, Christina, and Lucky, made some pottery figures that need to be fired, and generally just caught up on each other’s lives. In short, we had a blast!

We have got to go up to Rhode Island to visit the whole Douglas clan soon… Thank you to Scott and Lia for making the trip down to see us!

Scott and Lia Walk the Mississippi River Model


Lia Crosses the Mississippi

Lia Douglas at 10

Lia and Sadie

Breaking a Linux Software RAID 1 for Import using VMware Converter

I rarely post super-geeky stuff on here, but since I couldn’t find any good instructions to do this important and critical activity on the Internet, and my co-workers had to piece together a set of steps that worked for us, I wanted to share what we learned, hoping to save the next person all of the work.

Linux has supported the idea of software RAID, particularly RAID 1 (or disk mirroring) for a long time. Disk mirroring is a great way to gain some insurance against a single disk failure bringing down a critical system, as everything rewritten to one disk is also written to the other disk. Many servers use hardware RAID, which mirrors the disks at a lower level than the operating system can see, making it easier to gain this redundancy. However, hardware RAID has always been more expensive than software RAID, and so there are quite a few servers out there using software RAID to protect their data.

This week, we had one of our last physical (non-virtual) server (RedHat Linux 4 AS) that needed to be virtualized. Due to the size of the data stored on that system and how it uses an external disk array, it was important that we virtualize it in place, using excellent VMware’s Converter Standalone to import the running machine, so that there was no downtime while importing the data. However, the Converter Standalone will not import Linux systems using software RAID, due to problems accessing the underlying data structures of the disk through the metadevices presented by the software RAID. (You know you are having this problem when the Converter complains about not being able to access the /boot partition.) The best solution was to break the mirrored software RAID and boot the system off of one disk, so that all of the necessary partitions could be imported and the system could be virtualized.

Unfortunately, as important and seemingly common as breaking a mirrored software RAID is in Linux, I couldn’t find any good, comprehensive, working instructions on how to do it, and breaking a software RAID is a tricky business. It is very, very easy to end up with a non-booting system and no easy way to repair it. So, to help out the next person that runs into it, I’m posting the steps that we did to break the mirrored software RAID and set the system to boot off of only one disk, so that VMware’s Converter Standalone would work on it.

First, get an idea of what you are dealing with. Logged in as root, inspect the system:

[root@cr2 cr]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md2 456G 45G 389G 11% /
/dev/md0 487M 35M 427M 8% /boot
none 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 2.0T 560G 1.4T 30% /archive
[root@cr2 cr]# more /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
/dev/md2 / ext3 defaults 1 1
/dev/md0 /boot ext3 defaults 1 2
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/md1 swap swap defaults 0 0
/dev/sda1 /archive ext3 defaults 1 0
/dev/hda /media/cdrecorder auto pamconsole,exec,noauto,managed 0 0
[root@cr2 cr]# more /etc/mtab
/dev/md2 / ext3 rw 0 0
none /proc proc rw 0 0
none /sys sysfs rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/md0 /boot ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/sda1 /archive ext3 rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0

In this case, we have 3 metadevices (md0 as /boot, md1 as swap, and md2 as root /). You can get further details about your RAID configuration using the mdadm tool, as well as mdstat:

[root@cr2 cr]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdc2[1] sdb2[0]
 2200832 blocks [2/2] [UU]
md2 : active raid1 sdc3[1] sdb3[0]
 485668928 blocks [2/2] [UU]
md0 : active raid1 sdc1[1] sdb1[0]
 513984 blocks [2/2] [UU]
unused devices: <none>

You can also run these commands:

cat /proc/mdstat
mdadm --detail /dev/md2     (to inquire about the disk members of the /dev/md2 metadevice)

When you are finally ready to do this, make sure you have a full backup of your system, and then stop all running services, especially any that would possibly write data, such as databases.

Next, we need to use mdadm to set one of the disks (we chose /dev/sdc) to be set as “failed” and removed from the RAID array. To do this, we ran these commands:

mdadm --fail /dev/md2 /dev/sdc3
mdadm --remove /dev/md2 /dev/sdc3
mdadm --zero-superblock /dev/sdc3
mdadm --fail /dev/md1 /dev/sdc2
mdadm --remove /dev/md1 /dev/sdc2
mdadm --zero-superblock /dev/sdc2
mdadm --fail /dev/md0 /dev/sdc1
mdadm --remove /dev/md0 /dev/sdc1
mdadm --zero-superblock /dev/sdc1

At this point, the software RAID still exists, but the /dev/sdc disk has been removed from it. All of the data on /dev/sdc is set as standalone.

Next, we need to modify the partition table on /dev/sdc to change it from software RAID to standard Linux partitions.

fdisk /dev/sdc

Select “p” to change the partition table, then “t” to change the type of partition. Select the partition number from the list. We changed /boot and / to be standard ext3 partitions, which is code 83, and the swap partition was changed to 82. Be sure to select “w” at the end to write all of these changes to the /dev/sdc disk when you are done.

Next, we need to mount the / and /boot partitions of /dev/sdc so that we can change files on that filesystem, so that a reboot on /dev/sdc is possible.

mkdir /mntboot
mkdir /mntroot
mount /dev/sdc3 /mntroot/
mount /dev/sdc1 /mntboot/
vi /mntroot/etc/fstab

Change fstab to so that /dev/sdc partitions will be automounted upon boot, rather than the /dev/md devices. Also, move the mdadm.conf file on /dev/sdc out of the way, so that it cannot be used when booting /dev/sdc.

mv /mntroot/etc/mdadm.conf /mntroot/etc/mdadm.bak

Now, we need to modify grub so that the bootloader will load Linux using /dev/sdc and not the /dev/md device. Notice that I will be doing this on the existing /dev/md running filesystem.

vi /etc/grub.conf

Replace the /dev/md2 (or whatever your root partition is) references with /dev/sdc3 (in our case). Save the file and close it.

Also change /mntroot/boot/grub/grub.conf with the same information.

Next, we need to run mkinitrd to use grub to update the bootloader, so that /dev/sdc will be used on boot. To do this look at the kernel you are booting from in /etc/grub.conf. For us, the mdkinitrd command looked like this:

mkinitrd -f -v /boot/initrd-2.6.9-103.ELsmp.img 2.6.9-103.ELsmp

After you run that, you will the bootloader being reconfigured. When it is complete, you are ready to reboot the server. You should boot up on /dev/sdc and be ready to do the VMware Converter Standalone importer.

Beach to Bullring in Barcelona


Today, we started out at the Picasso Museum. In some ways, this has been a very Picasso-infused trip. From the 1890s as a boy through his death in 1973, he did his best work along the stretch of sea we’ve been experiencing the past few weeks. The Picasso Museum in Antibes explored his later work, especially pottery, while the Barcelona museum focuses on his earliest drawings, sketches, and paintings. Both are excellent, and I’m still not the biggest Picasso fan, I can definitely understand the genius behind some of the best art in the 20th Century.

Next, we headed down to the beach. I’m always going to be in favor of a beach that you can visit via subway. The Barcelona beach is exceptionally nice, a great place to stroll and take in the sea air.

The Barcelona Metro is the perfect teleportation machine to get around this diverse and busy city. Next up, we visited an old bullring that the Barcelonenos modified to be a surreal 5-story mall.

What a fun and unexpected city!



Gothic Barcelona


We started our first day in Barcelona at the must-see site for most of the 17 million visitors that visit the city every year, and as all of the other museums are closed on Monday, about half of that number descended on the Gaudi-designed masterpiece Sagrada Familia with us.

The church really is amazing and worth the admission price and crowds, but I’m not sure how “sacred” the place can feel with the crush of people all around. It has the feeling of a circus combined with an international photoshoot. I can’t imagine what it would be like in the real high-season, July and August.

We also explored Barcelona’s old gothic city, the marina, and Parc de Montjuic, with some tasty tapas on way back to the apartment. A fun day…









Katherine and I just made it into the apartment in Barcelona, and this is the view from our window. This is a famous market but is under renovation this year.

Many thanks to Laura, Antonio, Guisey, Toy, Chicho, and Antonella for hosting us and putting up with all of our questions (in English, no less!) over the past week! It was sad to leave Sicily, but we know we will return someday.



For a change of pace, Katherine, Antonio, and I took the ferry over to Panarea, a nearby island to Salina. We walked around, got some sun, ate a granita, hung out a a beach, and viewed an archeological site. Panarea is a beautiful place, much smaller than Salina.




Wonderful Salina


We are having a great time catching up with friends in Salina. A lot of relaxing going on here.


Laura organized a tour of a local winery, which was very interesting. They also produce capers, using this machine.


Otherwise, it has been a lot of sun, sand, cooking, eating, drinking, and just chilling. Just what we needed after France.