One Second Everyday – Last 3 Months of 2014, Plus Everything to Date

Happy New Year to everyone out there, especially those that I know that I haven’t had a chance to see yet in 2015! I hope the coming 12 months bring fun, laughter, and joy to you in everything you do throughout the year.

I’m still doing the 1 Second Everything thing, so here is the video of the last 3 months of 2014, plus a video of the entire year, as well as all seconds recorded thus far.

October 1 – December 31, 2014

All of 2014

August 4, 2013 to December 31, 2014 – All Seconds Thus Far

8th Annual Memphis Walk

IMG_20141025_230710

For eight years now (see here or here or here or here or here), Richie, Robert, and I have been doing this thing where we meet at the Mississippi River cobblestones at dawn during a Saturday in the fall and walk until nearly sunset, seeing a good chunk of Memphis at a relaxed pace through out the day. For this edition of the Memphis Walk, we were joined by John Christenson, with the goal of following urban rivers.

IMG_20141025_230821

This year, the Memphis Walk took us down Riverside Drive, past the finally-completed Beale Street Landing complex. I still think the cost of it was too high and time required for construction was too long, but at least it is here now and it is an impressive structure for visitors. I hope we are still using it 13 years from now, unlike the Pyramid. I’m sure we will still be paying for it then.

IMG_20141025_230911

IMG_20141025_230945

We walked through the downtown River Arts Festival before it opened for the day. I saw nearly as many food stands as artist booths, but I’m happy to see a vibrant and active South Main District.

IMG_20141025_081431

IMG_20141025_231046

IMG_20141025_231114

Heading south on Main, you eventually come across one of my favorite companies in Memphis, at least in terms of the name. I love the winking screw lady…

IMG_20141025_231222

Continuing south, we entered the area around I-55 and Crump Blvd that features a number of abandoned warehouses. I’ve driven past many of them before, and this was our chance to really peek inside and really take in how large, and sometimes beautiful, they are. (I even startled a homeless person that was living in the first floor of one of them.) We also came across some gems, such as an old fire engine next to a diesel repair shop.

IMG_20141025_231351

IMG_20141025_231420

IMG_20141025_231511

There is also a huge recycling center in this area, near Kansas St at Wisconsin Ave, that I had never seen before.

IMG_20141025_231546

IMG_20141025_231615

Eventually, we circled around past the French Fort Historic District, near the National Ornamental Metal Museum, and headed south again, bypassing the turn-off to President’s Island. All along the way, we could see cobblestone streets that had been covered up by multiple layers of concrete and then asphalt, which was gradually breaking up and revealing the original roadbed below.

IMG_20141025_231652

IMG_20141025_231733

Seeking a way to get away from the industrial area and stick closer to the river, we walked through the Riverside Golf Course, which offered bluff views of McKellar Lake and the Riverside Bayou.

IMG_20141025_231822

IMG_20141025_231907

Even while you are still in Martin Luther Riverside Park, you can see the steam? stacks of the Valero Memphis Refinery, just over the tree line.

IMG_20141025_232027

After passing the scrumptious smells of Jim Neely’s Interstate BBQ on Highway 61 (also South 3rd St), we turned off into an interesting area, bisected by levees and raised railroad bed and bridges that exists just to the south of I-55 and north of Nonconnah Creek. This grassy area is highly maintained and contains several gas and other fuel pipelines, as well as (possibly) the rumored direct Valero fuel pipeline that goes directly to the Memphis Airport and FedEx.

IMG_20141025_232119

Most intriguing, there are a number of small farm operations in this area which look like something you would find in the most remote parts of the southern US, complete with horses, chickens, and other livestock. One appeared to have power run to it, while another did not.

IMG_20141025_232149

IMG_20141025_232159

IMG_20141025_232243

Here, we are standing at the intersection of Nonconnah Creek and Cane Creek, trying to figure out how to cross either one of them.

IMG_20141025_232306

Eventually, we found a shallow section of Nonconnah Creek and waded across, watching out for all of the broken glass in the creek bed.

IMG_20141025_232345

IMG_20141025_232404

IMG_20141025_232423

We found what must be the boyfriend of the screw logo lady when we emerged from the woods around Nonconnah Creek in a modern warehouse district. Those two should definitely get together.

IMG_20141025_232438

IMG_20141025_232500

After a tasty and very filling meal at Uncle Lou’s Fried Chicken and observing a minor car wreck, we headed toward Winchester Road and points east.

IMG_20141025_232521

IMG_20141025_232612

The Airways Blvd at Democrat Rd area of Memphis has seen better days. At the Airways Transit Terminal, we just missed the bus downtown, so we called for a car from Uber, which got us to Bardog for a quick pint in less than 15 minutes.

IMG_20141025_230322

All in all, a fun and interesting way to spend the day.

Scott and Lia Come to Memphis!

Scott and Lia in the Swan

Scott and Lia spent a couple of days with us in Memphis last week. We explored the Mud Island River Park and Ornamental Metal Museum, ate Top’s beef brisket from a styrofoam cup, chased after Sadie holding a tennis ball, paddled plastic animals around a symbolic Gulf of Mexico, listened to Lia recite pi to about 20 places, chowed on some pizza with Dave, Christina, and Lucky, made some pottery figures that need to be fired, and generally just caught up on each other’s lives. In short, we had a blast!

We have got to go up to Rhode Island to visit the whole Douglas clan soon… Thank you to Scott and Lia for making the trip down to see us!

Scott and Lia Walk the Mississippi River Model

 

Lia Crosses the Mississippi

Lia Douglas at 10

Lia and Sadie

Breaking a Linux Software RAID 1 for Import using VMware Converter

I rarely post super-geeky stuff on here, but since I couldn’t find any good instructions to do this important and critical activity on the Internet, and my co-workers had to piece together a set of steps that worked for us, I wanted to share what we learned, hoping to save the next person all of the work.

Linux has supported the idea of software RAID, particularly RAID 1 (or disk mirroring) for a long time. Disk mirroring is a great way to gain some insurance against a single disk failure bringing down a critical system, as everything rewritten to one disk is also written to the other disk. Many servers use hardware RAID, which mirrors the disks at a lower level than the operating system can see, making it easier to gain this redundancy. However, hardware RAID has always been more expensive than software RAID, and so there are quite a few servers out there using software RAID to protect their data.

This week, we had one of our last physical (non-virtual) server (RedHat Linux 4 AS) that needed to be virtualized. Due to the size of the data stored on that system and how it uses an external disk array, it was important that we virtualize it in place, using excellent VMware’s Converter Standalone to import the running machine, so that there was no downtime while importing the data. However, the Converter Standalone will not import Linux systems using software RAID, due to problems accessing the underlying data structures of the disk through the metadevices presented by the software RAID. (You know you are having this problem when the Converter complains about not being able to access the /boot partition.) The best solution was to break the mirrored software RAID and boot the system off of one disk, so that all of the necessary partitions could be imported and the system could be virtualized.

Unfortunately, as important and seemingly common as breaking a mirrored software RAID is in Linux, I couldn’t find any good, comprehensive, working instructions on how to do it, and breaking a software RAID is a tricky business. It is very, very easy to end up with a non-booting system and no easy way to repair it. So, to help out the next person that runs into it, I’m posting the steps that we did to break the mirrored software RAID and set the system to boot off of only one disk, so that VMware’s Converter Standalone would work on it.

First, get an idea of what you are dealing with. Logged in as root, inspect the system:

[root@cr2 cr]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md2 456G 45G 389G 11% /
/dev/md0 487M 35M 427M 8% /boot
none 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 2.0T 560G 1.4T 30% /archive
[root@cr2 cr]# more /etc/fstab
# This file is edited by fstab-sync - see 'man fstab-sync' for details
/dev/md2 / ext3 defaults 1 1
/dev/md0 /boot ext3 defaults 1 2
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/md1 swap swap defaults 0 0
/dev/sda1 /archive ext3 defaults 1 0
/dev/hda /media/cdrecorder auto pamconsole,exec,noauto,managed 0 0
[root@cr2 cr]# more /etc/mtab
/dev/md2 / ext3 rw 0 0
none /proc proc rw 0 0
none /sys sysfs rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/md0 /boot ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/sda1 /archive ext3 rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0

In this case, we have 3 metadevices (md0 as /boot, md1 as swap, and md2 as root /). You can get further details about your RAID configuration using the mdadm tool, as well as mdstat:

[root@cr2 cr]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdc2[1] sdb2[0]
 2200832 blocks [2/2] [UU]
md2 : active raid1 sdc3[1] sdb3[0]
 485668928 blocks [2/2] [UU]
md0 : active raid1 sdc1[1] sdb1[0]
 513984 blocks [2/2] [UU]
unused devices: <none>

You can also run these commands:

cat /proc/mdstat
mdadm --detail /dev/md2     (to inquire about the disk members of the /dev/md2 metadevice)

When you are finally ready to do this, make sure you have a full backup of your system, and then stop all running services, especially any that would possibly write data, such as databases.

Next, we need to use mdadm to set one of the disks (we chose /dev/sdc) to be set as “failed” and removed from the RAID array. To do this, we ran these commands:

mdadm --fail /dev/md2 /dev/sdc3
mdadm --remove /dev/md2 /dev/sdc3
mdadm --zero-superblock /dev/sdc3
mdadm --fail /dev/md1 /dev/sdc2
mdadm --remove /dev/md1 /dev/sdc2
mdadm --zero-superblock /dev/sdc2
mdadm --fail /dev/md0 /dev/sdc1
mdadm --remove /dev/md0 /dev/sdc1
mdadm --zero-superblock /dev/sdc1

At this point, the software RAID still exists, but the /dev/sdc disk has been removed from it. All of the data on /dev/sdc is set as standalone.

Next, we need to modify the partition table on /dev/sdc to change it from software RAID to standard Linux partitions.

fdisk /dev/sdc

Select “p” to change the partition table, then “t” to change the type of partition. Select the partition number from the list. We changed /boot and / to be standard ext3 partitions, which is code 83, and the swap partition was changed to 82. Be sure to select “w” at the end to write all of these changes to the /dev/sdc disk when you are done.

Next, we need to mount the / and /boot partitions of /dev/sdc so that we can change files on that filesystem, so that a reboot on /dev/sdc is possible.

mkdir /mntboot
mkdir /mntroot
mount /dev/sdc3 /mntroot/
mount /dev/sdc1 /mntboot/
vi /mntroot/etc/fstab

Change fstab to so that /dev/sdc partitions will be automounted upon boot, rather than the /dev/md devices. Also, move the mdadm.conf file on /dev/sdc out of the way, so that it cannot be used when booting /dev/sdc.

mv /mntroot/etc/mdadm.conf /mntroot/etc/mdadm.bak

Now, we need to modify grub so that the bootloader will load Linux using /dev/sdc and not the /dev/md device. Notice that I will be doing this on the existing /dev/md running filesystem.

vi /etc/grub.conf

Replace the /dev/md2 (or whatever your root partition is) references with /dev/sdc3 (in our case). Save the file and close it.

Also change /mntroot/boot/grub/grub.conf with the same information.

Next, we need to run mkinitrd to use grub to update the bootloader, so that /dev/sdc will be used on boot. To do this look at the kernel you are booting from in /etc/grub.conf. For us, the mdkinitrd command looked like this:

mkinitrd -f -v /boot/initrd-2.6.9-103.ELsmp.img 2.6.9-103.ELsmp

After you run that, you will the bootloader being reconfigured. When it is complete, you are ready to reboot the server. You should boot up on /dev/sdc and be ready to do the VMware Converter Standalone importer.