Entries Tagged 'Storage' ↓

I found this article on Storagemojo. Seemingly, Apple has come up with a filesystem converter. A remarkable breakthrough! Just imagine the ease with which I could experiment, I have to keep aside an entire drive while trying out new things and this is a PITA. Instead this patent filed on the 1st of August is going to eliminate most of my problems (err this utility will be available on Linux right ?). I respect the fact that Apple is coming up with something like this and guess what, Apple is moving to ZFS ? I am of the opinion that Apple will have to mount its boot partition on something else as Sun itself has been unable to boot OpenSolaris off a ZFS partition. Still, by October, ZFS is going to be rid of that problem.
Lots of cool stuff happening in the storage domain.

I have returned from mukt.in a bit wiser. Mr. Raju Alluri from Sun Microsystems spoke about a new wave of innovation that promised to change the way the storage industry operated. ZFS- the new filesystem from Sun Microsystems. I was at my attentive best here. I began listening. ZFS presents a pool type storage (I didn’t know what this meant). He went on, “ZFS stores data as a pool and several filesystems, thousands, even more, can pull data from this pool”. Now I understand. The data speeds (I’m thinking I/O bandwidth, someone correct me if I understood wrong) are not limited to the speed of the individual device, in fact, the combined speed is available to pull data from this pool whatever be the device. WOW ! Then comes the shocker. ZFS allows one bloody exabyte as the max. file size ! Who has a file that large ? Then Mr. Alluri told us about the National Met. Org. in the US which has the largest db in the world. 300 Tb seemingly. Now met data gets constantly updated. A single backup is 300 TB, in a few years you could be touching a lot more. Again, you shouldn’t have problems trying to access the data. ZFS’s pool type system should make that possible.

I began thinking about incontiguous partitions that had been bugging us some time back. LVM is traditionally used to manage incontiguous partitions. But just imagine. With a pool type storage system, with filesystems accessing data from this pool directly, LVM might no longer be necessary at all ! And this is a big revolution !!!!!

Mr. Alluri then told us that ZFS was going to allow two operating systems to run side by side. I know this I thought. This is not a ZFS feature I thought. chroot allows me to do the same, I can install debian in gentoo etc. !

It turns out I am wrong again. ZFS will allow you to run two operating systems side by side all because of its pool type storage.

I am going to learn more about this excellent operating system over the next few weeks. I unfortunately cannot try ZFS as it is not supported on Linux and I have to learn about it only through wikipedia and other sources. MIT’s Athena Center has Solaris systems seemingly. Hopefully I’ll get to see ZFS in action there (if I make it).

I went to a friend’s place yesterday and within minutes found myself staring at the computer there (family friend, not mine exactly). I would have loved to listen to my music there as this guy’s hard rock ( soft/hard rock are oxymorons ) bugs me. I wanted to listen to Kitaro’s instrumental music or Billy Joel’s albums. I began clicking on the inbox link in Gmail over and over again expecting an interesting mail to come by. The interesting thing is that this guy has Linux installed ( Fedora ) but uses it once a month as it gives him a headache ( Anger !! ). A thought struck me then. What if I could store my mp3s in my gmail inbox ? I could then listen to them wherever I wanted to !

One google search later, I found gmailfs - a cool app that allows me to mount my gmail inbox on a local directory.

Back home, I installed gmailfs from the repos, edited /etc/gmailfs/gmailfs.conf to put my username and password there and I did:

mount.gmailfs none /mnt/google

Thats it ! I can now access my Gmail inbox from my shell.
Gmailfs was created by Richard Jones. This is a brilliant creation.

I believe that Gmailfs is needed, not because one wants to show off one’s python skills, but that my inbox is never going to face storage issues, despite the fact that I get 100 KB sized emails from comp.lang.python and comp.sys.ibm.pc.storage on a daily basis.  A gig is a lot of space. I can use this enormous space (Greedy ) to store stuff like my scripts or my mp3 files so that I can listen to them everywhere.

I found this research paper while looking for versioning filesystems (yeah, I’m obsessed with versioning).

What I found striking about this paper was the different way it approaches versioning. Most filesystems that I have covered take snapshots of files and in filesystems like ext3cow, one can access the previous version of the file using a certain syntax. PersiFS continuously versions files at every update. Considering that PersiFS operates over a network, the server should be able to dig out data from the fs as quick as it can on other orthodox filesystems. Persifs allows users to look at directory states as shown:

#current version:

/fs/now

#version of the filesystem on the 30th of February at 12:05 pm ( ) :

/fs/2007-2-30-12-05-00

Anyway, I have to cover the portion on optimizing read/write methods so I’ll put up a post on that when I get around to reading that. Till then, goodbye

We developers can’t do without versioning can we. We write crap html to test our parser and end up committing this as well to the server and then move back to a previous version so that those who use svn to get the bleeding edge version of the software do not get your useless html to laugh at. I was pondering over what advantages versioning could bring to our traditional way of work. I for one use sys-rescue to back up important files on my computer and append the version number to the filename. I just need to browse through this directory in case I screw up.

Anyway, I decided to have a look at operating systems which matched the performance of ext3 and had some form of versioning in them. My first stop was ext3cow from John Hopkins University. I was particularly attracted to it because it contained ext3 in its name.
I then installed ext3cow on a partition reading instructions from http://www.ext3cow.com/Applying%20the%20Patch.html .

I took me quite some time to understand what was going on. I approached ext3cow with the impression that one would have to revert back to a previous version of the fs by probably mounting it somewhere else (probably these are stored as .iso files somewhere so that they can be dumped on other media ?) . I was stumped to see something like this as I read:

[user@machine]$ echo "This is v1 of the file." > foo.txt
[user@machine]$ snapshot
Snapshot on . 
[user@machine]$ echo "This is v2 of the file." > foo.txt
[user@machine]$ cat foo@
This is v1 of the file.

Amazing. This means that I can access files at any given date in the same way as I access a file that currently exists in the filesystem. Now, there are certain drawbacks to this. Who in this world can remember the seconds since epoch the snapshot was taken ? And its not like I am going to access this snapshot seconds after I took one. There is a solution to all this, a gui interface to ext3cow. It should allow you to save these snapshots using a date-structure you want.

One of the drawbacks of the current version of ext3cow is the lack of the ability to delete a snapshot. It did exist in previous versions though.

After a terribly long period of using ext3cow, here is what I feel it needs:

1. A utility to convert ext3 to ext3cow.

2. The ability to delete a snapshot.

3. A better GUI for ext3cow

Here is my idea about versioning filesystems. Someone creates a versioning file-system that gives the user no control whatsoever over the frequency of snapshots taken and over the snapshots themselves. This would enable one to track let’s say the activities on an ATM machine ? Take the pic of a guy who enters through the door, save it, take a snapshot detailing the activities and done. If someone somehow manages to delete the current file, you can bring it back from the snapshot.

Well that’s how it is. I am going to read more about filesystems supporting versioning such as NILFS

My iPod began behaving oddly today. I plugged it in and tried to copy an entire album to /Music/Evanescence. To all those wondering, let me tell you that rockbox offers you to listen to music placed anywhere on the iPod’s filesystem. That way I don’t need to stick to the default iPod way of organising my music.
Upon clicking “Paste” in the Nautilus window where my iPod’s root directory was open, I got:

You don't have permission to write to this device

What the bloody hell! My iPod is fat32 formatted. Since when did fat32 have permissions. I tried twice and thrice in vain. Had the thing just cracked up ?
I did, “dosfsck -a /dev/sdb1″ (sdb1 on my laptop, check for the right location on your laptop. I suggest you use “blkid”
After a few minutes, it was done. My iPod is back to normal now. I still don’t understand what went wrong. Was it because of rockbox, was it because of rockbox’s id3-tag lookup app, “tagcache” ? I don’t know. Anyway, now you guys know what to do if you face this problem.

Trouble’s struck !!. Ubuntu’s libATA has stopped functioning and my partitions are now seen as IDE subsystems once again. Hence we have a conflict between udev rules and device names leading to automount getting <censored>. I seriously can’t be bothered by this. Considering that I try to behave as much as a sysadmin as possible, I tried to whip up a script that would mount the device specified as I have no idea what filesystems my 3 identical (in appearance) external drives have on them. I think one has reiserfs, another has ext3 and the third has fat32 on it. I tried to find out the command that would let me know about the fs I have on my external drive without actually mounting it. fdisk didn’t help and I was stuck with waiting for the new kernel 2.6.20-16-generic to be fixed. Later that day I did:

man -k locate

The first line in there was:

blkid (8)            - command-line utility to locate/print block device attributes

Was this command the solution to my troubles? It indeed was. I then whipped up a script in no time. Here it is:

 #!/usr/bin/python

#mountcreatingdir.py

#Author:Shriphani Palakodety

import commands

index = 1

while True:

 dir = "/mnt/usbdisk" + str(index)

 string = commands.getoutput('mkdir %s' % dir)

 if string == "":

  continue

 else:

  index = index + 1

 device = raw_input('Enter the name of the device you want to mount: ')

 a = commands.getoutput('blkid %s' % device)

 import string

 number = string.find(a, " T")

 print number

 requirement = a[number+6:]

 s = commands.getoutput('mount -t %s %s %s' % (requirement, device, dir))

 print s

That’s about it.

I got my feisty cds this morning. I installed feisty immediately and considering that one of the ubuntu devs on #ubuntu told me that feisty’s way of handling storage devices had changed, I immediately did:

shriphani@psp-laptop:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
#
proc            /proc           proc    defaults        0       0
# /dev/sda3
UUID=1a97343a-e3ed-48e2-9db4-ea67228fb1aa /               ext3    defaults,errors=remount-ro 0       1
# /dev/sda2
UUID=68033b3d-f469-4a8b-8ade-c3a9984f5281 /boot           ext2    defaults        0       2
# /dev/sda1
UUID=01107bb3-77bb-41c3-b271-6019bfa8ebb3 /home           ext3    defaults        0       2
# /dev/sda4
UUID=da271254-066a-46f4-b1a0-75ba2c4b494f none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0

What’s this !!. a UUID describing a device ? I was a bit confused. I also saw that my IDE drive (I have an acer travelmate 4500 laptop) was being recognised as SCSI. I immediately went to #ubuntu on freenode to figure out what was wrong. I was then given the link to this.
Wow! Surely a lot of thought going into that. So we will be dealing with just one subsystem from now on in Ubuntu land.