Posts from — August 2007

I found this article on Storagemojo. Seemingly, Apple has come up with a filesystem converter. A remarkable breakthrough! Just imagine the ease with which I could experiment, I have to keep aside an entire drive while trying out new things and this is a PITA. Instead this patent filed on the 1st of August is going to eliminate most of my problems (err this utility will be available on Linux right ?). I respect the fact that Apple is coming up with something like this and guess what, Apple is moving to ZFS ? I am of the opinion that Apple will have to mount its boot partition on something else as Sun itself has been unable to boot OpenSolaris off a ZFS partition. Still, by October, ZFS is going to be rid of that problem.
Lots of cool stuff happening in the storage domain.

Yes folks, I got a cd and that too from one of the least expected places, my neighbour! Amazing ain’t it? I finally get a more up to date os than dapper whose repos are hopelessly outdated. I suppose I can get back to coding now. I am working on a coordinate geometry module that deals with parabolas, ellipses and so on. It is going to include only a parabola at the moment (that is what they are teaching at school, so you know..). I am very worried because I got toefl coming up on the 12th of september, hence the tension is in full spate. By the way twincling is due to meet on the same day as my phase test (at FIIT-JEE). As they say, if it can go wrong, it will go wrong. I am also trying to learn lots of electronics as I can then design my dream LED powered fridge lighting woohoo. I had a lot of fun at mukt and learnt two things:
1. My hardware is crap as opposed to the hardware the general public owns,
2. Never ever trust an individual who gives you a copy of solaris and tells you that it is very fast.

Heh. I am bloody happy and hope that the world stays happy too. good luck and see you later.
Shriphani Palakodety

Well after an install fest in which my lappy had a taste of every distro available, I ended up installing ubuntu 6.06. It feels a bit odd. I am going to install django and certainly wait for the next release. phew.

There ! I code that is free from repetition. I do suppose that well written code has its uses. It looks beautiful . It accepts command line arguments too. All you need to do is install the BeautifulSoup module for Python and type, “python getstuff.py [extension]” (if you save this file as getstuff.py on your computer). That means if you want to download all the mp3 files on a webpage, type, “python getstuff.py .mp3″ and sit back and relax :).

#!/usr/bin/env python
#newproj.py
#Author: Shriphani Palakodety (PSP)
#Usage: python getstuff.py [extension]

import urllib
from urlparse import urlparse, urlunparse
from BeautifulSoup import BeautifulSoup
import sys

def linksFromSoup(soup, extension):
    """Iterator for all the valid links found."""
    for link in soup.findAll('a'):
        try:
            link_url = link['href']
        except KeyError:
            continue
        possible_link, filename = parseLink(url, link_url)

        if filename.endswith(extension):
            yield possible_link, filename

def parseLink(src_url, link_url):
    """Takes the page URL and a link from it and makes
    the absolute link and the filename."""

    (link_scheme, link_hostname, link_path, link_params, link_query, link_fragment) = urlparse(link_url)
    (src_scheme, src_hostname, src_path, src_params, src_query, src_fragment) = urlparse(src_url)
    filename = link_path.split('/')[-1]
    if link_hostname:
        possible_link = link_url

    else:
        full_path = '/'.join((src_path.rsplit('/', 1)[0], link_path))
        possible_link = urlunparse((src_scheme, src_hostname, full_path, link_params, link_query, link_fragment))
    return possible_link, filename

if __name__ == '__main__':
    extension = sys.argv[1]
    url = raw_input("URL you want to download things from: ")
    html = urllib.urlopen(url)
    print repr(html)
    soup = BeautifulSoup(html)
    for linkurl, filename in linksFromSoup(soup, extension):
        urllib.urlretrieve(linkurl, filename)

I have returned from mukt.in a bit wiser. Mr. Raju Alluri from Sun Microsystems spoke about a new wave of innovation that promised to change the way the storage industry operated. ZFS- the new filesystem from Sun Microsystems. I was at my attentive best here. I began listening. ZFS presents a pool type storage (I didn’t know what this meant). He went on, “ZFS stores data as a pool and several filesystems, thousands, even more, can pull data from this pool”. Now I understand. The data speeds (I’m thinking I/O bandwidth, someone correct me if I understood wrong) are not limited to the speed of the individual device, in fact, the combined speed is available to pull data from this pool whatever be the device. WOW ! Then comes the shocker. ZFS allows one bloody exabyte as the max. file size ! Who has a file that large ? Then Mr. Alluri told us about the National Met. Org. in the US which has the largest db in the world. 300 Tb seemingly. Now met data gets constantly updated. A single backup is 300 TB, in a few years you could be touching a lot more. Again, you shouldn’t have problems trying to access the data. ZFS’s pool type system should make that possible.

I began thinking about incontiguous partitions that had been bugging us some time back. LVM is traditionally used to manage incontiguous partitions. But just imagine. With a pool type storage system, with filesystems accessing data from this pool directly, LVM might no longer be necessary at all ! And this is a big revolution !!!!!

Mr. Alluri then told us that ZFS was going to allow two operating systems to run side by side. I know this I thought. This is not a ZFS feature I thought. chroot allows me to do the same, I can install debian in gentoo etc. !

It turns out I am wrong again. ZFS will allow you to run two operating systems side by side all because of its pool type storage.

I am going to learn more about this excellent operating system over the next few weeks. I unfortunately cannot try ZFS as it is not supported on Linux and I have to learn about it only through wikipedia and other sources. MIT’s Athena Center has Solaris systems seemingly. Hopefully I’ll get to see ZFS in action there (if I make it).

Have a look at this. I found this question in my neighbour’s CS book. He is a CS student and his college’s introductory CS course is taught in C (Sadly I don’t know C )

import nzmath.gcd

def crt(xl,ml):

    k = len(xl)

    m = ml[0]

    x = xl[0]

    for i in range(1,k):

        u,v,d = gcd.extgcd(m,ml[i])

        if d > 1:

            raise Error

            x = u*m*xl[i] + v*l[i]*x

            m = m*ml[i]

            x = x%m

    return x

There seemingly is an RSA based on the CRT but I don’t know about that. This question really got me to think.

I went to a friend’s place yesterday and within minutes found myself staring at the computer there (family friend, not mine exactly). I would have loved to listen to my music there as this guy’s hard rock ( soft/hard rock are oxymorons ) bugs me. I wanted to listen to Kitaro’s instrumental music or Billy Joel’s albums. I began clicking on the inbox link in Gmail over and over again expecting an interesting mail to come by. The interesting thing is that this guy has Linux installed ( Fedora ) but uses it once a month as it gives him a headache ( Anger !! ). A thought struck me then. What if I could store my mp3s in my gmail inbox ? I could then listen to them wherever I wanted to !

One google search later, I found gmailfs - a cool app that allows me to mount my gmail inbox on a local directory.

Back home, I installed gmailfs from the repos, edited /etc/gmailfs/gmailfs.conf to put my username and password there and I did:

mount.gmailfs none /mnt/google

Thats it ! I can now access my Gmail inbox from my shell.
Gmailfs was created by Richard Jones. This is a brilliant creation.

I believe that Gmailfs is needed, not because one wants to show off one’s python skills, but that my inbox is never going to face storage issues, despite the fact that I get 100 KB sized emails from comp.lang.python and comp.sys.ibm.pc.storage on a daily basis.  A gig is a lot of space. I can use this enormous space (Greedy ) to store stuff like my scripts or my mp3 files so that I can listen to them everywhere.

I found this research paper while looking for versioning filesystems (yeah, I’m obsessed with versioning).

What I found striking about this paper was the different way it approaches versioning. Most filesystems that I have covered take snapshots of files and in filesystems like ext3cow, one can access the previous version of the file using a certain syntax. PersiFS continuously versions files at every update. Considering that PersiFS operates over a network, the server should be able to dig out data from the fs as quick as it can on other orthodox filesystems. Persifs allows users to look at directory states as shown:

#current version:

/fs/now

#version of the filesystem on the 30th of February at 12:05 pm ( ) :

/fs/2007-2-30-12-05-00

Anyway, I have to cover the portion on optimizing read/write methods so I’ll put up a post on that when I get around to reading that. Till then, goodbye

We developers can’t do without versioning can we. We write crap html to test our parser and end up committing this as well to the server and then move back to a previous version so that those who use svn to get the bleeding edge version of the software do not get your useless html to laugh at. I was pondering over what advantages versioning could bring to our traditional way of work. I for one use sys-rescue to back up important files on my computer and append the version number to the filename. I just need to browse through this directory in case I screw up.

Anyway, I decided to have a look at operating systems which matched the performance of ext3 and had some form of versioning in them. My first stop was ext3cow from John Hopkins University. I was particularly attracted to it because it contained ext3 in its name.
I then installed ext3cow on a partition reading instructions from http://www.ext3cow.com/Applying%20the%20Patch.html .

I took me quite some time to understand what was going on. I approached ext3cow with the impression that one would have to revert back to a previous version of the fs by probably mounting it somewhere else (probably these are stored as .iso files somewhere so that they can be dumped on other media ?) . I was stumped to see something like this as I read:

[user@machine]$ echo "This is v1 of the file." > foo.txt
[user@machine]$ snapshot
Snapshot on . 
[user@machine]$ echo “This is v2 of the file.” > foo.txt
[user@machine]$ cat foo@
This is v1 of the file.

Amazing. This means that I can access files at any given date in the same way as I access a file that currently exists in the filesystem. Now, there are certain drawbacks to this. Who in this world can remember the seconds since epoch the snapshot was taken ? And its not like I am going to access this snapshot seconds after I took one. There is a solution to all this, a gui interface to ext3cow. It should allow you to save these snapshots using a date-structure you want.

One of the drawbacks of the current version of ext3cow is the lack of the ability to delete a snapshot. It did exist in previous versions though.

After a terribly long period of using ext3cow, here is what I feel it needs:

1. A utility to convert ext3 to ext3cow.

2. The ability to delete a snapshot.

3. A better GUI for ext3cow

Here is my idea about versioning filesystems. Someone creates a versioning file-system that gives the user no control whatsoever over the frequency of snapshots taken and over the snapshots themselves. This would enable one to track let’s say the activities on an ATM machine ? Take the pic of a guy who enters through the door, save it, take a snapshot detailing the activities and done. If someone somehow manages to delete the current file, you can bring it back from the snapshot.

Well that’s how it is. I am going to read more about filesystems supporting versioning such as NILFS

This script is slightly better in terms of what it does. At the moment it just kills all processes associated with eggdrops.

#!/usr/bin/python
#monitor application
#Author: Shriphani Palakodety
import os
from subprocess import *
process = Popen("ps aux", shell=True, stdout=PIPE)
output = process.communicate()
reqd_output = output[0]
for line in reqd_output.split('n')[1:-1]:
    list = line.split()
    pid = list[1].split(' ')[0]
    for number in pid.split('n'):
        print number
        pro_ass = Popen(["ps", "-p", number], stdout=PIPE)
        outputs = pro_ass.communicate()
        a = outputs[0]
        if a.find('eggdrop')== True:
             os.kill(pid)
             print "All banned applications have been killed."
        else:
             print "No dickface engaging in bs stuff."

Please ignore the last line. It was meant to be there for reasons like testing.