Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Monday, June 8, 2009

Wasting just a little bit more time: a tale of dzen and reddit

Are you tired of switching from the console to browser, and then pressing F5 to see what's new on reddit? So am I. There's got to be a better way of seeing new posts on reddit without leaving the comfort of the command line. And there is. If you run Linux with dzen, here's a possible solution. Simply use this script. It checks reddit and outputs the first ten titles in a way that can be used by dzen. It should be clear from the source how to make the script output more than ten titles or how to monitor something other than the main reddit (e.g., programming, pics, funny or others subreddits). Now all one needs to do is run this script every n seconds (depends on how often you expect new articles to arrive), you can use something like this. One last thing left to do is add a line to your .xsession that calls your checker script, e.g., I have the following line:


~/.dzen/checkReddit.sh | dzen2 -y 800 -ta c -l 10 -bg '#2c2c32' -fg 'grey70' -p -e 'entertitle=uncollapse;leavetitle=collapse' &

And here are the results (the reddit bar sits quietly at the bottom of your screen until you move your mouse over it, when you get the situation shown on the picture to the right):


Of course, once you see what's new you still have to switch to your browser if you want to actually follow the links, but this should at least save you the trouble of switching only to find out that there's nothing new. Note that there are many things left to be done here, this is just a little sample of how you can waste a few more minutes of your life.


Monday, April 7, 2008

Saving e-mails to disk (from a gmail account)

As I was trying to download all my spam messages to disk (I'm building a database of spam messages), I realised that this is no simple task. Well, one thing I could do is save the messages using Thunderbird or Outlook, but since I don't use either (I consider the Gmail web interface very nice and user-friendly) this one is out of the question. However, after a little browsing I discovered a wonderful Python package called libgmail. Long story short, here is the script I used to download all the messages from the spam folder:


#!/usr/bin/env python
'''
savemsg.py -- Download all messages from a specified folder
License: GPL 2.0
'''

import sys
from getpass import getpass
import libgmail

if __name__ == "__main__":
try:
name = sys.argv[1]
except IndexError:
name = raw_input("Gmail account name: ")

pw = getpass("Password: ")

ga = libgmail.GmailAccount(name, pw)

print "\nPlease wait, logging in..."

try:
ga.login()
except libgmail.GmailLoginFailure,e:
print "\nLogin failed. (%s)" % e.message
sys.exit(1)
else:
print "Login successful.\n"

FOLDER_list = {'U_INBOX_SEARCH' : 'inbox',
'U_STARRED_SEARCH' : 'starred',
'U_ALL_SEARCH' : 'all',
'U_DRAFTS_SEARCH' : 'drafts' ,
'U_SENT_SEARCH' : 'sent',
'U_SPAM_SEARCH' : 'spam',
}

FOLDER_list = raw_input('Choose a folder (inbox, starred, all, drafts, sent, spam): ')
folder = ga.getMessagesByFolder(FOLDER_list)

for thread in folder:
for msg in thread:
print "Downloading message %s " % msg.id
encIndexStart = msg.source.find('charset=')
if encIndexStart != -1:
encIndexEnd = (msg.source.find(' ', encIndexStart),\
msg.source.find('\n', encIndexStart),\
msg.source.find(';', encIndexStart),\
msg.source.find('"', encIndexStart+10))
encIndexEnd = [ind for ind in encIndexEnd if ind != -1]
encIndexEnd = min(encIndexEnd)
enc = msg.source[encIndexStart + 8:encIndexEnd]
enc = enc.replace('"', '').replace(';', '')
else:
enc = 'ascii'
print "Detected encoding %s\n" % enc
try:
f = open(msg.id + " " + msg.subject + ".txt", 'w')
except:
# message subject contains characters forbidden by the os in the
# file name, use just message id
f = open(msg.id + ".txt", 'w')
try:
f.write(msg.source.decode(enc).encode('utf-8'))
except:
f.write(msg.source)
f.close()
print "\n\nDone."


One could use the script to download messages from any gmail folder. The encoding of the message is automatically recognized and the message is saved in UTF-8 to facilitate later processing. Of course, you have to have libgmail installed to run the script. It is also very easy to adapt the script to use it for any other purpose (I actually wrote this script by changing one of the demo scripts that come with libgmail).








Thursday, January 31, 2008

Unifying expressions in Python

Most of you that had some AI course in college probably heard of unification of expressions. All of you who ever programmed (and I use the term loosly) in Prolog know what this is. For all others, just a note that I'm not going to explain here what is unification and how it's done. If you don't know what it is, check out, for example this WP article.
Here we can see the algorithm for finding the most general unifier of two expressions in pseudo-code (check out the bottom of the page). And here is my implementation of unifying expressions in Python. I'm not going to explain the code here---if you're interested, you can do this yourself. I'll just give a brief example of how you can use this. So, suppose that you have the following two expressions:
expr1 = P(f(a), g(y), f(w))
and
expr2 = P(x, g(f(x)), y),
where P is a predicate, g and f are functions, and x, y and w are variables (this is an actual assignment from a test we gave our students some two months ago). Lets solve this using Python:


import mgunifier as mg # just to make the listing shorter

# declare predicates, functions, and variables
P = mg.predicate("P")
g = mg.function("g")
f = mg.function("f")
x = mg.variable("x")
y = mg.variable("y")
w = mg.variable("w")
a = mg.constant("a")

expr1 = [P, [f, a], [g, y], [f, w]]
expr2 = [P, x, [g, [f, x]], y]
uni = mg.mgUnifier(expr1, expr2) # find the most general unifier
new1 = uni(expr1)
new2 = uni(expr2)
print "The most general unifier is ", uni
# check if the two expressions are the same after unification
print new1
print new2

That's all there is to it. If someone has an idea on how to make things more simple or improve the code in any way, I am always glad to hear from you. Now, those of you that had some experience with Prolog probably know that, although unification is the most basic tool of this language, there's a little bug/wart when using it (this applies to SWI Prolog implementation, I don't know how others handle it). Typing, for example,

?- X = f(X).

results in X = f(**), meaning that variable X is now f(f(f(f(...)))). This is, of course, wrong, because the left and the right side of the equal sign will never be the same (and if you don't believe me, check out the algorithm). Why exactly is Prolog doing this is beyond me. Anyway, try the following code:

import mgunifier as mg

P = mg.predicate("P")
g = mg.function("g")
f = mg.function("f")
x = mg.variable("x")
y = mg.variable("y")
w = mg.variable("w")
a = mg.constant("a")

first = [P, x, [g, y], z]
second = [P, [f, a], z, y]
uni = mg.mgUnifier(first, second)
print uni

Not only that you get an error, but you can also see exactly where the algorithm failed.
Now, what's the whole point of this post? To demonstrate some Python implementation of an algorithm no one cares about? Well, partially. I was looking through the web for something similar and found only this. As it didn't satisfy me, I wrote my own program that unifies expressions, so anyone looking for something like this doesn't have to. But, what I'd really like people to see (and read) is this (copy-paste from the source):

def mgUnifier(k1, k2):
"""Return the most general unifier of two expressions k1 and k2."""

if not k1 or not k2: return supstitution()
if isinstance(k1, (constant, variable, function, predicate)) or isinstance(k2, (constant, variable, function, predicate)):
if k1 == k2:
return supstitution()
if isinstance(k1, variable):
if k1 in k2:
return error(`k1` + " in " + `k2`)
else:
return supstitution(k2, k1)
if isinstance(k2, variable):
if k2 in k1:
return error(`k2` + " in " + `k1`)
else:
return supstitution(k1, k2)
if not isinstance(k1, variable) and not isinstance(k2, variable):
return error(`k1` + " and " + `k2` + " cannot be unified!")

alpha = mgUnifier(head(k1), head(k2))
if isinstance(alpha, error):
return alpha
k3 = alpha(tail(k1))
k4 = alpha(tail(k2))
beta = mgUnifier(k3, k4)
if isinstance(beta, error):
return beta
return alpha(beta)

This beautifully (IMHO) demonstrates the power and clarity of Python. Compare this code to the algorithm pseudo-code---it's almost the same. This is what I love about Python---I can simply copy the algorithm's pseudo-code and then just implement the "magic" that happens in the background.
Like I said, the purpose of this program is primarly educational, the code could probably be better written, but I stuck with the implementation that was most similar to the algorithm. Keeping that in mind, I appreciate any feedback on the code.














Monday, January 21, 2008

Real-life Python

I just came home all tired and the minute I saw the place it hit me---I didn't
tidy up in ages. There were books everywhere, clothes all over the place, dirty
dishes... As I'm always tired when I come home from work, I never feel like cleaning
the place up. Somehow Python came up in my head as the solution to my problem (well,
not really a solution:) ). What if we could execute Python code in real life. I
mean, wouldn't it be great if we could do something like:


apartment.sort()
dishes.wash()

As I of course didn't feel like cleaning, I decided I'd better spend my time writing
about how I would use Python in real life. Beside cleaning the house, I think Python
could be of great help in the following problems as well.

Understanding women
Did you ever have a fight with your girlfirend/wife (it's a rhetorical question,
of course you did)? Did you ever wonder "What is going on in that mind of hers"?
Well, think about this one:

print girlfriend.__doc__

And if that didn't help, wouldn't it be great to be able to do something like

import dis
print dis.dis(girlfriend.think)

This would not only help you understand her, but also enable you to predict how
will she react. That's right, no more "What did I do wrong"---just look at the
code and you will know.

The answer to life
For all those in doubt as to whether they are wasting their life going to church
every Sunday, here's an ellegant Pythonic solution:

import time
if life.endswith(death):
while alive:
party()
time.sleep(12*60*60)
die()
else:
import random
religions = [christianity, islam, buddhism, hinduism]
myReligion = random.choice(religions)
for day in allDays:
if day != sunday:
be_humble()
else:
repent_sins(myReligion)
#no more days, end of time is here
go_to_heaven(myReligion)


What would you use Python for in real life?






Monday, December 31, 2007

Naive fibonacci in pypy

As this is my last post this year, I'll just keep it short. I've been busy doing all sorts of things, so here's just a fast revisit to our old friend, the naive fibonacci algorithm, this time featuring Pypy. I downloaded the 1.0.0 version and ran the by now famous naive fibonacci. To get to the point, it took 184 seconds. Yep, you read it right---3 minutes. The same thing that took 27 seconds in CPython 2.5, or around 3 seconds using Psyco. Now, I didn't read much about pypy, but as I understand it, it should be faster than the CPython implementation... And AFAIK, the guys that made psyco abandoned that project to go work on bringing similar functionality to pypy. Now, I hate to bitch (no, no I don't:) ), but this really seems kinda slow. What's going on here? It's probably the function call overhead that slows it down. One should look at the code of pypy and see what exactly they put on the stack each time... The point being, I'll stick to CPython for a while more, at least until the naive fib runs under 20 seconds :) ('Cos what's more important in life than running a fast fibonacci algorithm?)
Happy holidays everyone





Saturday, December 22, 2007

Python 3000 - how mutable is immutable?

I just downloaded Py3k alpha 2 and decided to play with it a bit. Well, beside the print becoming a function, things seemed to be more or less the same as in 2.5. Then I decided to play with the more esoteric thingies in Python.
It has been long known that Python immutables aren't really immutable. For example, if you enter the following code in 2.5:


a = ([1], 2)
a[0] += [2]


you will get an error. However, printing a reveals that it has indeed been modified (a is now ([1, 2], 2)). Ok, so what happened to this little bug/hack/feature/gotcha/whatever-you-want-to-call-it in Py3k? Well, entering the code above also raises the same error, and again it actually modifies a. Now, I don't know is this "feature" is documented anywhere, but I really don't see any point in it being a part of the language (if someone can give me an example where it would be useful to have this kind of behavior, let me know). So, why is it still present in Py3k, when it's been known for a while now? I guess someone either thinks this is a good thing or they just forgot about it, either way---guys please change this, it's annoying me :) (and what better motivation to change the language than a rant by some guy on the Internet that you never met or heard of?)
But, the fun doesn't stop here kids! Try the following:


a = ({1:2}, 2)
a[0][2] = 3


This little piece of code gets executed without any problems, both in 2.5 and in Py3k. I wonder what is the rationale behind the fact that if we try to change a list (in place) that is part of a tuple, we get an error, while changing a dict happens silently. Ok, I do admit that doing


a = ([1], 2)
a[0].append(2)


also doesn't throw an error, but this only confuses me more. So, I'd like to know---are tuples mutable or immutable? Or better yet, how mutable are immutables?
As a final thought, consider the following code:


a = ({1:2}, 2)
a[0][2] = a
print a


In 2.5, this code will execute with no problems, and printing a will yield in
({1: 2, 2: ({...}, 2)}, 2)
thereby showing us that we have an infinite dictionary on our hands (or that's what I think these three dots stand for). However, doing the same thing in Py3k (i.e., running the above code and trying to print a---remember that in Py3k print is a function), we get

Traceback (most recent call last):
File "", line 1, in
TypeError: __repr__ returned non-string (type bytes)

Whoa! WTF?? What does this mean? What ever it means it tells me nothing about what is actually happening here. As far as I can see, there is something wrong with the __repr__ function---it isn't cut out to handle this kind of objects. So, the first thing that came to my mind was "Hey, cool, they won't print infinite dictionaries any more. They'll allow their construction, but when I try to print them, I'll get an error (a weird one, to say the least, but an error still). This kinda suxx, but ok, I guess I'll have to live with it." No. I was wrong. Try

print(a[0])

The infinite dict in the first position of the tuple prints with no problems. So, why is it a problem to print the tuple (which basically has one number, two parenthesis, and one comma extra)? No idea. Of course, trying to print a[0][3] yields an error, print a[0][3][0] goes without a problem, and so on. I guess that's why they call it alpha version :) I sincerely hope that this won't be a part of the language when a stable release of Py3k is out.






Saturday, December 8, 2007

Running fast code in Python

As I have suspected, the numbers I got for execution time of the Haskell fibonacci program (look at my previous posts) are wrong, in the sense that I compiled the program without optimisation, as most normal users will do in real applications. That is why I feel that I should redeem myself and give the numbers for the program with optimisation turned on. So, when compiling the program using

ghc -O2 fib.hs -o fib -no-recomp (thanks to Don Stewart)

execution time is 0.64s (mean time of five runs). This time is much closer to the one reported on the Haskell blog (0.48s). Ok, so my hat goes off to Haskell---it really is faster than Python some 50 times.
But, being a Python fan, I couldn't help but feel that something is wrong here. Is Haskell really that good? It's fast, easy to parallelize, pure functional and it probably cooks for you after a hard day at work. So, why should I use Python?
Well, the first thing that comes to my mind is the fact that ghc generated a 925kB exe file from a 1kB file that computes fibonacci numbers. In comparison, gcc (3.4.2) generated 16 kB for a fibonacci program that ran 0.75s (compiled using -O2). While it is still impresive that Haskell was faster than C, I cannot help but think that it is just too much to have an exe file almost 1000 times larger than the source. While this is certainly a drawback of Haskell, it still wasn't enough to return my faith in Python. Then, another thing came to my mind---if Haskell can use optimisations, why can't Python? I mean, if I used -O2 in ghc, why can't I do somehting similar in Python?
Of course, running Python with -OO gave no real speedup---it shook off about one second in execution time. Instead of 26s, I got 25s for the non-parallel case, and instead of 18s, I got 17s for the non-parallel case. But, the great thing about Python are its packages. I remember a friend of mine telling me about a thing called Psyco. It was some kind of a package that optimised execution of Python code, so I decided to give it a shot. After downloading and installing Psyco (http://psyco.sourceforge.net/), I just added two lines to my source (I give the full source here):


from time import time
import psyco
psyco.full()

def fib(n):
if n == 0 or n == 1:
return n
else:
return fib(n-1) + fib(n-2)

if __name__ == "__main__":
start_time = time()
for i in range(36):
print "n=%d => %d" % (i, fib(i))

print "Time elapsed: %f" % (time() - start_time)


So, the lines I added were import psyco, and psyco.full(). I ran this thing and voila---the time of execution was 1.93s (mean time of five runs). Wow! That was all I could think. Remember, this was the non-parallel case, so it went from 25 seconds to 1.9s. Or, in terms of Haskell speed, Haskell now beat Python by on 3x. And with no extra effort (ok, adding two lines to the source) and no parallelization.
Unfortunately, psyco didn't help the parallel version---it still took 18s. But still, I think this is pretty amazing and it definitely brought back faith in Python. So, it's three times slower, but the synatx is much cleaner and it doesn't generate a thousand times bigger executable. I'm sticking to Python :)
(Note: some parts of this post are biased and down right offensive to Haskell fans (just kidding about the last one:) ). My intention is not to start a flame war or anything like that, I just felt that someone should stand up and take Python's side:). Seriously though, if you have any objections on my methodology, I would like to hear from you.)





Monday, December 3, 2007

Holy shmoly Haskell doesn't smoke Python away (that much)

Recently, there has been some talk about how Haskell smokes Python and Ruby away in the naive fibonacci algorithm. While it is obvious that Haskell will beat both due to the code being compiled and not interpreted, I decided to see if I got the same results as reported on the Haskell hacking blog, and also see how Python handles parallel processing. Note that I will refer to the Haskell hacking blog simply as Haskell blog from now on.

Since I have no experience in Ruby, I will only compare Python and Haskell. So, here is the code I used to test the two.
Python:


from time import time

def fib(n):
if n == 0 or n == 1:
return n
else:
return fib(n-1) + fib(n-2)

if __name__ == "__main__":

start_time = time()
for i in range(36):
print "n=%d => %d" % (i, fib(i))

print "Time elapsed: %f" % (time() - start_time)


So, as we can see, this code is pretty much the same as that reported in the blog about Haskell.


Haskell:

import Control.Monad
import Text.Printf

fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n-1) + fib (n-2)
main = forM_ [0..35] $ \i ->
printf "n=%d => %d\n" i (fib i)


This is also the same as in the blog, the only difference being that I import what is needed to make it run (for some reason, the import statements aren't included in the Haskell blog). Since I have no idea how to measure time in Haskell (any hint on how to do that is appreciated), I used the ptime command for Windows (available at http://www.pc-tools.net/win32/ptime/).

So, here are the results of the test:

Python 2.5: 26.07 s
Haskell (GHC 6.8.1): 3.873 s (mean of 5 runs)

So, the results for Python are pretty much the same as reported in the Haskell blog (25.16 s is reported there), but the ones for Haskell are not, in fact the Haskell blog reports 0.48 s, which is some four times faster than I got on my computer. What is the reason for this? I have no idea. I compiled the program using "ghc fib.hs", maybe they used some code optimizations? Anyway, the Haskell advantage seems to be melting, although Haskell is still some 6-7 times faster. However, the Haskell blog proceeds to use parallel processing in Haskell, but uses no such thing in Python. So, I decided to see if Python can take advantage of my two cores to shake some time off this 26 seconds. I used the Parallel Python package (available at http://www.parallelpython.com/) to make use of my two cores. Here is the modified Python program:


from time import time
import pp

def fib(n):
if n == 0 or n == 1:
return n
else:
return fib(n-1) + fib(n-2)


if __name__ == "__main__":
start_time = time()
ppservers = ()
job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"
jobs = [(input, job_server.submit(fib, (input,), (), ())) for input in range(36)]
for input, job in jobs:
print "n=%d => %d" % (input, job())

print "Time elapsed: %f" % (time() - start_time)


The result: 17.86 s
So, Python went from 26.07s to 17.86s, which means it ran 31.5% faster than on only one core. Unfortunately, I had no success in running the parallel Haskell code reported on the Haskell blog (for some reason, I always got "`pseq` not in scope" error while compiling). However, I can use the numbers given on the Haskell blog, where Haskell went from 0.48s to 0.42s on two cores. This means a 12.5% reduction in execution time. Although I admit that is takes slightly more intervention from the programmer to make the parallelism in Python work, the gain in speed is also more significant. As a final note, let us compare 3.873s (Haskell) and 17.86s (parallel Python) - the ratio is around 4.6, let's say 5. So, Haskell doesn't smoke Python away THAT much.





Tuesday, November 20, 2007

Trie in Python

Here is my humble implementation of a trie in Python, just something to get me started.
Any comments on improving the code are welcome


import re, types
from collections import defaultdict

def tokenize(s, removePunctuation = True):
if removePunctuation:
p = re.compile('[\.,!?()\[\]{}:;"\'<>/ \n\r\t]')
return [el for el in p.split(s) if el]
else:
t = []
s = re.split('\s', s)
p = re.compile('(\W)')
for phrase in s:
words = p.split(phrase)
for word in words:
t.append(word)
return [el for el in t if el]

class Node(object):

def __init__(self):
self.freq = 0
self.next = defaultdict(int)

class Trie(object):

def __init__(self, inFile):

self.nodes = []
self.nodes.append(Node())
self.n = 1
self.numberOfWords = 0

for line in file(inFile):
words = tokenize(line)
for w in words:
currNode = 0
for char in w:
if self.nodes[currNode].next[char] == 0:
self.nodes.append(Node())
self.nodes[currNode].next[char] = self.n
currNode = self.n
self.n += 1
else:
currNode = self.nodes[currNode].next[char]
self.nodes[currNode].freq += 1
self.numberOfWords = len([node for node in self.nodes if node.freq != 0])

def __getitem__(self, word):
"""Return the frequency of the given word."""

if isinstance(word, types.StringTypes):
currNode = 0
for char in word:
if self.nodes[currNode].next[char] == 0:
raise AttributeError("No such word: %s" % word)
else:
currNode = self.nodes[currNode].next[char]
return self.nodes[currNode].freq
else:
raise TypeError()

def __len__(self):
"""Return the number of nodes in the trie."""

return self.n