tag:blogger.com,1999:blog-86880585512959465142024-03-14T11:11:16.749-07:00Daily rantA random blog about everything, mostly programmingFiloxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.comBlogger21125tag:blogger.com,1999:blog-8688058551295946514.post-55678534453585012010-05-23T09:44:00.000-07:002010-05-23T09:53:17.941-07:00Running separate xsession in UbuntuEvery time I upgrade to a new version of Ubuntu, it breaks my xsession (which runs bear Xmonad). And every time I have to mess around for half an hour to find what needs to be done to get it back. All that needs to be done is to modify the file /usr/share/xsessions/xmonad.desktop to look like this:<br /><br /><pre><br />[Desktop Entry]<br />Encoding=UTF-8<br />Name=XMonad<br />Comment=Lightweight tiling window manager<br />Exec=/etc/X11/Xsession<br />Icon=xmonad.png<br />Type=XSession<br /></pre><br /><br /><br /><script>reddit_url='http://filoxus.blogspot.com/2010/05/ubuntu-and-xsession.html'</script><script>reddit_title='Running separate xsession in Ubuntu'</script><script language="javascript" src="http://reddit.com/button.js?t=3"></script><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com0tag:blogger.com,1999:blog-8688058551295946514.post-6494428374413154502010-03-13T05:05:00.000-08:002010-03-13T05:14:36.594-08:00Twitter corpus v0.1I am pleased to announce that we have published the first version of our Twitter corpus. All the data was collected from Twitter's streaming API over a period of about two months (November 11th 2009 until February 1st 2010). You can download the corpus from our <a href="http://demeter.inf.ed.ac.uk">social media website</a> (which we just set up recently). There is an <a href="http://demeter.inf.ed.ac.uk/papers/petrovic10edinburgh.pdf">accompanying paper</a> which gives some statistics about the corpus. One things that might interest all the 13-year old girls out there is that it seems Justin Bieber > Nick Jonas (look at table 3 in the paper). I believe that the Twitter corpus will be of interest to anyone working in social media research and/or NLP. We do plan to release subsequent versions as we get more data (and we might release old data starting from April 2009, but more on this later).Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com0tag:blogger.com,1999:blog-8688058551295946514.post-30122299990336251142009-07-10T07:14:00.000-07:002009-07-11T05:56:23.261-07:00Limits in stdint.hI've recently had problems with compiling my code that used the numeric limits defined in <code>stdint.h</code>, UINT64_MAX in particular. The error I was getting was<br /><pre class="code"><br />error: ‘UINT64_MAX’ was not declared in this scope<br /></pre><br />What I didn't know is that simply including <code>stdint.h</code> wasn't enough. The macro __STDC_LIMIT_MACROS has to be defined before the point where <code>stdint.h</code> is included, otherwise the limits are not defined. The best way of defining this macro, IMHO, is by compiling with -D__STDC_LIMIT_MACROS, instead of manually defining the macro somewhere in code. Hope this post will save someone a few minutes (hours?) if they run into similar problems.<br /><br /><script>reddit_url='http://filoxus.blogspot.com/2009/07/limits-in-stdinth.html'</script><br /><script>reddit_title='Limits in stdint.h'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com1tag:blogger.com,1999:blog-8688058551295946514.post-54861981939472904702009-06-20T03:11:00.000-07:002009-06-20T03:29:03.793-07:00Do public prediction markets really fail?A few days ago I came across <a href="http://torontopm.wordpress.com/2009/06/11/why-public-prediction-markets-fail/">this article</a> explaining why public prediction markets fail. The article gives an example where three different PMs failed to pick a winner in American Idol (Betfair) or Britain's got talent (Hubdub and Intrade). While it was definitely an interesting read, I feel that the author didn't take some things into account.<br />First, the success of PMs depends on the notion of participants playing risk neutral strategies (see, for example, the <a href="http://linkinghub.elsevier.com/retrieve/pii/S016517650600022X">Manski 2005 paper</a>) -- when people play with fake money, this may well not hold (they will tend to risk more than usual because they have nothing to lose).<br />Also, PMs are said to be more accurate than other ways of aggregating opinions such as polls, or exert opinions. It would be nice if the author had compared these predictions with some predictions made by opinion polls or experts and shown how the predictions differ.<br />Then there's the thing with data from Hubdub. As noted in the article, Hubdub's market concerned with Britain's got talent failed to predict the true outcome, giving Susan Boyle 78% chance of winning. What was not taken into account is that there were in fact more markets on Hubdub concerned with Britain's got talent. The one used in the article can be found <a href="http://www.hubdub.com/m42972/Who_will_win_Britains_Got_Talent">here</a>. However, here's <a href="http://www.hubdub.com/m42543/Who_will_win_Britains_Got_Talent">another market</a> that accurately predicted that "Susan Boyle OR Diversity" will win (Diversity won). Note the following: the market that got Susan Boyle wrong had 25k$ of activity, whereas the one that got it right had twice as much activity. So, if you were to make any decisions based on PM predictions, you would probably go for the one with more activity and you would be right. I don't know if Intrade of Betfair had more markets for the same event.<br />Anyway, I agree in the bottom line that accuracy of PMs depends on how much information its participants have, and that, ultimately, they will fail some of the time. However, we should be careful about making statements like "public prediction markets fail", especially since there are so many examples when they don't (this might be a topic for another post). And even when they do fail, it's important to understand why.<br /><br /><script>reddit_url='http://filoxus.blogspot.com/2009/06/do-public-prediction-markets-really.html'</script><br /><script>reddit_title='Do public prediction markets really fail?'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com0tag:blogger.com,1999:blog-8688058551295946514.post-39882156812405247552009-06-08T14:41:00.000-07:002009-06-08T15:15:53.626-07:00Wasting just a little bit more time: a tale of dzen and redditAre you tired of switching from the console to browser, and then pressing F5 to see what's new on reddit? So am I. There's got to be a better way of seeing new posts on reddit without leaving the comfort of the command line. And there is. If you run Linux with dzen, here's a possible solution. Simply use <a href="http://montyphyton.googlepages.com/reddit.py">this script</a>. It checks reddit and outputs the first ten titles in a way that can be used by <a href="http://gotmor.googlepages.com/dzen">dzen</a>. It should be clear from the source how to make the script output more than ten titles or how to monitor something other than the main reddit (e.g., programming, pics, funny or others subreddits). Now all one needs to do is run this script every n seconds (depends on how often you expect new articles to arrive), you can use something like <a href="http://montyphyton.googlepages.com/checkReddit.sh">this</a>. One last thing left to do is add a line to your .xsession that calls your checker script, e.g., I have the following line:<br /><pre class="code"><br />~/.dzen/checkReddit.sh | dzen2 -y 800 -ta c -l 10 -bg '#2c2c32' -fg 'grey70' -p -e 'entertitle=uncollapse;leavetitle=collapse' &<br /></pre><br />And here are the results (the reddit bar sits quietly at the bottom of your screen until you move your mouse over it, when you get the situation shown on the picture to the right):<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBZegymqm44atZJGPJlQ2zKV_rJIzM2BlhSZfG08VOzu6EiAjRMGhveR5e1DEqmH2KIT-Pb4acAc8iRV8h_Ap4vdRYMBLrhJOg0tp4xqNh-JT15sEUJvEFu8nqFjWiyHvkt7xH4CNWUDU/s1600-h/reddit-still.png"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;width: 400px; height: 250px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBZegymqm44atZJGPJlQ2zKV_rJIzM2BlhSZfG08VOzu6EiAjRMGhveR5e1DEqmH2KIT-Pb4acAc8iRV8h_Ap4vdRYMBLrhJOg0tp4xqNh-JT15sEUJvEFu8nqFjWiyHvkt7xH4CNWUDU/s400/reddit-still.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5345081568749680290" /></a><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7OkiCzU_nLhN7yCmNIgBe4mgxn8TOZKaIKJ8smKKl98Pzzl7yrPf-UoFdmOSwcW9cUTCLaqu8kLEW-XbXuYLa2k9Idjtg2N7MJEC-Ky6VLTsgew0aHhsSqPZUYDr5p_qENpCYDA_WUrY/s1600-h/reddit-moneyshot.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 250px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7OkiCzU_nLhN7yCmNIgBe4mgxn8TOZKaIKJ8smKKl98Pzzl7yrPf-UoFdmOSwcW9cUTCLaqu8kLEW-XbXuYLa2k9Idjtg2N7MJEC-Ky6VLTsgew0aHhsSqPZUYDr5p_qENpCYDA_WUrY/s400/reddit-moneyshot.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5345082696450927986" /></a><br />Of course, once you see what's new you still have to switch to your browser if you want to actually follow the links, but this should at least save you the trouble of switching only to find out that there's nothing new. Note that there are many things left to be done here, this is just a little sample of how you can waste a few more minutes of your life.<br /><script>reddit_url='http://filoxus.blogspot.com/2009/06/wasting-just-little-bit-more-time-tale.html'</script><br /><script>reddit_title='Wasting just a little bit more time: a tale of dzen and reddit'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com0tag:blogger.com,1999:blog-8688058551295946514.post-84253885514820672622009-04-23T14:15:00.001-07:002009-04-23T14:15:11.733-07:00FlickrThis is a test post from <a href="http://www.flickr.com/r/testpost"><img alt="flickr" src="http://www.flickr.com/images/flickr_logo_blog.gif" width="41" height="18" border="0" align="absmiddle" /></a>, a fancy photo sharing thing.Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com0tag:blogger.com,1999:blog-8688058551295946514.post-83184450906191508002009-01-17T15:56:00.000-08:002009-01-17T15:58:20.648-08:00<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoyw1nh2TtysuK9QVwxBb8gMg4URf0ccOXcymKURCsRQQJS5kWyvb-OqkVNXsEAfdx1iKAqdDLO2BGb_aK5r2jcM78MM-lGzCAqvtCqQhwwpDfIWKHG5GE_SRzJH9p05VWPhgwIyTRYfo/s1600-h/reddit_funny.jpg"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;width: 400px; height: 250px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoyw1nh2TtysuK9QVwxBb8gMg4URf0ccOXcymKURCsRQQJS5kWyvb-OqkVNXsEAfdx1iKAqdDLO2BGb_aK5r2jcM78MM-lGzCAqvtCqQhwwpDfIWKHG5GE_SRzJH9p05VWPhgwIyTRYfo/s400/reddit_funny.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5292416048808409730" /></a>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com0tag:blogger.com,1999:blog-8688058551295946514.post-43483227092668759302008-06-17T15:57:00.000-07:002008-12-08T15:55:15.934-08:00Firefox 3 download day - Mozilla just got into politicsWell, Firefox 3 finally arrived and all is well in the world. I was just browsing the downloads map and was ready to call it a night when something caught my eye. Something was strange about that <a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-X3VBZHZTSA4WlbGekHEvzRhB-mnkWdJI7l4QtrJyLf4gqY1bnmVClaba3iiRcnM9F_Vn8T9nA_WeCaDR9eNACNCVcuZYkAwmhM-Pv5ycvJ_vBTkAFLh0xpbRcbbcizr2QTBgt8YsgAg/s1600-h/ff3.bmp"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-X3VBZHZTSA4WlbGekHEvzRhB-mnkWdJI7l4QtrJyLf4gqY1bnmVClaba3iiRcnM9F_Vn8T9nA_WeCaDR9eNACNCVcuZYkAwmhM-Pv5ycvJ_vBTkAFLh0xpbRcbbcizr2QTBgt8YsgAg/s400/ff3.bmp" border="0" alt=""id="BLOGGER_PHOTO_ID_5212993380931769922" />map</a>... Ah, that's it, there seems to be one state missing on the map. For all of you that don't know which one is missing, let me give you a hint -- it's a small state just north of Albania, south of Serbia. You guessed it, it's Kosovo. Yes, <a href="http://en.wikipedia.org/wiki/Kosovo">Kosovo</a>. The state that got its independence in February this year is not plotted as a separate country, but rather as a part of Serbia (of which it was a part until recently). Now, it may be considered overranting, but I wonder why isn't Kosovo listed as a separate country. After all, it is recognized by the US (country which Mozilla calls home), the UK, France, Italy, Germany... The list goes on. And it's not like the guys at Mozilla didn't have time to prepare, Kosovo got its independence in February which gave them some four months to add it to the countries list. So, why can't I see how many people from Kosovo downloaded Firefox 3? And more importantly, who decides if a country is relevant enough to be added to a company's list? It seems that nowadays a country has to be recognized not only by other countries, but also by companies if it wishes to make a meaningful existence...<br /><br /><script>reddit_url='http://filoxus.blogspot.com/2008/06/firefox-3-download-day-mozilla-just-got.html'</script><br /><script>reddit_title='Firefox 3 download day - Mozilla just got into politics'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com1tag:blogger.com,1999:blog-8688058551295946514.post-88457437535400357062008-04-07T01:49:00.000-07:002008-04-07T01:52:38.136-07:00Saving e-mails to disk (from a gmail account)As I was trying to download all my spam messages to disk (I'm building a database of spam messages), I realised that this is no simple task. Well, one thing I could do is save the messages using Thunderbird or Outlook, but since I don't use either (I consider the Gmail web interface very nice and user-friendly) this one is out of the question. However, after a little browsing I discovered a wonderful Python package called <a href="http://libgmail.sourceforge.net/">libgmail</a>. Long story short, here is the script I used to download all the messages from the spam folder:<br /><pre class="code"><br />#!/usr/bin/env python<br />'''<br />savemsg.py -- Download all messages from a specified folder<br />License: GPL 2.0<br />'''<br /><br />import sys<br />from getpass import getpass<br />import libgmail<br /><br />if __name__ == "__main__":<br /> try:<br /> name = sys.argv[1]<br /> except IndexError:<br /> name = raw_input("Gmail account name: ")<br /> <br /> pw = getpass("Password: ")<br /><br /> ga = libgmail.GmailAccount(name, pw)<br /><br /> print "\nPlease wait, logging in..."<br /><br /> try:<br /> ga.login()<br /> except libgmail.GmailLoginFailure,e:<br /> print "\nLogin failed. (%s)" % e.message<br /> sys.exit(1)<br /> else:<br /> print "Login successful.\n"<br /><br /> FOLDER_list = {'U_INBOX_SEARCH' : 'inbox',<br /> 'U_STARRED_SEARCH' : 'starred',<br /> 'U_ALL_SEARCH' : 'all',<br /> 'U_DRAFTS_SEARCH' : 'drafts' ,<br /> 'U_SENT_SEARCH' : 'sent',<br /> 'U_SPAM_SEARCH' : 'spam',<br /> }<br /><br /> FOLDER_list = raw_input('Choose a folder (inbox, starred, all, drafts, sent, spam): ')<br /> folder = ga.getMessagesByFolder(FOLDER_list)<br /><br /> for thread in folder:<br /> for msg in thread:<br /> print "Downloading message %s " % msg.id<br /> encIndexStart = msg.source.find('charset=')<br /> if encIndexStart != -1:<br /> encIndexEnd = (msg.source.find(' ', encIndexStart),\<br /> msg.source.find('\n', encIndexStart),\<br /> msg.source.find(';', encIndexStart),\<br /> msg.source.find('"', encIndexStart+10))<br /> encIndexEnd = [ind for ind in encIndexEnd if ind != -1]<br /> encIndexEnd = min(encIndexEnd)<br /> enc = msg.source[encIndexStart + 8:encIndexEnd]<br /> enc = enc.replace('"', '').replace(';', '')<br /> else:<br /> enc = 'ascii'<br /> print "Detected encoding %s\n" % enc<br /> try:<br /> f = open(msg.id + " " + msg.subject + ".txt", 'w')<br /> except:<br /> # message subject contains characters forbidden by the os in the<br /> # file name, use just message id<br /> f = open(msg.id + ".txt", 'w')<br /> try:<br /> f.write(msg.source.decode(enc).encode('utf-8'))<br /> except:<br /> f.write(msg.source)<br /> f.close()<br /> print "\n\nDone."<br /></pre><br /><br />One could use the script to download messages from any gmail folder. The encoding of the message is automatically recognized and the message is saved in UTF-8 to facilitate later processing. Of course, you have to have libgmail installed to run the script. It is also very easy to adapt the script to use it for any other purpose (I actually wrote this script by changing one of the demo scripts that come with libgmail).<br /><br /><br /><br /><br /><script>reddit_url='http://filoxus.blogspot.com/2008/04/saving-e-mails-to-disk-from-gmail.html'</script><br /><script>reddit_title='Saving e-mails to disk (from a gmail account)'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com3tag:blogger.com,1999:blog-8688058551295946514.post-66862565041173989872008-03-04T04:04:00.000-08:002008-12-08T15:55:16.216-08:00Reddit alien in LaTeXWhat to do on a lazy sunday afternoon? Well, there are probably a thousand useful<br />things one could do, but I chose to do something completely useless. And here it is, the reddit alien drawn in LaTeX (using PsTricks) :)<br /><pre class="code"><br />\documentclass{memoir}<br /><br />\usepackage{pstricks}<br />\usepackage{pst-all}<br /><br />\begin{document}<br /><br />\thispagestyle{empty}<br /><br />%\psset{linecolor=red}<br />\psset{linewidth=1pt}<br />\begin{pspicture}(6,9)<br />%\psgrid[subgriddiv=1,griddots=10,gridlabels=10pt](0,0)(12,12)<br />\psframe(0,0)(6,9)<br />% head<br />\psellipse(3,6)(1.8,1)<br />% ears<br />\psarc{-}(1.5,6.5){0.4}{38}{230}<br />\psarc{-}(4.5,6.5){0.4}{-50}{142}<br />% eyes<br />\pscircle[fillstyle=solid,linestyle=none,fillcolor=orange](2.3,6.3){0.3}<br />\pscircle[fillstyle=solid,linestyle=none,fillcolor=orange](3.7,6.3){0.3}<br />% mouth<br />\psbezier(2.3,5.6)(2.5,5.3)(3.5,5.3)(3.7,5.6)<br />% the tentacle thingie<br />\psline(2.99,6.99)(3.6,8)<br />\psline(3.581,7.993)(4.1,7.8)<br />\pscircle(4.35,7.65){0.3}<br />% arms<br />\psarc{-}(2.8,4.3){1.1}{130}{231}<br />\psarc{-}(3.15,4.3){1.1}{-48}{50}<br />% body<br />\psbezier(2.1,5.15)(2.0,4.7)(1.8,3.3)(2.6,2.7)<br />\psbezier(3.85,5.15)(4.1,4.65)(4.1,3.3)(3.4,2.7)<br />% feet<br />\psline(1.65,2.7)(4.3,2.7)<br />\psarc{-}(2.1,2.685){0.45}{70}{180}<br />\psarc{-}(3.85,2.685){0.45}{0}{105}<br />\end{pspicture}<br /><br />\end{document}<br /></pre><br />Note that I used the memoir document class, but you could also use article if you don't have memoir available (the results would be the same). The alien does not look exactly the same as the official reddit one, but who cares? And here is what you should get when you run this example through LaTeX:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjczlxmGW_wnm4Ag__w76uGBR84MPDJZ358IwI72evOhlCGB3xSlmZDZEB_MWEbhyphenhypheneY6sVf-kGRFAcA67MJMtcsQNU5Wra7FD_irbQG3Vyjkz_kTDfizQB8JWEFNc3vN93YzX5_upCq4ak/s1600-h/reddit.JPG"><img style="float:center; margin:0 10px 10px 0;cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjczlxmGW_wnm4Ag__w76uGBR84MPDJZ358IwI72evOhlCGB3xSlmZDZEB_MWEbhyphenhypheneY6sVf-kGRFAcA67MJMtcsQNU5Wra7FD_irbQG3Vyjkz_kTDfizQB8JWEFNc3vN93YzX5_upCq4ak/s400/reddit.JPG" border="0" alt=""id="BLOGGER_PHOTO_ID_5173856709876090802" /></a><br />And people say LaTeX suxx...<br /><br /><script>reddit_url='http://filoxus.blogspot.com/2008/03/reddit-alien-in-latex.html'</script><br /><script>reddit_title='Reddit alien in LaTeX'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com8tag:blogger.com,1999:blog-8688058551295946514.post-8292940727000738732008-02-25T01:43:00.000-08:002008-12-08T15:55:16.299-08:00Deferred printing in LaTeXEver written a textbook? Ever written any document that has questions/problems and answers to those problems? Usually when you do this, you want to have questions printed in one place (or questions in each section of the document), and answers printed at the end of the document (so your readers, which are usually students, would try to actually solve the problem rather than just look at the solution which is right under the problem).<br />Now, I don't know how most of you usually do this, but up until now the only way I knew to do this is the "brute force" approach---I would simply write the questions and then, at the end of the document, I would write the answers. The obvious drawback of this approach is that you have to look up each individual question when writing the answers (just to see what it was) and this can quickly become very boring. Or, if you have the questions/answers on paper, they are usually written in pairs question/answer so you have to first type all the questions (ignoring the answers), and then go back to the begining of the paper and type all the answers (ignoring the questions). Anyway, I hope you get the picture why I hate doing this.<br />Wouldn't it be great if I could somehow have the question and its answer in the same place in the source of the document (notice that we are not talking about WYSIWYG text processors here), but then defer the printing of answers in the output document (that is, print the answers at the end of the document)? The solution to my problem comes in a form of the LaTeX box mechanism. Just take a look at this minimal example:<br /><pre class="code"><br />\documentclass{article}<br /><br />\newbox\answerscollect<br />\newcounter{problem}<br />\setcounter{problem}{0}<br />\renewcommand{\theproblem}{\textit{Problem \arabic{problem}.}\,}<br /><br />\def\answer#1{\par\setbox\answerscollect=\vbox{\unvbox\answerscollect\vspace{5mm}\theproblem\,#1\par}}<br />\def\printanswers{\unvbox\answerscollect\setbox\answerscollect=\vbox{}}<br />\def\initbox{\setbox\answerscollect=\vbox{Answers:\par}}<br /><br />\newcommand{\problem}[1]{\addtocounter{problem}{1}\par\theproblem\textit{#1}}<br /><br />\begin{document}<br /><br />\initbox<br /><br />\section{Intro}<br />Some text before...<br /><br />\problem{What is the capital of the US?}<br />\answer{Washington, D.~C.}<br /><br />\problem{What is 2+2?}<br />\answer{4}<br /><br />\section{Foo}<br /><br />Some other text goes here...<br /><br />\section{Answers}<br /><br />\printanswers<br /><br />\end{document}<br /></pre><br />Now I know it looks a bit complicated, but it really isn't. What we are doing is creating a box and storing all the answers in it, and then, when the time is write, we simply print all the contents of that box using the <b>\printanswers</b> command. This way, we can simply have the problem and its solution (answer) in the same place in the source of the document, but have the answers appear in a totally different place in the output pdf (or postscript or whatever). Now, I'm not going to go into details of the LaTeX code itself here. Those who know LaTeX will pretty much understand the code, and those who don't should first learn the basics before trying to get this. Anyway, the code can be used out-of-the-box; if someone would like to change something but doesn't know how, you can contact me. Last, but not least, here is how the output pdf looks like:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiieIUTeFEUDMZV2PCNgMXuqIXM_FeeesIc-2A2krVuy9mqq0JVVwXPeNK8Tn1j9nZF_bvJ5UoEi5-v8SY6oDCTmsaXgl6CnD-GMN8ZRGxFUwKi8dtR5_HWF46Gx0EcwuhH71siVEoQHc/s1600-h/differed.JPG"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiieIUTeFEUDMZV2PCNgMXuqIXM_FeeesIc-2A2krVuy9mqq0JVVwXPeNK8Tn1j9nZF_bvJ5UoEi5-v8SY6oDCTmsaXgl6CnD-GMN8ZRGxFUwKi8dtR5_HWF46Gx0EcwuhH71siVEoQHc/s400/differed.JPG" border="0" alt=""id="BLOGGER_PHOTO_ID_5170851574559975234" /></a><br />UPDATE: As Evan noted, if you are going to use multiple files, don't use \include. Instead, use the \input command which does not create new .aux files.<br /><br /><br /><script>reddit_url='http://filoxus.blogspot.com/2008/02/deferred-printing-in-latex.html'</script><br /><script>reddit_title='Deferred printing in LaTeX'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com9tag:blogger.com,1999:blog-8688058551295946514.post-34950025332033700162008-01-31T01:09:00.000-08:002008-05-19T06:52:13.568-07:00Unifying expressions in PythonMost of you that had some AI course in college probably heard of unification of expressions. All of you who ever programmed (and I use the term loosly) in Prolog know what this is. For all others, just a note that I'm not going to explain here what is unification and how it's done. If you don't know what it is, check out, for example <a href="http://en.wikipedia.org/wiki/Unification">this WP article</a>.<br /><a href="http://starbase.trincoll.edu/~ram/cpsc352/notes/unification.html">Here</a> we can see the algorithm for finding the most general unifier of two expressions in pseudo-code (check out the bottom of the page). And <a href="http://montyphyton.googlepages.com/mgunifier.PY">here</a> is my implementation of unifying expressions in Python. I'm not going to explain the code here---if you're interested, you can do this yourself. I'll just give a brief example of how you can use this. So, suppose that you have the following two expressions:<br />expr1 = P(f(a), g(y), f(w))<br />and<br />expr2 = P(x, g(f(x)), y),<br />where P is a predicate, g and f are functions, and x, y and w are variables (this is an actual assignment from a test we gave our students some two months ago). Lets solve this using Python:<br /><pre class="code"><br />import mgunifier as mg # just to make the listing shorter<br /><br /># declare predicates, functions, and variables<br />P = mg.predicate("P")<br />g = mg.function("g")<br />f = mg.function("f")<br />x = mg.variable("x")<br />y = mg.variable("y")<br />w = mg.variable("w")<br />a = mg.constant("a")<br /><br />expr1 = [P, [f, a], [g, y], [f, w]]<br />expr2 = [P, x, [g, [f, x]], y]<br />uni = mg.mgUnifier(expr1, expr2) # find the most general unifier<br />new1 = uni(expr1)<br />new2 = uni(expr2)<br />print "The most general unifier is ", uni<br /># check if the two expressions are the same after unification<br />print new1<br />print new2<br /></pre><br />That's all there is to it. If someone has an idea on how to make things more simple or improve the code in any way, I am always glad to hear from you. Now, those of you that had some experience with Prolog probably know that, although unification is the most basic tool of this language, there's a little bug/wart when using it (this applies to SWI Prolog implementation, I don't know how others handle it). Typing, for example,<br /><pre class="code"><br />?- X = f(X).<br /></pre><br />results in X = f(**), meaning that variable X is now f(f(f(f(...)))). This is, of course, wrong, because the left and the right side of the equal sign will never be the same (and if you don't believe me, check out the algorithm). Why exactly is Prolog doing this is beyond me. Anyway, try the following code:<br /><pre class="code"><br />import mgunifier as mg<br /><br />P = mg.predicate("P")<br />g = mg.function("g")<br />f = mg.function("f")<br />x = mg.variable("x")<br />y = mg.variable("y")<br />w = mg.variable("w")<br />a = mg.constant("a")<br /><br />first = [P, x, [g, y], z]<br />second = [P, [f, a], z, y]<br />uni = mg.mgUnifier(first, second)<br />print uni<br /></pre><br />Not only that you get an error, but you can also see exactly where the algorithm failed.<br />Now, what's the whole point of this post? To demonstrate some Python implementation of an algorithm no one cares about? Well, partially. I was looking through the web for something similar and found only <a href="http://christophe.delord.free.fr/pylog/index.html">this</a>. As it didn't satisfy me, I wrote my own program that unifies expressions, so anyone looking for something like this doesn't have to. But, what I'd really like people to see (and read) is this (copy-paste from the source):<br /><pre class="code"><br />def mgUnifier(k1, k2):<br /> """Return the most general unifier of two expressions k1 and k2."""<br /> <br /> if not k1 or not k2: return supstitution()<br /> if isinstance(k1, (constant, variable, function, predicate)) or isinstance(k2, (constant, variable, function, predicate)):<br /> if k1 == k2:<br /> return supstitution()<br /> if isinstance(k1, variable):<br /> if k1 in k2:<br /> return error(`k1` + " in " + `k2`)<br /> else:<br /> return supstitution(k2, k1)<br /> if isinstance(k2, variable):<br /> if k2 in k1:<br /> return error(`k2` + " in " + `k1`)<br /> else:<br /> return supstitution(k1, k2)<br /> if not isinstance(k1, variable) and not isinstance(k2, variable):<br /> return error(`k1` + " and " + `k2` + " cannot be unified!")<br /><br /> alpha = mgUnifier(head(k1), head(k2))<br /> if isinstance(alpha, error):<br /> return alpha<br /> k3 = alpha(tail(k1))<br /> k4 = alpha(tail(k2))<br /> beta = mgUnifier(k3, k4)<br /> if isinstance(beta, error):<br /> return beta<br /> return alpha(beta)<br /></pre><br />This beautifully (IMHO) demonstrates the power and clarity of Python. Compare this code to the algorithm pseudo-code---it's almost the same. This is what I love about Python---I can simply copy the algorithm's pseudo-code and then just implement the "magic" that happens in the background.<br />Like I said, the purpose of this program is primarly educational, the code could probably be better written, but I stuck with the implementation that was most similar to the algorithm. Keeping that in mind, I appreciate any feedback on the code.<br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><br /><script>reddit_url='http://filoxus.blogspot.com/2008/01/unifying-expressions-in-python.html'</script><br /><script>reddit_title='Unifying expressions in Python'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com18tag:blogger.com,1999:blog-8688058551295946514.post-19906116756903187412008-01-21T01:15:00.000-08:002008-01-21T01:17:41.858-08:00Real-life PythonI just came home all tired and the minute I saw the place it hit me---I didn't<br />tidy up in ages. There were books everywhere, clothes all over the place, dirty<br />dishes... As I'm always tired when I come home from work, I never feel like cleaning<br />the place up. Somehow Python came up in my head as the solution to my problem (well,<br />not really a solution:) ). What if we could execute Python code in real life. I<br />mean, wouldn't it be great if we could do something like:<br /><pre class="code"><br />apartment.sort()<br />dishes.wash()<br /></pre><br />As I of course didn't feel like cleaning, I decided I'd better spend my time writing<br />about how I would use Python in real life. Beside cleaning the house, I think Python<br />could be of great help in the following problems as well.<br /><br /><b>Understanding women</b><br />Did you ever have a fight with your girlfirend/wife (it's a rhetorical question,<br />of course you did)? Did you ever wonder "What is going on in that mind of hers"?<br />Well, think about this one:<br /><pre class="code"><br />print girlfriend.__doc__<br /></pre><br />And if that didn't help, wouldn't it be great to be able to do something like<br /><pre class="code"><br />import dis<br />print dis.dis(girlfriend.think)<br /></pre><br />This would not only help you understand her, but also enable you to predict how<br />will she react. That's right, no more "What did I do wrong"---just look at the<br />code and you will know.<br /><br /><b>The answer to life</b><br />For all those in doubt as to whether they are wasting their life going to church<br />every Sunday, here's an ellegant Pythonic solution:<br /><pre class="code"><br />import time<br />if life.endswith(death):<br /> while alive:<br /> party()<br /> time.sleep(12*60*60)<br /> die()<br />else:<br /> import random<br /> religions = [christianity, islam, buddhism, hinduism]<br /> myReligion = random.choice(religions)<br /> for day in allDays:<br /> if day != sunday:<br /> be_humble()<br /> else:<br /> repent_sins(myReligion)<br /> #no more days, end of time is here<br /> go_to_heaven(myReligion)<br /></pre><br /><br />What would you use Python for in real life?<br /><br /><br /><script>reddit_url='http://filoxus.blogspot.com/2008/01/real-life-python.html'</script><br /><script>reddit_title='Real-life Python'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com0tag:blogger.com,1999:blog-8688058551295946514.post-49724259701682128812008-01-09T04:57:00.000-08:002008-12-08T15:55:16.420-08:00How to insert watermark in LaTeXFor my first post this year, I'll start off with something a bit different. This time we'll take a look at how to insert a watermark in a document using LaTeX. Note here that by watermark I mean some text that will appear on the background of each page of the document.<br />As with everything else in the wonderful world of LaTeX, there are more ways to do this. I know of two, which I will write about here. The first one is fairly simple, just include the package draftcopy and voila. Here's a minimal example of how it works:<br /><pre class="code"><br />\documentclass{article}<br />\usepackage{draftcopy}<br /><br />\title{Lorem ipsum}<br /><br />\begin{document}<br />\maketitle<br /><br />Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.<br /><br />\end{document}<br /></pre><br /><br />Running the previous example through LaTeX gives the following output:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCBwJQcyhOpJY9Gx9XhSH0tvwnmPru4evqpe-KajY6vRsbZyXJdSZ_ZacPNdyalwsjDKvCd8Pp3tnemBUqXz4ra5gqKWNYT98nbCWY1_QLUKT_19BOHhSs944hMxRmPBKfXe-z_voqEgQ/s1600-h/watermark.PNG"><img style="float:center; margin:0 10px 10px 0;cursor:pointer; cursor:hand;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCBwJQcyhOpJY9Gx9XhSH0tvwnmPru4evqpe-KajY6vRsbZyXJdSZ_ZacPNdyalwsjDKvCd8Pp3tnemBUqXz4ra5gqKWNYT98nbCWY1_QLUKT_19BOHhSs944hMxRmPBKfXe-z_voqEgQ/s400/watermark.PNG" border="0" alt=""id="BLOGGER_PHOTO_ID_5153461260030409682" /></a><br />Note that running pdflatex on this example will NOT insert the watermark. I use LaTeX->dvi->ps->pdf route to obtain the pdf.<br />Of course the draftcopy package gives a lot of options to customize the watermark, of which I believe \draftcopyName is the most useful. If you want the word ENTWURF to be printed instead of DRAFT, which is used by default, you would add the line "\draftcopyName{ENTWURF}{155}" in the preamble. The number 155 is the scaling factor for the font---play around with it to get the scaling you want.<br />For those that like to do it the hard way (like me:) ), here is a piece of code that does pretty much the same thing:<br /><pre class="code"><br /> \AddToShipoutPicture{%<br /> \setlength{\@tempdimb}{.5\paperwidth}%<br /> \setlength{\@tempdimc}{.5\paperheight}%<br /> \setlength{\unitlength}{1pt}%<br /> \put(\strip@pt\@tempdimb,\strip@pt\@tempdimc){%<br /> \makebox(600,-700){\rotatebox{45}{\textcolor[gray]{0.75}%<br /> {\fontsize{6cm}{6cm}\selectfont{DRAFT}}}}%<br /> }%<br /> }<br /></pre><br />To use this code, you have to include the following packages: graphicx, eso-pic, and type1cm. Here is a minimal example:<br /><pre class="code"><br />\documentclass{article}<br /><br />\usepackage{graphicx}<br />\usepackage{type1cm}<br />\usepackage{eso-pic}<br />\usepackage{color}<br /><br />\makeatletter<br />\AddToShipoutPicture{%<br /> \setlength{\@tempdimb}{.5\paperwidth}%<br /> \setlength{\@tempdimc}{.5\paperheight}%<br /> \setlength{\unitlength}{1pt}%<br /> \put(\strip@pt\@tempdimb,\strip@pt\@tempdimc){%<br /> \makebox(0,0){\rotatebox{45}{\textcolor[gray]{0.75}%<br /> {\fontsize{6cm}{6cm}\selectfont{DRAFT}}}}%<br /> }%<br />}<br />\makeatother<br /><br />\title{Lorem ipsum}<br /><br />\begin{document}<br />\maketitle<br /><br />Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.<br /><br />\end{document}<br /></pre><br />Be sure not to forget the \makeatletter and \makeatother commands. Unlike with draftcopy, running pdflatex on this example WILL insert the watermark in the pdf. I find this code nice because I can directly tinker with the various options. For example, if I want to change the angle of the text, I'll change the number inside the \rotatebox, if I want to move the text around the page, I'll change the two numbers inside \makebox, if I want to change the grayscale, I'll change the number inside \textcolor, etc.<br />UPDATE: The above code, as Ivijay pointed out, puts the word DRAFT on every page of the document. If you want it to appear on only one, you should use the <pre>AddToShipoutPicture*</pre> instead of <pre>AddToShipoutPicture</pre>. This command only adds the watermark to one page (first one if you leave the watermark code in your preamble).<br /><br />UPDATE: Neno wants to know how to use a picture instead of text. Well, it's just a matter of replacing the lines<br /><pre><br />\makebox(0,0){\rotatebox{45}{\textcolor[gray]{0.75}%<br />{\fontsize{6cm}{6cm}\selectfont{DRAFT}}}}%<br /></pre><br />with something like<br /><pre><br />\includegraphics{image.eps}<br /></pre><br />Of course, you might have to play around with adjusting the width and height of the picture, depending on its size. Also, you might want to change tempdimb and tempdimc to properly position the image.<br /><br />UPDATE: hcr suggests using draftwatermark package to do this sort of thing. I've just had a look at the package, and using it really seems simple. For most of the things I did here, there is a nice command that does that. However, some things are missing from the package. First, the color of the text is always grey (you can choose the grey scale, but can't have the watermark text in, say, red. There does not seem to be a way to change the typeface of the font for the watermark text, and the text is always centered, you can't move it around. And finally, there is no way you could insert a picture as a watermark (which is what Neno wanted to do), and IMHO this could be a big issue for many people who may want their company's logo as a watermark. So anyway, draftwatermark is a great choice if you just want to insert a word (centered) in the background of each page. If you need anything more sophisticated than that, you will have to do something in the line of what I described here.<br /><br />UPDATE: Amy wants to know is there a way to put the watermark only on some pages, preferably using something like \begin{watermark} and \end{watermark}. Well, Amy, there is no easy solution to your problem. One thing you could do is use \AddToShipoutPicture* that places the watermark only on one page, and then paste the code wherever you need it. Of course, this is ugly. I have managed to hack together some solution that seems to work, but the code is really ugly and I don't advise anyone to use it unless you really need to :). So, here goes:<br /><pre><br />\documentclass{article}<br /><br />\usepackage{graphicx}<br />\usepackage{type1cm}<br />\usepackage{eso-pic}<br />\usepackage{color}<br />\usepackage{everypage}<br /><br />\newenvironment{water}{\AddEverypageHook{\waterb}}{\AddThispageHook{\waterb}\AddEverypageHook{\watere}}<br /><br />\makeatletter<br />\newcommand{\waterb}{<br />\AddToShipoutPicture*{%<br /> \setlength{\@tempdimb}{.5\paperwidth}%<br /> \setlength{\@tempdimc}{.5\paperheight}%<br /> \setlength{\unitlength}{1pt}%<br /> \put(\strip@pt\@tempdimb,\strip@pt\@tempdimc){%<br /> \makebox(0,0){\rotatebox{45}{\textcolor[gray]{0.75}%<br /> {\fontsize{6cm}{6cm}\selectfont{DRAFT}}}}%<br /> }%<br />}}<br />\makeatother<br /><br />\makeatletter<br />\newcommand{\watere}{<br />\AddToShipoutPicture*{%<br /> \setlength{\@tempdimb}{.5\paperwidth}%<br /> \setlength{\@tempdimc}{.5\paperheight}%<br /> \setlength{\unitlength}{1pt}%<br /> \put(\strip@pt\@tempdimb,\strip@pt\@tempdimc){%<br /> \makebox(0,0){\rotatebox{45}{\textcolor[gray]{1}%<br /> {\fontsize{6cm}{6cm}\selectfont{DRAFT}}}}%<br /> }%<br />}}<br />\makeatother<br /><br />\title{Lorem ipsum}<br /><br />\begin{document}<br /><br />page one<br /><br />\newpage<br /><br />\begin{water}<br /> First watermarked page<br /> \newpage<br /> Second watermarked page<br /> \newpage<br /> Third watermarked page<br />\end{water}<br /><br />\newpage<br /><br />This page should not be watermarked<br />\end{document}<br /></pre><br />Well, Amy, let me know if this works for you...<br /><br /><script>reddit_url='http://filoxus.blogspot.com/2008/01/how-to-insert-watermark-in-latex.html#links'</script><br /><script>reddit_title='How to insert watermark in LaTeX'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com26tag:blogger.com,1999:blog-8688058551295946514.post-3424746359534864612007-12-31T05:55:00.000-08:002007-12-31T06:07:18.489-08:00Naive fibonacci in pypyAs this is my last post this year, I'll just keep it short. I've been busy doing all sorts of things, so here's just a fast revisit to our old friend, the naive fibonacci algorithm, this time featuring Pypy. I downloaded the 1.0.0 version and ran the by now famous naive fibonacci. To get to the point, it took 184 seconds. Yep, you read it right---3 minutes. The same thing that took 27 seconds in CPython 2.5, or around 3 seconds using Psyco. Now, I didn't read much about pypy, but as I understand it, it should be faster than the CPython implementation... And AFAIK, the guys that made psyco abandoned that project to go work on bringing similar functionality to pypy. Now, I hate to bitch (no, no I don't:) ), but this really seems kinda slow. What's going on here? It's probably the function call overhead that slows it down. One should look at the code of pypy and see what exactly they put on the stack each time... The point being, I'll stick to CPython for a while more, at least until the naive fib runs under 20 seconds :) ('Cos what's more important in life than running a fast fibonacci algorithm?)<br />Happy holidays everyone<br /><br /><script>reddit_url='http://filoxus.blogspot.com/2007/12/naive-fibonacci-in-pypy.html#links'</script><br /><script>reddit_title='Naive fibonacci in pypy'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com0tag:blogger.com,1999:blog-8688058551295946514.post-21963206964562091292007-12-22T11:26:00.000-08:002007-12-27T09:25:59.655-08:00Python 3000 - how mutable is immutable?I just downloaded Py3k alpha 2 and decided to play with it a bit. Well, beside the print becoming a function, things seemed to be more or less the same as in 2.5. Then I decided to play with the more esoteric thingies in Python.<br />It has been long known that Python immutables aren't really immutable. For example, if you enter the following code in 2.5:<br /><br /><pre class="code"><br />a = ([1], 2)<br />a[0] += [2]<br /></pre><br /><br />you will get an error. However, printing a reveals that it has indeed been modified (a is now ([1, 2], 2)). Ok, so what happened to this little bug/hack/feature/gotcha/whatever-you-want-to-call-it in Py3k? Well, entering the code above also raises the same error, and again it actually modifies a. Now, I don't know is this "feature" is documented anywhere, but I really don't see any point in it being a part of the language (if someone can give me an example where it would be useful to have this kind of behavior, let me know). So, why is it still present in Py3k, when it's been known for a while now? I guess someone either thinks this is a good thing or they just forgot about it, either way---guys please change this, it's annoying me :) (and what better motivation to change the language than a rant by some guy on the Internet that you never met or heard of?)<br />But, the fun doesn't stop here kids! Try the following:<br /><br /><pre class="code"><br />a = ({1:2}, 2)<br />a[0][2] = 3<br /></pre><br /><br />This little piece of code gets executed without any problems, both in 2.5 and in Py3k. I wonder what is the rationale behind the fact that if we try to change a list (in place) that is part of a tuple, we get an error, while changing a dict happens silently. Ok, I do admit that doing<br /><br /><pre class="code"><br />a = ([1], 2)<br />a[0].append(2)<br /></pre><br /><br />also doesn't throw an error, but this only confuses me more. So, I'd like to know---are tuples mutable or immutable? Or better yet, how mutable are immutables?<br />As a final thought, consider the following code:<br /><br /><pre class="code"><br />a = ({1:2}, 2)<br />a[0][2] = a<br />print a<br /></pre><br /><br />In 2.5, this code will execute with no problems, and printing a will yield in<br />({1: 2, 2: ({...}, 2)}, 2)<br />thereby showing us that we have an infinite dictionary on our hands (or that's what I think these three dots stand for). However, doing the same thing in Py3k (i.e., running the above code and trying to print a---remember that in Py3k print is a function), we get<br /><pre class="code"><br />Traceback (most recent call last):<br /> File "<stdin>", line 1, in <module><br />TypeError: __repr__ returned non-string (type bytes)<br /></pre><br />Whoa! WTF?? What does this mean? What ever it means it tells me nothing about what is actually happening here. As far as I can see, there is something wrong with the __repr__ function---it isn't cut out to handle this kind of objects. So, the first thing that came to my mind was "Hey, cool, they won't print infinite dictionaries any more. They'll allow their construction, but when I try to print them, I'll get an error (a weird one, to say the least, but an error still). This kinda suxx, but ok, I guess I'll have to live with it." No. I was wrong. Try<br /><pre class="code"><br />print(a[0])<br /></pre><br />The infinite dict in the first position of the tuple prints with no problems. So, why is it a problem to print the tuple (which basically has one number, two parenthesis, and one comma extra)? No idea. Of course, trying to print a[0][3] yields an error, print a[0][3][0] goes without a problem, and so on. I guess that's why they call it alpha version :) I sincerely hope that this won't be a part of the language when a stable release of Py3k is out.<br /><br /><br /><script>reddit_url='http://filoxus.blogspot.com/2007/12/python-3000-how-mutable-is-immutable.html#links'</script><br /><script>reddit_title='Python 3000 - how mutable is immutable?'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com4tag:blogger.com,1999:blog-8688058551295946514.post-8290602200664180482007-12-15T10:11:00.000-08:002007-12-15T10:16:33.595-08:00OMG my iPod likes Barbara StreisandA few days ago, I updated my iPod with some new songs (I was getting kinda bored listening the same ones for about two years now). I put most of these new songs in my 80's playlist (about a dozen new songs altogether). So, the last few days I had my iPod on every time I went home from work and noticed one strange thing - every time I listened to it, one of the songs was "The way we were" by Barbara Streisand (yes, I occasionly listen to Barbara, so what of it?). What was strange about it is that the shuffle option was always on and that there were 95 other songs in the playlist. To give the exact numbers, I heard Barbara four out of four days in a row, where I listened about 15 songs each day (the commute home takes about 45 minutes - that's 3 minutes per song), and, as I said, there were 96 songs in total to choose from.<br />This somehow seemed odd to me, not to say wierd. That's why I decided to see just what is the probability to observe this strange behaviour. So, back to basic statistics. The probability that any song will be selected from a playlist of 96 songs is 1/96. However, as one song finishes, the next one is selected from the remaining 95 songs (I assume that this is how the iPod's shuffle works - it shouldn't select a song that was already played until all others are played). So the probability that any song will be selected if two songs are played is 1/96 + 1/95. You get the picture how the story goes from here... So, the probability that you heard a song after playing 15 of them (in a playlist of 96 songs) equals to 1/96 + ... + 1/82 = 0.1689, that is, around 17%. This means that I have a 17% chance to hear "The way we were" on one of my trips home. However, I heard it four days in a row. Since hearing a particular song on one of my trips can be regarded as an experiment with two outcomes (I either heard it or not), and since I start the new shuffle every day (meaning that each experiment is independent of the previous one), we can say that we are talking about a binomial distribution. In such a setting, we can easily compute the probability that I hear Barbara four days in a row - it's simply 0.1689^4 = 8.145 * 10^-4. That is, there is an 8 in 10000 chance that I will hear Barbara four days in a row if I put my iPod on shuffle. Yet, there it was, I heard it. Four days in a row. As my knowledge of statistics ends here, I will not try to do a hypotesis test or anything like that. If there is someone out there that is willing and capable to do this, I would love to hear from him/her.<br />All this lead me to do some research online. Well, it turns out I'm not the only one having this shuffle issue. Many users have already complained how the "random" playing isn't random at all and how their iPods have souls. Here's just two references - <a href="http://forums.ilounge.com/archive/index.php/t-4575.html">http://forums.ilounge.com/archive/index.php/t-4575.html</a> <a href="http://www.blackwell-synergy.com/doi/pdf/10.1111/j.1540-4609.2007.00132.x?cookieSet=1">http://www.blackwell-synergy.com/doi/pdf/10.1111/j.1540-4609.2007.00132.x?cookieSet=1</a>. The second one is a really interesting paper with some more references to people with the same problem. So, it turns out that the machines have already started developing their own "mind". Yes, our machine overlords are taking over the world, and they have started with picking their favourite music. I'm just glad to know my iPod loves Barbara Streisand - I think it'll make an ok master.<br /><br /><br /><script>reddit_url='http://filoxus.blogspot.com/2007/12/omg-my-ipod-likes-barbara-streisand.html#links'</script><br /><script>reddit_title='OMG my iPod likes Barbara Streisand'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com0tag:blogger.com,1999:blog-8688058551295946514.post-40320409102353815972007-12-08T08:55:00.000-08:002007-12-15T03:30:44.658-08:00Running fast code in PythonAs I have suspected, the numbers I got for execution time of the Haskell fibonacci program (look at my previous posts) are wrong, in the sense that I compiled the program without optimisation, as most normal users will do in real applications. That is why I feel that I should redeem myself and give the numbers for the program with optimisation turned on. So, when compiling the program using<br /><br />ghc -O2 fib.hs -o fib -no-recomp (thanks to Don Stewart)<br /><br />execution time is 0.64s (mean time of five runs). This time is much closer to the one reported on the Haskell blog (0.48s). Ok, so my hat goes off to Haskell---it really is faster than Python some 50 times.<br />But, being a Python fan, I couldn't help but feel that something is wrong here. Is Haskell really that good? It's fast, easy to parallelize, pure functional and it probably cooks for you after a hard day at work. So, why should I use Python? <br />Well, the first thing that comes to my mind is the fact that ghc generated a 925kB exe file from a 1kB file that computes fibonacci numbers. In comparison, gcc (3.4.2) generated 16 kB for a fibonacci program that ran 0.75s (compiled using -O2). While it is still impresive that Haskell was faster than C, I cannot help but think that it is just too much to have an exe file almost 1000 times larger than the source. While this is certainly a drawback of Haskell, it still wasn't enough to return my faith in Python. Then, another thing came to my mind---if Haskell can use optimisations, why can't Python? I mean, if I used -O2 in ghc, why can't I do somehting similar in Python?<br />Of course, running Python with -OO gave no real speedup---it shook off about one second in execution time. Instead of 26s, I got 25s for the non-parallel case, and instead of 18s, I got 17s for the non-parallel case. But, the great thing about Python are its packages. I remember a friend of mine telling me about a thing called Psyco. It was some kind of a package that optimised execution of Python code, so I decided to give it a shot. After downloading and installing Psyco (<a href="http://psyco.sourceforge.net/">http://psyco.sourceforge.net/</a>), I just added two lines to my source (I give the full source here):<br /><br /><pre class=code><br />from time import time<br />import psyco<br />psyco.full()<br /><br />def fib(n):<br /> if n == 0 or n == 1:<br /> return n<br /> else:<br /> return fib(n-1) + fib(n-2)<br /><br />if __name__ == "__main__":<br /> start_time = time()<br /> for i in range(36):<br /> print "n=%d => %d" % (i, fib(i))<br /> <br /> print "Time elapsed: %f" % (time() - start_time)<br /></pre><br /><br />So, the lines I added were <span style="font-weight:bold;">import psyco</span>, and <span style="font-weight:bold;">psyco.full()</span>. I ran this thing and voila---the time of execution was 1.93s (mean time of five runs). Wow! That was all I could think. Remember, this was the non-parallel case, so it went from 25 seconds to 1.9s. Or, in terms of Haskell speed, Haskell now beat Python by on 3x. And with no extra effort (ok, adding two lines to the source) and no parallelization.<br />Unfortunately, psyco didn't help the parallel version---it still took 18s. But still, I think this is pretty amazing and it definitely brought back faith in Python. So, it's three times slower, but the synatx is much cleaner and it doesn't generate a thousand times bigger executable. I'm sticking to Python :)<br />(Note: some parts of this post are biased and down right offensive to Haskell fans (just kidding about the last one:) ). My intention is not to start a flame war or anything like that, I just felt that someone should stand up and take Python's side:). Seriously though, if you have any objections on my methodology, I would like to hear from you.)<br /><br /><script>reddit_url='http://filoxus.blogspot.com/2007/12/running-fast-code-in-python.html#links'</script><br /><script>reddit_title='Running fast code in Python'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com8tag:blogger.com,1999:blog-8688058551295946514.post-8493247003356948692007-12-05T06:04:00.000-08:002007-12-14T09:03:04.502-08:00Even xkcd is leaving Perl :)For those of you out there that don't read xkcd:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://imgs.xkcd.com/comics/python.png"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;width: 400px;" src="http://imgs.xkcd.com/comics/python.png" border="0" alt="" /></a><br clear=left><br /><br /><nobr><br /><script>reddit_url='http://filoxus.blogspot.com'</script><br /><script>reddit_title='Even xkcd is leaving Perl'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script></nobr>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com0tag:blogger.com,1999:blog-8688058551295946514.post-49455059752670409622007-12-03T06:13:00.000-08:002007-12-03T06:48:41.341-08:00Holy shmoly Haskell doesn't smoke Python away (that much)Recently, there has been some talk about how <a href="http://cgi.cse.unsw.edu.au/~dons/blog/2007/11/29#smoking">Haskell smokes Python and Ruby away</a> in the naive fibonacci algorithm. While it is obvious that Haskell will beat both due to the code being compiled and not interpreted, I decided to see if I got the same results as reported on the Haskell hacking blog, and also see how Python handles parallel processing. Note that I will refer to the Haskell hacking blog simply as Haskell blog from now on.<br /><br />Since I have no experience in Ruby, I will only compare Python and Haskell. So, here is the code I used to test the two.<br />Python:<br /><br /><pre class=code><br />from time import time<br /><br />def fib(n):<br /> if n == 0 or n == 1:<br /> return n<br /> else:<br /> return fib(n-1) + fib(n-2)<br /> <br />if __name__ == "__main__":<br /> <br /> start_time = time()<br /> for i in range(36):<br /> print "n=%d => %d" % (i, fib(i))<br /> <br /> print "Time elapsed: %f" % (time() - start_time)<br /></pre><br /> <br />So, as we can see, this code is pretty much the same as that reported in the blog about Haskell.<br /><br /><pre class=code><br />Haskell:<br /><br />import Control.Monad<br />import Text.Printf<br /><br />fib :: Int -> Int<br />fib 0 = 0<br />fib 1 = 1<br />fib n = fib (n-1) + fib (n-2)<br />main = forM_ [0..35] $ \i -><br /> printf "n=%d => %d\n" i (fib i)<br /></pre><br /><br />This is also the same as in the blog, the only difference being that I import what is needed to make it run (for some reason, the import statements aren't included in the Haskell blog). Since I have no idea how to measure time in Haskell (any hint on how to do that is appreciated), I used the ptime command for Windows (available at <a href="http://www.pc-tools.net/win32/ptime/">http://www.pc-tools.net/win32/ptime/</a>).<br /><br />So, here are the results of the test:<br /><br />Python 2.5: 26.07 s<br />Haskell (GHC 6.8.1): 3.873 s (mean of 5 runs)<br /><br />So, the results for Python are pretty much the same as reported in the Haskell blog (25.16 s is reported there), but the ones for Haskell are not, in fact the Haskell blog reports 0.48 s, which is some four times faster than I got on my computer. What is the reason for this? I have no idea. I compiled the program using "ghc fib.hs", maybe they used some code optimizations? Anyway, the Haskell advantage seems to be melting, although Haskell is still some 6-7 times faster. However, the Haskell blog proceeds to use parallel processing in Haskell, but uses no such thing in Python. So, I decided to see if Python can take advantage of my two cores to shake some time off this 26 seconds. I used the Parallel Python package (available at <a href="http://www.parallelpython.com/">http://www.parallelpython.com/</a>) to make use of my two cores. Here is the modified Python program:<br /><br /><pre class=code><br />from time import time<br />import pp<br /><br />def fib(n):<br /> if n == 0 or n == 1:<br /> return n<br /> else:<br /> return fib(n-1) + fib(n-2)<br /><br /><br />if __name__ == "__main__":<br /> start_time = time()<br /> ppservers = ()<br /> job_server = pp.Server(ppservers=ppservers)<br /> print "Starting pp with", job_server.get_ncpus(), "workers"<br /> jobs = [(input, job_server.submit(fib, (input,), (), ())) for input in range(36)]<br /> for input, job in jobs:<br /> print "n=%d => %d" % (input, job())<br /> <br /> print "Time elapsed: %f" % (time() - start_time)<br /></pre><br /><br />The result: 17.86 s<br />So, Python went from 26.07s to 17.86s, which means it ran 31.5% faster than on only one core. Unfortunately, I had no success in running the parallel Haskell code reported on the Haskell blog (for some reason, I always got "`pseq` not in scope" error while compiling). However, I can use the numbers given on the Haskell blog, where Haskell went from 0.48s to 0.42s on two cores. This means a 12.5% reduction in execution time. Although I admit that is takes slightly more intervention from the programmer to make the parallelism in Python work, the gain in speed is also more significant. As a final note, let us compare 3.873s (Haskell) and 17.86s (parallel Python) - the ratio is around 4.6, let's say 5. So, Haskell doesn't smoke Python away THAT much.<br /><br /><script>reddit_url='[URL]'</script><br /><script>reddit_title='[TITLE]'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com8tag:blogger.com,1999:blog-8688058551295946514.post-74510271955219718952007-11-20T03:15:00.000-08:002007-12-10T03:16:51.969-08:00Trie in PythonHere is my humble implementation of a trie in Python, just something to get me started.<br />Any comments on improving the code are welcome<br /><br /><pre lang="python"><br />import re, types<br />from collections import defaultdict<br /><br />def tokenize(s, removePunctuation = True):<br /> if removePunctuation:<br /> p = re.compile('[\.,!?()\[\]{}:;"\'<>/ \n\r\t]')<br /> return [el for el in p.split(s) if el]<br /> else:<br /> t = []<br /> s = re.split('\s', s)<br /> p = re.compile('(\W)')<br /> for phrase in s:<br /> words = p.split(phrase)<br /> for word in words:<br /> t.append(word)<br /> return [el for el in t if el]<br /><br />class Node(object):<br /> <br /> def __init__(self):<br /> self.freq = 0<br /> self.next = defaultdict(int)<br /><br />class Trie(object):<br /> <br /> def __init__(self, inFile):<br /><br /> self.nodes = []<br /> self.nodes.append(Node())<br /> self.n = 1<br /> self.numberOfWords = 0<br /><br /> for line in file(inFile):<br /> words = tokenize(line)<br /> for w in words:<br /> currNode = 0<br /> for char in w:<br /> if self.nodes[currNode].next[char] == 0:<br /> self.nodes.append(Node())<br /> self.nodes[currNode].next[char] = self.n<br /> currNode = self.n<br /> self.n += 1<br /> else:<br /> currNode = self.nodes[currNode].next[char]<br /> self.nodes[currNode].freq += 1<br /> self.numberOfWords = len([node for node in self.nodes if node.freq != 0])<br /><br /> def __getitem__(self, word):<br /> """Return the frequency of the given word."""<br /><br /> if isinstance(word, types.StringTypes):<br /> currNode = 0<br /> for char in word:<br /> if self.nodes[currNode].next[char] == 0:<br /> raise AttributeError("No such word: %s" % word)<br /> else:<br /> currNode = self.nodes[currNode].next[char]<br /> return self.nodes[currNode].freq<br /> else:<br /> raise TypeError()<br /><br /> def __len__(self):<br /> """Return the number of nodes in the trie."""<br /><br /> return self.n<br /></pre><br /><br /><p><br /><script>reddit_url='[URL]'</script><br /><script>reddit_title='[TITLE]'</script><br /><script language="javascript" src="http://reddit.com/button.js?t=3"></script><br /><br /><script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script><br /></p>Filoxhttp://www.blogger.com/profile/07052058313681392060noreply@blogger.com2