Tag: Tips

Executing a lot of SQL queries with Django’s db API? Watch out for one thing.

I recently had the need to write an indexer that processes a lot of text files and creates a huge number of database rows (19 million+). This is a background script for an internal django website so I ended up using the standard django psycopg2 library, but tried to minimize the overhead by executing queries directly using the db cursor. Even after this consideration and special ‘del’ statements to clean things out after regular intervals, to my surprise, the process was still taking up a lot of memory. So, I tweeted about this experience. Daniele Varrazzo, one of the authors of psycopg2, tweeted back with more info.

Read full post...

Boot Camp Windows running slow in VMware Fusion?

VMware Fusion Logo

Lately my Boot Camp based Windows instance had slowed down to a crawl. Booting it would take a few minutes and everything after I had logged in was just as painstakingly slow. So, after a few minutes of searching I ended up on this suggestion. Guess what? That brought things back to life. Boot up time went back to the more expected quicker time and programs started responding.

So, delete the VMware Fusion folder in ~/Library/Application Support/ while VMware is not running, restart VMware, let it create Boot Camp related setup and boot windows. You are good to go!

Cleaning up XML using VIM

Here is a quick function for cleaning up XML without any line breaks. This is a quick and dirty solution with some minor issues (e.g. turning <test></test> to <test>\n</test>), but the goal of this it not to be too accurate, but to quickly put a non-readable XML into a readable form for reference.

function! PrettifyXML()
	set ft=xml
	:norm =G

Put this in your vimrc file and call it using :call PrettifyXML()

Quick vim tip: Copying matching lines to the end of file

I recently had the need to cleanup a long diff file and move certain lines to the end of the file for further analysis. The diff file that I got was something like the following, only much longer (produced using diff -qr xxx yyy).

Files xxx/abc1 and yyy/abc1 differ
Only in xxx: cde1
Files xxx/abc3 and yyy/abc3 differ
Files xxx/abc4 and yyy/abc4 differ
Only in xxx: cde2
Only in xxx: cde5
Files xxx/abc5 and yyy/abc5 differ
Only in xxx: cde3

I didn’t care as much about the “Only…” lines, but the files that differed needed more attention. To accomplish this I wanted to get this file in the following format.

Only in xxx: cde1
Only in xxx: cde2
Only in xxx: cde5
Only in xxx: cde3
Files xxx/abc1 and yyy/abc1 differ
Files xxx/abc3 and yyy/abc3 differ
Files xxx/abc4 and yyy/abc4 differ
Files xxx/abc5 and yyy/abc5 differ

I could go in and copy and paste all the lines that matched, but that would be a lot of manual work. So, I looked around for a minute and came up with the following command.


Vim is awesome!

New Songbird Extension: Delete from Disk

One thing I have missed in OS X is a powerful media player like MediaMonkey. iTunes gets a good part of the job done, but doesn’t have everything that I need. Plus, I hate the fact that it doesn’t display the ratings that I have on the files from my Windows machine. So, I have been exploring my options and one possibility is Songbird. It it better, but still lacks in some places (once again, ratings). I am still playing around with Songbird because, compared to iTunes, it is a lot easier to create add-ins for Songbird.

The first thing that I noticed missing from Songbird was the ability to delete songs that are not “managed” by Songbird. This is pretty much how iTunes behaves as well. I did not like this because I like control. If I added a file to a playlist then I would also like to have the ability to delete it from both the playlist and the file system.

So, instead of using the prescribed “1-start rating” to denote deletions I decided to explore Songbird extension development. I am glad to report that I was able to figure out the framework and actually just published an extension in the official repository to do just that: Delete from Disk. If you are a Songbird user looking for this functionality then you don’t need to look any further. Browse to the extensions page in Songbird and search for Delete from Disk.

This is the new menu entry that the extension adds.

Now let’s figure out what we can do about the ratings…! I also wish that someone will fix the Apple media key support extension.

Low level hotkeys in MediaMonkey

MediaMonkey Logo

MediaMonkey (MM) is my music player of choice on Windows. I have a few global hotkeys setup (for instance WIN+ALT+B for next track), but after an upgrade a few months ago (probably starting with 3.0) I noticed that my shortcuts started sending the shortcut keys to the active applications. So, for instance, while in Outlook if I press WIN+ALT+B, MM will go to the next track, but in addition Outlook will switch views because ALT+B is a shortcut in Outlook.

After a quick post online I found the solution that I was looking for. Basically, I had to edit the MediaMonkey.ini and add “PreferLLKeysHook=1” to the “[options]” section. MediaMonkey.ini will be located in different places based on the operating system, you can find the details in this knowledge base article. Add “PreferLLKeysHook=1” to the “[options]” section in there and restart MM. From there on the shortcut keys will only be interpreted by MM and will not be forwarded to the active application.

Speeding up django’s development server on Windows

I am very picky about my development environment and I need it to be just right, otherwise the fun part of programming disappears. I have a dedicated Linux server in my office that is sharing and serving the development files. This is a solid server, fast enough that django’s development server refreshes as soon as I save the files, even before I have switched to my browser; and that’s how I like it! 🙂 Lately I have been on the road quite a bit so I have had to run the development environment on my tablet (when in Windows 7; runs excellent in Linux). The tablet has OK specs: 1.4GHz Core Duo with 2GB RAM and a 7200 RPM drive (generally the bottleneck). But for some reason django’s development server seems especially slow at serving the files. The refreshes after changes are OK, not fast, but OK. It is the media that it is very slow at serving (understandably so).

I did a lot of research on my options to speed this up. I am using the standard CPython distribution on Windows. I saw a lot of references to unladden-swallow, but there weren’t a lot of benchmarks to prove the speed gain yet. I realize that this is still under very heavy development, but the one benchmark that I found really excited me so I decided to give it a try. Unfortunately, after hunting for a number of source code packages necessary for compilation and still not succeeding I concluded that it wasn’t worth the time yet 😐 I decided to rule out pypy because of the possibility of compatibility issues, I wanted something that I could plug into the existing system. For some of my projects I am using external libraries, which might not work with pypy.

Anyways, my solution ended up involving Apache. Based on the console output of django’s dev server I had an idea that it was slow at serving multiple files. So I decided to serve the media, which generally is the majority of the files in a given view, using Apache and let django’s server deal only with the views. Microsoft’s IIS is also an option, but I had Apache setup for another project so I decided to use that. Below is a part of my dev_settings.py that makes this change.

import socket
# ...

# System specific dev settings.
if socket.gethostname() == "mystic":
	MEDIA_URL = 'https://thebitguru.com/projectname_media/'

With this new combination and using instead of localhost now my dev environment on Windows is fast enough to keep things interesting.

Windows not going to sleep?

Recently my laptop stopped going into sleep mode, which it used to do without any hesitation. I restarted because many times restarting takes care of minor issues, but that didn’t do it. Next try to get it to sleep resulted in the same behavior, the display would turn off, but the system would stay on. A quick look at the event viewer didn’t reveal any issues. A quick search online didn’t reveal much either. So, I was back on my own to solve this issue.

Read full post...

What extra steps do you take to increase your chance of getting an interview?

In the past whenever applying to a job I have tried to go a step beyond what the employer might have requested. In a few cases the requirement was C++, so what better way to demonstrate your skills than writing a C++ program? After some thinking I decided that a C++ based resume would be a good sample. So, I ended up writing a program that renders my resume almost exactly as it would have been rendered by Microsoft Word. Instead of hard coding the complete resume I wrote a text rendering engine that reads an array of string and renders it on screen based on some simple rules.

To make it simple for the hiring manager to run this program I embedded the text in a string array so they can just pass the exe around without having to worry about sending multiple files. The program works just as well reading from a text file. Here is a quick screenshot of this program.

This program renders my resume just as Microsoft Word would have rendered it.

So, what extra steps have you taken to increase your chance of getting an interview?

Cannot unlock iPhone: baseband 05.11.07, bootloader >05.09 (e.g. 06.04)

2009-11-03, update: GeoHot released the unlock for baseband 05.11.07! Check it out.

Recently a friend purchased an iPhone off of craigslist that he wanted to unlock for use with a different carrier. The phone was running iPhone OS 3.1, which the dev team has released a jailbreak for. Unfortunately, what he didn’t know was that the boot loader (or bootloader) version mattered a lot 🙂

He wanted a good, clean unit, and this was a newly refurbished one, jailbroken, very clean, no scratches, and 0 minute usage. He thought all he had to do was install ultrasn0w and he would be good to go. Well, he tried, but no luck. After a few tries he brought it to me and I started my research. Apparently, because it is a newly refurbished unit, it had the latest version of everything: iPhone 3.1 (seller might have upgraded it), latest baseband, and latest boot loader. Here is what my research turned up.

  1. Baseband 05.11.07 cannot be unlocked at the time of this post (October 2nd, 2009).
  2. You can downgrade baseband 05.11.07, but only if the boot loader is version 05.08 or older. In this case the easiest solution is to use fuzzyband downgrader.
  3. If you have baseband 05.11.07 and a boot loader newer than 05.08 then, unfortunately, you are out of luck because you cannot unlock your iPhone.

If you are buying an iPhone for use with a carrier other than AT&T then make sure that your device can be unlocked. Also, be careful once you do get the new, compatible iPhone because upgrading to iPhone OS 3.1 using the official ipsw will upgrade the baseband! and possibly the boot loader; you will be stuck without an unlock! (Hopefully you are not reading this post after the fact :|)

The easiest way to check the boot loader and baseband versions is to install Fuzzyband using Cydia. Do this before the purchase. You want a boot loader that is 05.08 or older, and a baseband that is 05.11.07 or older (if it is 05.11.07 then you will have to downgrade to an older version, which is possible assuming that the boot loader version is correct).

What if you have an “un-unlockable” iPhone?

Basically, you need an AT&T subscriber 🙂 Here are a few (realistic!) possibilities.

  1. The cleanest option is to sell it to someone who is an AT&T subscriber. There are AT&T subscribers looking for iPhones. My guess is that it is because they don’t want to extend their contract.
  2. A rare case would be an AT&T subscriber with an unlockable iPhone looking for an iPhone for a family member who is not on AT&T. In this case they can unlock their phone and give it to the family member, and use the new device themselves (on AT&T).
  3. Swap it with an AT&T subscriber who is using an unlockable iPhone. You will have to give them some incentive to do this though, e.g. if your iPhone is newer, cleaner than theirs, or maybe offer coffee, lunch, etc. 🙂