Recently in tech Category

I now have some working code for my dirhash dynamic memory allocation Google Summer of Code project. It is currently being reviewed and needs further testing, but if you're feeling brave you can grab a patch against HEAD from my project page on the FreeBSD wiki.

What the new code does is add an event handler for vm_lowmem events. When one of these events happens, the kernel will try to delete dirhashes that have not been used for DH_RECLAIMAGE seconds. This is currently set to five seconds, but I have no idea if this will be an optimal time. A bit of benchmarking and testing will be necessary to sort that out. If all dirhashes have been used more recently than DH_RECLAIMAGE, instead the first unlocked dirhash on the TAILQ list used to keep track of them all will be deleted. Due to how the existing dirhash code organizes this list based on a scoring system, the first hash on the list should be a mix of the the least recently used and least frequently used.

If my code turns out to work as it should, then the default maximum memory for dirhash can be safely increased quite a bit. If the system has plenty of spare memory, then lots will be available for dirhashes. Then if the system becomes low on memory, some of this dirhash memory will automatically be freed up for use by whatever else needs it. Hopefully we'll see some performance improvements overall on systems with lots of memory that need to deal with large directories, but I haven't done the benchmarks yet.

The Summer of Code officially started a couple of weeks ago, so I've been getting started on my project. I've not written any actual code yet, but have been busy with other aspects of the project. First of all I have been reading a bit of documentation and code relevant to dirhash itself and FreeBSD memory management. I have a lot to learn, especially about the memory stuff. The two approaches I am looking at, mentioned in the USENIX paper by Ian Dowse and David Malone (my SoC mentor), are to either use the buffer cache for dirhash's memory so the VM can take care of providing memory as it is available, or to use a method for allocating memory where the kernel can signal for some to be freed in low memory situations, perhaps using the slab allocator.

I have also been learning a little bit about Perforce, the version control system used by FreeBSD for experimental projects. I have not used Perforce before this, and it is a bit different from the VCSs I've used before (CVS and Subversion). Also, I have done a bit of experimentation with the current dirhash implementation just to be sure that it can free memory it no longer needs. This was just a simple test where I created a directory with a couple of million entries, checked the vfs.ufs.dirhash_mem sysctl, removed the directory, then checked vfs.ufs.dirhash_mem again to make sure the memory used had decreased. It had. Finally, I sent off a brief proposal to do a talk about this project at EuroBSDCon 2008. Obviously I don't have much to create a talk with yet, but I should by the end of the summer. Plus, the second day of EuroBSDCon this year happens to fall on my birthday, and I think hanging out with fellow BSD nerds in France could be a fun birthday activity :)

My Google Summer of Code Project has been approved! I am excited that I will be supported by Google to work on FreeBSD. It should be a fun time.

I've reworked my site once again. This time I'm using Movable Type to power everything. I've imported old entries I used to have on a now-defunct Typo powered blog, as well as more recent posts from my LiveJournal.

I found this cool Berlin subway map for iPhone/iPod Touch that Randy Reddig made. It was inspired by an NYC subway map by Khoi Vinh, and in turn inspired me to make one for SL's tunnelbana (metro/subway), lokalbanor (local train), and pendeltåg (commuter train) map of Stockholm. Download it here, unzip, and take a look at the readme.txt file to see how to put it on your iPod or iPhone.

I'd like to make one of these for SL's bus system too, but have not yet found a suitable map to chop up. The best Stockholm city bus map I've found doesn't have night busses on it. Most of the time when I'm taking the bus it's only because the t-bana is closed for the night, so having the night bus routes would be really useful!

The other day I decided to finally try and figure out why it is that Safari and other browsers do not offer to remember passwords for me on certain web sites. I regularly encounter this problem on some banking web sites, and the Yahoo login page for Flickr. Looks like the common way of keeping browsers from remembering passwords is the non-standard autocomplete HTML form attribute (assigned a value of "off"). This originally was a feature added to Microsoft Internet Explorer, but Firefox and Safari support it now, along with probably lots of other browsers.

The idea is presumably to keep users from storing passwords in an insecure manner on their computers. However, I use the built in keychain on Mac OS X, so it stores these remembered passwords in a fairly secure manner. Firefox's password manager also can encrypt stored passwords with a master passphrase if you tell it to. I don't memorize most of my passwords, so whenever I log in to a site that won't let me store a password in my keychain I have to go and decrypt a GnuPG encrypted text file I store all my passwords in. This essentially means I do the same thing as using the keychain would, but it requires doing pointless extra stupid work.

I should get to decide what my computer will do. I don't need web sites telling me how to secure things. This kind of behavior encourages users to choose crap passwords just so they can remember them, or re-use the same password for many sites/accounts. Both are, of course, bad for security.

Does anyone have any suggestions on how I can make Safari ignore the autocomplete form attribute? So far the only solutions I've seen for working around this involve patching WebCore, which seems scary and like total overkill. I don't want to use some other password manager either--I'd rather use the one built in to the OS. I tried using Privoxy with some filter rules, but gave up because it was taking too much effort and time to configure. I'm considering switching back to Firefox. I think it at has some extensions that will make it ignore this damn form attribute, or at least GreaseMonkey can do it for me. But it would be cool if I could just make Safari ignore this autocomplete attribute.

Well, I've got a new problem now. My math fonts for LaTeX are all kinds of screwed up, as described here and many other places. Sometimes if I view a PDF with LaTeX generated equations in it in Preview everything looks fine, but if I just change focus away from Preview and go back it could get screwed up. Here's a video of what happens:

Googling told me I should try wiping out ATS databases with the following commands:

atsutil databases -removeUser
sudo atsutil databases -remove

(Gotta log out after running just the removeUser option, and reboot if you do the full on remove as root). This seems to provide temporary reprieve, but I start experiencing font breakage again after not too long of messing around with generating PDFs from LaTeX.

So annoying! I really hope this is just a display bug and my PDFs are actually fine when I email them to people or print them. It would be nice if this somehow gets fixed soon.

Update: I reinstalled Leopard, creating a new home directory and everything from scratch, and this problem magically disappeared. Go figure.

I bought the newest release of Mac OS X a little while back and installed it on my Macbook. I have mixed feelings about it.

First some things I like. Overall the way it looks is nice. A lot of people hate the new dock and transparent menubar, but they don't really bug me. I really like the rounded corners on contextual menus. All apps looking the same is awesome too. No more mix of brushed metal and pinstripes.

Spaces is awesome. I didn't really expect I'd like it as much as I did. I have not really used virtual desktops much in recent years of using X11 on unix systems, so I didn't think I'd be as stoked about Spaces as I am.

New iChat is cool. Lots of little things have been added that I was like "oh man I wanted this but didn't even really know it." Actually that can be said about a number of other applications too. iCal, Preview, Terminal, and others have a lot of these little changes that overall have a big improving effect.

The improvements to the Finder are also quite excellent. I haven't used Cover Flow much but it's cool. I really like Quick Look. The auto-discovery of network shares in the sidebar is handy, as are smart folders.

Now some things that make me both happy and disappointed. One of the biggest things I was looking forward to were improvements to Mail. Having RSS feeds in my mail reader makes me happy. Even better are how they sync up the read status with RSS feeds you have in Safari. Interestingly, I'd never used Safari for RSS feeds until now. I like that I can seamlessly switch between Mail and Safari for reading feeds now, depending on which one I'm in the mood to read with. I was a little bummed that my feeds all showed up as plain text in Mail, though, because I'd previously used defaults write com.apple.mail PreferPlainText -bool TRUE to force my email to always be plain text if possible (I do not like HTML email). I had to unset this in order to have HTML in my RSS feeds, which makes them look much nicer, but now I get HTML email too. It would be cool if we could get a separate PreferPlainTextMail and PreferPlainTextFeeds or something. It was also annoying how Safari and Mail can't import feeds from an OPML file.

Notes and to dos in Mail are cool, and I was relieved that you can change the default font away from being Marker Felt in the Mail preferences. I'm glad that to dos are shared between Mail and iCal, because sometimes I'd rather edit them in iCal yet still want them synched with my IMAP server by Mail. However, I was annoyed to discover just today that to dos I have synched with IMAP do not get synched to my iPod! You can apparently only sync to dos on local calendars with iPods. This is less annoying to me than it would be if I regularly used more than one Mac, but since I don't really need to use IMAP to synch to dos for now I just moved them all back to a local calendar, fixing iPod synching. I wish notes would sync to my iPod though, rather than me having to continue to manually drag files into its Notes folder.

Time Machine was another feature I was greatly looking forward to. I'm really glad to have backups I don't have to think about now. But, I was disappointed that you cannot choose to only have it back up, say, your home directory. I thought this would be possible. You can specifically exclude folders to back up, thankfully, but I really only care about backing up my own personal files. I try to keep everything I can in my home directory. Oh well, I have an external hard drive that is big enough that I can afford to back up pretty much the whole disk. I also wish Time Machine made block differential snapshots rather than file based, so you wouldn't need a fresh copy of a huge file every time just a few bits were changed, but I knew I wasn't going to get this.

The worst thing, though, is that Time Machine and FileVault don't play well together. Since this is a laptop, I have been using FileVault to keep my personal files encrypted in case someone jacks my computer. I'm somewhat paranoid. But if you use FileVault with Time Machine, it will back up your home directory as just the single encrypted sparse dmg that FileVault uses as its backing store. And it says it only backs it up when you're logged out. Presumably every time you change a file in your home directory, the whole huge dmg will get backed up anew. What I was hoping for was that I could have Time Machine back up only my home directory, and then do so only when I was logged in and the files were decrypted (that's the only time files are going to change in it anyway). This way I'd get nice incremental snapshot backups as usual, but could still keep the files all encrypted on my laptop's internal disk. Boo. For now I've just turned FileVault off because having easy backups with Time Machine is more important than my paranoia.

Now for things that totally suck. I did a fresh clean install, wiping my disk and just copying back my old home directory, because I wanted to clear off old crap. Despite this, I'm seeing more instability than I used to. Applications seem to crash more often, especially Mail.app. I also have been annoyed to find that with Python 2.4.4 installed from MacPorts I can't use pdb (Python debugger). Every time I just do pdb.set_trace(), python immediately crashes. This is likely not Apple's fault and the OS provided Python works fine, but I use MacPorts to easily install some other Python modules that Apple doesn't ship so it is annoying. Then, earlier tonight my Macbook just rebooted itself randomly. This might be a hardware problem (which of course sucks, but isn't Leopard's fault), but if its the OS that is real bad.

Then there's the new firewall. Many people have talked about how much it sucks. I agree. The idea of being able to allow incoming network access to only use specified applications isn't bad, but having that be the only control is lame. Having signed apps or those running as root be automatically trusted so the firewall opens up for them is also shitty. I think these kinds of things should be allowed for users to set their firewall policy with, but I want to be able to filter explicitly by port too. I want to be able to filter out traffic that is trying to connect to local services running as root. And I want to be able to do this without having to use ipfw from the command line. I could do these things in Tiger, so why remove the functionality? It would be better if Apple had let you switch to the old way of configuring the packet filter if desired, or at least given some more options for tweaking the new one. Even better than that is if they'd made a firewall that you could set up to notify you any time an application (regardless of its signature or uid) that hadn't been allowed to do so before tried to start listening on a local socket, and ask the user if he or she wanted to open a hole in the firewall for that app on that port. Outgoing filtering would be incredibly useful too. I'd love if the OS supported asking your permission any time an application wanted to connect to some remote socket you hadn't already explicitly allowed, like Little Snitch does.

I'm altogether not too impressed with security features in Leopard. Aside from the firewall, I guess they're better than Tiger, but I wish Apple had made bigger improvements. These posts and others about Leopard security features seem to show that Apple had some good ideas for security enhancements but didn't really take them too far. The application signing stuff is pretty lame since all it apparently does is warn you if an app that was signed and has been modified wants to get to your keychain or poke a hole in the ineffectual firewall. The sandbox business looks like it has a ton of potential, but I guess nothing actually uses it yet and it's totally undocumented. I'm not surprised that the security features aren't dazzling though. Software security is something that always bums me out when I think about it.

All this being said, overall I'm pretty happy with the upgrade. I hope some of the nastier problems I've encountered get ironed out soon in software updates.

Even if you don't care about math, check out this pretty graph I made. I made the surface plot with Matlab, saved as an eps (encapsulated Postscript) file, then converted to a pdf with Preview.app on Mac OS X. In Matlab the figures never look this good, but when viewing it in Preview there's nice anti-aliasing and some awesome alpha transparency! Well, maybe it's only cool if you're a total nerd.

For those curious about what this is: I just did this homework assignment where I'm solving a simple parabolic partial differential equation in one spatial dimension, u_τ = u_ξξ. We used the Method of Lines to discretize the spatial dimension, turning the problem into a system of ordinary differential equations. We tried solving that system using some different ODE solvers, and this plot was made using the simple, inaccurate, and non-stiff Explicit Euler method. It turns out that if you don't make your time time steps smaller than about ½ (Δ ξ)^-2, where Δ ξ is the spatial step size, the solution is numerically unstable. This is why the u values are oscillating and blowing up to huge values near τ = 1 in this plot, making the actual solution impossible to see. If you're really curious, here is the assignment this is from.

Wow I'm amazed I could actually write that math just using HTML. Too bad everyone's browsers don't support some easy way of embedding LaTeX directly in HTML. Well, aside from jsMath, anyway, which I don't think I can use on LiveJournal since I'd have to embed JavaScript in my post.

2007-4-12 update: the version linked below now has been released under the GPL. The 0.1 version I first mentioned was released with a BSD license, but this may have not actually been allowed since it called functions from GPL code.

Quercus is Java implementation of PHP. It came to my attention as a possible way of avoiding many of the security vulnerabilities in PHP, particularly buffer overflows. I've been working on running MediaWiki with Quercus for the last couple of weeks, and one of the first hurdles I hit was that Quercus does not currently include implementations of PHP's LDAP functions. For a couple of wikis at work we use LDAP for authentication. I wrote a little Java library which implements the needed functions, which can be downloaded here and verified with this GPG signature. The source is also available under a BSD GPL license here (signature).

If you want to use the binary I provide, just drop quercus-ldap-0.1.jar into your web application's WEB-INF/lib directory, or your application server's shared lib directory. Some time in the next few days I'll post some more info about my experiences with MediaWiki under Quercus + Tomcat 6, including getting LDAP authentication to work.