Recently in soc Category

I now have some working code for my dirhash dynamic memory allocation Google Summer of Code project. It is currently being reviewed and needs further testing, but if you're feeling brave you can grab a patch against HEAD from my project page on the FreeBSD wiki.

What the new code does is add an event handler for vm_lowmem events. When one of these events happens, the kernel will try to delete dirhashes that have not been used for DH_RECLAIMAGE seconds. This is currently set to five seconds, but I have no idea if this will be an optimal time. A bit of benchmarking and testing will be necessary to sort that out. If all dirhashes have been used more recently than DH_RECLAIMAGE, instead the first unlocked dirhash on the TAILQ list used to keep track of them all will be deleted. Due to how the existing dirhash code organizes this list based on a scoring system, the first hash on the list should be a mix of the the least recently used and least frequently used.

If my code turns out to work as it should, then the default maximum memory for dirhash can be safely increased quite a bit. If the system has plenty of spare memory, then lots will be available for dirhashes. Then if the system becomes low on memory, some of this dirhash memory will automatically be freed up for use by whatever else needs it. Hopefully we'll see some performance improvements overall on systems with lots of memory that need to deal with large directories, but I haven't done the benchmarks yet.

The Summer of Code officially started a couple of weeks ago, so I've been getting started on my project. I've not written any actual code yet, but have been busy with other aspects of the project. First of all I have been reading a bit of documentation and code relevant to dirhash itself and FreeBSD memory management. I have a lot to learn, especially about the memory stuff. The two approaches I am looking at, mentioned in the USENIX paper by Ian Dowse and David Malone (my SoC mentor), are to either use the buffer cache for dirhash's memory so the VM can take care of providing memory as it is available, or to use a method for allocating memory where the kernel can signal for some to be freed in low memory situations, perhaps using the slab allocator.

I have also been learning a little bit about Perforce, the version control system used by FreeBSD for experimental projects. I have not used Perforce before this, and it is a bit different from the VCSs I've used before (CVS and Subversion). Also, I have done a bit of experimentation with the current dirhash implementation just to be sure that it can free memory it no longer needs. This was just a simple test where I created a directory with a couple of million entries, checked the vfs.ufs.dirhash_mem sysctl, removed the directory, then checked vfs.ufs.dirhash_mem again to make sure the memory used had decreased. It had. Finally, I sent off a brief proposal to do a talk about this project at EuroBSDCon 2008. Obviously I don't have much to create a talk with yet, but I should by the end of the summer. Plus, the second day of EuroBSDCon this year happens to fall on my birthday, and I think hanging out with fellow BSD nerds in France could be a fun birthday activity :)

My Google Summer of Code Project has been approved! I am excited that I will be supported by Google to work on FreeBSD. It should be a fun time.