tag:blogger.com,1999:blog-2914337944584630860.post8923755908399249304..comments2009-01-24T12:35:22.278-08:00Jon Harthttp://www.blogger.com/profile/02857880233692933624noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-2914337944584630860.post-38504051695338911012009-01-24T12:35:00.000-08:002009-01-24T12:35:00.000-08:00@postmodern:<BR/><BR/>The obstacle course is a great idea! I took a quick pass through it and it looks like the only part Hawler gets slightly confused on is the empty href tricks. I'm torn as to whether this a bug in Hawler or the fault of URI::merge, which is responsible for making new URIs out of the page being processed and the newly encountered "link". I've worked around this in the latest commit.<BR/><BR/>Also, its good to see familiar names working on cool projects. Congrats on Spidr!Jon Harthttp://www.blogger.com/profile/03410754059921403771noreply@blogger.comtag:blogger.com,1999:blog-2914337944584630860.post-36078773780469548792009-01-16T00:42:00.000-08:002009-01-16T00:42:00.000-08:00Hey, nice to see Hawler continue to grow into a handy website crawling toolkit.<BR/><BR/>I've also written a web spidering library, named <A HREF="http://spidr.rubyforge.org/" REL="nofollow">Spidr</A>. What could be useful to your Hawler project is that I've also written a <A HREF="http://spidr.rubyforge.org/course/start.html" REL="nofollow">Web-Spider Obstacle Course</A> for web-spiders. This course provides various foul HTML pages which the spider must navigate properly. There's also a <A HREF="http://spidr.rubyforge.org/course/specs.json" REL="nofollow">JSON file</A> that describes which links have to be followed/ignored/not-followed. I use the JSON file along with RSpec to test Spidr's ability at navigating rough HTML.<BR/><BR/>More of my code can be found on <A HREF="http://github.com/postmodern" REL="nofollow">GitHub</A>.postmodernhttp://houseofpostmodern.wordpress.com/noreply@blogger.com