Failure is inevitable


Microsoft Chart Control for ASP.NET and WinForms

Microsoft has apparently decided that we need another charting control.  Actually, I think this is pretty cool, I just wish it had come out before I was tied in to another vendor's control.  :|  In any case, you can see some pretty pictures, read documentation, and download some samples here. I wonder if this works with ASP.NET MVC...

Must-have software: VirtualWin

If you are like me and have to do a lot of multi-tasking, you may find yourself wishing that you had more room to work.  One thing I've found recently that's really helped is VirtualWin, a virtual desktop manager for Windows.  It's free and open-source and works with both XP and Vista.  To navigate between virtual desktops, you simply hold the Windows key while pressing one of the directional keys.  Right now, I have one desktop where all my test-related stuff is living, another where I'm doing a deployment, and one where I'm doing my development work.  I can shift fro... [More]

Calculating aggregates with Microsoft Parallel Extensions for .NET

The Parallel Extension for .NET (aka PFX) is a new API coming to .NET 3.5 (and included out-of-the-box with .NET 4.0) that greatly simplify and enhance the parallel-processing story in .NET.  Sure, we've got Thread and ThreadPool, but these classes are very lacking.  Fortunately, PFX is a huge improvement.  Be sure to check out their blog and the Parallel Computing Center on MSDN for more information. One of the things I ran into last week was the need to calculate an average of a series of numbers.  Each number was computed by a long-running method, and since each number ... [More]

Friday Grab-Bag

A severe lack of sleep and the "too many irons in the fire" syndrome has left me a tad frazzled, so instead of a well-written (I see you laughing!) post, you get a collection of random thoughts and observations: Too much lambda? I wonder if I'm relying too heavily on lambda expressions.  What do you think, is this going too far (extracted from one of my unit tests): 1: /// <summary> 2: /// Verifies that the correct number of graphs are returned 3: /// for various sizes. 4: /// </summary> 5: [Test] 6: pub... [More]

A long time ago, in a GridTreeView far, far away&hellip;

Some of you (and by some, I mean two) have been axiously awaiting the release of the code for the GridTreeView that I described a while back. Thanks to the generosity of my current employer, I’m happy to present the code, a compiled DLL, and a demo ASP.NET MVC site that you can use to test out the GridTreeView.  Enjoy! Binaries Source Code

Crawling results in DeepCrawler.NET

In the last post, I laid out DeepCrawler.NET's (primitive) strategy for finding search forms, populating them, and submitting their contents using WatiN and a heuristic search mechanism.  As I mentioned at the end of the previous post though, submitting a query is only the first step in a complicated process.  Assuming nothing goes wrong, submitting a query will get us back one or more pages of results.  The problem now becomes parsing and crawling the results.  The Current State First, let's look at what a typical result page consists of, using Google as an example.... [More]

DeepCrawler.NET: Alive and Kicking

Much to my surprise, getting DeepCrawler.NET up and working with basic functionality was surprisingly easy.  It's far from finished, and I haven't exhaustively tested it, but it does work.  In this post, I'll describe the current implementation with respect to how I've addressed some of the barriers raised in my last post. How do we decide which form contains a search form? Some sites (in particular, FedBizOpps) contain multiple forms.  As in this case, sites can contain login forms along side query forms, and we definitely don't want our crawler to try to log in with our vari... [More]

Rant: Automatic Updates

You know what's awesome?  When you leave a long-running process going overnight on your computer, and return the next day expecting to see the results, but instead realize that Automatic Updates rebooted your computer for you, killing the process and losing all of your process's progress!  Yeah, that's my bad for not remembering to disable Automatic Updates on this new computer, but this is terrible usability.  How hard would it have been to make Automatic Updates check my running processes to see if anything looked really, really busy before forcing a reboot?!?  Also, if... [More]

Deep-web crawling with .NET: Getting Started

Thanks go out to Sol over at for giving me some suggestions on things to watch out for.  If you want more background information on federated search or information retrieval, go check it out that site. In the last post, I introduced the idea of creating a deep-web crawler.  I laid out the basic requirements that I've given myself, and I touched on some of the barriers to meeting those requirements.  In this post, I'm going to introduce DeepCrawler.NET, my .NET-based (prototype-stage) crawler. DeepCrawler.NET is written in C# for Microsoft .NET 3.5. ... [More]

Creating a deep-web crawler with .NET: Background

For one of my graduate courses, I've decided to tackle the task of creating an intelligent agent for deep-web (AKA hidden-web) crawling.  Unlike traditional crawlers, which work by following hyperlinks from a set of seed pages, a deep-web crawler digs through search interfaces and forms on the Internet to access the data underneath, data that is typically not directly accessible via a hyperlink (and would therefore be missed by a traditional crawler).  It's a tough problem to crack.  No two search forms on the web are the same (example 1 and example 2), and there's no standard t... [More]