Tuesday, June 28, 2011

Rivers in the Sea

Global waterways are so bustling with activity that, maybe to a surprising extent, orderly paths and queuing patterns quickly emerge given enough data.  In an attempt to visually illuminate the actual routes taken by these clouds of vessels, rather than (or in addition to) prescribed shipping lanes, 3 million positions of transport vessels were fed into Visual Fusion's heat mapper, with some pretty sinuous results...

Here is a link to a skydrive folder where you can download the full-resolution images in their ridiculous original dimensions.

These images are reminiscent of Charles Joseph Minard's flow maps.  Minard pioneered the field of info-graphics, particularly flow maps.  In the example shown, lines depict the destination and volume of French wine exports (check out this compilation Minard's graphics).  A contemporary (and more geographically literal) implementation of this notion could be accomplished by weighting actual vessel locations by specific contents of their manifests.  Even French wine.

Desire Lines
The case for seeing where routes actually exist, vs. where they are prescribed has applications in the creation of more efficient global routes, effective management and security of well-beaten paths, and the overall notion of the market communicating back the more efficient and practical paths.  Take the desire lines of Detroit, for instance, described by James Griffioen...

As Detroit's populace retracts within itself, the legacy infrastructure doesn't adequately support the ad-hoc movements of its residents.  Desire lines are the well-worn actual-travel lines of pedestrians who have a better plan.  You tend to see this a lot on college campuses, as well -the smart institutions give them a coat of cement and the backwards ones post signs.

What are some of the methods used to create these shipping maps?  The vessels themselves come from an aggregate of about a dozen sources.  Only shipping and transport vessels were retained and redundant points were removed.  Using a vessel's orientation information, a point 5km fore and 5km aft were extrapolated and added to the point data set in order to enhance the definition of linear networks.  The result is a SQL Server database of about 6 million points.  These points were fed into Visual Fusion's clustering engine to isolate loaner vessels beyond a set geographic proximity of others in an effort to reduce "noise."  The resulting locations/clusters were processed as a heat map to visually denote areas of intense frequency.  This heat map was served up into a VFX client on top of Bing map tiles.  Then some post-processing to merge Bing's various tile styles.


  1. This is a really nice blog, John.

    I am a Visual Fusion developer. It is an excellent product for business users. However, as a developer, whenever we come across problems, the only way to resolve them is through support form IDV Solutions.

    While IDV Solutions is helpful, obtaining support is very costly for our organization. I know that other companies provide knowledge bases for paid customers, which compiles all the questions asked and answered. This is very helpful to save both time and money for both vendors and users because a lot of users probably have similar questions.

    I am only posting this comment here because under Contact Us my inquiry does not fall under any of the categories. IDV Solutions is growing rapidly, and with more and more users and developers will work with this software. A knowledge base would benefit all users and would attract more people to use it.

    Thank you for reading my suggestion,

  2. Thanks much, Helen! Its my hope that the blog has a bit for everyone, including VF developers like you.
    Did you know we recently started up a community page for just the reason you state? I hope it's a useful resource for you and, if you use it, it will only get better!

    Tell your fellow developers and thanks again for reading!

  3. Excellent work ... these tools to view the data .. is great!