Tuesday, June 9, 2015

Sightings

Poster print. If you scratch this poster and sniff it, does it smell like like the acrid energy of excited oxygen and ozone? Will it make the hair on your neck stand at attention? There is one way to find out.

Poster print. Does this poster glow in the dark? No. But if you hang the poster, stare at it for one minute, then immediately look at a blank white wall, you will receive complete consciousness.

This data comes from the National UFO Reporting Center, which is "dedicated to the collection and dissemination of objective UFO data." That, and some census data, and we are off to the races.

Ratios!
Of course, as is the case for any observation data, there is a strong tendency towards echoing a population map. This is certainly the case with this sighting data, as well. In order to visualize the actual sighting phenomenon, I needed to normalize by the underlying population.  The first, more prominent map shows a simple ratio of the sightings by population. A per-capita approach.  The second, smaller and slightly more complex map, shows a bi-variate mapping of sightings in the color dimension (dark slate for low-sightings and bright green for high-sightings) and population density in the opacity dimension (denser populations are more transparent). The result is a map that is more nuanced regarding the problem of variable populations and area sizes. Double normalized? Sort of. If you would like to traverse the rich and complex world of bi-variate mapping, check out this tour-de-force how-to by Joshua Stevens.

Poster print. Terrain of sightings, normalized by population (because without that, it would just be a population map). Blow your visitors' minds when they see this beauty hanging proudly on your, otherwise incomplete, wall. Regale them with a nuanced conversation about the subtle interplay of underlying populations and observed phenomena. Then point to your county and say, "Right there. That one's me. Let's have a few drinks; I'm not ready to talk about it quite yet."

Aggregation?
You may be thinking, "wait a second, I thought you said aggregating non-political data into political zones was lazy and pedestrian!" That's true. I am admittedly violating my own unrequested map tips 1, 5, and 6. But I am doing it with gusto.

Trending
One of those great questions that you have to ask yourself when making a visualization is, "compared to what?That's why the shape trending section was so interesting. Shape popularity compared to other shapes.  Compared to other times.  Physically analogous shapes like Disk and Egg? So mid-century materialist chic. Uncertainty? So late-century neo-relativism. Flaming lights? Such millennial clarity. No matter the phenomenon, crowd sourced data tends toward a first-order trend of how we see ourselves.

Poster print. Check out the USA's history of unidentified flying...fireballs? Triangles? Eggs? Feast your eyes on the distillations of thousands and thousands of reported sightings by what shape they were reported to be. Upon receipt of this poster, invite six friends over for a small dinner party. Reveal the chart and kick off the conversation with something like, "The postmodern age of uncertainty is eroding beneath our feed, my friends. Behold its pre-digital-age rise and abrupt fall at the hands of frenetically optimistic millennial certainty. Pedestrian notions of eggs and saucers? Robert Wise wants his breakfast back, thank you. Mbwa ha ha. Now then, who takes their absinthe with a cube of sugar?" So on and so forth.

Time
It isn't surprising that there is a clear spike in sightings on summer nights, but I wanted to see for myself.  As soon as somebody sends me some time-stamped data that quantifies when people are looking up, I'd be thrilled to normalize this by it. In the meantime, I still like seeing a picture of the terrain of observation. Like the shape trending, it may be more of a revelation about our own movements, than the schedule of unidentified flying objects, but there is value in revealing that structure. If you enjoy spurious correlations, feel free to indulge here as a first step.


Other print options.



View John Nelson's LinkedIn profile View John Nelson's profile

Tuesday, June 2, 2015

Migration

This is the annual migration pattern of the Rose-Breasted Grosbeak. A pretty fantastic voyage for a little thing -which they will undertake up to twelve times in their lives.  Each sighting (from the Cornell Lab of Ornithology's eBird project) is a little spotlight that reveals the underlying satellite imagery for that month. When seen together, the wonderful emergent properties of a species' migration illuminates a true bird's eye perspective of movement and structure on a continental-scale.

The big seasonal migration of a little bird. Each sighting reveals a spotlight of the underlying satellite image for that month. All together, they illuminate an annual ritual of chasing greenness. Click to open the full-blown version (could take a moment to fully load the animated gif).

Animal migration just blows my mind. The intrinsic commands wired into a bundle of brains and nerves such that they and all their pals are totally in the know and confident of the plan. Except that there is no 'plan.' It also makes me wonder how much of what I do and think in a day is intentional and novel, and how much is me holding a little point of light largely unaware of the larger structure I am a part of.  In many ways I'm not so different from that little bird (though less well-traveled).

Why the Spotlight Style?
When a creature like the Rose-Breasted Grosbeak feels the inner tuggings of the migratory urge, they are still just one element of a larger system. Any one of these birds only get to see what they themselves can see, and while their perspective might perch higher than ours, they have no map and no external device telling them which direction to fly and what the distance.  Their individual view is no bigger than one pinpoint of a spotlight at the continental scale, but they confidently charge ahead. I thought a visual method that revealed visited locations (rather than painting over them) would suit the phenomenon well, hopefully making the viewing a little more personal.

A month by month small multiple of the migration. Now your eyeballs can more efficiently observe the phenomenon rather than be held hostage by the beautiful tyranny of an animated sequence. Click to blow out the full resolution version.

Why?
This was a collaboration with a friend/cohort from Central Michigan University's Geography Department, David Patton. Dave is an actual birder (photographer, guitarist, scientist, gentleman poet), while I am...other things.  Also, special thanks to Daniel Huffman and Joshua Stevens, who both provided terrific cartographic advice along the way.
Dave has done work with the eBird data in the past, with his class, so he had already undertaken the painful process of downloading, parsing, and filtering a multi-gigabyte flat file with five years of observations. He noticed that the Rose-Breasted Grosbeak had a particularly impressive migration pattern.  I wondered if a subtractive, rather than additive, approach might be an interesting way to show this migration. Think scratch-off rather than dots.
[Insider scoop: The seed of this idea came from a concept we used years ago for an unnamed counter-terrorism division of a metropolitan police force.  We used a reverse-heatmap to show, etch-a-sketch style, the buildup of patrol unit movements over time.]

An early test of the scratch-off masking concept. Rather than covering the sighting locations, the sightings became, via masking, the only way to see the locations. You see what the bird sees, and that's it.

How?
First Look
Binning up large amounts of lat/long data (and parsing out dates into months, days, years, etc.) is most quickly and easily done in Excel. Yes, Excel. It is a handy first-pass look at the data, and it takes about two minutes.

An Excel pivot table of sightings by lat/long. You can filter by month, as well, to get a sense of movement over time with just a few clicks.

Here is an initial look at the strength of the migration -also a pivot table in Excel. The Y axis corresponds to latitude (North-Southiness), and the X axis breaks out sightings per month.

Satellite Imagery
The satellite imagery for each month was downloaded from the great NASA Visible Earth resource. They were projected into Lambert Conformal Conic (I tried other more ridiculous perspectives with a stronger horizon to try to echo the feel of looking down from above but good old Lambert was a way better balance of readability and perspective).

Masking
We separated out the sighting points by month in ArcMap, and exported them as PNGs with a transparent background.  Then, in Fireworks, they get a Gaussian glow and are used as a masking layer to reveal the satellite imagery.

Get that month's point cloud and satellite image, apply a Gaussian glow to the points, mask image by points.

Then we calculated the spatial mean of the sightings per month, and masked-out a state lines reference centered over that average location (for each month).

The spatial mean of each month's sightings provide the center of a faint elliptically masked-out state reference.

The combination of these layers, plus a very faint version of the satellite image to provide some context, formed a map for each month of the migration.  They were then stitched together into an animated gif, and small multiple (those things above the fold).

Putting all the layers together.

Charting
While the North/South shift in sightings was pretty obvious in the map animation, itself, I like the behind-the scenes view of proportionality I saw in Excel in the discovery process. So I added it to the map. Why hog all of that chart fun?

The "proportional sightings by latitude" charts were quick bar charts that I copied out of Excel and cleaned up via magic wand waving in Fireworks.

Data
Bird sightings
Dave wrote to the folks at eBird and got a mega-huge file.  Here is their website:
http://ebird.org/content/ebird/

Satellite Imagery
I am a devoted and compulsive user of the cloud-free mosaics from NASA's Visible Earth, made available to a world full of nerds.
http://visibleearth.nasa.gov/view_cat.php?categoryID=1484

State Reference
The US, Canadian, and Mexican state (province) linework came from the precise and generous folks at Natural Earth.
http://www.naturalearthdata.com/downloads/10m-cultural-vectors/

I hope you like it. It was a fun excuse to work with an old friend and a chance to re-visit an old visualization trick with the benefit of perspective; the fact that I am happy with the results is just gravy.


View John Nelson's LinkedIn profile View John Nelson's profile

Friday, May 29, 2015

Risk Archive Mesh

Historical Event Analysis Tool
We recently added for customers of VCC the ability to archive,search, and display, all of their incoming threat elements in order to build up a custom terrain of risk. Organizations aren't all threatened by the same sorts of things (Pennywise, for example, may have cast a long dark shadow over your childhood, or he was just Tim Curry dancing around in makeup), so letting customers define what determines a risk, then letting them add or remove those types of threats to build up a living mesh of relative danger, is a flexible way of conjuring the relative hot spots which may represent areas to avoid, reinforce, or they may be opportunities for improvement.

Historical incidents of a customer's feeds of terrorism, human trafficking, and gang activity.

Choosing a specific cell shows local trending along the sources of risk included in the terrain mapping.

A look at the local breakouts for extreme weather and natural disasters in the American Midwest.

A custom risk profile.

Risk trending within a selected cell.

Why all the risk, threat, danger? Because being aware of existential threats informs safe decisions, specifically for organizations that are responsible for the safety of their employees. Anyways, you know what they say about knowing.

Then what's up with all the hexagons?
First off, hexagons are awesome. Second off, they are nature's most efficient tessellation framework for maximum spatial variability within a tiled pattern on a Euclidean plane. But the Earth is not a Euclidean plane! So... Last off, teams of scientists have delivered on a multi-scale segmentation scheme for dividing the surface of the earth into roughly equal area cells (big thanks to Kevin M. Sahr of Southern Oregon University).

Cells?
Yes, cells! Because these hexagons (mostly hexagons) are vector cells, they are interactive to the user. So you can grab one and pull down all specific assets currently in that area or travelers destined to pass through that area, or export underlying data to Excel, or whatever. In any case, to an organization charged with protecting people and assets, the context of history is a welcome help.


View John Nelson's LinkedIn profile View John Nelson's profile


Wednesday, March 18, 2015

College Basketball Recruiting Footprints

On the cusp of full-blown March Madness, I took a look at the recruiting footprints of each of the top 25 men's college basketball teams (at the end of the regular season) and worked out their Air Mile Index, as is my way. If you're new to it, the Air Mile Index (likely correlated with the assistant coaches' divorce rate) is a rating of how far, overall, a team had to reach to lure players to their program. FYI, an Air Mile Index of ten is mega local (#2-ranked Villanova, are you kidding me?!) while thirty is really far-reaching (Gonzaga!!).

Air Mile Index Maps

Here are the Air Mile Indexes of the top 25:


























Distribution

And here are all those players from above, in one map. Of course there are some obvious city clusters, but this is by no means a rubber stamp of population.



Weird. Check out California, for instance. Players come from two cities in California. The sprawling megalopolis population of the eastern seaboard? Eh. Population big-shots Florida and Texas? Eh eh. It's like wherever corn is grown, so are top-25 collegiate basketball players. When an assistant coach is flying around for recruits, they might as well just look down, and if they see this:


 ...go ahead and jump out right there.

Were you curious about those threads in some of the team maps that extend beyond the map to players' distant hometowns? You can check out all that nitty gritty at the team roster lists. Speaking of distant hometowns, The Air Mile Index takes the square root of a player's hometown distance, so outlier players like the ones below have an ever-decreasing impact on the overall team average. Here is a world map of those national and international hometowns:



Data

I scraped the team rosters on ESPN.com to get player name, info, and hometowns. Then I Geocoded their hometown and their team venues. Then I used ArcMap to draw connections between those to-from coordinates, in a projection that preserves relative distance. Then I calculated their distances and whipped up the team Air Mile Indexes in Excel. Then I made maps that include these distances over a NASA satellite image.  I did it by exporting each layer as a PNG and composing them in Fireworks. Then I wrote about the maps here. Then I posted links to all this derived data: