Friday, July 29, 2011

Chalkboard Maps: United States of Craigslist


geometric pass at delineating areas within the United States potentially covered by each craigslist site -the United States of Craigslist.  Check out poster print options here.

Source Data Download
Here is a link to a skydrive folder with this data as a shapefile (.shp, .dbf, yada yada...) and a (spartanly formatted) kml file: https://skydrive.live.com/#!/?cid=2eb6aaf6c3ac1ebe&sc=documents&uc=2&id=2EB6AAF6C3AC1EBE%212242
With great power comes great responsibility.

Poster Print Available
If you want, you can go here and order a print of this sucker.  Or you can call your friend in the geology department with a huge plotter and sneak off a print after hours.

Why?
Locality is inherent to the value of craigslist; I go to craigslist.org but I get kicked over to the local instance of craigslist (my IP address sources me to somewhere in the illustrious Lansing, MI).  But how does craigslist know where to send me?   Some mysterious system of assigning a geocoded IP address to just the right site must be in place...I wonder what that map looks like.
When Ian Clemens proposed the idea, I looked around to find an existing map of craigslist sites-to-areas -maybe even find the lookup that they themselves use. I couldn't find anything like it.
Whether it matches their system well or not, here is a map that approximates geographic coverage to individual sites using a Voronoi process as a base (more info on process below).  It is at least a start at visualizing the geographic coverage and distribution of the community-driven instances of craigslist.  Shapes like this might provide some useful context for other data, demographic or market information, for instance.  Also, when pulled into VFX, it can serve as an input to some spatial querying on those other metrics.

We'll soon be releasing an interactive version of this data in VFX where you
can play with it in the context of other data and within alternate, though coupled,
charting and timeline dimensions.

Improvements
With access to web traffic data, one could compile a pragmatic view of coverage driven by the locations of actual website visitors (but this would just be the incestuous results of the current method craigslist uses to allocate visitors to sites).  That's ok, but it's more detective work than interesting data creation.
The use of openstreetmap data to weight the polygon drawing by travel time would improve the realism of the hypothetical zones considerably.  In that case, maybe it could be used to drive a more efficient assignment of craigslist.org visitors to their actual-nearest craigslist community.  That would rule.

Spin-off Functionality
Creating these areas was, in part, a helpful testbed for a generic region-building functionality that is in the skunk-works here at IDV.  The algorithmically inclined Abhinav Dayal has been crafting our drive-time service that is already doing a lot of the heavy lifting when it comes to Voronoi diagramming.  So, down the road we might see a new specialized tool that generates best-fit areas around an existing set of points -useful for some what-if scenarios around territory creation, available to the business user, not just the research scientist.

Nerdy Bits
• Scrape the list of Craig’s cities at http://www.craigslist.org/about/sites.
• Split joint-locations into individual locations (like "Odessa / Midland")
• Geocode place-specific locations.
• Manually position the more regional locations (like "Southeast Iowa").
• Divide locations into three geographically distinct regions (split by the Continental Divide along the spine of the Rockies and the Mississippi); duplicate any locations that meaningfully straddle a border, like St. Louis. I do this to introduce some true-cost of crossing either of those features, in the face of an algorithm that would otherwise treat the whole country as a smooth unfettered plain.
• Run Voronoi (Thiessen) algorithm to generate best-fit zones for the points, for all three regions.
• Clip Voronoi zones by a “land” shape to cut out the oceans and provide a common border between the three regions (my "land" was constructed from the Census Bureau's tracts file).
• Merge the 3 regional Voronoi sets into a unified nation-wide set.
• Dissolve boundaries between same-website Voronoi zones (to re-combine the joint-locations up there in step 2) into merged chunky polygons.
• Manually re-assign oddly-orphaned or split areas (common along complicated shorelines).

That's just about it.  Thoughts?  Ideas?  Outrage?  Incredulity?  Been done?  Guffaws?  Know a good dataset to improve this method?


Download the United States of Craigslist as a table of member Zip-Codes
Here's a map and lookup of the zip codes that correspond to these areas here: http://uxblog.idvsolutions.com/2011/12/craigszips.html



 

63 comments:

  1. Nice Voronois. However, the reason to employ Voronois is generally to accommodate a non-geographic pass at trade area delineation. Retailers have employed this to provide an objective (i.e. non ZIP code based) evaluation of a trade areas. In developing these service areas for craigs list, you've manually altered objectivity based on some fundamental regional understanding like the splits you mentioned. So, my question is, have you been true to the Voronoi method and do you think you've accurately portrayed these trade areas?

    ReplyDelete
    Replies
    1. Pedantic and condescending.

      Delete
    2. @Anonymous: Actually, there are several of us who found Joe's response -- and then John's follow-up -- quite interesting. What's the point of flaming a contributor (especially under an "Anonymous" ID)?? If you don't like it, ignore it. And BTW, I read an interesting article in The California Geographer about why they should NOT be called Thiessen polygons. Thiessen (1911) "fudged" his own example to yield remarkable "accurate" estimations or precipitations.

      Delete
    3. correction: "... his own example to yield remarkably 'accurate' estimations of precipitations." (--http://www.csun.edu/~calgeosoc/CG1999.html)

      Delete
    4. I agree, Mark; I was disappointed to see that anonymous response to a reasonable comment (I really hesitated doing the initial divide anyway). And Joe's comment is just the sort that improves and extents a discussion.
      To add a little more detail to my followup, since Craigslist transactions are largely drive-to-pickup transactions I would really love to incorporate some road friction in order to weight the zone boundaries by driving effort. I think this would result in a much more true-to-live divination of these economic zones. As an example of an input wishlist, check out this great work by Brandon Martin-Anderson at MIT...
      http://shortestpathtree.org/

      Delete
    5. https://www.empowernetwork.com/msabbie/blog/todays-core-commitment/ just trying to see if I can post in hear, cauase I wanted to post my own blogg but don;'t know how.. thanks and sorry

      Delete
    6. This is one great creation of United States of Craigslist zip code via user friendly indication.I really like it.

      Delete
  2. Hi Joe, thanks for the post!
    Actually, I'd love to inject plenty more bias into these areas. Like many models have to be, and why I can seldomly embrace economic models with much conviction, is that they, necessarily, remove variables from reality. When I can reintroduce variability based on prior understanding I have to consider it, particularly with social science data.
    The voronois are a useful general filling mechanism but with more time I'd like to first pre-split the country into the "where's George" economic zones by the folks from Northwestern. Then in addition to that, weight by drive-time.
    So in this case I'm relatively happy with the geometric heavy lifting Voronoi did, but I'm always looking for ways to introduce social and physical friction.
    John

    ReplyDelete
  3. Why not just have people add their address along with their post, not viewable to the public obviously for privacy, and then have sections based on peoples distance from one another. Just a simple 5, 10, 25, 50 miles away system based on the 2nd partys location

    ReplyDelete
  4. Or the tap or click method on a map to show where u r located

    ReplyDelete
  5. One way to dramatically improve this would be to use the BTS Journey To Work data.
    http://transtats.bts.gov/tables.asp?Table_ID=1321&SYS_Table_Name=T_CTPP2000_PART3_SELECTED

    It's Census-Tract based, so you'll need to get a shapefile for each Census tract (available online) and process it like you did the Zip Files. Then, you can get the Time-To-Work and Time-From-Work for any two tracts. Aggregation would probably be difficult, but should be possible.

    I did a similar process for a (now dead) mapping company. We would let you choose a house and we'd calculate the corresponding tract. Then we could map your commute by how long it would take to drive from home (and the reverse, where you should live to get a certain length commute from a known work).

    Hope this helps.

    ReplyDelete
  6. I wish there was a way to search within a certain radius of a zip code or certain region without having to click each town or area and retype what I am searching all over again.

    ReplyDelete
  7. ....*** CAN YOU PRETTY PLEASE ADD ON THE UPSTATE NY CL CITY/COUNTY OF ....SCHOHARIE COUNTY*** IT WOULD BE A GREAT ADDITION TO THE NEW YORK CL. ...PLEASE!!! :-)

    ReplyDelete
  8. ...*** from a LONG TIME USEER of CL...JENNA...I LOVE THIS SITE!!!.... PLEASE PLEASE ADD ... SCHOHARIE COUNTY IN NEW YORK'S CL. Without CL I would have never found my dog!! ...and she is the best in the world....I have found many great things on this site from apartments to my dog...and funiture... LOVE IT!! THANKS FOR ALL THE HARD WORK CL TEAM!!! KEEP IT UP, AND just remember to read customer comments to make the best service available!! :-)

    ReplyDelete
  9. What is the problem with entering the address and it bouncing back now?

    pers-xxxx-2955369389@craigslist.org

    doesnt work anymore???

    What is the new and improved address method?

    ReplyDelete
  10. Dear Various Anonymous Commenters,
    This is not a tool of Craigslist; it is not used by them; it was not made by them. It is a visualization of presumed/de-facto economic zones using craigslist sites as an input, on a sweet sweet chalkboard.

    ReplyDelete
  11. I love craigslist, a lot more than ebay. Except for the fornications hook ups, but besides that, you guys rule!

    ReplyDelete
    Replies
    1. I think they didn't read the part where you said you are not affiliated with CL - however, take it as a compliment?

      And YES! If CL would bail the x-rated junk and be a solid better-than-eBay site...the world would rejoice.

      Delete
  12. Very helpful! I was wondering why you didn't opt to include craigslist sub-regions? The Los Angeles craiglist, for example, has several sub-regions such as Long Beach. Probably adds all sorts of complexities but might be an interesting addition to v2.

    ReplyDelete
  13. This comment has been removed by a blog administrator.

    ReplyDelete
    Replies
    1. He can be reached at thatdudeintulsa@craigslist.org

      Delete
  14. I need to up;oad some pics in my add, but they never show HELP!! please reply to my e-mail gaby37_nice@hotmail.com

    ReplyDelete
  15. how in the world do i add my phone number to my cl account so i can post?

    ReplyDelete
  16. Why can't I see the classes I post. I know I made some mistakes when I started using CL this past week, but none of my friends told me about multiple similar events. Now, when I try to post correctly, I cannot see any of the dates I posted on; I can only scroll backwards form dates I try to preview. I am told someone may be trying to sabotage my classes. Ugh!

    ReplyDelete
  17. This is a wonderful post! is really informative for me. I liked it very much.

    ReplyDelete
  18. I went to my account today and saw two things posted there that weren't mine, and one of the things I had posted was completely gone. Does this mean someone is trying to get my identity? What should I do??

    ReplyDelete
  19. On cragslist under car/ truck by owner the pictures on the front page is not work well it takes too long to load the page. Put it back please thank you.

    ReplyDelete
  20. I have been trying to start a moving buisness I have the cheapest professional rates but it still dosn't seem to be cheap enough it really seems like everyone thats looking for any kind of service wants it done for basically free I offer all sorts of discounts like for police and firemen, political officals,church officals, seniors, people with disabilities and then I offer other people a a discount for putting a nice review on yellowpages.com nothing seems to work

    ReplyDelete
    Replies
    1. Why inthe world would you give dscounts to some of these groups? they make enough money and get enough handouts as it is

      Delete
  21. Humm,

    As I worked for a few years early on responding to those requesting CL cities sites, I find this map interesting. I would tell some folks they could not or should not have their own CL site as they were too close to existing CL sites.

    I tried clicking on an individual area like the Inland Empire which is Riverside and San Bernardino Counties in CA (except for areas near Palm Springs that has its own CL site as they met the criteria (international destination, has international airport, a computer connected population)

    twodogkd
    Riverside CA

    ReplyDelete
  22. I can comment more about some regions when I CAN SEE THE DETAILS of the map produced to better see what areas ended up within the same sections.

    It may be that some things do not make sense such as for people who live on the Nevada side of the Sierra Nevada where they live on a geographic north south corridor along US HIGHWAY 395, so chopping that up into sections going into Nevada may not make sense since folks along that road either drive up towards tahoe/reno to shop or drive down to shop.

    When I considered geographic areas that should be included in a Craiglist site, I did quite a bit of research using the wikipedia for the City, County, State, etc as needed to learn about the city/site and surrounding areas AND ALSO looked at the nearest Craigslist cities/sites.

    One issue with Craigslist is ONCE they GO SMALL it is hard to broaden out.

    I believe some Craigslist sites get TOO SMALL such that what would be ones "COMMUNITY AREA" or the area folks are normally drive and travel in the normal course of their lives may vary from area to area. In the Inland Empire (Riverside and San Bernardino Counties) many of us tend not to think much of traveling 60 to 80 miles and do so much more frequently and for some daily. Also there is much crossover between Riverside and San Bernardino with many Riverside citizens attending college in San Bernardino for example.

    Craiglist is about COMMUNITY so the geographic areas selected should reflect that to the extent it makes sense to do so.

    ReplyDelete
  23. I think it's funny that this is to be used to sell things but everytime we advertise we have pit puppies we get flagged and deleted hummmmm somethings just not right with this page you can't put up post's about puppies but you can advertize for sex that's pretty bad.

    ReplyDelete
  24. hello....someone want to buy something from me with a high value.He want me to give my paypal info,so he could send money imid. and calc. shipping since he is not from my area.it is a gmail.com email and im not sure if i should do it or if its scam

    ReplyDelete
  25. I live in the D/FW area of texas and in the last week you quit posting how long the listing has been live... I cannot see that now so I do not know if it is worth pursuing a curb alert or a free item... Is there a specific reason for this or is it a glitch that you are working on

    ReplyDelete
  26. Do you think that there would be any distortion in the map based on the difference between regions where Craigslist is the leading online classifieds service (e.g. Washington) and regions where it is not (e.g. Utah)?

    I suppose it would depend on whether Craigslist strives for larger zones where fewer people use Craigslist, or whether Craigslist is usage-agnostic when it decides how to divide up an area into zones.

    ReplyDelete
  27. I made a post I like the posts I see BUt where Is mY post

    ReplyDelete
  28. It would be interesting to see how this map over laps with area codes or zip codes.

    ReplyDelete
    Replies
    1. You can find that map and reference list here: http://uxblog.idvsolutions.com/2011/12/craigszips.html

      Delete
  29. It would be nice to see Delaware spit in 2 areas. One north and one south. I am in Maryland on the Eastern Shore and Northern De. is over 100 miles away.

    ReplyDelete
  30. Why is it so easy for a person to flag a post?it's as simple as if they don't like it,then they can rid of it.does that make sense..CL needs to do something about that..

    ReplyDelete
  31. This comment has been removed by the author.

    ReplyDelete
  32. I heard that I have bad things written about me on a blog.. How do I check

    ReplyDelete
  33. I have been looking at Kindles, for sale, on the Santa Fe, Taos, NM site. Two different people had good deals posted, but both had been sold. Both sellers sent me emails telling me they had purchased their on one of those auction sites, which are kind of like gambling, and said I should try it. They both said they always win, by waiting until the end of the bidding. This seems like a scam to me.

    ReplyDelete
  34. I never heard of the Voronoi (Thiessen) algorithm.
    Is this exclusive to Craigslist or is it in use by other companies?

    ReplyDelete
  35. will you please contact me regarding a problem with my replys to free stuff

    469 735 2856 I can not get the reply's to go through or some reason and your help site does not help at all!

    e-mail stoutbetty46@yahoo.com

    ReplyDelete
  36. i have an antique saddle listed and a guy from N.C. wants to buy it through paypal and pay to have a shipper come and get it from Washington state. How can i do this safely

    ReplyDelete
  37. http://1d142aoieuj0ms1pf4-50bfj3c.hop.clickbank.net/

    ReplyDelete
  38. We are manufacturer of clothing for children such as: wholesale boy clothing, wholesale girl clothing, smocked clothing, girl smocked dresses, boy smocked long-all, smocked short-all, Applique clothing,

    ReplyDelete
  39. look i dont understand why ppl have to flag my stuff when i have it on here for free to a good home it bs and it seam to me that they have nothing better to do with there time

    ReplyDelete
  40. I like this map and hope it becomes interactive soon!

    ReplyDelete
  41. Hello everyone,

    I am new to the list and need help listing some appliances. So help!!!!!!

    ReplyDelete
  42. Looking forward to an interactive map.

    https://www.avantidataproducts.com/

    ReplyDelete
    Replies
    1. SPAM

      However, I do provide a link to a KML file, which you can interact with in any mapping site.

      Delete
  43. Why not do something about all the flagging going on, an honest person cannot list a posting without being flagged every single time. I live in southeast missouri and cannot place any posting do to flaggers, this is childish and stupid.I have placed an ad for selling a few chickens, and looking for a certain breed of chicken and one for wanting an incubator, I even placed an ad to sell my laptop or trade it in computers, all flagged and removed every day. this is getting out of hand. DO SOMETHING TO STOP THIS FLAGGING!

    ReplyDelete
  44. Why not do something about all the flagging going on, an honest person cannot list a posting without being flagged every single time. I live in southeast missouri and cannot place any posting do to flaggers, this is childish and stupid.I have placed an ad for selling a few chickens, and looking for a certain breed of chicken and one for wanting an incubator, I even placed an ad to sell my laptop or trade it in computers, all flagged and removed every day. this is getting out of hand. DO SOMETHING TO STOP THIS FLAGGING!

    ReplyDelete
  45. I use craigslist looking for bikes and bike parts. I also use it for looking for cars, trucks and such. In both these catagories I often find that I get stuck looking at a bunch of parts when I am looking for a whole bike or whole car etc... Would be great if they added a couple of catagories for parts.

    ReplyDelete
  46. Cool map. Why the hell does everyone think you are craigslist tech support?

    ReplyDelete
  47. Cool map. Why the hell does everyone think you are craigslist tech support?

    ReplyDelete
  48. Ran into this while searching for a way to map out craigslist "city" categories to zip codes, mainly for the latitude longitude coordinates. I seen previously mentioned the kml files are interactive maps, i wonder if this could help me figure out what lat/long for each craigslist city. Either way great work mapping this out! I am especially thankful for the table associating zipcodes with craiglist cities.

    Thanks Again!!

    PS: no idea why so many people get the idea this is craigslist related, completely different domain to begin with.

    ReplyDelete