Monday, 1 February 2016

A few notes on building Real Estate of Mind - All City Edition

In case you missed it here's the viz


I sourced the data for this viz from which gives the latest rolling 12 months. This is a really great source for NYC property data, in fact I think have done a pretty fantastic job all around of making city data public. For historic data back to 2003 you can go here, that might be my next task!

The data comes separately for each borough, so I just downloaded them all and stuck them all into a single Excel sheet, noting the various borough codes (1=Manhattan, 2=Bronx, 3=Brooklyn, 4=Queens and 5=Staten Island).

This dataset includes ALL property sales, including commercial property and entire buildings. So to try and identify which sales were single residential units I filtered to the building categories below and then applied an extra filter to try and pick out apartments versus whole buildings:

I know for a fact that this wasn't 100% successful, but I think it was a pretty good way to filter. I also filtered out the 17,000 sales less than $50,000, because I just don't believe that could ever be real.

One wish I have for this dataset would be better sq ft data, its currently listed for less than half of properties, and bedrom counts.


The data provided by gives street addresses and zip codes, but I wanted to be able to map the building points exactly. Last time I used a geocoding tool by Texas A&M University. This time I searched around again and found a site called which seemed to offer a combination of very reasonable pricing with a friendly user interface.

To do the geocoding I simply uploaded a csv file including the street addresses and the zip code. In fact the first time I did it I also included the city name of New York for all points but this put everything in Manhattan. The folks at were kind enough to help me out quickly and return my credits, I was very impressed with the customer service they provided.

Now let's talk about accuracy of geocoding. For the most part it was pretty good, but there were some weird results too, for example check out this map showing all points I geocoded:

The zip code 11363 is in Queens, so I'm not sure how this point ended up in Arkansas. Fortunately the few big mistakes are easy to get rid of using the Tableau lasso tool.

More frustrating are the near misses, for example some points ended up in the wrong borough:

and some of the 'famous' high end buildings in Manhattan were in the wrong spot. For example:

I realize this might seem picky, but when you are looking at NYC real estate the $ difference between the top and the bottom of Central Park, for example, is HUGE. I'd say 95% of the data points are pretty spot on, but unfortunately the mistakes do cause some problems, particularly when trying to zoom to a particular neighborhood. do provide accuracy scores, but sometimes clearly wrong locations (like the wrong state) are scored 100%. Having said that, I would use the service again as I don't think any service has mastered batch geocoding perfectly.


For the design of this viz I wanted to create a unique look and a nice color palette. I searched around for NYC graphics and found this apple, and from here I built out the palette.

To do this I used a site called Its a really nice system for building 5-color palettes. I locked in the yellow, red green and grey from the apple and then hit the space bar to generate the fifth color until I was happy, easy peasy!


To be honest, the viz loads more slowly than I would like. At first I was using data blending to bring in the latitudes and longitudes, so I switched this to joins to try and speed things up. Unfortunately this didn't work, so I think the slowness is down to the number of mapped points, the use of medians and the high res images. I don't really want to lose any of these, so please be patient :-)

I hope you enjoy the viz, I like doing work with real estate data and will probably do some more in future.


  1. This comment has been removed by a blog administrator.

  2. I sent a mail regarding a couple of minor things and the Dove mountain property manager was surprisingly fast and emailed me back the next day inquiring after my schedule.

  3. Your post is simply great. its completely manage crm real estate

  4. Very informative and getting to know about the facts of real estate. Thanks for the sharing.

    properties for sale in coimbatore | coimbatore real estate

  5. free parking software
    violation management
    Free Parking Software for Town Houses, Condominiums, and Businesses Solves Parking Problems
    Searching for a parking space is such as frustrating routine activity for lots of people in cities all over the world. This search is burning over a million barrels of oil of the world each day. There are also 4-parking spaces every vehicle in the US and almost all streets are most of the time, empty, which leads to urban environments that are ironically doing far more to accommodate the vehicles than people. As the global population continuously urbanize, without a convenience-driven and well-planned retreat from cars, these sort of parking problems will worsen. software is the first step in the right decision. It involves using low-cost sensors, mobile phone-enabled, and real time data collection automated payment systems enabling people to reserve parking in advance or predict accurately where they can find a spot. When deployed as a system, free parking software thereby reduces car emissions in the urban centers by means of reducing the necessity for the people to circle the city blocks needlessly searching for parking. Furthermore, it permits the cities to manage their parking supply carefully.
    This free parking software is now being developed in many different states and cities around the United States and some other countries. For instance, in LA, smart meters and low-power sensors are tracking the occupancy of parking spaces across the Hollywood district, one of the most congested areas. The users will be able to access this occupancy data in order to determine the availability of the spots and then pay for them with their mobile phones. Other than the environmental benefits and lending convenience, free parking software is improving the utilization of the current parking, which lead to greater revenue for parking owners.
    These programs will be able to make great differences on a neighborhood level, but more widespread development and deployment is necessary for free parking software to change the cities and to contribute to the transportation sector pollution reductions greenhouse gas. One problem is that there are no citywide solutions all over the fragmented private and public parking providers. The occupancy data has a tendency to have many owners and is not accessible or standardized in a way that may enable software developers to turn into user-friendly applications. Thereby, individual smart parking efforts are so far successful locally, but uncoordinated, and operates in their own entrepreneurial or bureaucratic vacuums without a need to take gap between current free parking software and more widespread transportation system planning is an enormous missed opportunity for the cities to reduce the transportation related emissions.
    Moreover, free parking software has been hindered by a lack of insight into the complete benefits of this software, specifically when compared to the cost of building extra parking spaces. Lack of collaboration between communities with the parking software programs, as well as lack of coordination between hardware providers, municipalities, and developers is also contributing to the slower adoption of smart parking. Nevertheless, it is possible to overcome all these issues. Cities will be able further accelerate these advantages by means of updating the land use and building codes policies to reflect the reduced need for parking.

    1. Cities will be able further accelerate these advantages by means of updating the land use and building codes policies to reflect the reduced need for parking. hassan car

  6. I sent your articles links to all my contacts and they all adore it including me.
    house facts realty


  8. round rock web design
    web design round rock

    We're your local web design company in Round Rock & Austin, creating effective website designs that engages with your target audience & increases leads & sales. Call me >>Phone: 512-994-0191

  9. Trek removals is the best international moving company London, if you are looking Commercial Removals London, Commercial Storage London, Commercial moving company in London, Commercial space for storage purpose at very cheap price then please contact to us. Our contact no is- 0207 160 2059.
    Commercial storage London


Note: only a member of this blog may post a comment.