Geocode your data using, R, JSON and Google Maps' Geocoding API

First and foremost, I absolutely love the topic of Location Analytics (Geo-Spatial Analysis) and see tremendous business potential in not so distant future.  I would go out on a limb to predict that the Location Analytics will soon go viral in the enterprise space because it has the capability to WOW us. Look no further than your iPhone or an Android phone and count how many location aware apps you have. We all have at lease one app - Google Maps.  Mobile is one of the strongest catalyst for enterprise adoption of Location aware apps. All right, enough of business talk, let's get dirty with the code.


Over the last year and half, I have faced numerous challenges with geocoding the data that I have used to showcase my passion for location analytics.  In 2012, I decided to take thing in my control and turned to R.  Here, I am sharing a simple R script that I wrote to geo-code my data whenever I needed it, even BIG Data.


To geocode my data, I use Google's Geocoding service which returns the geocoded data in a JSON. I will recommend that you register with Google Maps API and get a key if you have large amount of data and would do repeated geo coding.

Here is function that can be called repeatedly by other functions:

getGeoCode <- function(gcStr)

{

  library("RJSONIO") #Load Library

  gcStr <- gsub(' ','%20',gcStr) #Encode URL Parameters

 #Open Connection

 connectStr <- paste('http://maps.google.com/maps/api/geocode/json?sensor=false&address=',gcStr, sep="") 

  con <- url(connectStr)

  data.json <- fromJSON(paste(readLines(con), collapse=""))

  close(con)

#Flatten the received JSON

  data.json <- unlist(data.json)

  lat <- data.json["results.geometry.location.lat"]

  lng <- data.json["results.geometry.location.lng"]

  gcodes <- c(lat, lng)

  names(gcodes) <- c("Lat", "Lng")

  return (gcodes)

}

Let's put this function to test:

geoCodes <- getGeoCode("Palo Alto,California")

> geoCodes
           Lat            Lng 
  "37.4418834" "-122.1430195" 

You can run this on the entire column of a data frame or a data table:

Here  is my sample data frame with three columns - Opposition, Ground.Country and Toss. Two of the columns, you guessed it right, need geocoding.

> head(shortDS,10)

     Opposition              Ground.Country Toss

1      Pakistan            Karachi,Pakistan  won

2      Pakistan         Faisalabad,Pakistan lost

3      Pakistan             Lahore,Pakistan  won

4      Pakistan            Sialkot,Pakistan lost

5   New Zealand    Christchurch,New Zealand lost

6   New Zealand          Napier,New Zealand  won

7   New Zealand        Auckland,New Zealand  won

8       England              Lord's,England  won

9       England          Manchester,England lost

10      England            The Oval,England  won

To geo code this, here is a simple one liner I execute:

shortDS <- with(shortDS, data.frame(Opposition, Ground.Country, Toss,

                  laply(Ground.Country, function(val){getGeoCode(val)})))

> head(shortDS, 10)
    Opposition           Ground.Country Toss  Ground.Lat  Ground.Lng
1     Pakistan         Karachi,Pakistan  won   24.893379   67.028061
2     Pakistan      Faisalabad,Pakistan lost   31.408951   73.083458
3     Pakistan          Lahore,Pakistan  won    31.54505   74.340683
4     Pakistan         Sialkot,Pakistan lost  32.4972222  74.5361111
5  New Zealand Christchurch,New Zealand lost -43.5320544 172.6362254
6  New Zealand       Napier,New Zealand  won -39.4928444 176.9120178
7  New Zealand     Auckland,New Zealand  won -36.8484597 174.7633315
8      England           Lord's,England  won     51.5294     -0.1727
9      England       Manchester,England lost   53.479251   -2.247926
10     England         The Oval,England  won   51.369037   -2.378269

Happy Demoing and Coding!

Enterprise Software Spending to Slow Down - Business Analytics to the Rescue?

Few months ago, I floated this hypothesis that the software spending generally has a lag of 1-2 quarters to hardware spending and given that hardware spending is slowing down now with Cisco, Juniper, Brocade, EMC, NetApp, (and chip companies prior to that) all coming out with revenue and EPS warnings, software spending could slow as well further down the road.

Now, if ORCL’s  warnings from last night and following quote from an analyst were to be taken seriously, this hypothesis is unfortunately is coming true.  

                   Jason Maynard, an analyst at Wells Fargo Securities, said in a Dec. 19 report that corporate spending on hardware and software may fall 8 percent in the first quarter, a steeper drop than the average 7.3 percent average decline during the quarter in the past 10 years. (Source: Business Week )

The Enterprise Software Industry has enjoyed 12-13 quarters of continuous growth and it is a well-known fact that the spending is cyclical in nature.  May be, the industry should prepare for couple of quarters of slow growth (or no growth.) 

I am off the opinion that a full blown contraction in software spending will not occur. There is a pent up demand and those demand dollars are shifting to the cloud for SaaS, PaaS, IaaS and all other types of aaS as these XaaS become a preferred choice. That is precisely what may have caused the bloody hiccups (the reaction on Oracle's stock in financial markets) at Oracle.

This may be just an aberration for the tech industry and it may require new economy companies to prove that is just an aberration and not a trend . (Please see this blog - Oracle earnings - an aberration or a trend? )

Coming to the Analytics topic - in good times or bad times, more so in bad times, business analytics has become a tool of necessity, a must-have weapon to understand what levers to pull to run the business more effectively, more efficiently and identify the right resources to be delivered to grow and optimize the business in tough times.  


Data is a strategic asset and Business Analytics provides tactical tools to exploit that asset, companies will mine data even deeper with more sophisticated tools to get even more deeper insights if the signs of slow down loom on the horizon.

It is yet to be seen that the business spending on analytics will slow as well.  I will take a different stance here and will form another hypothesis that the spending will likely increase over the next couple of quarters.

Closing the loop on Pervasive Location Analytics - an enlightening personal journey for sure!

When I started working on Google Maps deal at SAP in February of this year, I had no clue where it will end and what is next once the deal is done. I fell in love with this Location Analytics/Geo Data Visualization topic, and turned it into an opportunity to discuss this topic and also generate excitement in various different camps along the way.

Five sessions spread across three continents, 200+ attendees,1000 views and numerous downloads later, this topic became more than just a personal interest. I met great people along the way and worked with very smart and driven people to co-present from the likes of Ryan from Centigon Solutions, Nimish from FreshDirect and Brendan from ThinkSmart Technologies. (See links to slides and session evaluation below)

A proud moment arrived this morning when an alert from SlideShare popped up indicating that this topic is hot on Facebook and as a result this topic is being put on SlideShare home page. Wow!


Pervasive Location Analytics: The Next Frontier to Fall in The Enterprise Software?

Session Evaluations Results

Thank you - my next two blogs will be presenting my thoughts on Moblie Analytics and Agile BI - two topics I have spent significant amount of time from strategy, market, customer, competition and product point of view.

Best quotes from Forbes Article SAP-Google Maps Partnership

SAP Partnership with Google Maps Indicates a New Openness

"SAP is overcoming a legacy of an insular engineering culture that could accurately be accused of suffering from a “not invented here” complex in the past."


"The only problem with SAP’s pride in its history is that it has sometimes shut the company’s eyes to new ways of creating software. It appears that this announcement may mark a turning point to increased awareness and use of outside components. If SAP becomes truly open to using more and more outside components, and learns how to use them to create stable, reliable software, SAP could accelerate the pace of change, keeping the stable parts of its applications, but adding the best of what has newly arrived."


There is something more at stake for Google than money - it is likely that Google Maps will be adapted by Google to better meet the needs of enterprise applications.


"What’s next? What other cloud components will SAP start to incorporate? The second is: Is a bigger partnership possible? Google’s mission is “to organize the world’s information and make it universally accessible and useful.” Much of the information that runs the world is in SAP. Why aren’t Google and SAP working together to make it more universally accessible and useful?"