Geocode your data using, R, JSON and Google Maps' Geocoding API

First and foremost, I absolutely love the topic of Location Analytics (Geo-Spatial Analysis) and see tremendous business potential in not so distant future.  I would go out on a limb to predict that the Location Analytics will soon go viral in the enterprise space because it has the capability to WOW us. Look no further than your iPhone or an Android phone and count how many location aware apps you have. We all have at lease one app - Google Maps.  Mobile is one of the strongest catalyst for enterprise adoption of Location aware apps. All right, enough of business talk, let's get dirty with the code.


Over the last year and half, I have faced numerous challenges with geocoding the data that I have used to showcase my passion for location analytics.  In 2012, I decided to take thing in my control and turned to R.  Here, I am sharing a simple R script that I wrote to geo-code my data whenever I needed it, even BIG Data.


To geocode my data, I use Google's Geocoding service which returns the geocoded data in a JSON. I will recommend that you register with Google Maps API and get a key if you have large amount of data and would do repeated geo coding.

Here is function that can be called repeatedly by other functions:

getGeoCode <- function(gcStr)

{

  library("RJSONIO") #Load Library

  gcStr <- gsub(' ','%20',gcStr) #Encode URL Parameters

 #Open Connection

 connectStr <- paste('http://maps.google.com/maps/api/geocode/json?sensor=false&address=',gcStr, sep="") 

  con <- url(connectStr)

  data.json <- fromJSON(paste(readLines(con), collapse=""))

  close(con)

#Flatten the received JSON

  data.json <- unlist(data.json)

  lat <- data.json["results.geometry.location.lat"]

  lng <- data.json["results.geometry.location.lng"]

  gcodes <- c(lat, lng)

  names(gcodes) <- c("Lat", "Lng")

  return (gcodes)

}

Let's put this function to test:

geoCodes <- getGeoCode("Palo Alto,California")

> geoCodes
           Lat            Lng 
  "37.4418834" "-122.1430195" 

You can run this on the entire column of a data frame or a data table:

Here  is my sample data frame with three columns - Opposition, Ground.Country and Toss. Two of the columns, you guessed it right, need geocoding.

> head(shortDS,10)

     Opposition              Ground.Country Toss

1      Pakistan            Karachi,Pakistan  won

2      Pakistan         Faisalabad,Pakistan lost

3      Pakistan             Lahore,Pakistan  won

4      Pakistan            Sialkot,Pakistan lost

5   New Zealand    Christchurch,New Zealand lost

6   New Zealand          Napier,New Zealand  won

7   New Zealand        Auckland,New Zealand  won

8       England              Lord's,England  won

9       England          Manchester,England lost

10      England            The Oval,England  won

To geo code this, here is a simple one liner I execute:

shortDS <- with(shortDS, data.frame(Opposition, Ground.Country, Toss,

                  laply(Ground.Country, function(val){getGeoCode(val)})))

> head(shortDS, 10)
    Opposition           Ground.Country Toss  Ground.Lat  Ground.Lng
1     Pakistan         Karachi,Pakistan  won   24.893379   67.028061
2     Pakistan      Faisalabad,Pakistan lost   31.408951   73.083458
3     Pakistan          Lahore,Pakistan  won    31.54505   74.340683
4     Pakistan         Sialkot,Pakistan lost  32.4972222  74.5361111
5  New Zealand Christchurch,New Zealand lost -43.5320544 172.6362254
6  New Zealand       Napier,New Zealand  won -39.4928444 176.9120178
7  New Zealand     Auckland,New Zealand  won -36.8484597 174.7633315
8      England           Lord's,England  won     51.5294     -0.1727
9      England       Manchester,England lost   53.479251   -2.247926
10     England         The Oval,England  won   51.369037   -2.378269

Happy Demoing and Coding!

Enterprise Software Spending to Slow Down - Business Analytics to the Rescue?

Few months ago, I floated this hypothesis that the software spending generally has a lag of 1-2 quarters to hardware spending and given that hardware spending is slowing down now with Cisco, Juniper, Brocade, EMC, NetApp, (and chip companies prior to that) all coming out with revenue and EPS warnings, software spending could slow as well further down the road.

Now, if ORCL’s  warnings from last night and following quote from an analyst were to be taken seriously, this hypothesis is unfortunately is coming true.  

                   Jason Maynard, an analyst at Wells Fargo Securities, said in a Dec. 19 report that corporate spending on hardware and software may fall 8 percent in the first quarter, a steeper drop than the average 7.3 percent average decline during the quarter in the past 10 years. (Source: Business Week )

The Enterprise Software Industry has enjoyed 12-13 quarters of continuous growth and it is a well-known fact that the spending is cyclical in nature.  May be, the industry should prepare for couple of quarters of slow growth (or no growth.) 

I am off the opinion that a full blown contraction in software spending will not occur. There is a pent up demand and those demand dollars are shifting to the cloud for SaaS, PaaS, IaaS and all other types of aaS as these XaaS become a preferred choice. That is precisely what may have caused the bloody hiccups (the reaction on Oracle's stock in financial markets) at Oracle.

This may be just an aberration for the tech industry and it may require new economy companies to prove that is just an aberration and not a trend . (Please see this blog - Oracle earnings - an aberration or a trend? )

Coming to the Analytics topic - in good times or bad times, more so in bad times, business analytics has become a tool of necessity, a must-have weapon to understand what levers to pull to run the business more effectively, more efficiently and identify the right resources to be delivered to grow and optimize the business in tough times.  


Data is a strategic asset and Business Analytics provides tactical tools to exploit that asset, companies will mine data even deeper with more sophisticated tools to get even more deeper insights if the signs of slow down loom on the horizon.

It is yet to be seen that the business spending on analytics will slow as well.  I will take a different stance here and will form another hypothesis that the spending will likely increase over the next couple of quarters.

Growing Bigger - QlikView Faces Headwinds? One Analysts Thinks That is the Case.

QlikView (Ticker: QLIK)  has been a poster child of BI since 2006 growing at a record double digit pace and becoming the envy of many large BI players (by not direct competing with them but still out-pacing on sales and customer growth). Many executives at large firms (un)proudly speak of QlikView as a strong competitor. Pretty good achievement!

 Take a look at the following graphics below that captures all company vitals - 

As QLIK grows, it faces an interesting challenge as faced by every company standing at the gates on an inflection point - transitioning to a larger company. Here are some of the issues identified by identified by Gartner for QLIK - ( I am pretty confident that QLIK will continue to execute well in near-term but it definitely has to resolve all of these issues mentioned below) 

QlikTech offers limited metadata management. As QlikView grows into larger BI deployments spanning the enterprise, the lack of an enterprise semantic layer becomes a more pressing issue. Filling this gap requires additional cost and effort in the management of metadata to lock down common definitions and calculations, and to conform dimensions for cross-functional analysis across QlikView applications.

QlikTech's focus on analysis and usability for end users delivers significant advantages. However, its lack of a number of broad BI platform capabilities (high volume enterprise reporting, planning/financially oriented OLAP, Microsoft Office integration, scorecarding and predictive modelling) means that it will almost always need to be used alongside another BI platform.

QlikView is increasingly seen as expensive — almost a third of its customers surveyed (31.4% vs. 26.1% in the whole sample) see this as its main barrier to wider use. Its pricing model often does not sit well with larger deployments to more users, nor does the investment in RAM required to support the increasing numbers of concurrent users.