Updated Sentiment Analysis and a Word Cloud for Netflix - The R Way!

The Netflix investors must be happy and cheerful as the stock is up more than 78% since the beginning of the year (YES, 78%, Source: Yahoo Finance!).  I am not going to talk about what turned the stock around after a much talked/hyped about Netflix debacle of the late 2011 that earned Reed Hastings quite a few UNWANTED title and every one demanded his resignation from the top post.  Not so fast, Mr. Bear!  Reed Hastings must be smiling!  After a stellar performance this year including carefully released stats on viewership, streaming hours as well as a solid Q4'11 earnings, Netflix is back and most importantly viewers are back!

Well, is is not coincidental that the sentiment for Netflix is also improving, 68% of the tweets now have positive sentiment.  See the table below:

Total	Positive	Negative	Average	Total	Sentiment
Tweets  Fetched	Tweets	Tweets	Score	Tweets
499	171	80	0.281	251	68%

*Make sure you understand and interpret this analysis correctly. This analysis is not based on NLP. 

I updated the sentiment analysis that I did last year, http://goo.gl/fkfPy ,  (I was then just beginning to play with Twitter and Text Mining packages in R) and used advanced packages like "TM" and  "WordCloud".  The new analysis is based on more than 6,800 words which are most commonly prescribed in various sentiment analysis blogs/books. (Check out Hu and Liu http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html)

I came across this excellent blog by Jeffrey Bean, @JeffreyBean, (http://goo.gl/RPkFX) and his tutorial. Thank you Mr. Bean!  Please follow the instructions from Bean's slides and the R code listed there as well as the R code here:

Here is the updated R code snippets -
#Populate the list of sentiment words from Hu and Liu (http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html)

huliu.pwords <- scan('opinion-lexicon/positive-words.txt', what='character', comment.char=';')

huliu.nwords <- scan('opinion-lexicon/negative-words.txt', what='character', comment.char=';')

# Add some words

huliu.nwords <- c(huliu.nwords,'wtf','wait','waiting','epicfail', 'crash', 'bug', 'bugy', 'bugs', 'slow', 'lie')

#Remove some words

huliu.nwords <- huliu.nwords[!huliu.nwords=='sap']

huliu.nwords <- huliu.nwords[!huliu.nwords=='cloud']

#which('sap' %in% huliu.nwords)

twitterTag <- "@Netflix"

# Get 1500 tweets - an individual is only allowed to get 1500 tweets

 tweets <- searchTwitter(tag, n=1500)

  tweets.text <- laply(tweets,function(t)t$getText())

  sentimentScoreDF <- getSentimentScore(tweets.text)

  sentimentScoreDF$TwitterTag <- twitterTag

# Get rid of tweets that have zero score and seperate +ve from -ve tweets

sentimentScoreDF$posTweets <- as.numeric(sentimentScoreDF$SentimentScore >=1)

sentimentScoreDF$negTweets <- as.numeric(sentimentScoreDF$SentimentScore <=-1)

#Summarize finidings

summaryDF <- ddply(sentimentScoreDF,"TwitterTag", summarise, 

                 TotalTweetsFetched=length(SentimentScore),

                 PositiveTweets=sum(posTweets), NegativeTweets=sum(negTweets), 

                 AverageScore=round(mean(SentimentScore),3))

summaryDF$TotalTweets <- summaryDF$PositiveTweets + summaryDF$NegativeTweets

#Get Sentiment Score

summaryDF$Sentiment  <- round(summaryDF$PositiveTweets/summaryDF$TotalTweets, 2)

Saving the best for the last, here is a word cloud (also called tag cloud) for Netflix built in R-

I will be putting the R code up here for building a word cloud after scrubbing it.

Happy Analyzing!

Geocode your data using, R, JSON and Google Maps' Geocoding API

First and foremost, I absolutely love the topic of Location Analytics (Geo-Spatial Analysis) and see tremendous business potential in not so distant future.  I would go out on a limb to predict that the Location Analytics will soon go viral in the enterprise space because it has the capability to WOW us. Look no further than your iPhone or an Android phone and count how many location aware apps you have. We all have at lease one app - Google Maps.  Mobile is one of the strongest catalyst for enterprise adoption of Location aware apps. All right, enough of business talk, let's get dirty with the code.


Over the last year and half, I have faced numerous challenges with geocoding the data that I have used to showcase my passion for location analytics.  In 2012, I decided to take thing in my control and turned to R.  Here, I am sharing a simple R script that I wrote to geo-code my data whenever I needed it, even BIG Data.


To geocode my data, I use Google's Geocoding service which returns the geocoded data in a JSON. I will recommend that you register with Google Maps API and get a key if you have large amount of data and would do repeated geo coding.

Here is function that can be called repeatedly by other functions:

getGeoCode <- function(gcStr)

{

  library("RJSONIO") #Load Library

  gcStr <- gsub(' ','%20',gcStr) #Encode URL Parameters

 #Open Connection

 connectStr <- paste('http://maps.google.com/maps/api/geocode/json?sensor=false&address=',gcStr, sep="") 

  con <- url(connectStr)

  data.json <- fromJSON(paste(readLines(con), collapse=""))

  close(con)

#Flatten the received JSON

  data.json <- unlist(data.json)

  lat <- data.json["results.geometry.location.lat"]

  lng <- data.json["results.geometry.location.lng"]

  gcodes <- c(lat, lng)

  names(gcodes) <- c("Lat", "Lng")

  return (gcodes)

}

Let's put this function to test:

geoCodes <- getGeoCode("Palo Alto,California")

> geoCodes
           Lat            Lng 
  "37.4418834" "-122.1430195" 

You can run this on the entire column of a data frame or a data table:

Here  is my sample data frame with three columns - Opposition, Ground.Country and Toss. Two of the columns, you guessed it right, need geocoding.

> head(shortDS,10)

     Opposition              Ground.Country Toss

1      Pakistan            Karachi,Pakistan  won

2      Pakistan         Faisalabad,Pakistan lost

3      Pakistan             Lahore,Pakistan  won

4      Pakistan            Sialkot,Pakistan lost

5   New Zealand    Christchurch,New Zealand lost

6   New Zealand          Napier,New Zealand  won

7   New Zealand        Auckland,New Zealand  won

8       England              Lord's,England  won

9       England          Manchester,England lost

10      England            The Oval,England  won

To geo code this, here is a simple one liner I execute:

shortDS <- with(shortDS, data.frame(Opposition, Ground.Country, Toss,

                  laply(Ground.Country, function(val){getGeoCode(val)})))

> head(shortDS, 10)
    Opposition           Ground.Country Toss  Ground.Lat  Ground.Lng
1     Pakistan         Karachi,Pakistan  won   24.893379   67.028061
2     Pakistan      Faisalabad,Pakistan lost   31.408951   73.083458
3     Pakistan          Lahore,Pakistan  won    31.54505   74.340683
4     Pakistan         Sialkot,Pakistan lost  32.4972222  74.5361111
5  New Zealand Christchurch,New Zealand lost -43.5320544 172.6362254
6  New Zealand       Napier,New Zealand  won -39.4928444 176.9120178
7  New Zealand     Auckland,New Zealand  won -36.8484597 174.7633315
8      England           Lord's,England  won     51.5294     -0.1727
9      England       Manchester,England lost   53.479251   -2.247926
10     England         The Oval,England  won   51.369037   -2.378269

Happy Demoing and Coding!

Enterprise Software Spending to Slow Down - Business Analytics to the Rescue?

Few months ago, I floated this hypothesis that the software spending generally has a lag of 1-2 quarters to hardware spending and given that hardware spending is slowing down now with Cisco, Juniper, Brocade, EMC, NetApp, (and chip companies prior to that) all coming out with revenue and EPS warnings, software spending could slow as well further down the road.

Now, if ORCL’s  warnings from last night and following quote from an analyst were to be taken seriously, this hypothesis is unfortunately is coming true.  

                   Jason Maynard, an analyst at Wells Fargo Securities, said in a Dec. 19 report that corporate spending on hardware and software may fall 8 percent in the first quarter, a steeper drop than the average 7.3 percent average decline during the quarter in the past 10 years. (Source: Business Week )

The Enterprise Software Industry has enjoyed 12-13 quarters of continuous growth and it is a well-known fact that the spending is cyclical in nature.  May be, the industry should prepare for couple of quarters of slow growth (or no growth.) 

I am off the opinion that a full blown contraction in software spending will not occur. There is a pent up demand and those demand dollars are shifting to the cloud for SaaS, PaaS, IaaS and all other types of aaS as these XaaS become a preferred choice. That is precisely what may have caused the bloody hiccups (the reaction on Oracle's stock in financial markets) at Oracle.

This may be just an aberration for the tech industry and it may require new economy companies to prove that is just an aberration and not a trend . (Please see this blog - Oracle earnings - an aberration or a trend? )

Coming to the Analytics topic - in good times or bad times, more so in bad times, business analytics has become a tool of necessity, a must-have weapon to understand what levers to pull to run the business more effectively, more efficiently and identify the right resources to be delivered to grow and optimize the business in tough times.  


Data is a strategic asset and Business Analytics provides tactical tools to exploit that asset, companies will mine data even deeper with more sophisticated tools to get even more deeper insights if the signs of slow down loom on the horizon.

It is yet to be seen that the business spending on analytics will slow as well.  I will take a different stance here and will form another hypothesis that the spending will likely increase over the next couple of quarters.

Mobile Analytics - A Game Changer!

Mobile Analytics (a.ka. Mobile BI) has been the hottest strategic topic and a top focus for many enterprise software organizations as customers, small and large, grapple with the big data onslaught and throw everything at it to become even more efficient, both on top-line growth and bottom-line optimization, in an economy struggling to grow and a continent unable to stop a contagion from spreading and once again threatening the global economy.  

Customer's perennial struggle and in-turn a cost-saving approach translates into big analytics opportunity for enterprise software companies to shift customers from traditional analytics solutions to Mobile and Cloud based analytics solutions.

On the premise explained above, I did a business case about 9 months ago to develop a FULL picture of Mobile Analytics market.  I used a ton of research and analyst reports and interviews and invested upwards of hundreds of hours to develop and present a complete story on Mobile Analytics including developing my own proprietary models related to assessing the size of this opportunity. 

I am summarizing my findings at a very high level in following bullet points and have made available the synthesis slides on slideshare (link is printed below).

• Big Data - According to IDC, data is doubling every two years and is expected to reach 1.8 ZB (a trillion GB) in 2011.
• Eight mobility related mega trends are locked in a virtuous cycle and will be the bedrock for growth and adoption of Mobile BI solutions and for  the overall Enterprise Mobility.
• Mobile BI market could grow at 20% plus CAGR over the next 5 years and could likely become over a $2 billion market by 2015.
• According to Gartner, more than 33% of Analytics will be consumed using mobile devices, a prediction well supported by the 8 mobility related mega trends discussed here.
• Therefore, by 2015 more than 15% of Analytics revenues could come from Mobile Analytics solutions. This should be a serious strategic priority for every Analytics vendor if not already.
• Advanced Analytics including geo-spatial for  Mobile Consumers is growing as computing power and form factor of mobile devices change rapidly. 
• Shift to “active production model” from a “passive consumption model”  is expected to happen allowing mobile business users to assemble dashboards and produce/edit reports on the go.

Download slides from Slideshrae - Mobile Analytics (Mobile BI) - A Game Changer

Special thanks goes to Gartner, IDC, Boris Evelson of Forrester, Cindy Howson of BI Scorecard, and Howard Dresner of Dresner Advisory Services for producing excellent research on this topic and answering all my questions and to all my colleagues and friends across the world. 

Upcoming blog on Agile Analytics

Closing the loop on Pervasive Location Analytics - an enlightening personal journey for sure!

When I started working on Google Maps deal at SAP in February of this year, I had no clue where it will end and what is next once the deal is done. I fell in love with this Location Analytics/Geo Data Visualization topic, and turned it into an opportunity to discuss this topic and also generate excitement in various different camps along the way.

Five sessions spread across three continents, 200+ attendees,1000 views and numerous downloads later, this topic became more than just a personal interest. I met great people along the way and worked with very smart and driven people to co-present from the likes of Ryan from Centigon Solutions, Nimish from FreshDirect and Brendan from ThinkSmart Technologies. (See links to slides and session evaluation below)

A proud moment arrived this morning when an alert from SlideShare popped up indicating that this topic is hot on Facebook and as a result this topic is being put on SlideShare home page. Wow!


Pervasive Location Analytics: The Next Frontier to Fall in The Enterprise Software?

Session Evaluations Results

Thank you - my next two blogs will be presenting my thoughts on Moblie Analytics and Agile BI - two topics I have spent significant amount of time from strategy, market, customer, competition and product point of view.

Tale of Two Companies - SFSF and RNOW - Why would anyone compare SAP-SuccessFactors deal with Oracle-RightNow deal?

First and foremost, a masterstroke from SAP, I generally don't say that but this is a very smart and timely move. Read my other blog on why this a solid grab by SAP here

Facts: 

• SAP is proposing to pay $3.4 B to acquire SuccessFactors(SFSF), a multiple of 10.2 on expected 2011 revenue of $332M.  
• Oracle paid $1.4B to acquire RightNow (RNOW), a multiple of 6.2 on expected 2011 revenue of $226M.

Since Saturday, every other person is commenting that SAP overpaid including this article in WSJ.

Now what my friends in other circuits don't do is to double click on the deal itself which I did in my previous blog on the business rationale. In this blog, I will use a set of visuals to illustrate that SFSF is a far superior pick on financials. Let's start and discuss tale of two companies:

Tale of Two Companies: SFSF is a better revenue story with CAGR more than DOUBLE than that of RNOW:

SFSF is a far better growth story than RNOW:

SFSF has far better cost structure than RNOW even though SFSF has grown revenues more than TWICE as fast:

And my last point – SFSF has better operating structure and is rapidly becoming more efficient with every dollar it spends on its operating cost:

Both the growth in revenue and 15m subscriber base across the globe has come at a cost in net income but it is very quickly turning around: 

I hope that my friends can withdraw their criticism because both qualitatively and quantitatively this is an astute move from SAP.  Making money from cloud apps has been tough but this is very quickly starting to change. As always, time will tell who read this right! 

SuccessFactors - An amazing tech story through its financials and a solid grab by SAP!

I will take a slight detour from Analytics and talk about SAP's acquisition of  #1 cloud company SuccessFactors (SFSF). Announcement

The combination of SAP & SFSF will produce a cloud powerhouse in the cloud segment of the enterprise software market  and that is just starting to take off…

Strong business rationale:

·         Gartner - HCM to be a $10B by 2015, Talent Management alone will be a $4.5B with 75% of it coming from cloud based apps

·          SFSF is:

o    #1 HCM solution in the cloud

o   has 15m users from company of all sizes (CRM has only 3m users) in diverse 60 industries from across the globe (Example: Siemens has 450K seats)

o    60% recurring revenues from existing customers

o    90% of the growth is organic as oppose to Salesforce

o   Has just 14% overlap with SAP customers – a tremendous upside for both companies (with total addressable market of 500m employees of all SAP customers)

·         For SAP, SFSF will be a top-line acquisition with less emphasis on cost-synergies…

·         Deal will be slightly dilutive on EPS in 2012 but will be accretive in 2013 with significant upside to our revenues in 2013

Financials:

• SAP paid $3.4 B to acquire SFSF which is not profitable yet.
• SAP is paying ~10x for 2011 revenues, a multiple HP paid for Autonomy
• For SFSF, street expects $332M in 2011 revenues; SFSF had $230M YTD revenues for the first nine months with $91M coming in Q3’11
•   As of Sep, 2011, SAP had $5.2B in cash. The SFSF deal is all cash with $2B coming off SAP's own war chest and ~$1.4B of debt. 

 Taleo with 2011 expected revenues of $324M is barely profitable. Workday is on track to $320 million in billings in 2011, and is nearing profitability. Workday is preparing for an IPO.

Now let us talk about SFSF’s amazing growth over the past 9 years:

SFSF – a company which delivered a PERFECT hockey stick growth since 2002:

A revenue growth story that is enviable:

Operating structure has shown substantive improvement over the past 5 years:

Net net for SAP, a solid acquisition and timing couldn’t have been right. The ride has just begun…

Source: Company Financials and Analyst Calls