How to Use Web Scraping to Understand Your Market

Referral programs are a central part of the growth plan for many SaaS companies. They are also a staple for any ‘growth hacker’ when they are trying to create an environment for viral growth.

Referral programs are everywhere, from the famous startup examples of Paypal and Dropbox, all the way to phone apps like Candy Crush.

If your SaaS company has a referral system you are certainly following a best practice, but are you really getting value from it?

Well depending on the type of SaaS you are running you are going to have very different success metrics than other companies. Some companies will focus on the number of sign-ups created, while others focus on metrics like activation and installation rates.

One of the main challenges that come with referral programs is attributing the overall growth of your company to your newly referred users considering there are many different measures of success. For instance, when you start matching up referred user data with average user data(those acquired from other channels) you can see increases like higher average customer lifetime value and stronger trial to paid conversion rates.

In this article, I go over how you can measure market share of a public-facing widget like an E-Commerce tool and how you can use that data to measure against your other growth initiatives. Helping you understand how your growth stacks up against your competitors and which one of them you should keep a closer eye on.

Ways to Measure Market Share

There are a couple of ways to measure market share: Percent of industry revenue and Percent of industry unit sales. These are great measurements but it can take quite a while to get real data on how your industry is doing as a whole.

Services like BuiltWith can provide a lot of valuable data, but are also expensive. Another benefit with this service is the amount of historical data you get in their reports. You can take a look at the markets’ trends and use that information to position yourself as you establish a foothold.

Most of the information they are using is on the web, So why not go get the data yourself?

I use web scraping to find market share quickly. If your SaaS product is one with a public-facing Javascript tag there are ways that we can scrape the web to measure how many installs you and your competitors have.

Scraping across the live web at a massive scale is unrealistic, so finding a service that has a cached copy of the web helps. Zillabyte is helpful because we have copies of the web and have the machines to let you crawl over millions of web pages in a few hours.

Basics of Scraping

If you are going to use web scraping to track market share growth you need to know the scraping tools out there.  My go-to is Nokogiri, a scraper and XML parser in Ruby. Beautiful Soup is great if you know Python.

If you need to have a more interactive scraper use PhantomJS. As its’ name implies, it is written in Javascript. If you scrape using Zillabyte we have several examples and currently support Ruby and Python. Zillabyte uses a scripting language that makes it easy to use multiple tools on your scraper, and lets it run in parallel over our machines.

Getting The Data

In this app we can scrape Zillabytes’ pre-cached copies of the web to count the amount of installed public-facing javascript widgets for your market. Scraping this data every week or month can give you a great sense of how you and your competitors are growing their products’ installation base.

require ‘zillabyte’

require ‘csv’

app = Zillabyte.app “ecommerce_tools”

input = app.source “select * from web_pages”

tool_use_stream = input.each{

  prepare do

   @cat_snippets = CSV.parse(File.open(‘widgets.csv’), :headers => ["company","snippet"])

  end

  execute do |page|

   @cat_snippets.each do |row|

    if page["html"].to_s.include?(snippet.downcase)

     emit :url => page["url"], :company => company

    end

   end

  end

} 
tool_use_stream.sink do

 name “ecommerce_tool_finder”

 column “url”, :string

 column “company”, :string

end 

Just populate a CSV file with your competitors’ widgets so it knows what to scrape for and then you can correlate your new users with your list of referred users.

For a more detailed guide check out our documentation. Run the app as often as you want. Our cached copies of the web are refreshed weekly, so you can track your market share closely.

Using Your Data

Now you have a few ways to measure how much you and your competitors are growing. Additionally, you can look to segment this data against your own analytics to see which growth initiatives are leading to active installs.

One method is to segment your competitors’ user base.  By segmenting them you can see which user each company best serves. As you acquire new users you can see if you are pulling users from particular segments. How you segment users is best left to you, as nobody knows your customers like your team.

Another way is to look at the market share. By checking the distributed market share on a monthly, weekly or daily basis you can get a strong sense of how you and your competitors’ are growing(by number of installed customers). This will be helpful as you can keep tabs on how fast each company is growing and can warrant some additional snooping to their specific tactics.

At the end of the day these are just tools that give you more data and attribution to your overall growth. What you do with the data is up to you. If you have questions the team at Referral SaaSquatch has great ideas that can help push you to the next level.

Image courtesy of bplanet / freedigitalphotos.net

Christian Block works as a growth marketer at Zillabyte – A cloud platform for data analysis. He lives in San Francisco and enjoys learning about web scraping and content marketing.