Deciphering the genome of ... startups!

Mapping and exploring Data


The initial survey created by Max and Bjoern and the rest of the StartupGenome team was exploratory in nature. It contained a long list of questions covering many areas that can describe a startup to an outside observer.
In more formal language, the survey took a static view of the world and tried to capture a snapshot of a startup's developmental stage using a list of questions. The team had understood that a static view will probably provide very limited actionable information for startups, since the current stage of a startup does not contain all of its historical information. We had, however, to start somewhere.

The goal of the survey was to give us enough information to formulate a model that we can later focus on and test in future surveys. The future surveys are planned to take a more dynamic view of startup development. Although the SG team had an initial model in mind, it was still unclear if the model is correct, and what the answers to the questions would look like.

The first thing we had to do, given survey answers, was to describe them in a succinct and parsimonious manner. (On a side note, did you ever notice how people like to use "parsimonious" to say "simple" in a non-simple manner?).

The result of describing the data is called "descriptive statistics". The typical descriptive statistics are averages, standard deviations and counts of various aspects of the data. In our process we created a huge amount of different cuts, views, cross tabulations, graphs, means and statistical significance tests to check relationships and correlations in our data. Once we had those, we leaned back, looked at them from afar and tried to ask ourselves "do these results make sense".
The sanity check stage was very important, it showed us where we had inaccuracies and other problems in the data, where the people who answered the survey may have misunderstood the questions, and also where initial correlations show a promising direction for deeper research.

An interesting question was what tool to use to perform data analysis. As uncool as it may be, I still believe Microsoft Excel (yes, even the Mac version) is by far the most powerful tool to work with with raw data if the amount is not too large (up to 1 million rows of data or so).
It lets you easily filter, change, transform, graph, cross tabulate and do many other tasks very simply.
Another option would have probably been using SPSS for that manner.
Google spreadsheets is getting there, but until a week ago had no pivot table support, which was our main tool for generating insights and reports.

Some interesting things we noticed in this process were:

• There was a lot of useful information that could be gleaned even from the simple descriptive statistics. When we showed it to outsiders, the typical response was "wow, that is something I wish I had known before".
• Some of the survey questions could have been answered in several different ways for the same answer. The main reason was the different startups viewed their market potential, and potential customers in a subjective manner which can be interpreted in different ways. This gave rise to the concept of a startup "Type" which says that startups belong to underlying families. Startups within a family seem similar, but they behave very differently across families. This is one of the main themes in our initial report.
• It is very hard to tell from a static snapshot of a startup whether it is successful or not. That is, even given all of the details about the founders, their current stage, whether they raised funding and whether their product looks like it has a good fit with the market, it is still hard to differentiate a winning from a struggling startup without knowing more of how it evolves over time. One conclusion is that the current investment process in startups is done "half blindly" because of the tendency to do a static analysis.

Modeling


Once we had our descriptive statistics, we wanted to start and see if we can tell which startups face more difficulties than others, and whether we can create measures that startups can use to benchmark themselves against best practices.
I remember that in our first day of working on the data, Max had a pretty solid understanding of the stages model and what is the current common knowledge in industry about what stages startups go through, and how. I served as a sort of "devil's advocate" and "question guy", constantly asking questions about the model, and whether answers from the surveys could be interpreted as a one-to-one matching to the different stages in the model. Another thing that had to be done was transform the abstract stages model into something mathematical, or something measurable. Markov chains came as a natural model, and more on them is described later.

We then looked for questions in the data that will identify a stage. Identification (in the academic sense) has a very specific meaning, but intuitively, it says that a specific piece of observed answer or data can only mean a specific value of some unobserved information.
To put things more concretely, in our case the observed data are the answers to questions to the survey, and the unobserved information is the stage a company is in.
Here's an illustrative example (but it's not exactly what we did):
When firms were supposedly in stage 3 or above in the SG model (known as the Efficiency stage), they were supposed to have already achieved a fit between their product and their market, and were also supposed to scale their sales and marketing by pouring money into it.
As a result, we used to questions analyzing the product/market fit and marketing expenditures to determine whether a startup was in stage 3.
We performed this process for every stage in the model, and for each startup we had a Yes/No answer to whether they passed through each of the stage.

We also calculated a "score" for each firm, depending on its recorded answers and clustered firms according to similar score areas, and hence received the aforementioned startup "Personality Types".

Suddenly, we had a serendipitous moment.

We noticed some startups didn't go through all of the stages (according to their answers). We called these startups "inconsistent" and went to their detailed answers to see what inconsistency means. We found out that inconsistent startups were much more likely to lag behind and face difficulties in their developments. Inconsistency looked like a good predictor (or correlator) with a struggling startup.

[Source: StartupGenome Report]

This was an initial indication that the stages model makes sense as a view of the world.
The second thing we did regarding our modeling is assuming our stages model is correct. As a result, we received a model where a (consistent) firm can only move from one stage to the other sequentially. The entire model can then be seen as a Markovian process, or specifically, a POMDP.

If I were to explain what a POMDP is in one paragraph, think of a person making decisions which you observe, but you do not observe the underlying information the person has. Since the person uses the unobserved information to make decisions, one cannot analyze the chain of decisions at face value, but rather needs to have a "belief" about what the decision maker knows and does not know. If we then assume that startups of the same personality type are similar (homogeneous), we can assume the date we observe in the world is an equilibrium of sorts of a process for generating startups and having them move through stages.

All this babble mean we can treat the data for different startups in the same type as if it was almost the same startup in different stages, and reach some nice conclusions.

To simplify things, however, we just assumed we "know" the underlying stage (given the answers to survey questions), and tried to see what actions carried on by entrepreneurs help them move through actions.
Actions, as an example, can be hiring employees, pivoting more, raising money, consulting mentors and more. They do not indicate the stage of a startup (in our model), but can help a startup advance in its stage.

The technical method we used to estimate this model is a "simple" ordered probit, where firms generate "value" which is then converted to a stage by an outside observer. The software (for the geeky oriented) was Stata.
Our initial results seem promising and will be published once we formulate their presentation style (they are complex to read) and also finish all of the statistical testing.

Analysis


In many of my previous encounters with big data analysis projects, both in my VC years as well as in other positions, I noticed that most people stop after step 1, some continue to step 2, and almost no one does the important job - that of step 3.
Step 1 and 2 are descriptive steps - they tell us how the world behaves and what it looks like. It is interesting, and as people, we love reading graphs, seeing statistics and digesting information.
The reason (and this is purely a conjecture of mine) is that as people, we do not like uncertainty, and statistics brings some order to how we grasp the world.

The main question, I think, is what you do with the statistics and the data. The goal is to provide actionable results for startups, and although we have initial results, we still have a lot of work to do.
The analysis step is (in my opinion) where the true creativity of the team will come out. What we do is first take a step back, and look at the results. We analyze if they make sense and what they mean.
Then we ask a simple question: "What would happen if we actively changed something, or did some action". To test this, we can run a survey, perform an experiment, or run a simulation assuming our model is correct.

I guess our next step with the project is to focus more and more on the modeling and analysis phases, until we will reach the exciting conclusions we aim for.

If you've reached this far down the post, I hope you enjoyed it. As a summary, just remember that every data analysis task, as daunting as it may look has only 3 typical steps, and that the best results, in my opinion, are achieved only when you get past the Analysis phase.

A note on statistics: Many of the results presented in the report are based on somewhat small samples, mostly for the personality types. The majority are statistically significant using simple tests. We have tried to specify all the interesting cases where conclusions might not be statistically significant. During our work we have refined many questions in the surveys and are now collecting more data to validate the results and make inference more accurate.

More is yet to come. To keep tabs on future results, just sign-up for an email update from this blog, or follow me on Twitter.