When the 2015 Compass Global Ecosystem Rank report came out, many were disappointed to learn that Toronto had dropped to 17th, down from 8th in the 2012 report. And that Waterloo fell out of the top 20 entirely. At MaRS Data Catalyst, we intimately know the startup data in Ontario, so we decided to pull back the covers on the data analysis to help others understand why.

The details matter

When comparing the changes in rankings between years, we logically assume the methodology and data collection is consistent in each report.  However, in the Compass reports, there were significant changes between 2012 and 2015. The data collection methodology used in the 2012 report relied heavily on sources with great coverage in the Silicon Valley and other U.S. cities, but less coverage in other cities across the globe.  At the time, Toronto actually had good coverage in those databases compared to most other cities outside the U.S. In 2015 however, the data sources better represented cities outside the U.S., introducing into the rankingsthose cities whose data was previously not properly considered.  So as their source data expanded and improved, the previously underrepresented cities climbed while the well represented cities from 2012 “corrected” in a sense.

In addition to the change in source data, we also noted that in 2015, Compass changed their report methodology. A significant parameter impacting our rankings is that they no longer include Startup Density (the number of startups per capita) in their performance index.  With this change, Waterloo – a globally renowned engineering powerhouse with one of the highest per-capita densities of tech startups in the world – tumbled out of the rankings. Compass also changed the way they calculated the number of startups in a region to be more accurate in the 2015 report, allowing cities that ranked lower in the last report to rise up due to better data. Curiously, Waterloo is counted separately from Toronto while cities like Berkeley and Livermore are included in Silicon Valley. Compass claims to use a 100km city radius in their methodology and Waterloo and Toronto are within range of each other.

But doesn’t Toronto have a strong presence in the startup scene?

Yes, and this brings up another very important detail affecting Toronto’s rankings.  In order to be comparable with their largest data set (Silicon Valley IT tech startups), Compass specifically excludes biotech and cleantech startups. In Toronto, almost half of the startups and over two-thirds of the venture capital that supports them comes from those two sectors. Since the 2012 report, the biggest increase in startup funding volume in Toronto has come from cleantech ventures. So, when Compass calculated Toronto’s Performance Index, which is heavily influenced by funding events and exit valuations, this increased funding volume didn’t get considered.

There is the old adage that says that things are never as good, or as bad, as they seem and the 2012 and 2015 rankings are textbook examples of both.  Although Toronto cheered and congratulated themselves on their 8th place ranking in 2012, we now understand that due to limited data sets, that was an overestimate. Likewise, Toronto shouldn’t beat themselves up over their 2015 ranking because the strong performance of their diverse ecosystem wasn’t fully captured. But this doesn’t mean that Toronto can rest on its laurels.  The pace of startup entrepreneurship is accelerating around the world, and Toronto needs to keep the pressure on themselves in order to compete at a global level.

Don’t hang your hat on a single index

Let’s be clear — Compass has a rigorous methodology and their research team involves some of the biggest brains in entrepreneurship research.  The fact is that their rankings measure against Silicon Valley’s IT-centric startups, and cities like Toronto, which also develop diversified startups in biotech, cleantech and advanced materials will may never rank well in an index that singles out one sector.

Do we need a better index?  That’s a fair question. As humans we generally dislike complexity. So, when it comes to measuring complex things like entrepreneurial ecosystems, we like when it can be simplified for us.  And we really like it when it can be summed up in a single index. The problem is that the more diversity there is in the profiles and sectors of the underlying ventures, the less useful a single index becomes for comparing them.  Looking at multiple indicators and constantly improving the underlying quality of the data is the only way to really understand and assess the performance of a startup ecosystem.  It’s often said that startups are hard.  These reports and the underlying interpretation of them proves that measuring them is just as hard.