In my last blog I explained how Gartner Magic Quadrant (GMQ) works. In this column I set out some of the criticism that has been levelled at the GMQ. Gartner aims to deliver “the technology-related insight necessary for our clients to make the right decisions, every day” and the GMQ evaluates the technologies “best suited to the evolving needs of Gartner’s clients as buyers in the market.”
So, let’s review Gartner Magic Quadrant criticism. What are some of the concerns that have been raised about the GMQ?
Many of the criteria that Gartner uses to assess vendors are of little importance to technology buyers. The following criteria would be of more interest to investors than buyers: a clearly differentiated marketing strategy; a well-defined sales strategy; a valid business model; a distinct geographic market strategy; and successful marketing execution. If you wanted to invest in tech firms, this info could be quite important. But for technology buyers and users, there is little value in them.
Gartner is a for-profit organisation quoted on the NY stock exchange (ticker = IT). Investors in Gartner expect to make money. Gartner’s client list includes technology outfits such as IBM, SAP, Oracle, and salesforce.com. These companies also feature in many GMQ’s. This creates a prima facie conflict of interest. If Gartner delivers negative reviews of these companies’ products, there may be an adverse consequence for Gartner’s relationship with them, and the associated revenue streams. Many companies, of course, experience these sorts of conflicts that they resolve by establishing ‘Chinese walls’ that act to prevent information flow between the company teams that might be conflicted.
It isn’t always clear how Gartner collects independent information about the technologies. One source of information is the tech firms’ reference sites. These are technology buyers that have agreed to vendor requests to act as referees for other companies considering the technology. Would tech firms direct Gartner analysts to user sites where the buyer is discontent? Probably not.
Small vendors don’t get a fair representation in GMQ’s. Gartner admits that MQ’s do not provide an exhaustive analysis of every vendor in a market. Gartner says it considers a vendor’s market share, revenue, number of clients, installed base, types of products/services, target market or other defining characteristics before adding them to the dataset. Some critics assert that tech firms that have Analyst Relations people on the payroll do better. Small tech firms typically don’t have the funds for these people.
Accusations of pay-to-play. The Internet has been awash with rumors that you don’t get to feature well in GMQ’s unless you are a Gartner client. The tech firm Netscout Systems Inc. filed a 2104 legal case against Gartner asserting pay-to-play. I don’t have any personal knowledge of this very damaging assertion, and it is important to acknowledge that Gartner has consistently denied any such misdemeanor.
There is no satisfactory explanation about how the various criteria are measured and weighted. For example, how does an analyst measure Market Understanding that is defined by Gartner as “the ability of a vendor to understand buyer needs and translate these needs into products and services. A vendor that shows the highest degree of vision listens and understands what a buyer wants and needs, and can use that information to shape or enhance the relationship.” Is any analyst able to really get an intimate understanding of this issue without spending several weeks talking to a sample of clients about their relationship with the vendor, examining the history of product and service launches, and checking whether those launches are based on intuition or insight into client requirements? Gartner has reports to sell, money to make, and this sort of detailed analysis takes time and expertise. The more comprehensive the analysis, the less margin Gartner makes. In a 2009 court case (ZL Technologies, Inc. v. Gartner Group, Inc. (US District Court)), ZL claimed it has been defamed by Garter’s evaluation, but Gartner said in defence that the position of vendors in a MQ is based on “pure opinion”. Essentially, that means that there is not necessarily an objectively factual basis for the analyst’s judgement.