AIMultiple Benchmark Methodology & Its Rationale

AIMultiple benchmarks aim to make relevant measurements in a transparent and objective manner.

Transparent, data driven benchmarks of product performance are rare. Legacy industry analysts like Gartner and Forrester relied on opaque assessments where only these pieces of data were published:

– high-level qualitative (i.e. market understanding) and quantitative criteria that products would be evaluated against¹

– high-level assessments of these criteria without disclosing the values driving the assessment

These assessments relied on data that was provided by vendors which have undisclosed commercial relations with analysts.

Therefore the results are subject to numerous issues such as:

• Analyst bias: Analysts would evaluate vendor representatives’ responses including qualitative responses. Vendor representatives with commercial relationships with the industry analyst have the chance to build relationships with them by scheduling calls over the year. However, vendor representatives without such commercial relationships would present their product over a single call.
• Conflict of interest: For these assessments, vendor representatives would be asked questions about their private data (e.g. revenues, features, roadmap etc.). Since it would be clear which responses lead to better outcomes for the vendor (e.g. higher product revenues are likely to result in a higher rank), vendor representatives face a conflict of interest.

How does AIMultiple ensure objectivity?

To ensure that AIMultiple does not favor any solution and does not rely on other sources of income to run the benchmark: Each participant will

• pay a participation fee
• provide free access to their solution during the assessment.

AIMultiple will support the objectivity of its work with transparency since it is likely to reduce corruption and improve quality of results.

How does AIMultiple ensure transparency?

Detailed results of the assessments (excluding any human judgement where possible) will be shared with all participating parties.

For example, if the assessment involves measuring a value using automated systems, the participants will all receive these values:

• Assessment timestamps for all participants
• Measured metrics for their product and the average product

Why should you join?

Marketing & sales

To have a document that benchmarks your solution in the market. If the benchmark supports your marketing messages, you can back up marketing claims with 3rd party data gathered via an objective and transparent process.

Product

To understand your product’s strengths and weaknesses in a data-driven manner.

Why should you not join?

To get leads. AIMultiple is either the global leader or one of the leaders in digital audience in the domains where it runs benchmarks. However AIMultiple benchmarks are technical documents and are not geared towards a large audience.

Reach out to AIMultiple team via [email protected] if you would like to have an AIMultiple benchmark in your domain.

1. How Markets and Vendors Are Evaluated in Gartner Magic Quadrants. Gartner. January 22, 2016. Retrieved June 21, 2023.