Research
Saved answers. Counted carefully.
A small, documented research programme on which businesses AI assistants recommend. This page explains the method before any result exists, so you can judge the results by it.
01
The planned pilot
The first study is a small pilot. It is a plan, not a result. Its scope, including the country and the categories, is still subject to owner approval and has not been fixed publicly.
| Element | Count |
|---|---|
| Categories | 5 |
| Fixed unbranded buying questions per category | 5 |
| Assistants (via provider APIs) | 3 |
| Repetitions (rounds) | 2 |
| Planned answers | 150 |
5 categories x 5 questions x 3 assistants x 2 repetitions = 150 planned answers. Planned is not the same as successful: any failed request is reported separately, never silently dropped.
02
How answers are collected
- Answers come from search-enabled provider APIs. They are not replicas of the consumer chat apps, which may answer differently.
- Each study records the requested pinned model ID. Provider-returned model metadata, when supplied, is preserved in the downloadable observations.
- Gemini answers come from the Google Gemini API. That is not Google AI Overviews in search results, and we do not treat it as such.
- One fixed country and location for the whole study. Prompt text is written once and left unchanged.
- Every request is independent, with no shared conversation history, and is timestamped.
- Two rounds, so we can see whether the same assistant repeats itself.
- Source links the answer actually cites are kept apart from sources the search tool retrieved but the answer did not cite.
- Only valid HTTP(S) citations with identifiable domains are counted. For Google grounding redirects, the provider's source-domain label is used; destinations are not independently verified.
03
How answers are reviewed
- A person reviews each successful answer and annotates which businesses or brands are named, and which are actually recommended.
- Each annotation is backed by verbatim evidence from the answer. That evidence is saved privately for audit, not published.
- We do not publish full answer text, and nothing on this page is an endorsement from any assistant.
04
What we count, and how
- Brands and domains are counted once per answer. A business named five times in one answer counts as one answer.
- Percentages use successful answers as the denominator, and the denominator is always shown next to the figure.
- Values that cannot be computed are shown as not available, never as zero.
Agreement across assistants
For each prompt and round, take the reviewed set of recommended businesses from each assistant. For every pair of assistants compute Jaccard similarity: shared items divided by all distinct items in either set. Report the mean over all pairs.
Repeat stability within an assistant
Same calculation, but comparing one assistant's round 1 answer with its round 2 answer to the same prompt.
Pair rules
- Both sets empty: pair excluded.
- One set empty, the other not: scored 0.
- Either answer missing or failed: pair excluded.
- The number of comparisons is always shown with the mean.
05
Limitations, stated upfront
- Categories are self-selected, not a random sample of the economy.
- One location only. Answers elsewhere may differ.
- Two repetitions show some variation, not its full range.
- Results are not representative market share and support no causal claims about why a business is or is not recommended.
- Nothing here guarantees a ranking, traffic or revenue change for anyone.
06
What never goes into research
Research data is collected separately from the product. Customer scans, email addresses and report IDs are never used in, or published as part of, any study.
Published studies
No studies published yet.
This is not a zero score. Collection has not produced reviewed, publication-approved results, so there is nothing to show. When a study is published it will appear here with its full data files.