ChatGPT's interface vs its API: a 100-question study of brands and sources

We asked ChatGPT 100 buying questions through its own web interface, twice, and the same questions through OpenAI's API three ways. The brands came out mostly the same. The sources did not: the API cited fewer than half as many sites, leaned on the brands' own pages, and shared only 8% of its cited pages with what a reader of chatgpt.com was shown.

What we found

• The interface is its own noise floor. Asked the same question twice in the same minute, ChatGPT named the same brands 78% of the time and cited the same pages only 30% of the time.
• The API names roughly the right brands, in a different order. Brand overlap with the interface was 65% for the API with web search, against 78% for the interface with itself. It picked the same top brand 57% of the time, against 73%.
• The sources are a different list. The API's cited pages overlapped the interface's 8% of the time — under a third of the interface's agreement with itself.
• The API cites the brands; the interface cites reviewers. 64% of the domains the API cited belonged to a brand in the category, against 41% in the interface. TechRadar and Tom's Guide were cited in 18% vs 3% and 16% vs 3% of answers respectively.
• Per-brand figures can be far apart. In running shoes, the interface named Hoka in 80% of answers and the API in 10%.

The interface does not agree with itself

Before comparing the API with the interface, we compared the interface with itself. Every question went to chatgpt.com twice, within seconds of each other, from the same location and with web search on. That pair is the yardstick for everything else: a difference between the API and the interface only means something by however much it exceeds the difference between two interface answers.

The two interface answers named an average of 3.8 tracked brands each and shared 78% of them. They chose the same brand first 73% of the time. Their sources moved much more: 47% of cited domains were shared, and only 30% of cited pages. One answer is a sample, not a measurement — which is why any single screenshot of "what ChatGPT says about us" should be read as one draw from a distribution.

Brands: close, but not the same order

Brand overlap is the share of brands named by either answer that both named. Each API arm is compared with both interface answers and the two figures averaged. The interval is a 95% bootstrap interval over the 100 questions.

Every API arm agreed with the interface less than the interface agreed with itself, but not by a wide margin, and one result ran against expectation: the smaller model with search tracked the interface slightly more closely than the larger one. Its answers were longer and named more brands — 3.9 on average against 2.9 — and a longer list simply has more chances to overlap. The larger API model wrote 196 words on average; the interface wrote 589.

Sources: where the two part ways

The interface cited 7.6 domains per answer; the API with search cited 3.2. Measured the same way as brands:

The difference is not only how many sources, but which kind. Counting a cited domain as brand-owned when it carries the name of a brand in the category, 64% of the API's citations went to the brands themselves, against 41% of the interface's. The interface leaned on review publishers and comparison articles the API rarely touched:

Among the domains the interface cited and the API never did: dupple.com (8%), technologyadvice.com (7%), brevo.com (6%), surfshark.com (6%), dickssportinggoods.com (5%), itechguides.com (5%), dreametech.com (5%), reddit.com (4%).

Category by category

Share of voice here is the share of answers in a category naming a brand. The gap is the average, over every brand named, of the absolute difference between two arms' shares. With ten questions per category these are directional, and the interface-against-itself bar says how much of any gap is noise.

In password managers, CRM, email marketing, accounting, web hosting and AI visibility tools, the API's gap was within a few points of the interface's own. In project management, VPNs, running shoes and robot vacuums it was close to double, and the brands that moved most — Monday, Surfshark, Hoka, Eufy — were named far more often by the interface than by the API.

What it means if you track AI visibility

• A brand-level number from the API is a rough guide. It will usually find the same leaders, and it can be badly wrong about one brand in one category — which is the brand you are paying to watch.
• A source list from the API is not the one readers see. If the plan is to get onto the pages ChatGPT cites, the API points at brand sites, and the interface points at the reviewers and comparison articles that actually carry the answer.
• One answer is not a measurement. Even the interface disagrees with itself on most of its sources. A figure is only as good as the number of answers behind it, and should come with an interval. We wrote about how we count that in what AI search visibility is.

Method and limits

• Questions. 100 buying questions, ten in each of ten categories — CRM, project management, email marketing, password managers, VPNs, running shoes, robot vacuums, small-business accounting, web hosting and AI visibility tools — phrased as people ask them: "best", "alternatives to", "X vs Y", "cheapest".
• Five answers per question, 500 in all. Two from the chatgpt.com interface, captured by DataForSEO's LLM Scraper with web search forced, located in the United States in English; the interface reported its model as gpt-6. Three from the API through OpenRouter: gpt-6-sol with OpenAI's native web search, gpt-5.6-luna with the same search, and gpt-6-sol with no search. The API answers were sent the question alone, with no system prompt and no location. All were collected on 12 October 2026.
• Brands. Each category has a fixed list of 16 to 20 brands, matched by the same deterministic detector our product uses, with aliases. A brand missing from a list is invisible to every arm equally, so it understates the counts without favouring an arm.
• Sources. The interface's sources are the citations a reader is shown; the API's are its URL citations. Tracking parameters were stripped before comparing pages. "Brand-owned" is a name match on the domain and is approximate.
• Limits. One engine, one country, one language, one day. The interface does not say which gpt-6 tier it runs, so the main API arm is our best match, not a certain one. Ten questions per category is enough to see a pattern, not to rank brands.
• Cost. The whole run cost $12.94, most of it the larger API model with search.

The full dataset — every question, arm, model, brand named and cited URL, 500 rows — is free to download as CSV. You are welcome to use it with a link back to this page.