Why automatic competitor-site comparison was removed

Sep 18, 2026First-party research14 min

“Have the AI read competitors’ sites and put the differences from us in a table” is a request that always comes up when you are building a marketing-support AI. I built it too. I connected it to strategy on September 11, 2026, and removed it from the normal path of recommendation generation on September 17.

The reason is not cost. Within what could be confirmed in production, not a single comparison succeeded.

I am building Shoryaku AI ( https://shoryaku.net ), a marketing-support SaaS, as an indie developer. This article is a record of what I built, where it got stuck, and why I chose removal instead of a repair. The second half is how to choose the materials so that someone doing competitor analysis in ChatGPT or Gemini can avoid the same walls.

What I was building

Inside recommendation generation (the feature that produces moves to raise revenue from the numbers of connected ads and GA4), there was a stage of external research.

Collect the competitors registered in the business settings, and competitor candidates found by search
Fetch the competitors’ official sites ourselves, following robots.txt
The AI structures the target customer, the appeal, the price, and the path from excerpts of the body, on the same axes
At the strategy stage, reflect the differences from us and how we can win

On screen, the design was a competitor-positioning figure in the strategy chapter, and “competitors’ appeal words” in the search-market field.

Wall 1: The search that looks for competitors could not be used in production

The design used the Google Custom Search JSON API to search for competitor candidates. When I checked production, the Google Cloud project the key belonged to had no entitlement to use it, and the request was rejected with HTTP 403. Google’s official notice was also announcing the end of provision to new customers, so it was not something that would be fixed by registering the key again.

At that point I switched the policy once, from “find them by search” to “read only the competitors the user registered.” There was no screen for registering competitors, so I added an input field to the business settings. Even without the search API, fetching and comparison should work as long as there is a URL.

Wall 2: The comparison output was cut off midway through the first company

On the day after I added the registration screen, I checked a production run on the real screen, in the logs, and in the saved model usage.

It received 2 registered competitors, and fetching the body of the 2 official sites succeeded
The comparison extraction failed, at 0 companies
The call to the lightweight model that handles extraction was 9,311 input tokens, 2,048 output tokens, and 5.8 seconds

2,048 output tokens is the ceiling that had been set for this use, itself. I could judge that the comparison-result JSON was cut off midway through the first company, and there was not a single completed competitor that could be rescued from the broken JSON.

This failure alone can be fixed. Raise the output ceiling to 4,096, and narrow the number of items per company.

Wall 3: Even if it is fixed, you cannot measure whether it worked

The problem was what sits past the fix.

Even if the output ceiling is raised, the failure points remain: robots.txt on the site being fetched, pages whose body cannot be fetched because they are rendered with JavaScript, the size of the HTML, changes in the site structure, and the AI extraction being cut off midway. If a feature with many failure points stays on the main path, it needs an effect that matches that.

I could not confirm that effect. Of 8 runs since September 13 in which the logs of the competitor stage could be checked, the number of runs in which a comparison succeeded for even 1 company was 0. Of the 2 runs that had registered competitors, 1 had 0 site fetches, and 1 had 2 fetches and 0 comparisons. Before measuring whether the comparison result raised the accuracy of the recommendations, the comparison result itself never came out even once.

Cost did not become a reason. The failed extraction call was about $0.002, and that whole run was about $0.06. In a measurement a little earlier as well, external research (search and extraction) was about 12% of all the tokens in one run, a smaller share than had been assumed.

The reason for removal is not the amount. It is that it increased wait time and complexity while supplying not a single piece of evidence.

“It can be repaired” and “it is worth keeping” are separate decisions

What I decided is the following 5.

Remove automatic “fetch the competitor site, then compare” from the normal path of recommendation generation
Do not treat the search-market check as “partially failed” just because competitor comparison is absent. If the Keyword Planner fetch succeeds, treat it as complete
Remove “fetch competitor official sites” and “competitor comparison” from the progress display
Do not fill the 3C competitor field with a guess. Competitor names and notes the user entered are treated as “user input,” and are not treated as a fact that the site body was read
If it is brought back later, do not build it into every recommendation generation. Make it a separate feature the user runs explicitly, measure the success rate and the adoption rate, and then decide

I think the 4th is the most important. Even after the comparison feature is removed, the AI can fill the competitor field with “a general picture of the industry.” To stop that, items with no material are left blank and shown as “things not yet confirmed.”

What remained as external research is only the check of search demand, competition, and bid prices through the Google Ads Keyword Planner. That API is stable, and the numbers themselves are the evidence.

If you do competitor analysis in ChatGPT, change the materials you pass first

From the experience of building it and then removing it, there are 4 things I can say to someone doing competitor analysis with a free AI.

1. What you can take from an official site stops at “the appeal they publish.” The target customer, the price, the path, and the words they emphasize. Weaknesses, revenue, and customer satisfaction beyond that cannot be known from public information. If you ask the AI, it will fill them in, but that is a guess. The procedure for having it write facts and hypotheses separately is summarized in How to do a 3C analysis with AI.

2. Paste one company at a time, and narrow the pages. The top page and the pricing page are about enough. If you paste several companies together, the same things I hit happen: it gets cut off midway, and the later part gets sloppy.

3. Have it write which sentence it judged from. If it says “Competitor A is for beginners,” have it also write which expression on the pricing page that judgment came from. An item whose basis cannot be written is treated as a guess.

4. Before the competitors, pass the customer’s words and the search demand. In my implementation, what could be supplied stably as evidence for a recommendation was Keyword Planner search demand. The other one, the customer’s words that appeared in inquiries and reviews, is the only material the AI cannot replace with a guess. Adding a comparison of competitor sites is not too late after those two are in place. How to read search demand is written in How to find search keywords with little competition.

Summary

I built automatic comparison of competitor sites, and removed it while not a single comparison had succeeded in production
The direct causes were the search API being unavailable, and the output being cut off at the ceiling. Both could have been repaired
The reason for removal was not cost. It was that an effect matching a feature with many failure points could not be confirmed
Even after comparison is removed, items with no material are left blank, so the AI does not fill the competitor field with a guess
If you do the competitor analysis yourself, do it one company at a time, with the source sentence, and pass the customer’s words and the search demand first

“Have the AI read the competitors” was a feature that is easy to build and hard to confirm is working. If you decide, before you build it, what to use instead when a comparison result does not come out, the decision to remove it within 1 week comes sooner, as it did for me.