Yash Kanoria is a Professor in the Decision, Risk, and Operations division at Columbia Business School. He specializes in the design and optimization of marketplaces including matching platforms and AI-enabled e-commerce. In the MBA program, he teaches an elective on Digital Marketplaces, and the Business Analytics core class.
Kanoria obtained a BTech from IIT Bombay and a PhD in Electrical Engineering from Stanford. He has won several research and practice awards including the MSOM Young Scholar Prize, the Lanchester Prize, the Sigecom Test of Time award, the NSF Career Award, and a Best OR Paper Award as an Amazon Scholar for developing highly efficient and scalable supply chain algorithms. A scheduled speaker at the June 16 Platform Leaders conference on the future of digital platforms and AI, he agreed to speak to The Innovator about what happens to platforms, marketplaces and e-commerce in general when a growing share of users are leveraging AI assistants or agents.
Q: Tell us a little about your research on AI shopping assistants
YK: AI tools like ChatGPT and Google’s Gemini are already influencing shopping habits, so we wanted to find out what happens if you ask them to open up a marketplace website and, for example, find a suitable tube of toothpaste. What do they choose when presented with a display of products? The last two or three generations of models will always get the right answer to a question with one right answer, but shopping is not like that. Trade-offs inevitably emerge between attributes like price, ratings, number of reviews, platform tags, and product features and brand.
The main question we were asking in our paper is: can we get an overall quantitative sense of how AI tools make these trade-offs? We introduced a framework for systematic evaluation of the choice behavior of an AI shopping tool and when we used it to evaluate the models of the leading providers –ChatGPT, Gemini and Claude. My collaborators Amine Allouah and Josue Figueroa from MyCustomAI, my Columbia colleague Omar Besbes, and our former PhD student Akshit Kumar and I found some things which were quite surprising.
Q: What were your findings?
YK: We evaluated the shopping behavior of AI shopping agents through randomized experiments. One thing we uncovered is that when showing AI agents product displays, for instance a display with two rows and four columns, they strongly prefer certain positions over others — which challenges the notion of them being perfectly rational. The first GPT model we evaluated was three times more likely to pick the product in the top left than random chance would predict, in a hypothetical situation where listing position is the only differentiator between products.
One might wonder whether this reflects training on human data, since humans also focus on the top left. So, we checked what other models were doing. We discovered that Gemini and Claude models in that generation have different, also strong positional biases — one favoring the top right, another the top center. We wondered whether different providers were training their models differently and whether that was playing a role.
Looking at four generations of models, we found that successive generations differ widely in how they make choices and trade-offs between attributes, so it is not just about the provider and how they are training their models. There is a significant idiosyncratic component to model behavior that appears to be a byproduct of the so-called post-training process of AI models and is largely outside the control of the person building the model. They can post-train their model to control certain narrow aspects of its behavior, but the more aspects they try to control, the more the model is likely to go wild in other ways. This idiosyncratic behavior shows up not just in position biases, which vary from model to model across model generations and providers, but also in how they weigh price, ratings, number of reviews, platform tags and so on. Just when you think you see a pattern, the next generation model comes out with some different and unforeseen behavior.
Q: What about agents’ response to brands?
YK: We see that, like humans, AI models have strong preferences for certain products and brands, but these preferences are also idiosyncratically changing from one model to the next and one generation to the next. Companies selling online should be prepared for this kind of unpredictable volatility. That is not to say there is nothing they can do, but they should expect marketplace shocks with each new model release as AI becomes more mainstream and plays a larger role in helping people shop.
Q: What can companies do to respond to this idiosyncratic behavior? Is there a kind of SEO equivalent that allows you to tweak things and make sure your product rises above the others?
YK: Yes, absolutely. It is called GEO — Generative Engine Optimization. Anybody who cares about their search rankings now also needs to care about how chatbots and AI tools perceive them. AI tools are already playing a big role in the search and discovery part of the shopping journey of customers. It worth noting that Google’s search bar and most search bars are also AI powered tools at this point. As a seller, you want an AI shopping tool, when a relevant query comes up, to surface what you are selling and rank it highly.
In the short run, the main lever companies have is the product listing. Companies can potentially have different flavors of listings for the same product — those prepared for humans versus those prepared for AI agents. Things that are easy for humans to parse may not be easy for these AI tools, and vice versa. For example, AI tools may find it easier than humans to parse comparisons with competitor products and may place value in such comparisons. My colleague Tianyi Peng at Columbia Business School and his co-authors have written an early paper on how to do systematic GEO for selling online. Their paper uncovers features sellers should incorporate in their listings to make them more attractive. It also posits that the seller should not be trying to execute these principles manually — companies can and should use AI tools and datasets of user queries to optimize their listings automatically. Their paper describes how to do this.
Q: Does this mean companies will need separate marketing materials for humans and for AI agents?
YK: Yes. A company that wants to be ahead of the curve — or even competitive in the near future — will need a GEO strategy. Not everyone on every shopping occasion will rely on an AI tool, so you still need to convince human shoppers that you have something to offer. In that sense, you will need two strategies.
Q: What are the most important tips for fine-tuning your listings to appeal to AI agents?
YK: One really important thing is that the listing needs to be made available in a format easily scannable by the AI tool. The formats that work best may not be the same as for humans.
You are also going to want to ask an AI tool to rewrite your listing description using a rewriting prompt. That prompt should instruct the AI to retain factuality, rank the product highly, anticipate user intent, draw comparisons with competing products, use a compelling and persuasive narrative tone, adopt an authoritative voice, focus on unique selling points, and convey a sense of urgency or scarcity. These are among the themes emerging from research into prompts that work well. Interestingly, standard prompt optimization techniques automatically surface these themes, without the need for extensive human input in crafting the prompt.
Listings will increasingly be customized not just for humans versus AI, but for specific AI tools. Just as product descriptions are now often AI-generated and targeted to individual human shoppers, the same will happen with AI shopping tools.
In terms of the process of customizing listings, I would distinguish two types of sellers. Those with a large, diverse portfolio of products — many of them niche and low-volume — should focus on optimizing the rewriting prompt itself. Human intuition is becoming less reliable for catering to AI assistive tools. You start with an initial prompt, test the descriptions it generates against a dataset of user queries to see how highly the target AI shopping assistant is ranking your products, then feed that data back to your AI seller assistant and ask it to improve the rewriting prompt. You repeat this a few times. It sounds like a lot, but this kind of thing is becoming standard practice — much like SEO techniques are standard practice. Incidentally, prompt optimization is a broadly useful approach, not just for e-commerce, and one that is likely to remain valuable for some time.
The other kind of seller, those with a narrower product line might instead optimize the listing descriptions directly. You test a description, see how it performs on your database of customer queries in terms of rankings by the target AI buyer assistant, give that data back to the AI seller assistant, and ask it to generate a better description. You repeat this separately for each AI shopping tool you are trying to reach, since slightly different things may work best for the latest GPT model versus the latest Gemini or Claude.
Q: Most brands want a direct relationship with their customers. Doesn’t having an agent in the middle complicate things?
YK: Yes, and I am learning about this alongside your readers. Brands will need new skills and will have to navigate a new reality. These tools do care about brands — you want to be a brand that the tools like, trust, and recommend, even as some of that is not fully under your control.
Some second-tier brands may be in for a struggle, because AI tools tend to concentrate on a few brands or products. One student project group in my marketplace elective at Columbia studied the role of AI tools in the luxury space. They found that certain heritage luxury brands were actively favored by these tools, while some second-tier luxury brands were essentially invisible to them. There is also a perception question for the most exclusive brands: will their customers still see them as exclusive if they have any relationship with chatbots? These are some of the new complexities I have been noticing, though I am by no means an expert on this topic.
Q: How do you advise companies to prepare for agentic commerce?
YK: Make sure your products are visible with quality real-time information in a way that is scannable by AI tools. It sounds simple but there are real complexities, especially for sellers with wide, diverse inventories. We were speaking recently with folks from Nestlé at Columbia Business School, and they highlighted how challenging it is to clean up all that data in real time and make it visible to AI tools.
Think about how to build your database of representative user queries in a way that is maintainable, because we know that the market changes over time. Establish an automatic pipeline for optimizing your listings that leverages this database, followed by A/B testing on the ground to get even better.
If you are a large business, you would want to be on top of your database of representative queries yourself. You may also want to develop the capability to optimize listing descriptions and refine your rewriting prompts in-house. SMBs should look out for relevant offerings from their SaaS provider, switch providers if necessary to get access to the best tools, adopt platform-native AI tools early, and figure out how to provide the highest quality, most accurate raw data for the platform’s models to work with.
Finally, expect new kinds of volatility — not all of it will be under your control. Be prepared to adjust, be nimble, and try to stay ahead of the curve as best you can. The AI transformation is poised to create new winners and losers.
This article is content that would normally only be available to subscribers. Become a subscriber to see what you have been missing
