Editorial Team Let’s be honest, page views are a pretty weak metric for conversational AI. For our “Cognitive Commerce Connect” campaign, we had to move past simple visits and start measuring what really mattered: deep AI agent engagement. A user who spends five minutes working with an AI assistant to build out a custom product is way more valuable than a quick bounce from a static page. We needed to figure out how to measure these new interactions and, more importantly, how to get better at them.
Key Takeaways
- Our AI-assisted product configs had a 27% higher conversion rate than the old-school landing page funnels.
- We ran sentiment analysis on the AI chats and found a direct link: a positive sentiment score over 0.7 meant a 15% jump in purchase intent.
- Retargeting users who finished an AI flow with personalized offers killed it, delivering a ROAS of 4.2x, blowing past the campaign’s overall 3.5x ROAS.
- We tightened up the AI agent scripts and cut the average interaction time by 18 seconds, which in turn boosted user satisfaction scores by 10%.
Campaign Teardown: Cognitive Commerce Connect
We ran the “Cognitive Commerce Connect” campaign for six months, from Jan to June 2026, and put a $750,000 budget behind it to really understand how AI-driven customer journeys work. Our objective was straightforward: drive more qualified leads and direct sales for our high-value, customizable products by using AI agents during the pre-purchase phase.
Strategy and Core Hypothesis
We basically bet that users who engaged with an intelligent AI agent to customize a product would have much higher purchase intent and convert better than people fumbling through static product pages. Our AI agent, which we built on a proprietary NLP framework, walked users through a Q&A to nail down their needs for a complex B2B software solution. From there, it served up tailored configurations and offered to either book a live demo with sales or spit out a direct quote. The goal was to enrich the lead qualification process, sending warmer, better-informed prospects to our sales team.
We went after decision-makers and tech buyers at medium-to-large companies (500+ employees) in manufacturing and logistics. Our CRM data showed these were the exact people getting lost and bouncing from our dense spec pages. The AI agent was designed to cut through that noise for them.
Creative Approach and Targeting
Our ad creative was all about problem-solution. We showed how our AI could simplify really complex decisions, running a mix of 15-30 second video ads on LinkedIn Marketing Solutions and programmatic display across trade publications. The CTA was always direct, sending people to a landing page with the AI agent front and center. We used specific calls to action like “Configure Your Solution Now” or “Talk to Our AI Assistant” instead of the generic “learn more.”
The targeting was surgical. On LinkedIn, we filtered by job titles like “Head of Operations” or “Supply Chain Director,” industry, and company size. For programmatic, we did some IP-based targeting for specific corporate campuses and used third-party data to find companies looking into B2B software. We started by focusing on industrial hubs in the Midwest and Southeast US, especially around cities like Atlanta, Georgia, and Chicago, Illinois, where our sales team already had a foothold.
What Worked: Data-Driven Insights
The data proved our core hypothesis was right, with the campaign hitting several key goals. Our overall Return on Ad Spend (ROAS) came in at 3.5x, and while that’s a respectable number, it doesn’t even begin to show what the AI agent was really doing for us.
Enhanced Conversion Rates
People who went through a full AI agent interaction, we defined this as answering 80% of the config questions and getting a recommendation, had a conversion rate of 5.8%. That’s a huge improvement compared to the 4.6% conversion rate from users who hit the same pages but didn’t talk to the bot. That 27% conversion uplift for AI-assisted journeys was the number that got everyone’s attention. Even better, the cost per qualified lead (CPL) for these AI-engaged users was only $125, way down from the $180 CPL for leads coming from our old-fashioned forms.
Deeper Engagement Metrics
We started tracking a whole new set of metrics beyond just page views to actually measure AI engagement:
- Average Interaction Duration: Users were actively talking to the AI for an average of 3 minutes and 45 seconds. This blew away the average time on page for our static product sheets, which was a paltry 1 minute and 20 seconds.
- Question Completion Rate: A surprising 72% of users who started a chat with the AI stuck around to complete at least 80% of the questions. That showed us they were genuinely committed to the process.
- Sentiment Score: We plugged sentiment analysis into the bot’s backend. This was a huge lightbulb moment for us, we found that conversations with an average sentiment score of 0.7 or higher (on a -1 to 1 scale) were 15% more likely to request a demo. A good experience with the bot meant a better quality lead was coming down the pipeline.
Retargeting Success
One of our biggest wins was segmenting and retargeting users based on what they did with the AI. For users who configured a solution but then ghosted without asking for a demo, we hit them with display ads showing their specific configuration along with a limited-time discount code. The ROAS on this segment was a fantastic 4.2x. The CTR of 1.1% on these retargeting ads was nearly double the general campaign CTR of 0.65%. These people had already put in the time with the AI, so the personalized offer was incredibly effective.
What Didn’t Work and Optimization Steps
It wasn’t all perfect. Our first draft of the AI agent script was too long-winded and we saw people dropping off. The average interaction time in month one was 4 minutes and 10 seconds, which felt too long. We also got feedback that some of the questions were too technical for someone just starting their research.
Script Optimization
We immediately started A/B testing the AI scripts, simplifying the language, shortening questions, and creating more dynamic paths. For example, if a user seemed confused by a term, the AI would offer a quick explanation instead of just plowing ahead. Through this iterative tuning, we got the average interaction time down to 3 minutes and 27 seconds, an 18-second improvement. It sounds like a tiny change, but shaving off those 18 seconds bumped our user satisfaction scores (from post-chat surveys) by 10%.
Initial Ad Creative Misalignment
Some of our first video ads were way too broad, talking about “digital transformation” instead of showing what the AI actually did. The results were predictable: a low initial CTR of 0.4% in the first month. We pivoted fast, creating new ads that showed the AI agent in action, simplifying a complicated task. One ad that worked really well showed a split-screen of a user breezing through an ERP integration config with the AI versus someone looking frustrated while flipping through a thick manual. This new creative approach pushed the average CTR up to 0.7% for the rest of the campaign.
Cost Per Conversion Analysis
While the CPL for leads was solid, our overall cost per conversion (CPC) for a closed deal started out higher than we wanted, at $2,500. It pointed to a clunky handoff to the sales team. Turns out, sales wasn’t getting the full context from the AI chat and were asking prospects questions they’d already answered. We fixed this by automatically piping the full AI chat transcript and the AI-generated summary right into the lead’s CRM record. Giving sales that context reduced their initial qualification time by about 15% and helped bring the final CPC down to $1,800 by the end of the campaign.
Looking Ahead
The “Cognitive Commerce Connect” campaign proved that AI agent engagement is a serious driver for qualified leads and sales. We’re now prioritizing metrics like interaction duration, sentiment scores, and completion rates for specific AI flows because they paint a much clearer picture of user intent than just counting traffic. As marketers, this means we have to stop obsessing over the click and start focusing on the quality of the conversation.
Figuring out the true business impact of all this requires better analytics. To get a sense of the new challenges, check out AI Attribution: GA4 Challenges for Marketers in 2026. It really gets into the weeds of attributing success in an AI-driven world.
And of course, with AI creating more content and having these interactions, the quality has to be there to maintain user trust. We’re keeping a close eye on this, and you can see why by reading AI Content Quality: 70% Need Editing in 2026.
What do you mean by “AI agent engagement” in marketing?
It’s when a user is actively interacting with an AI system, like a chatbot or virtual assistant, on your site or in an ad. They aren’t just passively reading. They are providing input, getting personalized responses, and being guided through a specific process like a product configuration.
How is AI engagement different from a page view?
A page view just tells you someone loaded your page. That’s it. AI engagement measures the depth and quality of what they did there. We’re looking at things like how long the conversation was, if their comments were positive or negative, and if they completed a task, which tells you a lot more about their actual intent.
What are the most important metrics for AI agent engagement?
The essentials are average interaction duration, completion rates for key flows (like a full configuration), sentiment scores from the chat, conversion rates after the interaction, and the cost per qualified lead that the AI generates. Together, these metrics tell you if the AI is actually working.
Do AI agents actually help conversion rates for complex products?
Absolutely. For complicated products, an AI agent acts as a guide, simplifying the endless options and answering specific questions on the spot. It cuts down on the user’s decision fatigue, builds their confidence that they’re picking the right thing, and directly leads to higher purchase intent and more conversions.
How does sentiment analysis help an AI marketing campaign?
Sentiment analysis gives you a real-time read on how users are feeling during the AI chat. If sentiment is positive, you know they’re engaged and likely interested. If it’s negative, it’s a red flag telling you that a part of your AI script is confusing or frustrating, or that it might be time to offer a human handoff. This data lets you make adjustments on the fly.
