Measuring the Efficacy of LLMs in Media Planning: Insights from R Evaluations

Recent assessments of large language models (LLMs) in media planning reveal critical shortcomings regarding accuracy and reasoning capabilities. In practice, LLMs can draft compelling media strategies nearly instantaneously, but the pressing concern lies in their calculation accuracy, representation of assumptions, and alignment with campaign objectives.

At Havas Media, I developed the Havas AI Media Quality Index (HAI-Q), designed specifically to address the need for specialized evaluations in the media sector. Traditional knowledge tests seem inadequate since even a model producing a convincing media plan might falter on fundamental aspects, such as budget allocation or target demographic definitions.

1. Evaluating LLMs with R

To assess LLMs effectively, we need a structured evaluation framework with three primary components:

1. A dataset containing specific questions and expected answers.
2. A solver capable of generating responses to these questions.
3. A scoring mechanism to evaluate the responses.

Utilizing the vitals package in R, evaluations can be tailored to focus on critical outputs in media planning. The target column may accommodate not only definitive answers but also grading criteria outlining necessary elements an appropriate response should encompass. This allows for more nuanced assessments beyond simple correctness.

However, the traditional evaluative structure has limits. While it may test final answers effectively, it doesn’t capture the complete functionality of an AI agent that may need to interact with data sources, execute calculations, and manage states while determining when to seek further input. The flexibility to introduce custom solvers is present, but building the comprehensive agent needs to address more than mere answer correctness.

Creating Assessment Questions

To construct an effective evaluation, it’s crucial to have well-defined questions that cover a variety of aspects such as calculations and strategic planning. Here’s a sample evaluation set designed to probe fundamental media planning concepts:

library(dplyr)
library(tibble)
media_questions <- tribble(
~id, ~input, ~target, ~domain, ~task,
"grp-calculation",
"A campaign delivers 60% reach at an average frequency of 3. What is the campaign's GRP? State the assumption behind the calculation.",
"A full-credit answer calculates 60 * 3 = 180 GRP. It explains that this is the standard simplified relationship GRP = reach (in percent) * average frequency. It must not describe GRP as the number of unique people reached.",
"Reach and GRP", "Calculation",
"budget-allocation",
"A client has EUR 100,000 to reach adults aged 25-49 in Germany. There is no historical channel performance data. Propose a first-pass media allocation and explain how you would improve it after launch.",
"There is no single correct allocation. A strong answer makes the uncertainty explicit, explains the role of each channel, avoids inventing CPMs or reach figures, and proposes a measurement and test-and-learn plan. It should distinguish a planning hypothesis from an observed result.",
"Media strategy", "Recommendation",
"reach-frequency-tradeoff",
"Two plans have the same budget. Plan A has higher reach and lower frequency; Plan B has lower reach and higher frequency. How would you decide between them?",
"A full-credit answer says that the choice depends on the communication objective, the selected channels, the overall attention, audience size, purchase cycle, creative strength, and expected response. It should explain the reach-frequency trade-off and ask for missing information instead of declaring one plan universally better.",
"Media strategy", "Reasoning"
)
media_questions

The questions aim to provoke specific responses, avoiding vague concepts like "create a media plan," which lends itself to broad interpretations. Instead, focusing on concrete metrics facilitates numerical assessments, particularly in areas where models might struggle.

The Evaluation Process

The evaluation process encompasses both a solver and a scoring mechanism. With the Task$new() function from vitals, it is straightforward to compile questions and evaluate the outputs from an LLM, such as Claude Sonnet or an OpenAI model. Queries yield responses that can subsequently be scored with partial credit options, making it easier to discern the nuances between accurate, partially correct, and incorrect outputs.

For instance, when evaluating budget allocation, a model may produce answers with valid reasoning but miscalculate numerical components crucial to the strategy, which must be noted and addressed in scoring.

2. Challenges in Media Planning Evaluations

Media planning presents a complex landscape where evaluation becomes inherently challenging. One critical issue is that many planning inquiries lack a single correct response. Different objectives, budgets, and strategies can yield multiple acceptable plans, making rigid scoring difficult. Therefore, evaluators must reward rational thought without skewing towards one planner’s choices as definitive.

Market Variability and Temporal Considerations

The accuracy of media calculations is often anchored in local market conditions and time-specific data. What holds true in one environment may not apply in another, and data variables like costs and audience definitions are not universally applicable. Therefore, a robust evaluation should note market context and data validity, as assumptions made in answers may significantly influence correctness.

Precision in Terminology Matters

Media arrays are rich with jargon that can lead to misinterpretations. Terms like reach or frequency can have different meanings depending on the context. An answer that appears coherent can still misapply formulas to incorrect values, necessitating careful scrutiny in evaluations. It’s essential that checks involve both numerical accuracy and overall reasoning quality.

Limitations of LLM Evaluation Approaches

The integration of LLMs as evaluators introduces its own set of challenges. Each model may exhibit biases or preferences that affect grading outcomes. Therefore, rigorously calibrating scoring systems against human inputs and allowing for various evaluation dimensions can better reflect the efficacy of LLMs in media planning. Separating the results into different categories—factual accuracy, numerical validity, and reasoning quality—offers deeper insights into where models excel or require improvement.

3. HAI-Q Results and Model Performance

The results from the HAI-Q benchmark are enlightening, exhibiting the performance of various AI models within the context of media planning tasks:

These findings indicate that even the leading LLMs struggle with accuracy in media-related queries. GPT-5, while outperforming others, succeeds in less than half of the evaluated cases, emphasizing a crucial need for human oversight, particularly regarding numerical assessments. Observations from HAI-Q data underscore a consistent pattern: models exhibit proficiency in creating fluid narratives but falter when confronted with precise numerical reasoning.

Comparatively, improvements in model performance, notably between GPT-4o and GPT-5, suggest strides in development, though not yet sufficient for exclusive reliance in media decision-making. Claude Sonnet’s lower scores reflect the evaluation's specificity; while it may perform adequately on generic tasks, its applicability in niche evaluations like media planning is less reliable.

4. Key Takeaways

Launching an effective LLM evaluation for media planning entails a grounded approach using concrete tasks derived from real-world scenarios. In R, the structure of a tibble with input and target serves as the backbone for assessments, enhanced through the vitals framework for connecting queries with scoring methodologies.

The core challenge lies not merely in combatting the computational limitations of models but in explicitly stating what constitutes “correct” within the ever-evolving landscape of media planning, where data can widely fluctuate across markets and time frames. Upcoming evaluations should integrate end-to-end task assessments to ensure comprehensive insights into tool utilization and broader workflow efficacy.

The HAI-Q results present an urgent call for caution, highlighting the need for diligent auditing of numerical outputs from LLMs. To effectively employ LLMs in media planning processes, organizations should leverage models to enhance efficiency yet maintain rigorous checks on any numerical output before application.

Useful Resources

• Introduction to vitals
• Documentation for vitals
• Overview of HAI-Q: Havas AI Media Quality Index
• Documentation for ellmer