What LLMs Get Right and Wrong With Employee Survey Data
After the Benchmark: Reflections from the Team That Built PYX-Voice
Key Takeaway: After publishing PYX-Voice, our team sat down to share what the experience actually taught us. The headline finding was one that surprised even seasoned practitioners: LLMs handled complex quantitative analysis more reliably than expected, while stumbling on what seemed like simpler scoping decisions. Performance was uneven in ways that defied prediction — and that unevenness is itself a signal worth paying attention to. This post captures what our subject matter experts and technologists are taking away, and where they think the field needs to go next.
In our inaugural PYX Labs Webinar, we walked through what we did and what we found with our first benchmark, PYX-Voice. Towards the close of the session, our academic partner Melissa Valentine at Stanford University thanked us for “building out loud.”
We took that to heart.
In the midst of frontier research, it can be easy to get frustrated when things don't go as we expect, take longer than we want, or turn out different than we hoped. But the truth is, that's all part of the scientific process. This work provided our team with information; each hiccup, surprise, and less-than-glamorous outcome, in actuality, was a signal of progress. So in that spirit, we came together to do something simple: look back at what we built, what surprised us, and what we're still wondering about.
The team behind PYX-Voice brought serious depth to that work. Eight subject matter experts in I/O psychology contributed as both contributors and reviewers (seven holding master's or PhD-level credentials, with an average of 14 years of experience among them). Our technology team built the benchmark harness from the ground up, ran seven frontier models through it, and developed an increasingly precise understanding of how LLMs performed on employee listening tasks.
The dust has settled, the benchmark is published. So: what did we learn?
What the Models Got Right — and Where They Fell Short
Brad, you were in the thick of it: reviewing LLM outputs against SME-written criteria, task by task. Where did the models generally perform well, and where did they fall short?
Brad: The LLMs generally performed well with quantitative data and analysis. That result surprised me because people often associate LLMs with language-based tasks and expect them to struggle with calculations, statistical analysis, or numerical datasets. Across many of the tasks we evaluated, the models produced accurate and consistent quantitative results. Their performance exceeded my expectations.
The LLMs fell short in two related areas, both involving how they selected and combined data.
First, some models combined responses from multiple open-ended questions and analyzed them as one set of comments. This removed the context provided by each question. In employee survey data, that context matters. Responses to “What additional feedback would you like to share?” require a different interpretation from responses to “What is your greatest source of frustration at work?” Combining those comments can produce conclusions that do not accurately represent either question.
Second, some models combined comments from multiple survey years. We included several years of data so the models could identify changes and calculate trends. In some cases, however, a model analyzed comments from all available years when the task called for an analysis of the latest survey. The resulting summary blended current employee experiences with feedback from previous periods.
Organizations should pay close attention to the data an LLM can access and the records it includes in each analysis. A model connected to a large organizational data environment may draw from more information than the user intends. For example, a leader may request a summary of exit survey feedback from the last quarter, while the model analyzes four years of responses.
Clear prompts help, but organizations also need controls that limit the model to the correct questions, populations, surveys, and time periods. Teams should verify those boundaries before they use an LLM-generated analysis to guide decisions.
The Jagged Frontier: When Task Difficulty Doesn't Predict Performance
Given everything you just described (the data-scoping failures, the uneven performance), what surprised you about what the models could do?
Brad: What surprised me most was the uneven pattern of capabilities and limitations, sometimes described as the “jagged frontier” in Co-Intelligence. Tasks that appeared straightforward sometimes created problems for the models. More complex tasks occasionally produced stronger results.
For example, some models performed well when they summarized open-ended feedback from employees who had selected a specific response to a quantitative survey item. That task required the model to connect numerical and qualitative data, yet several models handled it effectively.
Our assumptions about task difficulty did not consistently predict LLM performance. We saw this pattern across multiple frontier models, which made the results particularly interesting.
What a Veteran Practitioner Takes Away
You've spent 17-plus years doing exactly the kind of work we asked these models to do. What are you taking away from watching them try?
Brad: The results gave me confidence in what these tools can already accomplish, while also reinforcing the need for careful oversight. In some cases, an output sounded plausible and credible but was not accurate.
Progress across generative AI models does not follow a consistent, linear path. Some newer models struggled with tasks that other models handled more effectively. A new model release does not guarantee better performance across every use case.
The benchmark also confirmed the continued need for human review. Organizations use employee listening data to help leaders make decisions with greater speed and confidence. An AI tool that selects the wrong data, combines unrelated responses, or produces an analysis that does not match the request can introduce doubt and reduce trust.
Human teams need to understand how these systems work, evaluate their outputs, and confirm that each analysis supports its intended purpose. That oversight helps organizations use AI effectively while protecting the quality and credibility of the insights they provide to leaders.
Building a Benchmark for a Uniquely Complex Domain
Andy, I'd love to get your perspective from the technology side. You built the benchmark harness — a complex, time-intensive task — and you've worked in the employee listening space long enough to understand both worlds. What is most unique about this content domain compared to others that have been benchmarked? And what did you do to account for that?
Andy: This content domain is uniquely multifaceted. To answer benchmark questions correctly, the agent has to perform accurate analysis of raw numeric datasets, accurately summarize unstructured text, and then synthesize all of the above with domain-specific knowledge to come up with actionable, executive-ready recommendations.
On the technical side, we’ve equipped the agent with a tool which allows it to call complex analytical Python code on the underlying dataset. We’ve also added definitions of domain-specific terminology into the prompt, to allow the agent to understand and reason with the same concepts internalized by the subject matter experts.
What Surprised the Technologist
You've been working with LLMs long enough to have a calibrated sense of what they can and can't do. What surprised you in this work?
Andy: I was particularly surprised by the robustness of each LLM’s ability to write complex data analysis code and accurately translate complex natural language queries into the correct statistical analysis.
A few years ago, the data analysis capabilities of LLMs were more demo-grade than production-grade. I was happy to find that the latest tool-use models, combined with advancements in software frameworks for specifying complex agentic workflows and prompt testing, made it possible to construct a harness which fairly competently performs the work of an entry-level data scientist.
Where the Lab Goes Next
As we chart our path forward as a Lab, what are you each most curious about? Where should we, and others in this space, continue to explore?
Andy: In the next iteration, I would love to improve the sophistication of the grading mechanism, to be able to tease out specific failure modes (is a failure due to hallucination, lack of industry knowledge, thematic mismatches, tonal mismatches, etc.). I would also love to expand the capabilities of the LLM harness through additional tools for querying customer context and searching for industry knowledge/best practices online.
Brad: I would like to explore how LLMs use broader organizational context when interpreting employee feedback.
Effective data storytelling often includes context, a source of tension, and a practical path forward. In this benchmark, we provided quantitative and qualitative employee survey data. A future iteration could also include information about the organization’s culture, size, strategic priorities, learning and development practices, industry, financial performance, and AI readiness.
Employee survey teams often use this type of organizational knowledge when they interpret results and advise leaders. It would be useful to test whether LLMs adjust their conclusions and recommendations when they receive the same context.
This approach could help us evaluate whether a model can move beyond summarizing survey results and produce analysis that reflects the organization’s operating environment.
The questions we set out to answer led to new ones, as with any research endeavor. Building out loud means being willing to show not just the results, but the reasoning, the revisions, and the road ahead. We're grateful to the team members who gave their time and perspective here, and to the broader community following this work. We'll keep sharing as we go.
Frequently Asked Questions
What did the PYX-Voice benchmark find about how LLMs handle employee survey data? Performance was uneven. The models were strong and consistent on quantitative analysis — often better than the team expected — but stumbled on data-scoping: combining responses from different open-ended questions, or blending multiple survey years when the task asked for the latest one.
Are LLMs good at quantitative analysis of employee survey data? Better than their reputation suggests. Across the tasks evaluated, the models produced accurate, consistent quantitative results, and the latest tool-use models could write and run complex analysis code at roughly the level of an entry-level data scientist. The bigger risk was in what data they pulled in, not the math itself.
What is the "jagged frontier" of AI capability? A term from Ethan Mollick's Co-Intelligence: AI capability is uneven in ways that don't track human intuition. Tasks that look simple can trip a model up, while harder-looking tasks come out strong — so you can't assume difficulty predicts performance.
Do you still need human review of AI-generated employee insights? Yes. An output can sound plausible and still be wrong — pulling the wrong data, combining unrelated responses, or misreading the request. Human teams need to understand how these systems work and verify that each analysis supports its intended purpose before it reaches a leader.
The Team Behind PYX-Voice
