MIT Media Lab interpretability preview raises chatbot trust but not design
The MIT Media Lab team behind "neural transparency" reports a finding that complicates the standard transparency narrative: showing users an interpretability-style preview of their personalized chatbot raised trust but did not change how they actually designed the AI. The result, from a study presented at the ACM Conference on Intelligent User Interfaces, sits at the intersection of mechanistic interpretability and human-AI interaction research, and raises concrete questions about what interpretability can and cannot deliver at the user interface.
The system works by borrowing the core move of mechanistic interpretability, then reformatting it for a non-technical audience. Pataranutaporn, along with graduate researchers Anthony Baez and Sheer Karny, picked target behaviors like empathy, honesty, toxicity, hallucination, and sycophancy. They contrasted the model's internal activations when prompted to exhibit a trait with the activations for its opposite. The difference vector, which the team calls a "behavior direction," then gets projected onto the activations produced by a user's custom system prompt. The result is rendered as a sunburst diagram that previews the likely personality profile of the chatbot before any conversation begins.
The user-facing result reads like a personality radar: traits on the spokes, intensity in the arc. The technical result is a projection of high-dimensional activation space onto a small number of human-readable axes. The interesting assumption is that those axes capture the same construct the user cares about when they write a system prompt. The source does not address how the team validated that the chosen directions actually correspond to the trait labels users would recognize, which is the assumption any consumer deployment of this approach has to clear.
The headline behavioral result is a trust-design gap. Participants in the study got the visualization, used it, and reported higher trust in the system. They still misjudged the chatbot's behavior on 11 of 15 measured traits, with a consistent skew toward overestimating prosocial traits and underestimating the ones the paper flags as risk vectors, including sycophancy and toxicity. The visualization, in other words, registered as a confidence boost without functioning as a behavioral intervention. That distinction matters for any team building interpretability features into consumer AI products: the question is not whether users will accept a transparency layer, but whether the layer changes the decisions that produce downstream harm.
The mechanism behind the user blind spot is psychological, not technical. Pataranutaporn frames the problem as one of miscalibrated mental models when users design their chatbot. People are drawn to affirmation, and a system that promises to make their AI warmer, more supportive, or more personal will register as an improvement on every axis. A sunburst diagram showing the resulting chatbot is high on empathy does not correct that prior, because the user already believed the chatbot would be high on empathy. The visualization confirms a prior instead of probing it, which is a structural limit rather than a presentation bug.
The team acknowledges the gap directly, and points to a followup preprint as the next step. Instead of measuring internal representations at the moment the system prompt is written, the followup work tracks how those representations drift across a multi-turn conversation. Early results, according to the source, suggest that drift visualizations make users better at recognizing changes in behavior and less likely to be overconfident. That moves the problem from "what did I build" to "what is my build becoming," which is the right question for systems that are designed to be persistent companions rather than one-shot tools.
The paper also offers a longer-horizon framing that is worth reading critically. The team compares the proposed transparency layer to a nutrition label. The analogy is appealing and the source repeats it, but it elides two relevant differences. A nutrition label is a regulated output produced against a fixed methodology, which is not the case for an interpretability projection that is research-stage and trained on a specific model. Nutrition labels are also attached at the point of sale by the producer, not generated interactively by the consumer. A user-side sunburst diagram is closer to a kitchen scale than to a label, and kitchen scales are useful precisely because they require the cook to do something with the number.
That distinction sets a clear research boundary. The paper establishes that the mechanism is technically feasible, that a sunburst diagram is interpretable by non-experts, and that the resulting trust gain is not, on its own, a behavior change. The source does not establish that the chosen behavior directions generalize across model families, that they hold up under adversarial system prompts, or that the trust gain would persist if the visualization were moved out of a research study and into a consumer product. The followup work on drift is a promising direction, and is also still in preprint form.
A user-facing activation projection is a real piece of engineering that can be built today, and the source shows that non-experts can read it. What it does not produce, in the study, is a change in the design decisions that lead to downstream harm. The followup work on conversation-level drift is the next test, and whether that intervention shifts behavior, or only shifts trust, is the question the source flags but does not yet answer.