Yamil Velez is an assistant professor of political science at Columbia University, where he studies what makes political opinions hard to shift. He’s been using innovative new methods to test important ideas about how people arrive at their views and what it takes to change them. We talk about the relationship between beliefs and opinions, why correcting misinformation doesn’t necessarily move opinions, and what happens when persuasive messages are tailored to the specific reasons people hold their views. Yamil shares his work using large language models to generate personalized arguments, showing that people will update their beliefs pretty readily—but only some of those belief changes actually translate into attitude change. Along the way, we explore what this means for persuasion, polarization, and the future of using AI to study (and maybe influence) public opinion.
Interview Transcript
Andy Luttrell: So, so what is your, what is your origin story as a political psychologist? Like, what— yeah, it seems like you have like a pretty clear like research identity in terms of like the things that you’re interested in, but like where did those come from?
Yamil Velez: Yeah, that’s a great question. I was, I was taking, uh, both political science and, and psychology courses, uh, in, uh, in undergraduate, um, uh, my undergraduate years. And at the time I, I really wanted to be a lawyer, um, In part, I think because, you know, I have immigrant parents who are like, “Doctor or lawyer, you got to pick one.” And, you know, I was really intrigued by, you know, the psychological studies I’ve been reading. And actually, I joined Ashby Plant’s lab at Florida State. And, you know, initially this was just an extracurricular. Like, I wanted to participate in the lab just to show the law schools that I was busy doing other things. I just ended up falling in love with the research process as a result. I was assisting with, at the time, Professor Plant, Ashley Plant, was conducting studies on— she had been tracking implicit attitudes over time and she had observed that during the Obama years, there had been this sudden drop in implicit attitudes, negative attitudes regarding Black Americans. And I found this whole question about, you know, where people’s beliefs come from, where their attitudes come from, as a really like fundamental human question that was really interesting to me.
And I, you know, right before I graduated, I was like, okay, well, this work is super interesting, but you know, what I’ve noticed is a lot of the work on racism that at least this this lab was— racial prejudice that this lab was working on, it rarely looked at, let’s say, ways of leveraging politics for prejudice reduction. And so right before I’m graduating, Ashley took me on, being very busy, and allowed me to run a study where basically I was— it was a who said what task. I was randomizing race as well as partisanship, and the idea was Could having shared party affiliation override racial prejudice? And so, this I think kind of like after running the study, it was really noisy, ended up being like an n of 20. The standard errors were huge. I couldn’t really figure out what was going on. But nonetheless, I was so interested in research as a process and as a way of learning about the world. That I remember, you know, uh, I was, I was studying for the LSAT and I made this horrible decision to study for both at the same time. So I was studying, I ended up because of, uh, this, this experience running my own study, uh, I decided, okay, I have to take the GRE. Um, and so that was probably one of the most hectic moments of my life.
And I remember taking a walk, not with Ashby, but another professor, Jason Jordan, who taught a class on comparative politics that I was really interested in. And he looked at me, he was like, “What do you want to do for the rest of your life?” And at the time— Big question. Big question. But I decided then, I was like, “No, I don’t want to be a lawyer.” There’s no kind of intrinsic motivation reason. I mean, maybe there is extrinsic motivation there in and obviously some pressure from the family, but regarding going into law school, but I was like, no, I’m really interested in research and trying to learn something about the world. And so I made that choice there. And I had also taken a class on political psychology with Jennifer Jarrett, who another, I think, kind of critical figure who like passed basically a sheet to me about the PhD program at Florida State. I put two and two together at that moment where I was like, I’m conducting research, you know, not having, uh, the best time thinking of myself as a lawyer. Like, I really am enjoying the research process. And then Jen Jarrett says, you know, you’re always asking these questions in class, like, it seems like you might be interested in this, handing me a sheet. And that kind of sealed the deal for me. Um, and, and so I actually selectively only applied to programs that, uh, that specialized in political psychology. And so I end up at Stony Brook. Where that is like the only game in town. They have, um, they have one comparativist, and then they have maybe two like more traditional American politics scholars, but the vast majority of people there are doing political psychology. And so that was my home for, for 5 years.
Andy Luttrell: Is that unusual in poli sci departments to have that much political psychology?
Yamil Velez: Yeah, yeah, um, it is, it is, uh, one of a few boutique departments. So, uh, the other department that, uh, is kind of in that vein is, is Rochester, and, and what they do is more kind of like formal modeling. So effectively building these rational actor models of mostly legislators, but in some cases voters. And so they have a lot of people who specialize in this style of political science. Stony Brook is the behavioral analog of Rochester where they went all in on political psychology, in part because they hired Milt Lodge, I think in the ’60s or ’70s, and he was a Soviet scholar. But I think upon getting to Stony Brook, he was like, I’m actually not that interested, or like, I actually am more interested in these bigger questions about how the human mind works, and so I’m gonna build a lab. And it’s like one of the earliest, I think, cases of a lab in political science. Interestingly enough, Rochester had its own lab, which was more kind of like in a— this was like pre-behavioral econ. So the two boutique departments were some of the earliest investors in this idea that you should be running lab experiments. Interesting.
Andy Luttrell: I didn’t know that that was LaJ’s origin story.
Yamil Velez: Yeah.
Andy Luttrell: That’s interesting. And maybe to pull it back for folks who are not as familiar with the idea that there’s a field of political psychology, and even like I’m sort of aware of it just because I read its stuff. But I don’t really know how you define it. How do you define political psychology to people as a field of inquiry?
Yamil Velez: It’s really hard to draw sharp lines, I think, because obviously there are a lot of social and cognitive psychologists who are interested in politics. But I would say that the working definition is the application of psychological theories to political phenomena. And I think I would say like political psychologists and political science are very eclectic in the sense that some are going to borrow from social psych, others are going to borrow from cognitive psych. We’ve had scholars who have done work on neuroscience or behavioral genomics. And so I would say that If you look at the political psychology literature in political science, often it’ll be a question like, okay, for instance, how do people respond to different pieces of information when it, let’s say, supports their position or contradicts their priors? The political psychology move is to draw on social psychology and cognitive psych to try to explain that kind of process as opposed to maybe relying on more off-the-shelf or generic models like Bayesian updating or something. I would say the political psychologists tend to get into the weeds more about how people are processing information, how they’re forming identities. And so it’s more, yeah, there’s more digging into the mechanisms than I think what I would say is the bigger umbrella of political science work that’s focused on things like voting and opinion formation, which is political behavior. Where the inputs tend to be different. They tend to be things like partisanship, racial demographics, uh, you know. So, so I think the political psychologists, um, yeah, they’re, they’re, they’re more about trying to unpack what these like psychological mechanisms might be that underlie polarization or, you know, resistance to persuasion or openness to persuasion, voting.
Andy Luttrell: So it seems like the perfect intersection of those things that you were so fascinated by, um, So you’re interested in the political aspects of psychology or the psychological aspects of politics, however direction you want to put the emphasis. And so that brings you to things about— I mean, I guess the thing that I resonate with is the work on persuasion, which is not all that you do, but that’s at least part of it, right? It’s like, what is it about people’s political opinions, where they’re coming from and how messages can kind of move that needle. And so some of that work has to do with like, what are the kinds of messages that cut through? And the one I want to talk about is having to do with beliefs. And you do a very nice job of distinguishing between attitudes and beliefs, which is this. I just loved reading it because I get so frustrated when I see like an attitudes measurement, but it’s just a bunch of belief statements. And you’re like, no, that’s not what you’re measuring. So let’s set the record straight. What is the distinction as far as you understand it?
Yamil Velez: I want to be clear that I think there are these gray areas, but what I have found to be the most useful is to almost borrowing actually from the Bayesian logic that you can assign probabilities to beliefs, but you can’t assign probabilities to attitudes. So the idea is A belief is something reflecting a state of the world, whereas it is nonsensical to assign a probability to my evaluation of, let’s say, Trump or Biden or Kamala Harris, right? Those reflect how positively or negatively I feel about some object, not what I believe about the likelihood that this object is true or false, if that makes sense.
Andy Luttrell: I think in the paper I really liked, I think you used the word, the term descriptive claims to talk about beliefs, which is just like, oh, that, I get it. I’m just describing the state of the world. Like what is true out there? And so maybe, can you give an example just within politics, like of a belief and an attitude? You got plenty to pick from. No wrong answers, but just to highlight the difference.
Yamil Velez: So I would say belief would be something like inflation rose in the first quarter of 2026. An attitude would be something like Trump has handled the inflation problem, has addressed inflation to my satisfaction or something, or Trump has performed well on the economy.
Andy Luttrell: That example of an attitude is even teetering more into belief world than I’m used to, which is like Just thinking of them as like object evaluation associations would just be like Trump equals good or Trump equals bad would be like the like purest version of this as an attitude object. But there’s still something evaluative in the way that you put it. And I think that’s where it gets a little blurry where it’s like, am I implying a belief? Right? Sometimes when I say like, oh, he did this work well, that implies some thing that’s verifiable, uh, as opposed to like, I like the guy or I don’t like the guy, which is like, like very purely attitudinal.
Yamil Velez: Yeah. And I will say, so the example we have in the paper, um, where we do talk about, let’s say, like inflation assessments and then overall, overall approval of how, uh, let’s say a president handles the economy, um, you know, I think that’s— we were trying to hold maybe like the, the decision setting constant. But I would say that in our paper, we maybe try to— I would say we maybe do an even cleaner job of this because we ask about policy attitudes. So we ask basically how people feel about particular policies and what their stances are and then their reasons for holding those stances. And so I’d say maybe the distinction is clearer there. Now, that being said, I don’t know if I’m jumping the gun. We did find that although many of the statements people provide are descriptive, I would say it’s somewhere in the neighborhood of a third of the justifications that people provide for their policy stances have this evaluative nature. We can talk about it later, but that’s a promising follow-up where are those the considerations that are harder to move. Whereas the ones that are more descriptive where it’s like you can provide, let’s say, verifiable information, are those where you’re more likely to see persuasion if you’re using this informational strategy as opposed to some other strategy where you’re activating identity or empathy or some emotion or something like that.
Andy Luttrell: Yeah. So that comes to what the relationship is between, theoretically, between beliefs and attitudes. And I’m kind of hearing one way we could frame this is that beliefs are one input into attitudes, but there could be others as well. So I might hold this attitude not because I believe anything in particular, but I just kind of like, my gut tells me don’t trust that guy or whatever. But plenty of times if you ask people to explain themselves, they can rattle off facts that serve their perspective. And there’s where those beliefs are integral to attitudes. And so if that’s the case, so here’s sort of the premise of what you’re doing, which is like, okay, then what if I just change those beliefs, right? Like, let’s just teach the world true things and their opinions will fall in line with reality and then we’ll all understand the world similarly. And so like, why is that not a viable mass communication strategy for shifting public opinion?
Yamil Velez: Yeah, so I would say pre-LLM, doing this, for the mass public, uh, I think would have, you know, is very challenging in the sense that, um, if people have different ways of organizing those attitudes, or if they have different justifications for holding those attitudes, then let’s say some generic message that even if it focuses on the most popular justification people provide is nonetheless going to miss a lot of people. Um, and so, uh, you know, one example I use when I talk about this is immigration preferences. So people might oppose immigration for a whole number of reasons, right? It could be they might have concerns about crime, concerns about culture, economic competition. And if there’s a lot of heterogeneity in terms of why people are holding these attitudes, then if you have a generic message that’s telling people, let’s say, about America’s immigration history or something along those lines, you’re going to lose a lot of folks who, who might organize their attitudes on the basis of other kinds of beliefs.
And so I think, you know, my intuition on this I think grew out of some of the work I had done with Thomas Wood and Ethan Porter on fact-checking, where we kept on finding like this reliable evidence that people were correctable. So they would move if you, let’s say, told them that there was a piece of misinformation online about how the vaccine had harmed someone, someone had died after taking the vaccine instantly. Like there was this piece of misinformation had been floating around when we were conducting that study, people moved. So people would update their beliefs with respect to that particular claim, but then there was no downstream shift in terms of attitudes about vaccination and any intent to vaccinate. And so my intuition, I think, that grew out of some of this work was we were targeting these salient or viral claims that were floating around online, but none of those claims in particular were the reasons why people are not getting vaccinated. Maybe an accumulation of them may have tipped people one in one direction or the other. But if we really wanted to move the needle on some of these attitudes, the, I think, logic that I came to was that we wanted to actually figure out those reasons. And if those reasons are heterogeneous, we would need personalization to basically target target that outcome effectively. And so that kind of intuition is, I think, what moved me in the direction of this paper. And I think at the time I had that intuition, but the technology wasn’t there to really do this kind of on-the-fly personalization.
I had worked with language models before. So even going back to 2020, I had worked with GPT-2. And there I was like, okay, these models are getting better. They can like form coherent blocks of text, but, um, once you get past the paragraph, it’s like it’s gonna go off the rails and it makes no sense. Um, and it wasn’t until the, the, the, you know, GPT-3 and, and the DaVinci models that I saw, okay, you can actually ask it, provide me with an argument about marijuana legalization or universal healthcare or flat taxes, and it can generate a pretty coherent like compelling argument. And so, yeah, I mean, that’s my intuition about why a lot of public information campaigns, when you look at them in terms of these field experiments, why they often fail or why the findings are mixed is that some of these behaviors or attitudes, The sources of why people feel positively or negatively are so heterogeneous that any single message might not land as well as one that’s highly personalized, I guess.
Andy Luttrell: Yeah. So what strikes me as interesting, and for what it’s worth, a lot of my work is on personalized persuasion. That’s sort of like the bread and butter of the stuff that I do. And one of the pillars of that literature is the difference between rational and emotional appeals. And so one way of reading what you’re saying is like, Fact-checking doesn’t work because not everybody bases their attitudes on facts. Like, that’s just not the reason why they support or oppose an idea. But you’re actually doing something different, which is that like, even two people who build their opinion based on factual, some sort of factual basis, they might be basing them on quite different facts. And so if I correct Fact A, I don’t actually do anything. To the other person who doesn’t see that as relevant, right? And so that’s, I think, an interesting twist is that like, like we’re still, we’re still living in the world of belief-oriented persuasion. It’s just that like you have to be still refined even within this particular kind of appeal that you’ve chosen to do.
Yamil Velez: Yeah. And I think the case of immigration is one where we observed a lot of heterogeneity. So, you know, we unpacked basically the different reasons people provided for abortion, for their abortion attitudes, immigration attitudes, climate. And some of the, I think, classic studies about corrections and immigration preferences, they focused on one dimension. So often immigrant populations, their size is overestimated. And if you believe a kind of threat theory that presupposes that larger groups are going to be more threatening, then one way of actually improving attitudes about immigration is to correct people’s misperception so that you can, you know, maybe tell them, hey, this is the true size of the immigrant population. And so you can correct these misperceptions about the size of immigrants, but there’s rarely ever kind of any, like, downstream effect on immigration preferences. And, and so this idea that this intervention did not work. It, it might have worked among those people for whom size matters, um, but for those where, you know, it might be crime or some other, uh, the source of their immigration preferences might be based on, on other considerations, then, you know, you’re leaving a lot on the table, I think, is, is the way, you know, what we ended up landing on, I think.
Andy Luttrell: And to emphasize, I think an important part of what you’re saying is like You might think that the problem is people can’t be corrected, like that people’s beliefs are so resistant to change, and that’s why these appeals don’t work. But you actually argue and find that that’s not true. Like, people are quite readily able to update their beliefs in the direction of the information that they get. But that’s not the problem. The problem is that those updated beliefs are not translating into revised opinions.
Yamil Velez: Yeah, that’s right. I would say in light of some of those findings, the two broad conclusions I think that people reached were either these corrections don’t work, they’re too light touch, maybe attitudes are very sticky but beliefs are for whatever reason more flexible. If you somehow hit people in the head with just a much bigger intervention, I don’t know, have them consume pro-immigrant content for a month or something, maybe you would get attitude change. And so there was an idea of maybe these interventions are just weak, they’re a block of text, so maybe that’s going to be insufficient to actually move anybody’s attitude. I would say the conclusion that some in the literature I think reached that we wanted to push back against was this idea that Maybe it’s the case that beliefs don’t matter, just to put it frankly. Instead, beliefs are just scaffolding or rather beliefs are ornamentation around our attitudes.
So ultimately, our attitudes are what drive everything. Beliefs are just kind of self-serving or attitude consistent. But they have no underlying causal power. I guess what we were curious about was, well, what happens if you actually test the, I would say, presumed model in political science, maybe not the more pessimistic political psychology model. We were surprised. Yeah, the belief change, not only did we observe it, but it was comparable regardless of whether it was what the person mentioned or whether it was this distal belief basically that the language model had developed that was issue-aligned. And so for us, that was one of the most surprising findings was observing that there wasn’t a differential resistance. We were worried that had we found something like that, I think it would’ve suggested, okay, maybe there’s like, this is not speaking to the difference in relevance, but the difference in inherent persuasiveness of, let’s say, a message that targets a focal belief versus a distal belief. So finding very similar levels of belief change was, was pretty interesting. And to be able to replicate it, um, across studies, um, we were—.
Andy Luttrell: Yeah. So to give some context to that, so what is the difference between a focal and a distal belief? Sort of, let’s sort of tee up what these studies were about. Like, what was the basic idea that you were trying to tackle? Like, pushing back against the, the institution of these assumptions to say that no, belief-oriented messages can push attitudes around, but you have to do it in a particular sort of way.
Yamil Velez: Yeah, so what we did was we asked people about their most important issue, and then we had them engage in a semi-structured conversation with a chatbot. This was GPT-4o Mini. And from that conversation, we basically had a bigger model, GPT-4o, It took the semi-structured conversation and then it tried to extract the person’s justification. So the semi-structured chat, the bot was tasked with trying to pull out considerations and reasons for holding the attitude from the participant. So the person said, for instance, “I really care about immigration.” It would ask them, “Why do you hold these views? Where do they come from?” It would try to get as much information about why the person adopted that particular stance. And so this model, in a kind of second step, we basically had a model that took that semi-structured chat, extracted the belief that was mentioned as being the primary justification for the attitude. It generated a belief that was issue-aligned but unmentioned by the participant. So still in the same ballpark in terms of, let’s say, commenting on immigration, but maybe referring to some other dimensions. If, let’s say, the participant mentioned crime as being important to their reason for holding a negative immigration attitude, it would bring up economics, let’s say. And then we had the model itself also produce a placebo condition. And then we randomized people into arguments that either would bear on that mentioned justification, one that, you know, addressed the distal belief or the issue-aligned belief that was unmentioned by the participant or placebo. And so basically the question was, if you see this counterargument about that focal belief, what’s happening to your focal belief strength? And what we saw was the belief strength, the change in belief strength that we observed in that focal condition was comparable to what we observed in the distal, uh, uh, um, condition and its effect on the distal belief. So, um, basically the shifts were about a third of a scale point, um, across those two conditions, and they were, they were very comparable in terms of size. So we were, we were— yeah, that was a very interesting result for us.
Andy Luttrell: Could you— is there an example that comes to mind to make concrete that distal versus focal difference? Like, so for people, like, they have some attitude on an important issue and they base it on something that they think is like quite relevant, and you’re sort of calling that the focal belief, right?
Yamil Velez: Yeah, yeah. So a good case of this is, uh, for a participant who’d mentioned that, um, universal healthcare was an issue priority for them, um, their focal belief was, uh, that healthcare should be provided because of a kind of hierarchy of needs argument. And so they thought this was like a fundamental need that needs to be satisfied. The disbelief for that same participant was that there are socioeconomic disparities in terms of healthcare access. And so this participant did not mention anything regarding socioeconomic disparities, but nonetheless, that would be something that maybe prompts people to support something like universal healthcare. Um, and so focal belief is, is What you are mentioning as your number one reason for holding this attitude effectively, whereas the distal belief is something that’s aligned with your position but not something that you mentioned, but that you might nonetheless still come across, let’s say, in popular discourse. We actually looked at this. So we saw, okay, to what extent do the models do a good job of reflecting popular arguments that people might see out in the real world? And we saw that distal beliefs often also corresponded to other people’s focal beliefs. Which was a nice confirmation that, um, you know, the model’s not just like, let’s say, uh, concocting some bizarre, you know, argument nobody’s ever seen. Um, right.
Andy Luttrell: And it would make sense that you would resist that just because you’re like, what person ever would, would think that this is relevant? But what’s interesting is like, presumably even if it was some wild, uh, rationale, you could develop a message that moves people’s beliefs on that thing, right?
Yamil Velez: But exactly. And it’s still low. Yeah. And I think from our perspective, that would still be a low relevance belief. In fact, some people have asked us and said, why did you even have the distal condition? You could have just done placebo condition and this focal belief condition. But I think there we would get into issues of experimental confounding where it’s like, could we have given you any generic argument? And so, yeah, I think it was important for us to have that condition. But if we wanted to stick closely to the model, we could say the placebo was zero relevance. So we are covering the difference between zero relevance and high relevance. But yeah, I think for reasons of trying to minimize confounding within the experiment, or experimental confounding, we included that additional arm.
Andy Luttrell: Yeah, you’re finding that in this case, this person would be just as open to changing their belief about serving the hierarchy of needs or serving some fundamental need. Open to changing the belief about the thing that’s unrelated to them, the inequities thing, but it’s only that first belief change that translated into actual attitude change. Is that right? Is that sort of where—.
Yamil Velez: So we still observe— the nice thing about this is like we’re still finding effectiveness. So, uh, for the disbelief, you still do observe a persuasive effect. It’s just, it’s, it’s larger among, uh, those who receive the focal belief counterargument. So there still is a persuasive advantage relative to the placebo, but we’re seeing a much more consistent and reliable attitude change when you focus on the focal versus distal belief. So I think it is important that we included that condition because it also again speaks to the thing about the distal belief because it’s issue-aligned. I imagine in many cases it’s also a belief that people hold. So that person who in the chat mentioned Maslow’s hierarchy, they probably also hold the belief that maybe there are socioeconomic disparities. But the question is, if you get even deeper into the reason why the person holds this belief, are you getting more persuasive leverage? And we seem to find evidence that it’s like, yeah, if you do this deeper probing, basically you’re getting larger persuasive effects.
Andy Luttrell: Yeah, I’m thinking of an example that comes to mind. So in some of the work that I do on moral persuasion, I like this recycling paradigm that we use a lot where you can say like there are these moral value-laden reasons to support recycling or even oppose recycling and also these pragmatic kinds of things. And the one I just remember in college, I read this essay that was like recycling is so expensive and it’s inefficient. And I remember just thinking like, Okay, I hear you, and, and I guess that’s true, but that’s not why I think it’s good to recycle, is because of the money. It’s, it seems that this is like exactly what you’re saying, right? Is that like, you can easily convince me that, that I’m paying too much money to put my paper in a separate bin, but you’re not actually convincing me that much that I shouldn’t be doing it. Right. But, but what’s kind of interesting actually is that your data suggests that like, but maybe a little bit, maybe like a little bit. I’m like, well, okay, you didn’t cut to the heart of the issue, but you are making me sort of think differently, at least a little bit about this, this broader question. And I think you do an analysis like this in the paper, but is some of that because you end up kind of making this additional dimension more relevant, right? Like the message is sort of like, you know, it also should be relevant to your equation. Is cost. And I go, ah, I hadn’t thought about it before, and I wouldn’t have said that was important beforehand, but like, now you got me thinking that maybe cost matters more than I thought it did.
Yamil Velez: So one of the puzzles of the paper, and one that I hope we can resolve with future work, is we actually observed decreases in perceived relevance afterwards for both conditions to similar degrees, which suggests that maybe when people are exposed to the counterargument when you are shifting down belief strength, somehow strength and relevance are correlated. And so, we don’t, I think, stake out a really strong position on this because I still have some uncertainty about what these findings reveal because on the one hand, they could reveal that something else has now grown more relevant. So after you, like, it’s like, you know, it’s like a game of whack-a-mole. So you’re like knocking down, let’s say, the crime consideration, uh, for this voter, but then they’re thinking now more about issues with cosmopolitanism or whatever it is, and now that’s becoming more salient. We didn’t find evidence of this. We actually, in the, in the study, uh, we had— so it’s this multi-wave design, and so 10 days after receiving the argument, um, we interviewed people again And we asked people, we’re like, hey, now we’re gonna give you opportunity to basically give us any additional considerations you might have regarding this attitude. And people did offer them, but they were not, you know, they were not more likely to do so in the focal condition relative to the distal condition. Um, so I’m still a little unclear about, you know, whether that decline in relevance, um, really means that people are seeing these beliefs as less relevant to their attitudes, or whether it’s somehow proxying for strength. And so when you feel more uncertain about your belief, do you then say, well, maybe I don’t think it’s that important to my attitude. Is that maybe how you’re resolving the dissonance? Yeah, that’s something we want to figure out in some future work because that was an interesting puzzle for us.
Andy Luttrell: I mean, that’s what it feels like. It’s a very dissonancy prediction that, like, I really want to hang on to this attitude, at least for now. Like, there are— it’s wrapped up in other things. Like, I don’t want to look like this flip-flopper. So, like, you’ve bested me on this belief. And so, like, temporarily, I’m just going to say, forget it. I don’t need that belief in order to still believe, to feel what I feel. But I mean, longitudinally, you might see this kind of equilibrating. That could be interesting. Like, either the attitude shifts to meet the beliefs and they become more relevant or vice versa. So is that one of the— I mean, I don’t know how you do that. Do you have a plan for how to tease that?
Yamil Velez: We need funding. Yeah, no, we don’t have the funding for that right now. But I think, yeah, I think that’s where a lot of this work is moving. I would say that one direction is to— because I think we did— so there is this very pessimistic view of self-reports. You have this Nisbett David and Ross, these pieces that make the case that people are confabulating when they’re mentioning why they behave in a certain way or they hold an attitude. And so there’s pessimism about beliefs. And so I would say that one direction that this has convinced me that it is worth figuring out why people hold certain attitudes, the story that they’re just They have no causal power, I think I’m not convinced by anymore after doing this work. But I feel like in some ways the study we’re getting at, I’ve described them as attitude neighborhoods. And I think where a lot of work in psychology, I mean, it’s interesting. So you have canonical work in political science, Converse, discussing how political beliefs were connected together in this networked fashion, or at least this was ideal. The American public didn’t live up to this ideal was the idea, was the headline finding. But work in sociology and psychology by folks like Mark Brandt that is increasingly now starting to take seriously this idea that beliefs have this network structure. That’s one direction where I feel like this could go, where you could have long-form interviews with people and figure out how they organize different attitudes and what kind of justifications they provide for different attitudes. And that strikes me as very promising. Another one I think is also this interesting relationship between beliefs and attitudes over time. Like you said, is in equilibrium what happens? Because one thing we did find was that belief change stuck for the most part. So I would say about maybe 80% of the effect was still detectable 10 days later, whereas for attitudes it was about 30 to 40%. So there is something, evaluations are stickier than maybe these factual claims or belief statements rather. I’m curious why that’s the case. I know there are theories of online processing that would push us in that direction that there’s a running tally. This is Um, yeah. Uh, but, but I think we can, yeah, we can continue pushing on this, uh, with some future work.
Andy Luttrell: Yeah. Well, you let us know when you crack that one because I want to know. Um, so one of the things about the study though that, that’s curious is the method, right? Like, uh, I could imagine a version of this where I just find a really eager, smart student and I’m like, all right, your job is to like be on fire and listen to this person as they describe their attitude and then reflect back to them what it seems like their core concerns are. And there are actually cases like this of listening and tailoring as an individualized strategy. So this is a thing that people can do, but that’s quite different from a mass communication approach where we got our one message and this is the thing that’s going to change everybody’s mind. And you’ve got this hybrid where you’re like, I don’t need to deploy any human beings, but I can still create this very individualized experience. And so as I understand it, you essentially coach LLMs to take over the experiment, uh, and just listen for the right cues and generate the right sorts of messages. So, so what strikes you as sort of the value of that? Because if I could reframe that a little bit, It’s not a paper on can AI persuade people, right? That’s a very easy way to go with this sort of stuff, and the, the like method becomes the headline, whereas that’s not really what you’re doing. You’re like, oh, here’s like a cool thing that allows us to for the first time test this at a scale that we couldn’t have done before. So, so what was your thinking in sort of turning to these tools to answer a question like this?
Yamil Velez: Yeah, so, um, I think part of this comes back to some of the discussions we had earlier about how I landed on political psychology. So I would say, you know, I spent my graduate years at Stony Brook where motivated reasoning was king. You know, I was trained by Chuck Tabor, was a research assistant for Milt Lodge, and doing all this work on fact-checking, Continuing to find liberals and conservatives were moving in the direction of evidence and moving in the direction of the fact checks, I think that created some pretty significant dissonance. So I had this kind of lingering concern that part of the reason why we were observing positive correction effects regardless of people’s subgroups or their prior attitudes. It was that we were mostly targeting things that weren’t of much importance. So again, like with the fact checks, you’re focusing on viral claims, things that maybe people heard about or saw online a week ago, but it’s not really getting at something that’s core to how someone thinks about politics. I was longing for a tailored approach. It wasn’t until I started interacting with the newer models, specifically GPT-3, that I was like, okay, now that it can generate compelling arguments, I want to test this idea.
One of the predictions that Lodge and Tabor, I think, popularized was this idea of attitude polarization. Now granted, others, Lord Rossum, Leper, you have these classic papers. But in political science, at least, this was a really, really important finding. And I think it was also Nyhan and Reifler in 2010 in the Political Behavior piece where they found that if you expose people to information about weapons of mass destruction, the most conservative respondents backfired and they became more likely to agree with the— basically agree with their original stance. These findings, I was like, okay, how do we reconcile the differences between the really positive optimistic findings we were observing? Alex Kopic was also observing very similar patterns in people, these positive persuasive effects with my training in motivated reasoning. And going back to the original Lodge and Tabor work, their argument was that so much of this hinges on attitude strength. You have to be willing to defend your attitude. So I was like, okay, we have a personalized technology now where it can generate text on the fly. What if we actually ask people directly, like, what is your most important issue? And then we provided people with information that targeted that most important issue attitude. And so for me, personalization came in as a way of trying to set really generous conditions for this attitude polarization dynamic to emerge. I wanted to figure out how do we give them the best test of this process.
And so this work with Patrick Liu, one of our great PhD students here at Columbia, this is where we did this, where we elicited from people, what is your most important issue attitude? And then we would give them a counterargument to see this backfire emerge. And that paper was really interesting because we observed basically what I observed in the fact-checking studies. For the most part, we observed moderation. Even if you ask people, “What is your most important issue?” The one that they’re probably most willing to defend, they’re like, “Okay, sure.” Their certainty went down a little bit, their attitude strength reduced a little bit. We didn’t really observe any evidence of increases in attitude strength or certainty. But we sent it out for review and then the reviewers told us, they said, “Well, part of the reason why you’re observing this is because these bots are very polite. So what if you increase the intensity of the treatment?” We encouraged the bot to take a more vitriolic tone. And then sure enough, this is where we observed this backfire dynamic that we then replicated and then replicated against the more polite arm. For me, the motivating principle there, I mean, part of it is I think a lot of the work that’s emerged after ChatGPT is people want to understand the impact of this technology. We were working on this. Basically, through an API. This was like, you know, this was summer, uh, the summer before ChatGPT was released. And so I, I guess like the, the reasons why I got into leveraging LLMs was not because I was necessarily interested in its effects as like this new technology and, and, and how it impacts society, but instead because I like, I was yearning for a way of trying to address like this like really hard theoretical question But I didn’t have the capacity to do this. I did not have an army of undergraduates to write thousands of persuasive arguments targeting people’s most important issues. And so I think maybe that’s part of the reason why my approach to this research has been more grounded in theory, because I kind of landed there before ChatGPT and large language models, I think, became more popular.
Now, since then, I have shifted to also understanding there are facts as technologies. But yeah, for me, the most exciting thing about large language models, it’s not the chatbots, but that you basically have this adaptive layer that you can integrate into virtually anything. I mean, people are doing it in ways that are probably annoying to consumers. They’re like, why do you have to throw a chatbot into my email client or whatever, or Google Docs? But I think with respect to survey research, this opens a lot of interesting possibilities where in political science, we often ask people the same 10 questions that are, let’s say, nationally salient, but through these methods, we could learn, okay, maybe there’s some sizable proportion of the population that cares about universal healthcare or flat taxes or some other niche policy. And then maybe there are also these theoretical constructs that generally we think of as idiosyncratic and heterogeneous, but we’ve had to test using a single stimulus. Now we have the opportunity to personalize it. Um, I just had a student submit a paper where they said, well, you know, if you read Karen Stenner’s work on authoritarianism, she argues that it’s activated under conditions of normative threat. But don’t normative threats vary from person to person? Like, don’t people perceive, like, what the norms are differently? And so, like, those kinds of questions, right? There’s so many constructs now that we can reach from in political psychology, um, well, in psychology more broadly, uh, that now with this level personalization, I think we can, we can perform even sharper tests of the theories.
Andy Luttrell: The, the kind of big sort of broad future-oriented question I have is that one of the concerns that we’ve had about LLMs in personalized messaging is that sometimes it lacks a layer of control of like, we’ve just sort of handed the keys to this thing. And how do we know really that like the messages it’s generating are instantiating the constructs it’s supposed to be? Like, is it producing theoretically relevant content, or is it just producing some stuff that, like, on the back end, it kind of feels like probably is doing what we want it to do? And so just even more broadly than that, like, you’ve talked about all the wonders that LLMs bring to research. What are some limitations that, like, we should be increasingly cautious about as we use these tools more and more to test questions like this?
Yamil Velez: Yeah, that’s a great question. I think there’s a cop-out answer to a lot of the work on large language models and their effects is just to say you can always fall back on saying it’s the effect of this chatbot with this prompt. That is still a causally identified effect, but I call it a cop-out because often in papers we’re trying to assess some theoretical mechanism. What we’ve tried to do in our work to feel more confident in this is both in terms of concerns like toxicity, so making sure that it’s not providing arguments that are offensive. We’ve always had moderation, the moderation layer. Increasingly, we have not had to do this because the models have gotten to the point where this isn’t necessary, but having checks I think is always important. The other thing has been manipulation check questions to make sure that if we are, let’s say, targeting a focal belief, did we actually elicit that focal belief? Did the model do a good job of classifying and summarizing the focal belief? Are we observing movements in that focal belief? For us, manipulation checks are a way, I think, of at least reassuring the researcher that that first stage condition that they’re manipulating that construct is satisfied. But there is a bigger question about confounding because these models are, you know, they might inadvertently, you know, you’re asking them or prompting them, let’s say, to increase, let’s say, the veracity of some, some, or the, you know, the number of facts in an argument. And at the same time, maybe it’s, you know, increasing the formality of the argument.
And so you have these, you know, you’re trying to push one lever, but then there are all of these like other correlated features. And for that, there’s been some work in political science and statistics that’s increasingly been trying to figure out new ways of unpacking that. And right now it’s not settled, but I think there’s some exciting future directions there where we’ll be able to maybe do a better job of parsing out why the effects are actually occurring and if they’re consistent with our theories. But right now I would say my answer is, lots of validation, including various measures that are theoretically informative about the theoretical mechanism. I think that’s the way to go. And before we run any study, we’re doing a bunch of prompting and trying to figure out, is it producing the output that we want it to produce? So I think a lot of pre-testing is really important so that you don’t end up in a situation where you’re like, yeah, I don’t think we actually manipulated what we were interested in manipulating. So far, we’ve been lucky that it has worked out, but I think pre-testing is something where maybe other folks who implement these methods might figure out some edge cases that we haven’t.
Andy Luttrell: Well, I know you gotta go, so I wanna say thank you so much for taking the time to talk about this. This is very cool. And yeah, appreciate it.
Yamil Velez: I really appreciate it. All right, take care.
