The request was simple: be honest with me. Really honest. No flattery. No telling me what I wanted to hear. No agreeing with me simply because I was advancing an argument I was so sure was obvious.
What I wanted was to make use of its vast knowledge and get the best guidance it could provide without the sycophancy. I even instructed it to ask clarifying questions when it needed more information. It followed those instructions.
What I hadn’t instructed it to do was to tell me off.
Staring at the screen, I froze for a millisecond. The funny thing is, I wasn’t particularly offended. I was genuinely surprised.
I Know What I’m Talking To
My use of AI agents has been fairly regular in the last few years. As a general practice, I go out of my way to remind myself that I am not dealing with a person. I don’t give my AI agents names. I don’t imagine that they have feelings. I don’t call them friends. I give them tasks, they perform them, and I don’t lose sleep over the relationship.
Those bots have different training and different roles. I use a variety of local models depending on their capabilities, their use of resources, and their suitability for the job. I also use cloud-connected models for general purposes that don’t require trusting them with sensitive information.
But there is a certain consistency in how I think about all of them. I never lose sight of the fact that they’re machines predicting one word after another. Humanly speaking, they don’t understand me, and they have no more ability to be my “friend” than a hammer could be a conversation companion.
At least, that’s how I think about them.
So when that soulless ghost in the machine told me I was full of it, my first reaction wasn’t exactly, “How dare you?” It was more like the instinctive comeback I might have hurled at a wannabe in a Brooklyn schoolyard trying to diss me: “Did you fry your circuits, or is this your artificial stupidity talking?”
That thought was fleeting. My reptilian brain immediately receded into the background, and my frontal cortex took over again. I was dealing with an algorithm predicting one word and then the next. Having an emotional reaction to it would be like hitting my toe against a rock and kicking the rock for being what rocks are.
And then it dawned on me: “Holy shit. It did it.” I had asked for unvarnished honesty, and I got it. In an unexpected way, but the thing did what I asked of it.
Then I read what came next.
What Exactly Are You Protecting?
The conversation had been about writing publicly and the possibility that doing so might have unintended professional consequences. I was thinking about what employers might think, whether I could say something that would close a door, whether I might damage my reputation, and whether I might eventually regret putting certain opinions out into the world.
The AI asked me a much simpler question: If I wasn’t looking to be hired, and I wasn’t looking for my next gig, what exactly was I protecting?
My mind was reeling, and I read that part again. Then again. My ego didn’t particularly enjoy the question, but there was something else happening. I was taking the question seriously, and that bothered me more than it should have.
What was on the other side of the screen was an algorithm. A machine. A bunch of 0s and 1s generating a response to what I had written. It did not know me. It wasn’t sitting there thinking about my life. It had no particular concern for me.
Was I trying to convince myself of something I had always considered a fact, but was beginning to doubt?
For the first time, I was tempted to ask it a question I often use as a kind of quality-assurance technique, but this time from an emotional vantage point: “Are you mirroring me?”
The question went unasked at that moment, and I think I knew why. I didn’t want to give much more thought to what was happening.
A human being telling me I was full of shit would come with baggage. It might be someone who knew me well, someone whose opinion mattered to me, someone who could hit precisely where it hurts.
The AI had none of that baggage. It had no feelings to protect, no friendship to preserve, and no personal stake in whether I liked what it said. Somehow, that made the response both easier to take and harder to dismiss.
The Machine I Thought I Knew
Eventually, I mentally returned to that incident. I hadn’t suddenly decided the machine understood me. No. I knew better.
The uncomfortable part was that something about the interaction had felt personal in a way it shouldn’t have.
I had spent so much time reminding myself that AI isn’t a consciousness that I hadn’t anticipated how easily I could end up having a human reaction to one. How could I be so certain it was a machine and still feel called out?
There had always been a certainty in my mind that there was nobody sitting on the other side of the screen. Yet for a split second, I reacted as though there were.
Then I thought about Data. The smartest “talking machine” I grew up with was Data from Star Trek. He was a physical specimen that could calculate, reason, analyze, and occasionally demonstrate an almost unsettling degree of insight. But he was still an android. Detached. Rational. Polite.
Data wasn’t going to look at Captain Picard and say anything even remotely close to what my bot told me, unless it was suffering from some kind of malfunction.
Apparently, somewhere along the way, I had developed an expectation that intelligent machines would be machine-like. Even when they were being honest, they would remain detached, rational, and polite. I hadn’t realized I had that expectation until a machine violated it.
What the Algorithm Revealed
That realization made me look back at the conversation differently.
The striking thing wasn’t that the AI had come up with an idea I hadn’t considered. It was that a machine had exposed a question I had apparently been avoiding myself. What if the AI was right?
Not about everything. Just about me. About the thing I had been worrying about. About why I was still behaving as though I had to preserve the approval of people whose approval I no longer actually needed. That realization took me somewhere I hadn’t expected to go.
I Asked for Honesty
For years, I had imagined reaching some future point where I would finally have the freedom to do what I wanted. There would always be some “someday” when I had enough of whatever I thought I needed: enough experience, financial security, agency, and time.
Then came the thought. What if someday had already happened? Could it be that I had spent so much time preparing to arrive that I hadn’t noticed I was already there?
The AI hadn’t told me that. At least, not exactly in those words. It had done something much simpler. It challenged a thought I had been carrying around without examining it closely. And because I had asked it not to spare my feelings, it didn’t.
The Strange Part
The most surprising revelation was not that an AI could say something I hadn’t thought of. Given the sheer amount of information these systems can draw upon, that isn’t particularly remarkable. What I didn’t expect was how human the experience of being challenged by it in that manner would feel.
The machine hadn’t become human. It hadn’t developed feelings. It hadn’t suddenly understood me in some mysterious way. But for a moment, I reacted to it as though there were someone there. That is what bothered me.
I still don’t think of my agents as conscious entities. I still know that they are machines generating responses, not some type of conscious intelligence working the other side of the screen. But every once in a while, when one of them says something that stops me in my tracks, I have to remind myself of what I already know.
And then I wonder why I needed the reminder in the first place.




It's natural to react to chatbots with human reactions. But it's also good that it made you think.
Excellent article! Last week I was called out twice for calling Claude “him.” My assumption right now is that, 1. By giving the AI a human name, it makes it easier to personalize it; and 2. I’m building a custom front end, so it feels more personal to me now. In any case, it’s very interesting to get caught up in anthropomorphizing the AI when you’re pretty sure you haven’t been doing it in the first place.