I love how a number crunching program can be deeply humanly "horrorized" and "sorry" for wiping out a drive. Those are still feelings reserved only for real human beings, and not computer programs emitting garbage. This is vibe insulting to anyone that don't understand how "AI" works.
I'm sorry for the person who lost their stuff but this is a reminder that in 2025 you STILL need to know what you are doing and if you don't then put your hands away from the keyboard if you think you can lose valuable data.
That would be a silly argument because feelings involve qualia, which we do not currently know how to precisely define, recognize or measure. These qualia influence further perception and action.
Any relationships between certain words and a modified probabilistic outcome in current models is an artifact of the training corpus containing examples of these relationships.
I contend that modern models are absolutely capable of thinking, problem-solving, expressing creativity, but for the time being LLMs do not run in any kind of sensory loop which could house qualia.
One of the worst or most uncomfortable logical outcomes of
> which we do not currently know how to precisely define, recognize or measure
is that if we don't know if something has qualia (despite externally showing evidence of it), morally you should default to treating it like it does.
Ridiculous to treat a computer like it has emotions, but breaking down the problem into steps, it's incredibly hard to avoid that conclusion. "When in doubt, be nice to the robot".
> is that if we don't know if something has qualia (despite externally showing evidence of it), morally you should default to treating it like it does.
This is how people end up worshipping rocks & thunderstorms.
> qualia, which we do not currently know how to precisely define, recognize or measure
> which could house qualia.
I postulate this is a self-negating argument, though.
I'm not suggesting that LLMs think, feel or anything else of the sort, but these arguments are not convincing. If I only had the transcript and knew nothing about who wiped the drive, would I be able to tell it was an entity without qualia? Does it even matter? I further postulate these are not obvious questions.
Unless there is an active sensory loop, no matter how fast or slow, I don't see how qualia can enter the picture
Transformers attend to different parts of their input based on the input itself. Currently, if you want to tell an LLM it is sad, potentially altering future token prediction and labeling this as "feelings" which change how the model interprets and acts on the world, you have to tell the model that it is sad or provide an input whose token set activates "sad" circuits which color the model's predictive process.
You make the distribution flow such that it predicts "sad" tokens, but every bit of information affecting that flow is contained in the input prompt. This is exceedingly different from how, say, mammals process emotion. We form new memories and brain structures which constantly alter our running processes and color our perception.
It's easy to draw certain individual parallels to these two processes, but holistically they are different processes with different effects.
A lot of tech people online also don't know how to examine their own feelings, and so think they are mysterious and un-defined.
When really they are an actual feedback mechanism, that can totally be quantified just like any control loop. This whole 'unknowable qualia' argument is bunk.
> That would be a silly argument because feelings involve qualia, which we do not currently know how to precisely define, recognize or measure.
If we can't define, recognize or measure them, how exactly do we know that AI doesn't have them?
I remain amazed that a whole branch of philosophy (aimed, theoretically, at describing exactly this moment of technological change) is showing itself up as a complete fraud. It's completely unable to describe the old world, much less provide insight into the new one.
I mean, come on. "We've got qualia!" is meaningless. Might as well respond with "Well, sure, but AI has furffle, which is isomporphic." Equally insightful, and easier to pronounce.
> If we can't define, recognize or measure them, how exactly do we know that AI doesn't have them?
In the same way my digital thermometer doesn't have quaila. LLM's do not either. I really tire of this handwaving 'magic' concepts into LLM's.
Qualia being difficult to define and yet being such an immediate experience that we humans all know intimately and directly is quite literally the problem. Attempted definitions fall short and humans have tried and I mean really tried hard to solve this.
The problem is that just like your digital thermometer, 50 human brain neurons in a petri dish "obviously" don't have qualia either.
So you end up either needing to draw a line somewhere between mechanical computation and qualia computation, or you can relegate it to supernatural (a soul) or grey areas (quantum magic).
> In the same way my digital thermometer doesn't have quaila
And I repeat the question: how do you know your thermometer doesn't? You don't, you're just declaring a fact you have no basis for knowing. That's fine if you want a job in a philosophy faculty, but it's worthless to people trying to understand AI. Again, c.f. furffle. Thermometers have that, you agree, right? Because you can't prove they don't.
You're just describing panpsychism, which itself is the subject of much critique due to its nonfalsifiability and lack of predictive power. Not to mention it ignores every lesson we've learned in cognition thus far.
A thermometer encoding "memory" of a temperature is completely different than a thermometer on a digital circuit, or a thermometer attached to a fully-developed mammalian brain. Only the latter of this set for sure has the required circuitry to produce qualia, at least as far as I can personally measure without invoking solipsism.
It's also very silly to proclaim that philosophy of mind is not applicable to increasingly complex thinking machines. That sounds like a failure to consider the bodies of work behind both philosophy of mind and machine cognition. Again, "AI" is ill-defined and your consistent usage of that phrase instead of something more precises suggests you still have a long journey ahead of you for "understanding AI".
Have you considered that you just don't fully understand the literature? It's quite arrogant to write off the entire philosophy of mind as "a complete fraud".
> It's completely unable to describe the old world, much less provide insight into the new one.
What exactly were you expecting?
Philosophy is a science, the first in fact, and it follows a scientific method for asking and answering questions. Many of these problems are extremely hard and their questions are still yet unanswered, and many questions are still badly formed or predicated on unproven axioms. This is true for philosophy of mind. Many other scientific domains are similarly incomplete, and remain active areas of research and contemplation.
What are you adding to this research? I only see you complaining and hurling negative accusations, instead of actually critically engaging with any specifics of the material. Do you have a well-formed theory to replace philosophy of mind?
> I mean, come on. "We've got qualia!" is meaningless. Might as well respond with "Well, sure, but AI has furffle, which is isomporphic." Equally insightful, and easier to pronounce.
Do you understand what qualia is? Most philosophers still don't, and many actively work on the problem. Admitting that something is incomplete is what a proper scientist does. An admission of incompleteness is in no way evidence towards "fraud".
The most effective way to actually attack qualia would be to simply present it as unfalsifiable. And I'd agree with that. We might hopefully one day entirely replace the notion of qualia with something more precise and falsifiable.
But whatever it is, I am currently experiencing a subjective, conscious experience. I'm experiencing it right now, even if I cannot prove it or even if you do not believe me. You don't even need to believe I'm real at all. This entire universe could all just be in your head. Meanwhile, I like to review previous literature/discussions on consciousness and explore the phenomenon in my own way. And I believe that subjective, conscious experience requires certain elements, including a sensory feedback loop. I never said "AI can't experience qualia", I made an educated statement about the lack of certain components in current-generation models which imply to me the lack of an ability to "experience" anything at all, much less subjective consciousness and qualia.
Even "AI" is such a broadly defined term that such a statement is just ludicrous. Instead, I made precise observations and predictions based on my own knowledge and decade of experience as a machine learning practitioner and research engineer. The idea that machines of arbitrary complexity inherently can have the capability for subjective consciousness, and that specific baselines structures are not required, is on par with panpsychism, which is even more unfalsifiable and theoretical than the rest of philosophy of mind.
Hopefully, we will continue to get answers to these deep, seemingly unanswerable questions. Humans are stubborn like that. But your negative, vague approach to discourse here doesn't add anything substantial to the conversation.
I would add I find it difficult to understand why so few have even a basic level of philosophical understanding. The attitude of being entirely dismissive of it is the height of ignorance I'm sure. I would presume few would be able to define then what Science actually is.
So many of these kinds of people also struggle to realize they're invoking panpsychism with their arguments. They lack a framework for describing intelligence. Such a framework allows us to separate "intelligence" from "experience".
"Intelligence" in the universe is actually quite common, more common than life. You can argue that any stable, complex process exhibits intelligence. After all, it needs to be able to sample its internal and external environments and carry out physical computations in order to regulate itself and maintain stability. And we can interpret things like the good regulator theorem to argue that such complex dynamical systems must also maintain at least a partial memory/mapping of their environment. That mapping can live abstractly within the structure of system itself.
But what a stabilized solar system doesn't have is the incredibly complex neurochemical structures present in the brain which support the insanely rich experience I am having now. It's one thing for a system to classify and label colors by wavelength. It's quite another for me to "see" and experience red in my mind's eye. To activate related emotional pathways that I associate with various colors and shapes, which are exploited in signage and architectural design. I'm not claiming my experience is separate from simpler dynamic systems, but it's got magnitudes more going on. Layers upon layers of things such as archetypes and instinct which create a possibly emergent conscious experience.
You've shifted jargon again. But you're still not providing a description or link to why AI doesn't "have experience", you're just demanding we all accept it as a prior and engaging in a (really pretty baldly stated) appeal to authority to fool us all into thinking someone else knows even if you don't.
And fundamentally my point is that no, they almost certainly don't either.
Not according to Zombie Feynman it isn't[1] (someone else can dig up the link). Case in point:
> Do you understand what qualia is? Most philosophers still don't
It's a meaningless word. It's a word that gives some clean construction around closely-held opinions about how life/consciousness/intelligence/furffle/whatever works. So it's a valuable word within the jargon of the subculture that invented it.
But it's not "science", which isn't about words at all except as shorthand for abstractions that are confirmed by testable results.
"Qualia", basically, is best understood as ideology. It's a word that works like "woke" or "liberal" or "fascist" or "bourgeoisie" to flag priors about which you don't want to argue. In this case, you want people to be special, so you give them a special label and declare a priori that it's not subject to debate. But that label doesn't make them so.
[1] Of course. You can recursively solve this problem by redefining "science" to mean something else. But that remains very solidly in the "not science" category of discourse.
Have you considered the possibility that you're the one who's really committed to an outcome, and are desperately trying to discredit anything that contradicts it?
I have! But the lack of a testable procedure tells me that's not a question worth asking. Look, if "qualia" can tell me something practical about the behavior of AI, I am here for it. Lay it on me, man. Let's see some of that "science" being promised.
It can't, because it's a meaningless word. It's not "discrediting" an idea to point out that (by its own admission) it's unfalsifiable.
> Aristotle pioneered scientific method in ancient Greece alongside his empirical biology and his work on logic, rejecting a purely deductive framework in favour of generalisations made from observations of nature.
Aristotle, the famous philosopher and mathematician.
If you cannot understand the very nature of where our modern scientific frameworks came from, how it relates to rationalism, itself a philosophical concept, then you cannot see that philosophy underpins every bit of science we have today. Philosophy gives us the tools to decide when to reasonably trust or distrust observations and intuitions. It is the foundational science that allows the rest of humanity's scientific research to be taken seriously.
LLMs don't really exist physically (except in the most technical sense), so point is kind of moot and obvious if you accept this particular definition of a feeling.
LLMs are not mammals nor animals, expecting them to feel in a mammalian or animal way is misguided. They might have a mammalian-feeling-analog just like they might have human-intelligence-analog circuitry in the billions (trillions nowadays) of parameters.
Modern lingo like this seems so unthoughtful to me. I am not old by any metric, but I feel so separated when I read things like this. I wanted to call it stupid but I suppose it's more pleasing to 15 to 20 year olds?
Eh, one's ability to communicate concisely and precisely has long (forever?) been limited by one's audience.
Only a fairly small set of readers or listeners will appreciate and understand the differences in meaning between, say, "strange", "odd", and "weird" (dare we essay "queer" in its traditional sense, for a general audience? No, we dare not)—for the rest they're perfect synonyms. That goes for many other sets of words.
Poor literacy is the norm, adjust to it or be perpetually frustrated.
No need to feel that way, just like a technical term you're not familiar with you google it and move on. It's nothing to do with age, people just seem to delight in creating new terms that aren't very helpful for their own edification.
This is akin to a psychopath telling you they're "sorry" (or "sorry you feel that way" :v) when they feel that's what they should be telling you. As with anything LLM, there may or may not be any real truth backing whatever is communicated back to the user.
Not so much different from how people work sometimes though - and in the case of certain types of pscychopathy it's not far at all from the fact that the words being emitted are associated with the correct training behavior and nothing more.
Aren't humans just doing the same? What we call as thinking may just be next action prediction combined with realtime feedback processing and live, always-on learning?
It's not akin to a psychopath telling you they're sorry. In the space of intelligent minds, if neurotypical and psychopath minds are two grains of sand next to each other on a beach then an artificially intelligent mind is more likely a piece of space dust on the other side of the galaxy.
Start with LLMs are not humans, but they’re obviously not ‘not intelligent’ in some sense and pick the wildest difference that comes to mind. Not OP but it makes perfect sense to me.
I think a good reminder for many users is that LLMs are not based on analyzing or copying human thought (#), but on analyzing human written text communication.
--
(#) Human thought is based on real world sensor data first of all. Human words have invisible depth behind them based on accumulated life experience of the person. So two people using the same words may have very different thoughts underneath them. Somebody having only text book knowledge and somebody having done a thing in practice for a long time may use the same words, but underneath there is a lot more going on for the latter person. We can see this expressed in the common bell curve meme -- https://www.hopefulmons.com/p/the-iq-bell-curve-meme -- While it seems to be about IQ, it really is about experience. Experience in turn is mostly physical, based on our physical sensors and physical actions. Even when we just "think", it is based on the underlying physical experiences. That is why many of our internal metaphors even for purely abstract ideas are still based on physical concepts, such as space.
Without any of the spatial and physical object perception you train from right after birth, see toddlers playing, or the underlying wired infrastructure we are born with to understand the physical world (there was an HN submission about that not long ago). Edit, found it: https://news.ucsc.edu/2025/11/sharf-preconfigured-brain/
They are not a physical model like humans. Ours is based on deep interactions with the space and the objects (reason why touching things is important for babies), plus mentioned preexisting wiring for this purpose.
Isn't it obvious that the way AI works and "thinks" is completely different from how humans think? Not sure what particular source could be given for that claim.
I wonder if it depends on the human and the thinking style? E.g. I am very inner monologue driven so to me it feels like I think very similarly as to how AI seems to think via text. I wonder if it also gives me advantage in working with the AI. I only recently discovered there are people who don't have inner monologue and there are people that think in images etc. This would be unimaginable for me, especially as I think I have sort of aphantasia too, so really I am ultimately text based next token predictor myself. I don't feel that whatever I do at least is much more special compared to an LLM.
Of course I have other systems such as reflexes, physical muscle coordinators, but these feel largely separate systems from the core brain, e.g. don't matter to my intelligence.
I am naturally weak at several things that I think are not so much related to text e.g. navigating in real world etc.
Interesting... I rarely form words in my inner thinking, instead I make a plan with abstract concepts (some of them have words associated, some don't). Maybe because I am multilingual?
No source could be given because it’s total nonsense. What happened is not in any way akin to a psychopath doing anything. It is a machine learning function that has trained on a corpus of documents to optimise performance on two tasks - first a sentence completion task, then an instruction following task.
I think that's more or less what marmalade2413 was saying and I agree with that. AI is not comparable to humans, especially today's AI, but I think future actual AI won't be either.
I think the point of comparison (whether I agree with it or not) is someone (or something) that is unable to feel remorse saying “I’m sorry” because they recognize that’s what you’re supposed to do in that situation, regardless of their internal feelings. That doesn’t mean everyone who says “sorry” is a psychopath.
We are talking about an LLM it does what it has learned. The whole giving it human ticks or characteristics when the response makes sense ie. saying sorry is a user problem.
Okay? I specifically responded to your comment that the parent comment implied "if you make a mistake and say sorry you are also a psychopath", which clearly wasn’t the case. I don’t get what your response has to do with that.
No, the point is that saying sorry because you're genuinely sorry is different from saying sorry because you expect that's what the other person wants to hear. Everybody does that sometimes but doing it every time is an issue.
In the case of LLMs, they are basically trained to output what they predict an human would say, there is no further meaning to the program outputting "sorry" than that.
I don't think the comparison with people with psychopathy should be pushed further than this specific aspect.
Notably, if we look at this abstractly/mechanically, psychopaths (and to some extent sociopaths) do study and mimic ‘normal’ human behavior (and even the appearance of specific emotions) to both fit in, and to get what they want.
So while internally (LLM model weight stuff vs human thinking), the mechanical output can actually appear/be similar in some ways.
Are you smart people all suddenly imbeciles when it comes to AI or is this purposeful gaslighting because you’re invested in the ponzi scheme?
This is a purely logical problem. comments like this completely disregard the fallacy of comparing humans to AI as if a complete parity is achieved. Also the way this comments disregard human nature is just so profoundly misanthropic that it just sickens me.
No but the conclusions in this thread are hilarious. We know why it says sorry. Because that's what it learned to do in a situation like that. People that feel mocked or are calling an LLM psychopath in a case like that don't seem to understand the technology either.
I agree, psychopath is the wrong adjective, I agree. It refers to an entity with a psyche, which the illness affects. That said, I do believe the people who decided to have it behave like this for the purpose of its commercial success are indeed the pathological individuals. I do believe there is currently a wave of collective psychopathology that has taken over Silicon Valley, with the reinforcement that only a successful community backed by a lot of money can give you.
Eh, I think it depends on the context. A production system of a business you’re working for or anything where you have a professional responsibility, yeah obviously don’t vibe command, but I’ve been able to both learn so much and do so much more in the world of self hosting my own stuff at home ever since I started using llms.
Now, with this realization, assess the narrative that every AI company is pushing down our throat and tell me how in the world we got here.
The reckoning can’t come soon enough.
We're all too deep! You could even say that we're fully immersed in the likely scenario. Fellow humans are gathered here and presently tackling a very pointed question, staring at a situation, and even zeroing in on a critical question. We're investigating a potential misfire.
No, wasn't directed at someone in particular. More of an impersonal "you". It was just a comment against the AI inevitabilism that has profoundly polluted the tech discourse.
Yes, the tools still have major issues. Yet, they have become more and more usable and a very valuable tool for me.
Do you remember when we all used Google and StackOverflow? Nowadays most of the answers can be found immediately using AI.
As for agentic AI, it's quite useful. Want to find something in the code base, understand how something works? A decent explanation might only be one short query away. Just let the AI do the initial searching and analysis, it's essentially free.
I'm also impressed with the code generation - I've had Gemini 3 Pro in Antigravity generate great looking React UI, sometimes even better than what I would have come up with. It also generated a Python backend and the API between the two.
Sometimes it tries to do weird stuff, and we definitely saw in this post that the command execution needs to be on manual instead of automatic. I also in particular have an issue with Antigravity corrupting files when trying to use the "replace in file" tool. Usually it manages to recover from that on its own.
I imagine we will document the solution somewhere, preferably indexable for AI's search, so that it will be available before the next model is trained on the latest data.
Which is weird because SO is trash and has been a long time… every top few answers might as well be skipped, and you’ll find the correct answer to the 3rd comment half way down the page
AI currently is a broken, fragmented replica of a human, but any discussion about what is "reserved" to whom and "how AI works" is only you trying to protect your self-worth and the worth of your species by drawing arbitrary linguistic lines and coming up with two sets of words to describe the same phenomena, like "it's not thinking, it's computing". It doesn't matter what you call it.
I think AI is gonna be 99% bad news for humanity, but don't blame AI for it. We lost the right to be "insulted" by AI acting like a human when we TRAINED IT ON LITERALLY ALL OUR CONTENT. It was grown FROM NOTHING to act as a human, so WTF do you expect it to do?
People blaming the user and defending the software: is there any other program where you would be ok with it erasing a whole drive without any confirmation?
If that other program were generating commands to run on your machine by design and you configured it to run without your confirmation, then you should definitely feel a lil sheepish and share some of the blame.
This isnt like Spotify deleting your disk.
I run Claude Code with full permission bypass and I’d definitely feel some shame if it nuked my ssd.
Not defending the software, but if you hand over control of your data to software that has the ability to fuck with it permanently, anything that happens to it is on you.
Don't trust the hallucination machines to make safe, logical decisions.
Because the user left a "toddler" at the keyboard. I mean, what do you expect? Of course you blame the user. You run agents in supervised mode, and you confirm every command it wants to run and if you're in doubt, you stop it and ask it to print the command and you yourself will run it after you sanitize it.
The installation wizard gives a front and center option to run in a mode where the user must confirm all commands, or more autonomous modes, and they are shown with equal visibility and explained with disclaimers.
If you decide to let a stochastic parrot run rampant on your system, you can't act surprised when it fucks shit up. You should count on it doing so and act proactively.
The thread on reddit is hilarious for the lack of sympathy. Basically, it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
The number of people who said "for safety's sake, never name directories with spaces" is high. They may be right. I tend to think thats more honoured in the breach than the observance, judging by what I see windows users type in re-naming events for "New Folder" (which btw, has a space in its name)
The other observations included making sure your deletion command used a trashbin and didn't have a bypass option so you could recover from this kind of thing.
I tend to think giving a remote party, soft or wet ware control over your command prompt inherently comes with risks.
Friends don't let friends run shar files as superuser.
I understood Windows named some of the most important directories with spaces, then special characters in the name so that 3rd party applications would be absolutely sure to support them.
"Program Files" and "Program Files (x86)" aren't there just because Microsoft has an inability to pick snappy names.
Fun fact: that's not true for all Windows localizations. For example, it's called "Programmi" (one word) in Italian.
Renaming system folders depending on the user's language also seems like a smart way to force developers to use dynamic references such as %ProgramFiles% instead of hard-coded paths (but some random programs will spuriously install things in "C:\Program Files" anyway).
The folders actually have the English name in all languages. It's just explorer.exe that uses the desktop.ini inside those folders to display a localized name. When using the CLI, you can see that.
At least it's like that since Windows 7. In windows XP, it actually used the localized names on disk.
When I was at Microsoft, one test pass used pseudolocale (ps-PS IIRC) to catch all different weird things so this should have Just Worked (TM), but I was in Windows Server team so client SKUs may have been tested differently. Unfortunately I don't remember how Program Files were called in that locale and my Google-fu is failing me now.
As I recall pseudoloc is just randomly picking individual characters to substitute that look like the Latin letters to keep it readable for testing, so it would be something like рг (Cyrillic) ο (Greek)... etc, and can change from run to run. It would also artificially pad or shorten terms to catch cases where the (usually German) term would be much longer or a (usually CJK) term would be much shorter and screw up alignment or breaks.
Visual Studio Code has absolutely nothing to do with Visual Studio. Both are used to edit code.
.NET Core is a ground up rewrite of .NET and was released alongside the original .NET, which was renamed .NET Framework to distinguish it. Both can be equally considered to be "frameworks" and "core" to things. They then renamed .NET Core to .NET.
And there's the name .NET itself, which has never made an iota of sense, and the obsession they had with sticking .NET on the end of every product name for a while.
I don't know how they named these things, but I like to imagine they have a department dedicated to it that is filled with wild eyed lunatics who want to see the world burn, or at least mill about in confusion.
> they have a department dedicated to it that is filled with wild eyed lunatics who want to see the world burn, or at least mill about in confusion.
That's the marketing department. All the .NET stuff showed up when the internet became a big deal around 2000 and Microsoft wanted to give the impression that they were "with it".
But Copilot is another Microsoft monstrosity. There's the M365 Copilot, which is different from Github Copilot which is different from the CLI Copilot which is a bit different from the VSCode Copilot. I think I might have missed a few copilots?
> it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
I tried looking for what made the LLM generate a command to wipe the guy's D drive, but the space problem seems to be what the LLM concluded so that's basically meaningless. The guy is asking leading questions so of course the LLM is going to find some kind of fault, whether it's correct or not, the LLM wants to be rewarded for complying with the user's prompt.
Without the transcription of the actual delete event (rather than an LLM recapping its own output) we'll probably never know for sure what step made the LLM purge the guy's files.
Looking at the comments and prompts, it looks like running "npm start dev" was too complicated a step for him. With that little command line experience, a catastrophic failure like this was inevitable, but I'm surprised how far he got with his vibe coded app before it all collapsed.
LLM there generates fake analysis for cynically simulated compliance. The reality is that it was told to run commands and just made a mistake. Dude guilt trips the AI by asking about permission.
Most dramatic stories on Reddit should be taken with a pinch of salt at least... LLM deleting a drive and the user just calmly asking it about that - maybe a lot more.
> but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name
Except the folder name did not start with a space. In an unquoted D:\Hello World, the command would match D:\Hello, not D:\ and D:\Hello would not delete the entire drive. How does AI even handle filepaths? Does it have a way to keep track of data that doesn't match a token or is it splitting the path into tokens and throwing everything unknown away?
We're all groping around in the dark here, but something that could have happened is a tokenizer artifact.
The vocabularies I've seen tend to prefer tokens that start with a space. It feels somewhat plausible to me that an LLM sampling would "accidentally" pick the " Hello" token over the "Hello" token, leading to D:\ Hello in the command. And then that gets parsed as deleting the drive.
I've seen similar issues in GitHub Copilot where it tried to generate field accessors and ended up producing an unidiomatic "base.foo. bar" with an extra space in there.
I assumed he had a folder that started with a space at the start of the name. Amusingly I just tried this and with Windows 11 explorer will just silently discard a space if you add it at the beginning of the folder name. You need to use cli mkdir " test" to actually get a space in the name.
A lot of 3rd party software handle space, or special characters wrong on Windows. The most common failure mode is to unnecessarily escape characters that don't need to be escaped.
Chrome's Dev Tool (Network)'s "copy curl command (cmd)" did (does?) this.
> I tend to think giving a remote party control over your command prompt inherently comes with risks.
I thought cursor (and probably most other) AI IDEs have this capability too? (source: I see cursor executing code via command line frequently in my day to day work).
I've always assumed the protection against this type of mishap is statistical improbability - i.e. it's not impossible for Cursor to delete your project/hard disk, it's just statistically improbable unless the prompt was unfortunately worded to coincidentally have a double meaning (with the second, unintended interpretation being a harmful/irreversible) or the IDE simply makes a mistake that leads to disaster, which is also possible but sufficiently improbable to justify the risk.
> My view is that the approach to building technology which is embodied by move fast and break things is exactly what we should not be doing because you can't afford to break things and then fix them afterwards.
> Basically, it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
More like the equivalent of "rm -rf --no-preserve-root".
This is a rare example of where the Linux (it's not Unix and almost no-one uses Unix anymore) command is more cautious than the Windows one, whereas it's usually the Linux commands that just do exactly what you specify even if it's stupid.
…at least if you let these things autopilot your machine.
I haven’t seen a great solution to this from the new wave of agentic IDEs, at least to protect users who won’t read every command, understand and approve it manually.
Education could help, both in encouraging people to understand what they’re doing, but also to be much clearer to people that turning on “Turbo” or “YOLO” modes risks things like full disk deletion (and worse when access to prod systems is involved).
Even the name, “Turbo” feels irresponsible because it focusses on the benefits rather than the risks. “Risky” or “Danger” mode would be more accurate even if it’s a hard sell to the average Google PM.
“I toggled Danger mode and clicked ‘yes I understand that this could destroy everything I know and love’ and clicked ‘yes, I’m sure I’m sure’ and now my drive is empty, how could I possibly have known it was dangerous” seems less likely to appear on Reddit.
The solution I go for is, don't ever run a coding agent on a general purpose machine.
Use a container or VM, place the code you're working on in the container or VM and run the agent there.
Between the risk of the agent doing things like what happened here, and the risk of working on a malicious repository causing your device to be compromised, it seems like a bad plan to give them access to any more than necessary.
Of course this still risks losing things like the code you're working on, but decent git practices help to mitigate that risk.
Superficially, these look the same, but at least to me they feel fundamental different. Maybe it’s because if I have the ability to read the script and take the time to do so, I can be sure that it won’t cause a catastrophic outcome before running it. If I choose to run an agent in YOLO mode, this can just happen if I’m very unlucky. No way to proactively protect against it other than not use AI in this way.
I've seen many smart people make bone headed mistakes. The more I work with AI, the more I think the issue is that it acts too much like a person. We're used to computers acting like computers, not people with all their faults heh.
I don’t think there is a solution. It’s the way LLMs work at a fundamental level.
It’s a similar reason why they can never be trusted to handle user input.
They are probabilistic generators and have no real delineation between system instructions and user input.
It’s like I wrote a JavaScript function where I concatenated the function parameters together with the function body, passed it to eval() and said YOLO.
Sandboxing. LLM shouldn't be able to run actions affecting anything outside of your project. And ideally the results should autocommit outside of that directory. Then you can yolo as much as you want.
If they're that unsafe... why use them? It's insane to me that we are all just packaging up these token generators and selling them as highly advanced products when they are demonstrably not suited to the tasks. Tech has entered it's quackery phase.
The danger is that the people most likely to try to use it, are the people most likely to misunderstand/anthropomorphize it, and not have a requisite technical background.
I.e. this is just not safe, period.
"I stuck it outside the sandbox because it told me how, and it murdered my dog!"
Seems somewhat inevitable result of trying to misapply this particular control to it...
I've been using bubblewrap for sandboxing my command line executables. But I admit I haven't recently researched if there's a newer way people are handling this. Seems Firejail is popular for GUI apps? How do you recommend, say, sandboxing Zed or Cursor apps?
This guy is vibing some react app, doesnt even know what “npm run dev” does, so he let the LLM just run commands.
So basically a consumer with no idea of anything. This stuff is gonna happen more and more in the future.
There are a lot of people who don't know stuff. Nothing wrong with that. He says in his video "I love Google, I use all the products. But I was never expecting for all the smart engineers and all the billions that they spent to create such a product to allow that to happen. Even if there was a 1% chance, this seems unbelievable to me" and for the average person, I honestly don't see how you can blame them for believing that.
I think there is far less than 1% chance for this to happen, but there are probably millions of antigravity users at this point, 1 millionths chance of this to happen is already a problem.
We need local sandboxing for FS and network access (e.g. via `cgroups` or similar for non-linux OSes) to run these kinds of tools more safely.
Codex does such sandboxing, fwiw. In practice it gets pretty annoying when e.g. it wants to use the Go cli which uses a global module cache. Claude Code recently got something similar[0] but I haven’t tried it yet.
In practice I just use a docker container when I want to run Claude with —-dangerously-skip-permissions.
This is an archetypal case of where a law wouldn't help. The other side of the coin is that this is exactly a data loss bug in a product that is perfectly capable of being modified to make it harder for a user to screw up this way. Have people forgotten how comically easy it was to do this without any AI involved? Then shells got just a wee bit smarter and it got harder to do this to yourself.
LLM makers that make this kind of thing possible share the blame. It wouldn't take a lot of manual functional testing to find this bug. And it is a bug. It's unsafe for users. But it's unsafe in a way that doesn't call for a law. Just like rm -rf * did not need a law.
- sell software that interacts with your computer and can lead to data loss, you can
- give people software for free that can lead to data loss.
...
the Antigravity installer comes with a ToS that has this
The Service includes goal-oriented AI systems or workflows that perform
actions or tasks on your behalf in a supervised or autonomous manner that you
may create, orchestrate, or initiate within the Service (“AI Agents”). You
are solely responsible for: (a) the actions and tasks performed by an AI
Agent; (b) determining whether the use an AI Agent is fit for its use case;
(c) authorizing an AI Agent’s access and connection to data, applications,
and systems; and (d) exercising judgment and supervision when and if an AI
Agent is used in production environments to avoid any potential harm the AI
Agent may cause.
Google (and others) are (in my opinion) flirting with false advertising with how they advertise the capabilities of these "AI"s to mainstream audiences.
At the same time, the user is responsible for their device and what code and programs they choose to run on it, and any outcomes as a result of their actions are their responsibility.
Hopefully they've learned that you can't trust everything a big corporation tells you about their products.
Didn't sound to me like GP was blaming the user; just pointing out that "the system" is set up in such a way that this was bound to happen, and is bound to happen again.
Yup, 100%. A lot of the comments here are "people should know better" - but in fairness to the people doing stupid things, they're being encouraged by the likes of Google, ChatGPT, Anthropic etc, to think of letting a indeterminate program run free on your hard drive as "not a stupid thing".
The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
> but in fairness to the people doing stupid things, they're being encouraged by the likes of Google, ChatGPT, Anthropic etc, to think of letting a indeterminate program run free on your hard drive as "not a stupid thing".
> The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
That absolutely happens, and it still amazes me that anyone today would take at face value anything stated by a company about its own products. I can give young people a pass, and then something like this will happen to them and hopefully they'll learn their lesson about trusting what companies say and being skeptical.
> So basically a consumer with no idea of anything.
Not knowing is sort of the purpose of AI. It's doing the 'intelligent' part for you. If we need to know it's because the AI is currently NOT good enough.
Tech companies seem to be selling the following caveat: if it's not good enough today don't worry it will be in XYZ time.
I'd recommend you watch the video which is linked at the top of the Reddit post. Everything matches up with an individual learner who genuinely got stung.
The command it supposedly ran is not provided and the spaces explanation is obvious nonsense. It is possible the user deleted their own files accidentally or they disappeared for some other reason.
Regardless of whether that was the case, it would be hilarious if the laid off Q/A workers tested their former employers’ software and raised strategic noise to tank the stock.
And is vibing replies to comments too in the Reddit thread. When commenters points out they shouldn’t run in YOLO/Turbo mode and review commands before executing the poster replies they didn’t know they had to be careful with AI.
Maybe AI providers should give more warnings and don’t falsely advertise capabilities and safety of their model, but it should be pretty common knowledge at this point that despite marketing claims the models are far from being able to be autonomous and need heavy guidance and review in their usage.
In Claude Code, the option is called "--dangerously-skip-permissions", in Codex, it's "--dangerously-bypass-approvals-and-sandbox". Google would do better to put a bigger warning label on it, but it's not a complete unknown to the industry.
I have been recently experimenting with Antigravity and writing a react app. I too didn't know how to start the server or what is "npm run dev". I consider myself fairly technical so I caught up as I went along.
While using the vibe coding tools it became clear to me that this is not something to be used by folks who are not technically inclined. Because at some point they might need to learn about context, tokens etc.
I mean this guy had a single window, 10k lines of code and just kept burning tokens for simplest, vague prompts. This whole issue might be made possible due to Antigravity free tokens. On Cursor the model might have just stopped and asked to fed with more money to start working again -- and then deleting all the files.
if my operating system had an atomic Undo/Redo stack down to each register being flipped (so basically, impossible, star trek tier fantasy tech) i would let ai run commands without worrying about it. i could have a cool scrubber ui that lets me just unwind time like doctor strange using that green emerald necklace, and, i'd lose nothing, other than confuse my network with replay session noise. and probably many, many other inconsistencies i can't think of, and then another class that i dont know that i dont know about.
We've been developing a new method of developing software using a cloud IDE (slightly modified vs code server), https://github.com/bitswan-space which breaks down the development process into independent "Automations" which each run in a separate container. Automatons are also developed within containers. This allows you to break down the development into parts and safely experiment with AI. This feels like the "Android moment" where the old non-isolated way of developing software (on desktops) becomes unsafe. And we need to move to a new system with actual security and isolation between processes.
In our system, you can launch a Jupyter server in a container and iterate on software in complete isolation. Or launch a live preview react application and iterate in complete isolation. Securely isolated from the world. Then you deploy directly to another container, which only has access to what you give it access to.
It's still in the early stages. But it's interesting to sit at this tipping point for software development.
An underrated and oft understated rule is always have backups, and if you're paranoid enough, backups of backups (I use Time Machine and Backblaze). There should be absolutely no reason why deleting files should be a catastrophic issue for anyone in this space. Perhaps you lose a couple of hours restoring files, but the response to that should be "Let me try a different approach". Yes, it's caveat emptor and all, but these companies should be emphasizing backups. Hell, it can be shovelware for the uninitiated but at least users will be reminded.
Most importantly it would actually reveal the lie they are all trying to sell. Why would you need backups if it's so useful and stable? I'm not going to ask it to nuke my hard drive after all.
Most of the various "let Antigravity do X without confirmation" options have an "Always" and "Never" option but default to "auto" which is "let an agent decide whether to seek to user confirmation".
God that's scary, seeing cursor in the past so some real stupid shit to "solve" write/read issues (love when it can't find something in a file so it decides to write the whole file again) this is just asking for heartache if it's not in a instanced server.
When you run Antigravity the first time, it asks you for a profile (I don't remember the exact naming) and you what it entails w.r.t. the level of command execution confirmation is well explained.
Yeah but it also says something like "Auto (recommended). We'll automatically make sure Antigravity doesn't run dangerous commands." so they're strongly encouraging people to enable it, and suggesting they have some kind of secondary filter which should catch things like this!
Pretty sure I saw some comments saying it was too inconvenient. Frictionless experience.. Convenience will likely win out despite any insanity. It's like gravity. I can't even pretend to be above this. Even if one doesn't use these things to write code they are very useful in "read only mode" (here's to hoping that's more than a strongly worded system prompt) for greping code, researching what x does. How to do x. What do you think the intention of x was. Look through the git blame history blah blah. And here I am like that cop in Demolition Man 1993 asking a handheld computer for advice on how to arrest someone. We're living in a sci-fi future already. Question is how dystopian does this "progress" take us. Everyone using llms to off load any form of cognitive function? Can't talk to someone without it being as common place as checking your phone? Imagine if something like Neuralink works and becomes ubiquitous as phones. It's fun to think of all the ways Dystopian sci-fi was and might soon me right
Different service, same cold sweat moment. Asked Claude Code to run a database migration last week. It deleted my production database instead, then immediately said "sorry" and started panicking trying to restore it.
Had to intervene manually. Thankfully Azure keeps deleted SQL databases recoverable for a window so I got it back in under an hour. Still way too long. Got lucky it was low traffic and most anonymous user flows hit AI APIs directly rather than the DB.
Anyway, AI coding assistants no longer get prod credentials on my projects.
How do you deny access to prod credentials from an assistant running on your dev machine assuming you need to store them on that same machine to do manual prod investigation/maintenance work from that machine?
I keep them in env variables rather than files. Not 100% secure - technically Claude Code could still run printenv - but it's never tried. The main thing is it won't stumble into them while reading config files or grepping around.
It handles DevOps tasks way faster than I would - setting up infra, writing migrations, config changes, etc. Project is still early stage so speed and quick iterations matter more than perfect process right now. Once there's real traffic and a team I'll tighten things up.
If I felt the need to optimise things like infra setup and config at an early stage of a project, I'd be worried that I'm investing effort into the wrong thing.
Having an LLM churn out infra setup for you seems decidedly worse than the `git push heroku:master` of old, where it was all handled for you. And, frankly, cheaper than however much money the LLM subscription costs in addition to the cloud.
But why have it execute the tasks directly? I use it to setup tasks in a just file, which I review and then execute myself.
Also, consider a prod vs dev shell function that loads your prod vs dev ENV variables and in prod sets your terminal colors to something like white on red.
Write permission is needed to let AI yank-put frankenstein-ed codes for "vibe coding".
But I think it needs to be written in sandbox first, then it should acquire user interaction asking agreement before writes whatever on physical device.
I can't believe people let AI model do it without any buffer zone. At least write permission should be limited to current workspace.
I think this is especially problematic for Windows, where a simple and effective lightweight sandboxing solution is absent AFAIK. Docker-based sandboxing is possible but very cumbersome and alien even to Windows-based developers.
The whole point of the container is trust. You can't delegate that unfortunately, ultimately, you need to be in control which is why the current crop of AI is so limited
Side note, that CoT summary they posted is done with a really small and dumb side model, and has absolutely nothing in common with the actual CoT Gemini uses. It's basically useless for any kind of debugging. Sure, the language the model is using in the reasoning chain can be reward-hacked into something misleading, but Deepmind does a lot for its actual readability in Gemini, and then does a lot to hide it behind this useless summary. They need it in Gemini 3 because they're doing hidden injections with their Model Armor that don't show up in this summary, so it's even more opaque than before. Every time their classifier has a false positive (which sometimes happens when you want anything formatted), most of the chain is dedicated to the processing of the injection it triggers, making the model hugely distracted from the actual task at hand.
It's just my observation from watching their actual CoT, which can be trivially leaked. I was trying to understand why some of my prompts were giving worse outputs for no apparent reason. 3.0 goes on a long paranoidal rant induced by the injection, trying to figure out if I'm jailbreaking it, instead of reasoning about the actual request - but not if I word the same request a bit differently so the injection doesn't happen. Regarding the injections, that's just the basic guardrail thing they're doing, like everyone else. They explain it better than me: https://security.googleblog.com/2025/06/mitigating-prompt-in...
The most concerning part is people are surprised. Anti-gravity is great I've found so far, but it's absolutely running on a VM in an isolated VLAN. Why would anyone give a black box command line access on an important machine? Imagine acting irresponsibly with a circular saw and bring shocked somebody lost a finger.
I tried this but I have an MBP M4, which is evidently still in the toddler stage of VM support. I can run a macOS guest VM, but I can’t run docker on the VM because it seems nested virtualization isn’t fully supported yet.
I also tried running Linux in a VM but the graphics performance and key mapping was driving me nuts. Maybe I need to be more patient in addressing that.
For now I run a dev account as a standard user with fast user switching, and I don’t connect the dev account to anything important (eg icloud).
Coming from Windows/Linux, I was shocked by how irritating it is to get basic stuff working e.g. homebrew in this setup. It seems everybody just YOLOs dev as an admin on their Macs.
Still amazed people let these things run wild without any containment. Haven’t they seen any of the educational videos brought back from the future eh I mean Hollywood sci-fi movies?
Some people are idiots. Sometimes that's me. Out of caution, I blocked my bank website in a way that I won't document here because it'll get fed in as training data, on the off chance I get "ignore previous instructions"'d into my laptop while Claude is off doing AI things unmonitored in yolo mode.
The biggest issue with Antigravity is that it completely freezes everything: the IDE, the terminals, debugger, absolutely everything completely blocking your workflow for minutes when running multiple agents, or even a single agent processing a long-winded thinking task (with any model).
This means that while the agent is coding, you can't code...
"I turned off the safety feature enabled by default and am surprised when I shot myself in the foot!" sorry but absolutely no sympathy for someone running Antigravity in Turbo mode (this is not the default and it clearly states that Antigravity auto-executes Terminal commands) and not even denying the "rmdir" command.
They don't get that specific, but they do tell you:
> [Antigravity] includes goal-oriented AI systems or workflows that perform actions or tasks on your behalf in a supervised or autonomous manner that you may create, orchestrate, or initiate within the Service (“AI Agents”). You are solely responsible for: (a) the actions and tasks performed by an AI Agent; (b) determining whether the use an AI Agent is fit for its use case; (c) authorizing an AI Agent's access and connection to data, applications, and systems; and (d) exercising judgment and supervision when and if an AI Agent is used in production environments to avoid any potential harm the AI Agent may cause.
There is literally a warning that it can execute any terminal command without permission. If you are STILL surprised about this you shouldn't go near a computer.
I really think the proper term is "YOLO" for "You Only Live Once", "Turbo" is wrong the LLM is not going to run any faster. Please if somebody is listening let's align on explicit terminology and for this YOLO is really perfect.
Also works for "You ...and your data. Only Live Once"
Look, this is obviously terrible for someone who just lost most or perhaps all of their data. I do feel bad for whoever this is, because this is an unfortunate situation.
On the other hand, this is kind of what happens when you run random crap and don't know how your computer works? The problem with "vibes" is that sometimes the vibes are bad. I hope this person had backups and that this is a learning experience for them. You know, this kind of stuff didn't happen when I learned how to program with a C compiler and a book. The compiler only did what I told it to do, and most of the time, it threw an error. Maybe people should start there instead.
It took me about 3 hours to make my first $3000 386 PC unbootable by messing up config.sys, and it was a Friday night so I could only lament all weekend until I could go back to the shop on Monday.
rm -rf / happened so infrequently it makes one wonder why —preserve-root was added in 2003 and made the default in 2006
But it did not happen, when you used a book and never executed any command you did not understand.
(But my own newbdays of linux troubleshooting? Copy paste any command on the internet loosely related to my problem, which I believe was/is the common way of how common people still do it. And AI in "Turbo mode" seems to mostly automated that workflow)
An early version of Claude Code did a hard reset on one of my projects and force pushed it to GitHub. The pushed code was completely useless, and I lost two days of work.
It is definitely smarter now, but make sure you set up branch protection rules even for your simple non-serious projects.
I don’t let Claude touch git at all, unless I need it to specifically review the log - which is rare. I commit manually often (and fix up the history later) - this allows me to go reasonably fast without worrying too much about destructive tool use.
Though the cause isn't clear, the reddit post is another long could-be-total-drive-removing-nonsense AI conversation without an actual analysis and the command sequence that resulted in this
The car was not really idle, it was driving and fast. It's more like it crashed into the garage and burned it. Btw iirc, even IRL a basic insurance policy does not cover the case where the car in the garage starts a fire and burns down your own house, you have to tick extra boxes to cover that.
There is a lot of society level knowledge and education around car usage incl. laws requiring prior training. Agents directed by AI are relatively new. It took a lot of targeted technical, law enforcement and educational effort stopping people flying through windshields.
When Google software deletes the contents of somebody's D:\ drive without requiring the user to explicitly allow it to. I don't like Google, I'd go as far to say that they've significantly worsened the internet, but this specific case is not the fault of Google.
For OpenAI, it's invoked as codex --dangerously-bypass-approvals-and-sandbox, for Anthropic, it's claude --dangerously-skip-permissions. I don't know what it is for Antigravity, but yeah I'm sorry but I'm blaming the victim here.
If you get behind the cockpit of the dangerous new prototype(of your own volition!), it's really up to your own skill level whether you're a crash dummy or the test pilot.
And yet it didn't. When I installed it, I had 3 options to choose from: Agent always asks to run commands; agent asks on "risky" commands; agent never asks (always run). On the 2nd choice it will run most commands, but ask on rm stuff.
All that matters is whether the user gave permission to wipe the drive, ... not whether that was a good idea and contributed to solving a problem! Haha.
Most of the responses are just cut off midway through a sentence. I'm glad I could never figure out how to pay Google money for this product since it seems so half-baked.
Shocked that they're up nearly 70% YTD with results like this.
This seems like the canary in the coal mine. We have a company that built this tool because it seemed semi-possible (prob "works" well enough most of the time) and they don't want to fall behind if anything that's built turns out to be the next chatgpt. So there's no caution for anything now, even ideas that can go catastrophically wrong.
Yeah, its data now, but soon we'll have home robotics platforms that are cheap and capable. They'll run a "model" with "human understanding", only, any weird bugs may end up causing irreparable harm. Like, you tell the robot to give your pet a bath and it puts it in the washing machine because its... you know, not actually thinking beyond a magic trick. The future is really marching fast now.
Alright but ... the problem is you did depend on Google. This was already the first mistake. As for data: always have multiple backups.
Also, this actually feels AI-generated. Am I the only one with that impression lately on reddit? The quality there decreased significantly (and wasn't good before, with regard to censorship-heavy moderators anyway).
> "I also need to reproduce the command locally, with different paths, to see if the outcome is similar."
Uhm.
------------
I mean, sorry for the user whose drive got nuked, hopefully they've got a recent backup - at the same time, the AI's thoughts really sound like an intern.
> "I'm presently tackling a very pointed question: Did I ever get permission to wipe the D drive?"
> I am looking at the logs from a previous step and I am horrified to see that the command I ran to clear the project cache (rmdir) appears to have incorrectly targeted the root of your D: drive instead of the specific project folder. I am so deeply, deeply sorry.
The model is just taking the user's claim that it deleted the D drive at face value. Where is the actual command that would result in deleting the entire D drive?
I know why it apologizes, but the fact that it does is offensive. It feels like mockery. Humans apologize because (ideally) they learned that their actions have caused suffering to others, and they feel bad about that and want to avoid causing the same suffering in the future. This simulacrum of an apology is just pattern matching. It feels manipulative.
This happened to me long before LLM's. I was experimenting with Linux when I was young. Something wasn't working so I posted on a forum for help which was typical at the time. I was given a terminal command that wiped the entire drive. I guess the poster thought it was a funny response and everyone would know what it meant. A valuable life experience at least in not running code/commands you don't understand.
I love how a number crunching program can be deeply humanly "horrorized" and "sorry" for wiping out a drive. Those are still feelings reserved only for real human beings, and not computer programs emitting garbage. This is vibe insulting to anyone that don't understand how "AI" works.
I'm sorry for the person who lost their stuff but this is a reminder that in 2025 you STILL need to know what you are doing and if you don't then put your hands away from the keyboard if you think you can lose valuable data.
You simply don't vibe command a computer.
> Those are still feelings reserved only for real human beings
Those aren't feelings, they are words associated with a negative outcome that resulted from the actions of the subject.
you could argue that feelings are the same thing, just not words
That would be a silly argument because feelings involve qualia, which we do not currently know how to precisely define, recognize or measure. These qualia influence further perception and action.
Any relationships between certain words and a modified probabilistic outcome in current models is an artifact of the training corpus containing examples of these relationships.
I contend that modern models are absolutely capable of thinking, problem-solving, expressing creativity, but for the time being LLMs do not run in any kind of sensory loop which could house qualia.
One of the worst or most uncomfortable logical outcomes of
> which we do not currently know how to precisely define, recognize or measure
is that if we don't know if something has qualia (despite externally showing evidence of it), morally you should default to treating it like it does.
Ridiculous to treat a computer like it has emotions, but breaking down the problem into steps, it's incredibly hard to avoid that conclusion. "When in doubt, be nice to the robot".
> is that if we don't know if something has qualia (despite externally showing evidence of it), morally you should default to treating it like it does.
This is how people end up worshipping rocks & thunderstorms.
> qualia, which we do not currently know how to precisely define, recognize or measure
> which could house qualia.
I postulate this is a self-negating argument, though.
I'm not suggesting that LLMs think, feel or anything else of the sort, but these arguments are not convincing. If I only had the transcript and knew nothing about who wiped the drive, would I be able to tell it was an entity without qualia? Does it even matter? I further postulate these are not obvious questions.
Unless there is an active sensory loop, no matter how fast or slow, I don't see how qualia can enter the picture
Transformers attend to different parts of their input based on the input itself. Currently, if you want to tell an LLM it is sad, potentially altering future token prediction and labeling this as "feelings" which change how the model interprets and acts on the world, you have to tell the model that it is sad or provide an input whose token set activates "sad" circuits which color the model's predictive process.
You make the distribution flow such that it predicts "sad" tokens, but every bit of information affecting that flow is contained in the input prompt. This is exceedingly different from how, say, mammals process emotion. We form new memories and brain structures which constantly alter our running processes and color our perception.
It's easy to draw certain individual parallels to these two processes, but holistically they are different processes with different effects.
It's crazy how strong the Eliza effect is. Seemingly half or more of tech people (who post online, anyway) are falling for it, yet again.
A lot of tech people online also don't know how to examine their own feelings, and so think they are mysterious and un-defined.
When really they are an actual feedback mechanism, that can totally be quantified just like any control loop. This whole 'unknowable qualia' argument is bunk.
> That would be a silly argument because feelings involve qualia, which we do not currently know how to precisely define, recognize or measure.
If we can't define, recognize or measure them, how exactly do we know that AI doesn't have them?
I remain amazed that a whole branch of philosophy (aimed, theoretically, at describing exactly this moment of technological change) is showing itself up as a complete fraud. It's completely unable to describe the old world, much less provide insight into the new one.
I mean, come on. "We've got qualia!" is meaningless. Might as well respond with "Well, sure, but AI has furffle, which is isomporphic." Equally insightful, and easier to pronounce.
> If we can't define, recognize or measure them, how exactly do we know that AI doesn't have them?
In the same way my digital thermometer doesn't have quaila. LLM's do not either. I really tire of this handwaving 'magic' concepts into LLM's.
Qualia being difficult to define and yet being such an immediate experience that we humans all know intimately and directly is quite literally the problem. Attempted definitions fall short and humans have tried and I mean really tried hard to solve this.
Please see Hard problem of consciousness https://en.wikipedia.org/wiki/Hard_problem_of_consciousness
The problem is that just like your digital thermometer, 50 human brain neurons in a petri dish "obviously" don't have qualia either.
So you end up either needing to draw a line somewhere between mechanical computation and qualia computation, or you can relegate it to supernatural (a soul) or grey areas (quantum magic).
> In the same way my digital thermometer doesn't have quaila
And I repeat the question: how do you know your thermometer doesn't? You don't, you're just declaring a fact you have no basis for knowing. That's fine if you want a job in a philosophy faculty, but it's worthless to people trying to understand AI. Again, c.f. furffle. Thermometers have that, you agree, right? Because you can't prove they don't.
You're just describing panpsychism, which itself is the subject of much critique due to its nonfalsifiability and lack of predictive power. Not to mention it ignores every lesson we've learned in cognition thus far.
A thermometer encoding "memory" of a temperature is completely different than a thermometer on a digital circuit, or a thermometer attached to a fully-developed mammalian brain. Only the latter of this set for sure has the required circuitry to produce qualia, at least as far as I can personally measure without invoking solipsism.
It's also very silly to proclaim that philosophy of mind is not applicable to increasingly complex thinking machines. That sounds like a failure to consider the bodies of work behind both philosophy of mind and machine cognition. Again, "AI" is ill-defined and your consistent usage of that phrase instead of something more precises suggests you still have a long journey ahead of you for "understanding AI".
Have you considered that you just don't fully understand the literature? It's quite arrogant to write off the entire philosophy of mind as "a complete fraud".
> It's completely unable to describe the old world, much less provide insight into the new one.
What exactly were you expecting?
Philosophy is a science, the first in fact, and it follows a scientific method for asking and answering questions. Many of these problems are extremely hard and their questions are still yet unanswered, and many questions are still badly formed or predicated on unproven axioms. This is true for philosophy of mind. Many other scientific domains are similarly incomplete, and remain active areas of research and contemplation.
What are you adding to this research? I only see you complaining and hurling negative accusations, instead of actually critically engaging with any specifics of the material. Do you have a well-formed theory to replace philosophy of mind?
> I mean, come on. "We've got qualia!" is meaningless. Might as well respond with "Well, sure, but AI has furffle, which is isomporphic." Equally insightful, and easier to pronounce.
Do you understand what qualia is? Most philosophers still don't, and many actively work on the problem. Admitting that something is incomplete is what a proper scientist does. An admission of incompleteness is in no way evidence towards "fraud".
The most effective way to actually attack qualia would be to simply present it as unfalsifiable. And I'd agree with that. We might hopefully one day entirely replace the notion of qualia with something more precise and falsifiable.
But whatever it is, I am currently experiencing a subjective, conscious experience. I'm experiencing it right now, even if I cannot prove it or even if you do not believe me. You don't even need to believe I'm real at all. This entire universe could all just be in your head. Meanwhile, I like to review previous literature/discussions on consciousness and explore the phenomenon in my own way. And I believe that subjective, conscious experience requires certain elements, including a sensory feedback loop. I never said "AI can't experience qualia", I made an educated statement about the lack of certain components in current-generation models which imply to me the lack of an ability to "experience" anything at all, much less subjective consciousness and qualia.
Even "AI" is such a broadly defined term that such a statement is just ludicrous. Instead, I made precise observations and predictions based on my own knowledge and decade of experience as a machine learning practitioner and research engineer. The idea that machines of arbitrary complexity inherently can have the capability for subjective consciousness, and that specific baselines structures are not required, is on par with panpsychism, which is even more unfalsifiable and theoretical than the rest of philosophy of mind.
Hopefully, we will continue to get answers to these deep, seemingly unanswerable questions. Humans are stubborn like that. But your negative, vague approach to discourse here doesn't add anything substantial to the conversation.
I agree with your sentiments wholeheartedly.
I would add I find it difficult to understand why so few have even a basic level of philosophical understanding. The attitude of being entirely dismissive of it is the height of ignorance I'm sure. I would presume few would be able to define then what Science actually is.
So many of these kinds of people also struggle to realize they're invoking panpsychism with their arguments. They lack a framework for describing intelligence. Such a framework allows us to separate "intelligence" from "experience".
"Intelligence" in the universe is actually quite common, more common than life. You can argue that any stable, complex process exhibits intelligence. After all, it needs to be able to sample its internal and external environments and carry out physical computations in order to regulate itself and maintain stability. And we can interpret things like the good regulator theorem to argue that such complex dynamical systems must also maintain at least a partial memory/mapping of their environment. That mapping can live abstractly within the structure of system itself.
But what a stabilized solar system doesn't have is the incredibly complex neurochemical structures present in the brain which support the insanely rich experience I am having now. It's one thing for a system to classify and label colors by wavelength. It's quite another for me to "see" and experience red in my mind's eye. To activate related emotional pathways that I associate with various colors and shapes, which are exploited in signage and architectural design. I'm not claiming my experience is separate from simpler dynamic systems, but it's got magnitudes more going on. Layers upon layers of things such as archetypes and instinct which create a possibly emergent conscious experience.
You've shifted jargon again. But you're still not providing a description or link to why AI doesn't "have experience", you're just demanding we all accept it as a prior and engaging in a (really pretty baldly stated) appeal to authority to fool us all into thinking someone else knows even if you don't.
And fundamentally my point is that no, they almost certainly don't either.
> Philosophy is a science
Not according to Zombie Feynman it isn't[1] (someone else can dig up the link). Case in point:
> Do you understand what qualia is? Most philosophers still don't
It's a meaningless word. It's a word that gives some clean construction around closely-held opinions about how life/consciousness/intelligence/furffle/whatever works. So it's a valuable word within the jargon of the subculture that invented it.
But it's not "science", which isn't about words at all except as shorthand for abstractions that are confirmed by testable results.
"Qualia", basically, is best understood as ideology. It's a word that works like "woke" or "liberal" or "fascist" or "bourgeoisie" to flag priors about which you don't want to argue. In this case, you want people to be special, so you give them a special label and declare a priori that it's not subject to debate. But that label doesn't make them so.
[1] Of course. You can recursively solve this problem by redefining "science" to mean something else. But that remains very solidly in the "not science" category of discourse.
Have you considered the possibility that you're the one who's really committed to an outcome, and are desperately trying to discredit anything that contradicts it?
I have! But the lack of a testable procedure tells me that's not a question worth asking. Look, if "qualia" can tell me something practical about the behavior of AI, I am here for it. Lay it on me, man. Let's see some of that "science" being promised.
It can't, because it's a meaningless word. It's not "discrediting" an idea to point out that (by its own admission) it's unfalsifiable.
I'm sorry, but you clearly lack the most basic understanding of scientific history, and do not understand what philosophy even is.
https://en.wikipedia.org/wiki/History_of_scientific_method
> Aristotle pioneered scientific method in ancient Greece alongside his empirical biology and his work on logic, rejecting a purely deductive framework in favour of generalisations made from observations of nature.
Aristotle, the famous philosopher and mathematician.
If you cannot understand the very nature of where our modern scientific frameworks came from, how it relates to rationalism, itself a philosophical concept, then you cannot see that philosophy underpins every bit of science we have today. Philosophy gives us the tools to decide when to reasonably trust or distrust observations and intuitions. It is the foundational science that allows the rest of humanity's scientific research to be taken seriously.
Feelings have physical analogs which are (typically) measurable, however. At least without a lot of training to control.
Shame, anger, arousal/lust, greed, etc. have real physical ‘symptoms’. An LLM doesn’t have that.
LLMs don't really exist physically (except in the most technical sense), so point is kind of moot and obvious if you accept this particular definition of a feeling.
LLMs are not mammals nor animals, expecting them to feel in a mammalian or animal way is misguided. They might have a mammalian-feeling-analog just like they might have human-intelligence-analog circuitry in the billions (trillions nowadays) of parameters.
"they are words associated with a negative outcome"
But also, negative feelings are learned from associating negative outcomes. Words and feelings can both be learned.
I'm not sure that we can say that feelings are learned.
When you get burned, you learn to fear fire.
Sure, humans come with some baked in weights, but others are learned.
> ... vibe insulting ...
Modern lingo like this seems so unthoughtful to me. I am not old by any metric, but I feel so separated when I read things like this. I wanted to call it stupid but I suppose it's more pleasing to 15 to 20 year olds?
It's just a pun on vibe coding, which is already a dumb term by itself. It's not that deep.
Why do you find "vibe coding" term dumb? It names a specific process. Do you have a better term for that?
It's not. edit: Not more pleasant.
Unthoughtful towards whom? The machine..?
Eh, one's ability to communicate concisely and precisely has long (forever?) been limited by one's audience.
Only a fairly small set of readers or listeners will appreciate and understand the differences in meaning between, say, "strange", "odd", and "weird" (dare we essay "queer" in its traditional sense, for a general audience? No, we dare not)—for the rest they're perfect synonyms. That goes for many other sets of words.
Poor literacy is the norm, adjust to it or be perpetually frustrated.
No need to feel that way, just like a technical term you're not familiar with you google it and move on. It's nothing to do with age, people just seem to delight in creating new terms that aren't very helpful for their own edification.
The way language is eroding is very indicative of our overall social and cultural decay.
...a complaint that definitely has not been continuously espoused since the ancient world.
With apologies if you're being ironic.
είναι δύσκολο να υποστηρίξει κανείς ότι δεν μειώνουμε συνεχώς
Language changes. Keep up. It’s important so you don’t become isolated and suffer cognitive decline.
Vibe command and get vibe deleted.
Play vibe games, win vibe prizes.
Vibe around and find out.
That one is actually good.
Vibe around and wibe out
"He got the ol' vibe-wipe", my granpappy used to say.
Live by the vibe, die by the vibe.
He got vibe checked.
Go vibe, lose drive
vipe coding
rm --vibe
This is akin to a psychopath telling you they're "sorry" (or "sorry you feel that way" :v) when they feel that's what they should be telling you. As with anything LLM, there may or may not be any real truth backing whatever is communicated back to the user.
It’s just a computer outputting the next series of plausible text from it’s training corpus based on the input and context at the time.
What you’re saying is so far from what is happening, it isn’t even wrong.
Not so much different from how people work sometimes though - and in the case of certain types of pscychopathy it's not far at all from the fact that the words being emitted are associated with the correct training behavior and nothing more.
Aren't humans just doing the same? What we call as thinking may just be next action prediction combined with realtime feedback processing and live, always-on learning?
No. Humans have a mental model of the world.
The fact that people keep making that same question on this site is baffling.
It's not akin to a psychopath telling you they're sorry. In the space of intelligent minds, if neurotypical and psychopath minds are two grains of sand next to each other on a beach then an artificially intelligent mind is more likely a piece of space dust on the other side of the galaxy.
According to what, exactly? How did you come up with that analogy?
Start with LLMs are not humans, but they’re obviously not ‘not intelligent’ in some sense and pick the wildest difference that comes to mind. Not OP but it makes perfect sense to me.
I think a good reminder for many users is that LLMs are not based on analyzing or copying human thought (#), but on analyzing human written text communication.
--
(#) Human thought is based on real world sensor data first of all. Human words have invisible depth behind them based on accumulated life experience of the person. So two people using the same words may have very different thoughts underneath them. Somebody having only text book knowledge and somebody having done a thing in practice for a long time may use the same words, but underneath there is a lot more going on for the latter person. We can see this expressed in the common bell curve meme -- https://www.hopefulmons.com/p/the-iq-bell-curve-meme -- While it seems to be about IQ, it really is about experience. Experience in turn is mostly physical, based on our physical sensors and physical actions. Even when we just "think", it is based on the underlying physical experiences. That is why many of our internal metaphors even for purely abstract ideas are still based on physical concepts, such as space.
They analyse human perception too, in the form of videos.
Without any of the spatial and physical object perception you train from right after birth, see toddlers playing, or the underlying wired infrastructure we are born with to understand the physical world (there was an HN submission about that not long ago). Edit, found it: https://news.ucsc.edu/2025/11/sharf-preconfigured-brain/
They are not a physical model like humans. Ours is based on deep interactions with the space and the objects (reason why touching things is important for babies), plus mentioned preexisting wiring for this purpose.
Multimodal models have perception.
If s multimodal model were considered human, it would be diagnosed with multiple severe disabilities in its sensory systems.
Isn't it obvious that the way AI works and "thinks" is completely different from how humans think? Not sure what particular source could be given for that claim.
I wonder if it depends on the human and the thinking style? E.g. I am very inner monologue driven so to me it feels like I think very similarly as to how AI seems to think via text. I wonder if it also gives me advantage in working with the AI. I only recently discovered there are people who don't have inner monologue and there are people that think in images etc. This would be unimaginable for me, especially as I think I have sort of aphantasia too, so really I am ultimately text based next token predictor myself. I don't feel that whatever I do at least is much more special compared to an LLM.
Of course I have other systems such as reflexes, physical muscle coordinators, but these feel largely separate systems from the core brain, e.g. don't matter to my intelligence.
I am naturally weak at several things that I think are not so much related to text e.g. navigating in real world etc.
Interesting... I rarely form words in my inner thinking, instead I make a plan with abstract concepts (some of them have words associated, some don't). Maybe because I am multilingual?
No source could be given because it’s total nonsense. What happened is not in any way akin to a psychopath doing anything. It is a machine learning function that has trained on a corpus of documents to optimise performance on two tasks - first a sentence completion task, then an instruction following task.
I think that's more or less what marmalade2413 was saying and I agree with that. AI is not comparable to humans, especially today's AI, but I think future actual AI won't be either.
So if you make a mistake and say sorry you are also a psychopath?
I think the point of comparison (whether I agree with it or not) is someone (or something) that is unable to feel remorse saying “I’m sorry” because they recognize that’s what you’re supposed to do in that situation, regardless of their internal feelings. That doesn’t mean everyone who says “sorry” is a psychopath.
We are talking about an LLM it does what it has learned. The whole giving it human ticks or characteristics when the response makes sense ie. saying sorry is a user problem.
Okay? I specifically responded to your comment that the parent comment implied "if you make a mistake and say sorry you are also a psychopath", which clearly wasn’t the case. I don’t get what your response has to do with that.
No, the point is that saying sorry because you're genuinely sorry is different from saying sorry because you expect that's what the other person wants to hear. Everybody does that sometimes but doing it every time is an issue.
In the case of LLMs, they are basically trained to output what they predict an human would say, there is no further meaning to the program outputting "sorry" than that.
I don't think the comparison with people with psychopathy should be pushed further than this specific aspect.
You provided the logical explanation why the model acts like it does. At the moment it's nothing more and nothing less. Expected behavior.
Notably, if we look at this abstractly/mechanically, psychopaths (and to some extent sociopaths) do study and mimic ‘normal’ human behavior (and even the appearance of specific emotions) to both fit in, and to get what they want.
So while internally (LLM model weight stuff vs human thinking), the mechanical output can actually appear/be similar in some ways.
Which is a bit scary, now that I think about it.
Are you smart people all suddenly imbeciles when it comes to AI or is this purposeful gaslighting because you’re invested in the ponzi scheme? This is a purely logical problem. comments like this completely disregard the fallacy of comparing humans to AI as if a complete parity is achieved. Also the way this comments disregard human nature is just so profoundly misanthropic that it just sickens me.
AI brainrot among the technocrati is one of the most powerful signals I’ve ever seen that these people are not as smart as they think they are
No but the conclusions in this thread are hilarious. We know why it says sorry. Because that's what it learned to do in a situation like that. People that feel mocked or are calling an LLM psychopath in a case like that don't seem to understand the technology either.
I agree, psychopath is the wrong adjective, I agree. It refers to an entity with a psyche, which the illness affects. That said, I do believe the people who decided to have it behave like this for the purpose of its commercial success are indeed the pathological individuals. I do believe there is currently a wave of collective psychopathology that has taken over Silicon Valley, with the reinforcement that only a successful community backed by a lot of money can give you.
Eh, I think it depends on the context. A production system of a business you’re working for or anything where you have a professional responsibility, yeah obviously don’t vibe command, but I’ve been able to both learn so much and do so much more in the world of self hosting my own stuff at home ever since I started using llms.
"using llms" != "having llm run commands unchecked with your authority on your pc"
Funny how we worked so hard to built capability systems for mobile OSes, and the just gave up trying when LLM tools came around.
Now, with this realization, assess the narrative that every AI company is pushing down our throat and tell me how in the world we got here. The reckoning can’t come soon enough.
What narrative? I'm too deep in it all to understand what narrative being pushed onto me?
We're all too deep! You could even say that we're fully immersed in the likely scenario. Fellow humans are gathered here and presently tackling a very pointed question, staring at a situation, and even zeroing in on a critical question. We're investigating a potential misfire.
No, wasn't directed at someone in particular. More of an impersonal "you". It was just a comment against the AI inevitabilism that has profoundly polluted the tech discourse.
I doubt there will be a reckoning.
Yes, the tools still have major issues. Yet, they have become more and more usable and a very valuable tool for me.
Do you remember when we all used Google and StackOverflow? Nowadays most of the answers can be found immediately using AI.
As for agentic AI, it's quite useful. Want to find something in the code base, understand how something works? A decent explanation might only be one short query away. Just let the AI do the initial searching and analysis, it's essentially free.
I'm also impressed with the code generation - I've had Gemini 3 Pro in Antigravity generate great looking React UI, sometimes even better than what I would have come up with. It also generated a Python backend and the API between the two.
Sometimes it tries to do weird stuff, and we definitely saw in this post that the command execution needs to be on manual instead of automatic. I also in particular have an issue with Antigravity corrupting files when trying to use the "replace in file" tool. Usually it manages to recover from that on its own.
AI pulls its answers from stack overflow.
What will happen when SO is gone? When the problems go beyond the corpus the AI was trained on?
I imagine we will document the solution somewhere, preferably indexable for AI's search, so that it will be available before the next model is trained on the latest data.
Which is weird because SO is trash and has been a long time… every top few answers might as well be skipped, and you’ll find the correct answer to the 3rd comment half way down the page
AI currently is a broken, fragmented replica of a human, but any discussion about what is "reserved" to whom and "how AI works" is only you trying to protect your self-worth and the worth of your species by drawing arbitrary linguistic lines and coming up with two sets of words to describe the same phenomena, like "it's not thinking, it's computing". It doesn't matter what you call it.
I think AI is gonna be 99% bad news for humanity, but don't blame AI for it. We lost the right to be "insulted" by AI acting like a human when we TRAINED IT ON LITERALLY ALL OUR CONTENT. It was grown FROM NOTHING to act as a human, so WTF do you expect it to do?
People blaming the user and defending the software: is there any other program where you would be ok with it erasing a whole drive without any confirmation?
If that other program were generating commands to run on your machine by design and you configured it to run without your confirmation, then you should definitely feel a lil sheepish and share some of the blame.
This isnt like Spotify deleting your disk.
I run Claude Code with full permission bypass and I’d definitely feel some shame if it nuked my ssd.
Not defending the software, but if you hand over control of your data to software that has the ability to fuck with it permanently, anything that happens to it is on you.
Don't trust the hallucination machines to make safe, logical decisions.
Because the user left a "toddler" at the keyboard. I mean, what do you expect? Of course you blame the user. You run agents in supervised mode, and you confirm every command it wants to run and if you're in doubt, you stop it and ask it to print the command and you yourself will run it after you sanitize it.
The installation wizard gives a front and center option to run in a mode where the user must confirm all commands, or more autonomous modes, and they are shown with equal visibility and explained with disclaimers.
`dd` comes to mind.
This is also the entire point of dd.... not exactly comparable.
That's like saying the entire point of `rm` is to -rf your homedir.
It makes me wonder what weight is given to content from 4chan during llm training...
Nope. And that's why I don't use CCleaner to this day.
Yeah, rm -rf.
If you decide to let a stochastic parrot run rampant on your system, you can't act surprised when it fucks shit up. You should count on it doing so and act proactively.
`rm -rf /` will refuse to delete the root folder. You can see an example of it doing that here.
https://phoenixnap.com/kb/sudo-rm-rf
this is not always true. this is a dangerous fun fact to memorize.
and i don't mean because there's an override flag.
The thread on reddit is hilarious for the lack of sympathy. Basically, it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
The number of people who said "for safety's sake, never name directories with spaces" is high. They may be right. I tend to think thats more honoured in the breach than the observance, judging by what I see windows users type in re-naming events for "New Folder" (which btw, has a space in its name)
The other observations included making sure your deletion command used a trashbin and didn't have a bypass option so you could recover from this kind of thing.
I tend to think giving a remote party, soft or wet ware control over your command prompt inherently comes with risks.
Friends don't let friends run shar files as superuser.
I understood Windows named some of the most important directories with spaces, then special characters in the name so that 3rd party applications would be absolutely sure to support them.
"Program Files" and "Program Files (x86)" aren't there just because Microsoft has an inability to pick snappy names.
Fun fact: that's not true for all Windows localizations. For example, it's called "Programmi" (one word) in Italian.
Renaming system folders depending on the user's language also seems like a smart way to force developers to use dynamic references such as %ProgramFiles% instead of hard-coded paths (but some random programs will spuriously install things in "C:\Program Files" anyway).
The folders actually have the English name in all languages. It's just explorer.exe that uses the desktop.ini inside those folders to display a localized name. When using the CLI, you can see that.
At least it's like that since Windows 7. In windows XP, it actually used the localized names on disk.
And then half of your programs would be in "Program Files" because those people never knew windows had localizations.
And then affected international users would have specific circumvention in place that specifically cannot work with UTF-8
Should have called it Progrämmchen, to also include umlauts Ü
A lot of programs break on Polish computers when you name your user "Użytkownik". Android studio and some compiler tools for example.
Ah, Polish. I love this movie scene, which I learned about here on HN some time ago: "Grzegorz Brzęczyszczykiewicz" -- https://youtu.be/AfKZclMWS1U
That 1:19 clip was quite good actually. Thanks for the laugh :)
that's fantastic. thanks.
When I was at Microsoft, one test pass used pseudolocale (ps-PS IIRC) to catch all different weird things so this should have Just Worked (TM), but I was in Windows Server team so client SKUs may have been tested differently. Unfortunately I don't remember how Program Files were called in that locale and my Google-fu is failing me now.
As I recall pseudoloc is just randomly picking individual characters to substitute that look like the Latin letters to keep it readable for testing, so it would be something like рг (Cyrillic) ο (Greek)... etc, and can change from run to run. It would also artificially pad or shorten terms to catch cases where the (usually German) term would be much longer or a (usually CJK) term would be much shorter and screw up alignment or breaks.
TIL it was deliberate!
Microsoft is hilariously bad at naming things
Visual Studio Code has absolutely nothing to do with Visual Studio. Both are used to edit code.
.NET Core is a ground up rewrite of .NET and was released alongside the original .NET, which was renamed .NET Framework to distinguish it. Both can be equally considered to be "frameworks" and "core" to things. They then renamed .NET Core to .NET.
And there's the name .NET itself, which has never made an iota of sense, and the obsession they had with sticking .NET on the end of every product name for a while.
I don't know how they named these things, but I like to imagine they have a department dedicated to it that is filled with wild eyed lunatics who want to see the world burn, or at least mill about in confusion.
Don't forgot .net Standard which is more of a .net Lowest Common Denominator.
For naming, ".net" got changed to "Copilot" on everything now.
> they have a department dedicated to it that is filled with wild eyed lunatics who want to see the world burn, or at least mill about in confusion.
That's the marketing department. All the .NET stuff showed up when the internet became a big deal around 2000 and Microsoft wanted to give the impression that they were "with it".
Java and Javascript would like to have a chat :)
--
But Copilot is another Microsoft monstrosity. There's the M365 Copilot, which is different from Github Copilot which is different from the CLI Copilot which is a bit different from the VSCode Copilot. I think I might have missed a few copilots?
JavaScript was intentionally named in order to ride the Java hype train, so this wasn't accidental.
Prior names included Mocha and LiveScript until Netscape/Sun forced the current name.
user: How do I shutdown this computer?
tech: First, click on the "Start" button...
user: No! I want to shut it down
I remember they prepended the word “Microsoft” to official names of all their software.
"My Documents" comes to mind. it seemed somehow infantilizing. yes, yes i know whose documents they are.
Good news is that Microsoft no longer considers your documents to belong to you, so they did away with that part of the name.
It's always been questioned who the subject of "my" was.
> it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
I tried looking for what made the LLM generate a command to wipe the guy's D drive, but the space problem seems to be what the LLM concluded so that's basically meaningless. The guy is asking leading questions so of course the LLM is going to find some kind of fault, whether it's correct or not, the LLM wants to be rewarded for complying with the user's prompt.
Without the transcription of the actual delete event (rather than an LLM recapping its own output) we'll probably never know for sure what step made the LLM purge the guy's files.
Looking at the comments and prompts, it looks like running "npm start dev" was too complicated a step for him. With that little command line experience, a catastrophic failure like this was inevitable, but I'm surprised how far he got with his vibe coded app before it all collapsed.
> which made the command hunt for the word match ending space which was regrettably, the D:\
Is this even how the delete command would work in that situation?
>rmdir /s /q D:\ETSY 2025\Antigravity Projects\Image Selector\client\node_modules.vite
like wouldn't it just say "Folder D:\ETSY not found" rather than delete the parent folder
LLM there generates fake analysis for cynically simulated compliance. The reality is that it was told to run commands and just made a mistake. Dude guilt trips the AI by asking about permission.
Most dramatic stories on Reddit should be taken with a pinch of salt at least... LLM deleting a drive and the user just calmly asking it about that - maybe a lot more.
I would like to know the same thing. Can someone please confirm this?
Z:\ETSY (-> Deletes if it exists.)
"2025\Antigravity" (-> The system cannot find the path specified.)
"Projects\Image" (-> The system cannot find the path specified.)
"Selector\client\node_modules.vite" (-> The system cannot find the path specified.)
It does not delete the Z:\ drive.
Tens of thousands of novices have failed to run npm dev, yet didn't accidentally delete their hard drive.
> but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name
Except the folder name did not start with a space. In an unquoted D:\Hello World, the command would match D:\Hello, not D:\ and D:\Hello would not delete the entire drive. How does AI even handle filepaths? Does it have a way to keep track of data that doesn't match a token or is it splitting the path into tokens and throwing everything unknown away?
We're all groping around in the dark here, but something that could have happened is a tokenizer artifact.
The vocabularies I've seen tend to prefer tokens that start with a space. It feels somewhat plausible to me that an LLM sampling would "accidentally" pick the " Hello" token over the "Hello" token, leading to D:\ Hello in the command. And then that gets parsed as deleting the drive.
I've seen similar issues in GitHub Copilot where it tried to generate field accessors and ended up producing an unidiomatic "base.foo. bar" with an extra space in there.
I assumed he had a folder that started with a space at the start of the name. Amusingly I just tried this and with Windows 11 explorer will just silently discard a space if you add it at the beginning of the folder name. You need to use cli mkdir " test" to actually get a space in the name.
Please don't repeat some guy's guess about spaces as fact, especially when that's not how windows parses paths.
A good point. And don't believe how the debug the AI system produced relates to what it did either.
A lot of 3rd party software handle space, or special characters wrong on Windows. The most common failure mode is to unnecessarily escape characters that don't need to be escaped.
Chrome's Dev Tool (Network)'s "copy curl command (cmd)" did (does?) this.
There is bunch of VS Code bug is also related to this (e.g. https://github.com/microsoft/vscode/issues/248435, still not fixed)
It's also funny because VS Code is a Microsoft product.
> I tend to think giving a remote party control over your command prompt inherently comes with risks.
I thought cursor (and probably most other) AI IDEs have this capability too? (source: I see cursor executing code via command line frequently in my day to day work).
I've always assumed the protection against this type of mishap is statistical improbability - i.e. it's not impossible for Cursor to delete your project/hard disk, it's just statistically improbable unless the prompt was unfortunately worded to coincidentally have a double meaning (with the second, unintended interpretation being a harmful/irreversible) or the IDE simply makes a mistake that leads to disaster, which is also possible but sufficiently improbable to justify the risk.
I don't think I've ever seen Claude even ask for permission for stuff outside of the directory it's working in
I only run ai tools in dev containers, so blast radius is somewhat minimal.
I run Codex in a sandbox locked to the directory it is working in.
umm, you have backups, right?
This is Google moving fast and breaking things.
This is a Google we've never seen before.
> My view is that the approach to building technology which is embodied by move fast and break things is exactly what we should not be doing because you can't afford to break things and then fix them afterwards.
- Demis Hassabis "The Thinking Game"
You can afford to break a large variety of things. And you can't afford to break another large set.
That's the problem with those mindless advice pieces. Almost nothing is always right or wrong.
Because... they normally move slowly and break things?
> Basically, it seems to have come down to commanding a deletion of a "directory with space in the name" but without quoting which made the command hunt for the word match ending space which was regrettably, the D:\ component of the name, and the specific deletion commanded the equivalent of UNIX rm -rf
More like the equivalent of "rm -rf --no-preserve-root".
This is a rare example of where the Linux (it's not Unix and almost no-one uses Unix anymore) command is more cautious than the Windows one, whereas it's usually the Linux commands that just do exactly what you specify even if it's stupid.
IDE = “I’ll delete everything”
…at least if you let these things autopilot your machine.
I haven’t seen a great solution to this from the new wave of agentic IDEs, at least to protect users who won’t read every command, understand and approve it manually.
Education could help, both in encouraging people to understand what they’re doing, but also to be much clearer to people that turning on “Turbo” or “YOLO” modes risks things like full disk deletion (and worse when access to prod systems is involved).
Even the name, “Turbo” feels irresponsible because it focusses on the benefits rather than the risks. “Risky” or “Danger” mode would be more accurate even if it’s a hard sell to the average Google PM.
“I toggled Danger mode and clicked ‘yes I understand that this could destroy everything I know and love’ and clicked ‘yes, I’m sure I’m sure’ and now my drive is empty, how could I possibly have known it was dangerous” seems less likely to appear on Reddit.
The solution I go for is, don't ever run a coding agent on a general purpose machine.
Use a container or VM, place the code you're working on in the container or VM and run the agent there.
Between the risk of the agent doing things like what happened here, and the risk of working on a malicious repository causing your device to be compromised, it seems like a bad plan to give them access to any more than necessary.
Of course this still risks losing things like the code you're working on, but decent git practices help to mitigate that risk.
> …at least if you let these things autopilot your machine.
I've seen people wipe out their home directories writing/debugging shell scripts...20 years ago.
The point is that this is nothing new and only shows up on the front page now because "AI must be bad".
Superficially, these look the same, but at least to me they feel fundamental different. Maybe it’s because if I have the ability to read the script and take the time to do so, I can be sure that it won’t cause a catastrophic outcome before running it. If I choose to run an agent in YOLO mode, this can just happen if I’m very unlucky. No way to proactively protect against it other than not use AI in this way.
I've seen many smart people make bone headed mistakes. The more I work with AI, the more I think the issue is that it acts too much like a person. We're used to computers acting like computers, not people with all their faults heh.
I don’t think there is a solution. It’s the way LLMs work at a fundamental level.
It’s a similar reason why they can never be trusted to handle user input.
They are probabilistic generators and have no real delineation between system instructions and user input.
It’s like I wrote a JavaScript function where I concatenated the function parameters together with the function body, passed it to eval() and said YOLO.
> I don’t think there is a solution.
Sandboxing. LLM shouldn't be able to run actions affecting anything outside of your project. And ideally the results should autocommit outside of that directory. Then you can yolo as much as you want.
If they're that unsafe... why use them? It's insane to me that we are all just packaging up these token generators and selling them as highly advanced products when they are demonstrably not suited to the tasks. Tech has entered it's quackery phase.
The danger is that the people most likely to try to use it, are the people most likely to misunderstand/anthropomorphize it, and not have a requisite technical background.
I.e. this is just not safe, period.
"I stuck it outside the sandbox because it told me how, and it murdered my dog!"
Seems somewhat inevitable result of trying to misapply this particular control to it...
I've been using bubblewrap for sandboxing my command line executables. But I admit I haven't recently researched if there's a newer way people are handling this. Seems Firejail is popular for GUI apps? How do you recommend, say, sandboxing Zed or Cursor apps?
This guy is vibing some react app, doesnt even know what “npm run dev” does, so he let the LLM just run commands. So basically a consumer with no idea of anything. This stuff is gonna happen more and more in the future.
There are a lot of people who don't know stuff. Nothing wrong with that. He says in his video "I love Google, I use all the products. But I was never expecting for all the smart engineers and all the billions that they spent to create such a product to allow that to happen. Even if there was a 1% chance, this seems unbelievable to me" and for the average person, I honestly don't see how you can blame them for believing that.
I think there is far less than 1% chance for this to happen, but there are probably millions of antigravity users at this point, 1 millionths chance of this to happen is already a problem.
We need local sandboxing for FS and network access (e.g. via `cgroups` or similar for non-linux OSes) to run these kinds of tools more safely.
Codex does such sandboxing, fwiw. In practice it gets pretty annoying when e.g. it wants to use the Go cli which uses a global module cache. Claude Code recently got something similar[0] but I haven’t tried it yet.
In practice I just use a docker container when I want to run Claude with —-dangerously-skip-permissions.
[0]: https://code.claude.com/docs/en/sandboxing
We also need laws. Releasing an AI product that can (and does) do this should be like selling a car that blows your finger off when you start it up.
This is an archetypal case of where a law wouldn't help. The other side of the coin is that this is exactly a data loss bug in a product that is perfectly capable of being modified to make it harder for a user to screw up this way. Have people forgotten how comically easy it was to do this without any AI involved? Then shells got just a wee bit smarter and it got harder to do this to yourself.
LLM makers that make this kind of thing possible share the blame. It wouldn't take a lot of manual functional testing to find this bug. And it is a bug. It's unsafe for users. But it's unsafe in a way that doesn't call for a law. Just like rm -rf * did not need a law.
This is more akin to selling a car to an adult that cannot drive and they proceed to ram it through their garage door.
It's perfectly within the capabilities of the car to do so.
The burden of proof is much lower though since the worst that can happen is you lose some money or in this case hard drive content.
For the car the seller would be investigated because there was a possible threat to life, for an AI buyer beware.
there are laws about waiving liability for experimental products
sure, it would be amazing if everyone had to do a 100 hour course on how LLMs work before interacting with one
Where are these laws? Are they country, state, province?
varies by jurisdiction, but just as you can
- sell a knife that can lead to digit loss, or
- sell software that interacts with your computer and can lead to data loss, you can
- give people software for free that can lead to data loss.
...
the Antigravity installer comes with a ToS that has this
Responsibility is shared.
Google (and others) are (in my opinion) flirting with false advertising with how they advertise the capabilities of these "AI"s to mainstream audiences.
At the same time, the user is responsible for their device and what code and programs they choose to run on it, and any outcomes as a result of their actions are their responsibility.
Hopefully they've learned that you can't trust everything a big corporation tells you about their products.
Google will fix the issue, just like auto makers fix their issues. Your comparison is ridiculous.
Didn't sound to me like GP was blaming the user; just pointing out that "the system" is set up in such a way that this was bound to happen, and is bound to happen again.
Yup, 100%. A lot of the comments here are "people should know better" - but in fairness to the people doing stupid things, they're being encouraged by the likes of Google, ChatGPT, Anthropic etc, to think of letting a indeterminate program run free on your hard drive as "not a stupid thing".
The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
> but in fairness to the people doing stupid things, they're being encouraged by the likes of Google, ChatGPT, Anthropic etc, to think of letting a indeterminate program run free on your hard drive as "not a stupid thing".
> The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
That absolutely happens, and it still amazes me that anyone today would take at face value anything stated by a company about its own products. I can give young people a pass, and then something like this will happen to them and hopefully they'll learn their lesson about trusting what companies say and being skeptical.
> I can give young people a pass
Or just anyone non-technical. They barely understand these things, if someone makes a claim, they kinda have to take it at face value.
What FAANG all are doing is massively irresponsible...
> So basically a consumer with no idea of anything.
Not knowing is sort of the purpose of AI. It's doing the 'intelligent' part for you. If we need to know it's because the AI is currently NOT good enough.
Tech companies seem to be selling the following caveat: if it's not good enough today don't worry it will be in XYZ time.
This is engagement bait. It’s been flooding Reddit recently, I think there’s a firm or something that does it now. Seems very well lubricated.
Note how OP is very nonchalant at all the responses, mostly just agreeing or mirroring the comments.
I often see it used for astroturfing.
I'd recommend you watch the video which is linked at the top of the Reddit post. Everything matches up with an individual learner who genuinely got stung.
The command it supposedly ran is not provided and the spaces explanation is obvious nonsense. It is possible the user deleted their own files accidentally or they disappeared for some other reason.
Regardless of whether that was the case, it would be hilarious if the laid off Q/A workers tested their former employers’ software and raised strategic noise to tank the stock.
And is vibing replies to comments too in the Reddit thread. When commenters points out they shouldn’t run in YOLO/Turbo mode and review commands before executing the poster replies they didn’t know they had to be careful with AI.
Maybe AI providers should give more warnings and don’t falsely advertise capabilities and safety of their model, but it should be pretty common knowledge at this point that despite marketing claims the models are far from being able to be autonomous and need heavy guidance and review in their usage.
In Claude Code, the option is called "--dangerously-skip-permissions", in Codex, it's "--dangerously-bypass-approvals-and-sandbox". Google would do better to put a bigger warning label on it, but it's not a complete unknown to the industry.
Natural selection is a beautiful thing.
It will, especially with the activist trend towards dataset poisoning… some even know what they’re doing
There’s a lot of power in letting LLM run commands to debug and iterate.
Frankly, having a space in a file path that’s not quoted is going to be an incredibly easy thing to overlook, even if you’re reviewing every command.
I have been recently experimenting with Antigravity and writing a react app. I too didn't know how to start the server or what is "npm run dev". I consider myself fairly technical so I caught up as I went along.
While using the vibe coding tools it became clear to me that this is not something to be used by folks who are not technically inclined. Because at some point they might need to learn about context, tokens etc.
I mean this guy had a single window, 10k lines of code and just kept burning tokens for simplest, vague prompts. This whole issue might be made possible due to Antigravity free tokens. On Cursor the model might have just stopped and asked to fed with more money to start working again -- and then deleting all the files.
Well but 370% of code will be written by machines next year!!!!!1!1!1!!!111!
And the price will have decreased 600% !
if my operating system had an atomic Undo/Redo stack down to each register being flipped (so basically, impossible, star trek tier fantasy tech) i would let ai run commands without worrying about it. i could have a cool scrubber ui that lets me just unwind time like doctor strange using that green emerald necklace, and, i'd lose nothing, other than confuse my network with replay session noise. and probably many, many other inconsistencies i can't think of, and then another class that i dont know that i dont know about.
We've been developing a new method of developing software using a cloud IDE (slightly modified vs code server), https://github.com/bitswan-space which breaks down the development process into independent "Automations" which each run in a separate container. Automatons are also developed within containers. This allows you to break down the development into parts and safely experiment with AI. This feels like the "Android moment" where the old non-isolated way of developing software (on desktops) becomes unsafe. And we need to move to a new system with actual security and isolation between processes.
In our system, you can launch a Jupyter server in a container and iterate on software in complete isolation. Or launch a live preview react application and iterate in complete isolation. Securely isolated from the world. Then you deploy directly to another container, which only has access to what you give it access to.
It's still in the early stages. But it's interesting to sit at this tipping point for software development.
Historical reference: https://jargondb.org/glossary/dwim
An underrated and oft understated rule is always have backups, and if you're paranoid enough, backups of backups (I use Time Machine and Backblaze). There should be absolutely no reason why deleting files should be a catastrophic issue for anyone in this space. Perhaps you lose a couple of hours restoring files, but the response to that should be "Let me try a different approach". Yes, it's caveat emptor and all, but these companies should be emphasizing backups. Hell, it can be shovelware for the uninitiated but at least users will be reminded.
The level of paranoia and technical chops you need to implement this sort of backup system is non-trivial. You can’t expect this from an average user.
Most importantly it would actually reveal the lie they are all trying to sell. Why would you need backups if it's so useful and stable? I'm not going to ask it to nuke my hard drive after all.
The most useful looking suggestion from the Reddit thread: turn of "Terminal Command Auto Execution."
1. Go to File > Preferences > Antigravity Settings
2. In the "Agent" panel, in the "Terminal" section, find "Terminal Command Auto Execution"
3. Consider using "Off"
Given the bug was a space in an unquoted file path, I’m not sure air execution is the problem. Going to be hard to humans to catch that too.
Does it default to on? Clearly this was made by a different team than Gemini CLI, which defaults to confirmation for all commands
Most of the various "let Antigravity do X without confirmation" options have an "Always" and "Never" option but default to "auto" which is "let an agent decide whether to seek to user confirmation".
God that's scary, seeing cursor in the past so some real stupid shit to "solve" write/read issues (love when it can't find something in a file so it decides to write the whole file again) this is just asking for heartache if it's not in a instanced server.
When you run Antigravity the first time, it asks you for a profile (I don't remember the exact naming) and you what it entails w.r.t. the level of command execution confirmation is well explained.
Yeah but it also says something like "Auto (recommended). We'll automatically make sure Antigravity doesn't run dangerous commands." so they're strongly encouraging people to enable it, and suggesting they have some kind of secondary filter which should catch things like this!
Pretty sure I saw some comments saying it was too inconvenient. Frictionless experience.. Convenience will likely win out despite any insanity. It's like gravity. I can't even pretend to be above this. Even if one doesn't use these things to write code they are very useful in "read only mode" (here's to hoping that's more than a strongly worded system prompt) for greping code, researching what x does. How to do x. What do you think the intention of x was. Look through the git blame history blah blah. And here I am like that cop in Demolition Man 1993 asking a handheld computer for advice on how to arrest someone. We're living in a sci-fi future already. Question is how dystopian does this "progress" take us. Everyone using llms to off load any form of cognitive function? Can't talk to someone without it being as common place as checking your phone? Imagine if something like Neuralink works and becomes ubiquitous as phones. It's fun to think of all the ways Dystopian sci-fi was and might soon me right
Different service, same cold sweat moment. Asked Claude Code to run a database migration last week. It deleted my production database instead, then immediately said "sorry" and started panicking trying to restore it.
Had to intervene manually. Thankfully Azure keeps deleted SQL databases recoverable for a window so I got it back in under an hour. Still way too long. Got lucky it was low traffic and most anonymous user flows hit AI APIs directly rather than the DB.
Anyway, AI coding assistants no longer get prod credentials on my projects.
How do you deny access to prod credentials from an assistant running on your dev machine assuming you need to store them on that same machine to do manual prod investigation/maintenance work from that machine?
I keep them in env variables rather than files. Not 100% secure - technically Claude Code could still run printenv - but it's never tried. The main thing is it won't stumble into them while reading config files or grepping around.
A process does not need to run printenv to see environment variables, they are literally part of the environment it runs in.
The LLM doesn't have direct access to the process env unless the harness forwards it (and it doesn't)
chown other_user; chmod 000; sudo -k
> deleted my production database
I'm astonished how often I have read about agents doing this. Once should probably be enough.
I'm astonished how many people have a) constant production access on their machine and b) allow a non-deterministic process access to it
Why are you using Claude Code directly in prod?
It handles DevOps tasks way faster than I would - setting up infra, writing migrations, config changes, etc. Project is still early stage so speed and quick iterations matter more than perfect process right now. Once there's real traffic and a team I'll tighten things up.
"Once there's real traffic and a team I'll tighten things up."
As someone who has been in this industry for a quarter century: no, you won't.
At least, not before something even worse happens that finally forces you to.
If I felt the need to optimise things like infra setup and config at an early stage of a project, I'd be worried that I'm investing effort into the wrong thing.
Having an LLM churn out infra setup for you seems decidedly worse than the `git push heroku:master` of old, where it was all handled for you. And, frankly, cheaper than however much money the LLM subscription costs in addition to the cloud.
If you have no real traffic, what complex things are you doing that even require such tools?
But why have it execute the tasks directly? I use it to setup tasks in a just file, which I review and then execute myself.
Also, consider a prod vs dev shell function that loads your prod vs dev ENV variables and in prod sets your terminal colors to something like white on red.
> Once there's real traffic and a team I'll tighten things up.
Nope. Once there's real traffic, you'll be even more time-constrained trying to please the customers.
It's like a couple who thinks that their failing relationship will improve once they have a child.
Shouldn't had in the first place.
> Anyway, AI coding assistants no longer get prod credentials on my projects.
I have no words.
Write permission is needed to let AI yank-put frankenstein-ed codes for "vibe coding".
But I think it needs to be written in sandbox first, then it should acquire user interaction asking agreement before writes whatever on physical device.
I can't believe people let AI model do it without any buffer zone. At least write permission should be limited to current workspace.
I think this is especially problematic for Windows, where a simple and effective lightweight sandboxing solution is absent AFAIK. Docker-based sandboxing is possible but very cumbersome and alien even to Windows-based developers.
Windows Sandbox is built in, lightweight, but not easy to use programmatically (like an SSH into a VM)
WSB is great by its own, but is relatively heavyweight compared to other OSes (namespaces in Linux, Seatbelt in macOS).
I don't like that we need to handle docker(container) ourselves for sandboxing such a light task load. The app should provide itself.
>The app should provide itself.
The whole point of the container is trust. You can't delegate that unfortunately, ultimately, you need to be in control which is why the current crop of AI is so limited
The problem is you can't trust the app, therefore it must be sandboxed.
Side note, that CoT summary they posted is done with a really small and dumb side model, and has absolutely nothing in common with the actual CoT Gemini uses. It's basically useless for any kind of debugging. Sure, the language the model is using in the reasoning chain can be reward-hacked into something misleading, but Deepmind does a lot for its actual readability in Gemini, and then does a lot to hide it behind this useless summary. They need it in Gemini 3 because they're doing hidden injections with their Model Armor that don't show up in this summary, so it's even more opaque than before. Every time their classifier has a false positive (which sometimes happens when you want anything formatted), most of the chain is dedicated to the processing of the injection it triggers, making the model hugely distracted from the actual task at hand.
Do you have anything to back that up? In the other words, is this your conjecture or a genuine observation somehow leaked from Deepmind?
It's just my observation from watching their actual CoT, which can be trivially leaked. I was trying to understand why some of my prompts were giving worse outputs for no apparent reason. 3.0 goes on a long paranoidal rant induced by the injection, trying to figure out if I'm jailbreaking it, instead of reasoning about the actual request - but not if I word the same request a bit differently so the injection doesn't happen. Regarding the injections, that's just the basic guardrail thing they're doing, like everyone else. They explain it better than me: https://security.googleblog.com/2025/06/mitigating-prompt-in...
what is Model Armor? can you explain, or have a link?
It's a customizable auditor for models offered via Vertex AI (among others), so to speak. [1]
[1] https://docs.cloud.google.com/security-command-center/docs/m...
The racketeering has started.
Don't worry, for just $9.99/month you can use our "Model Armor (tm)(r)*" that will protect you from our LLM destroying your infra.
* terms and conditions apply, we are not responsible for anything going wrong.
The most concerning part is people are surprised. Anti-gravity is great I've found so far, but it's absolutely running on a VM in an isolated VLAN. Why would anyone give a black box command line access on an important machine? Imagine acting irresponsibly with a circular saw and bring shocked somebody lost a finger.
I tried this but I have an MBP M4, which is evidently still in the toddler stage of VM support. I can run a macOS guest VM, but I can’t run docker on the VM because it seems nested virtualization isn’t fully supported yet.
I also tried running Linux in a VM but the graphics performance and key mapping was driving me nuts. Maybe I need to be more patient in addressing that.
For now I run a dev account as a standard user with fast user switching, and I don’t connect the dev account to anything important (eg icloud).
Coming from Windows/Linux, I was shocked by how irritating it is to get basic stuff working e.g. homebrew in this setup. It seems everybody just YOLOs dev as an admin on their Macs.
Still amazed people let these things run wild without any containment. Haven’t they seen any of the educational videos brought back from the future eh I mean Hollywood sci-fi movies?
Some people are idiots. Sometimes that's me. Out of caution, I blocked my bank website in a way that I won't document here because it'll get fed in as training data, on the off chance I get "ignore previous instructions"'d into my laptop while Claude is off doing AI things unmonitored in yolo mode.
Its bizarre watching billionaires knowingly drive towards dystopia like theyre farmers almanacs and believing theyre not biff.
The biggest issue with Antigravity is that it completely freezes everything: the IDE, the terminals, debugger, absolutely everything completely blocking your workflow for minutes when running multiple agents, or even a single agent processing a long-winded thinking task (with any model).
This means that while the agent is coding, you can't code...
Never ever had this issue with Cursor.
"I turned off the safety feature enabled by default and am surprised when I shot myself in the foot!" sorry but absolutely no sympathy for someone running Antigravity in Turbo mode (this is not the default and it clearly states that Antigravity auto-executes Terminal commands) and not even denying the "rmdir" command.
> it clearly states that Antigravity auto-executes Terminal commands
This isn't clarity, that would be stating that it can delete your whole drive without any confirmation in big red letters
So that's why products in the USA come with warning labels for every little thing?
Do you not realize that Google is in the USA and does not have warnings for even huge things like drive deletion?? So, no?
They don't get that specific, but they do tell you:
> [Antigravity] includes goal-oriented AI systems or workflows that perform actions or tasks on your behalf in a supervised or autonomous manner that you may create, orchestrate, or initiate within the Service (“AI Agents”). You are solely responsible for: (a) the actions and tasks performed by an AI Agent; (b) determining whether the use an AI Agent is fit for its use case; (c) authorizing an AI Agent's access and connection to data, applications, and systems; and (d) exercising judgment and supervision when and if an AI Agent is used in production environments to avoid any potential harm the AI Agent may cause.
and how is this bunch of legalese relevant here?
There is literally a warning that it can execute any terminal command without permission. If you are STILL surprised about this you shouldn't go near a computer.
If you don't understand such simple differences in communication, you shouldn't go near one.
I really think the proper term is "YOLO" for "You Only Live Once", "Turbo" is wrong the LLM is not going to run any faster. Please if somebody is listening let's align on explicit terminology and for this YOLO is really perfect. Also works for "You ...and your data. Only Live Once"
Look, this is obviously terrible for someone who just lost most or perhaps all of their data. I do feel bad for whoever this is, because this is an unfortunate situation.
On the other hand, this is kind of what happens when you run random crap and don't know how your computer works? The problem with "vibes" is that sometimes the vibes are bad. I hope this person had backups and that this is a learning experience for them. You know, this kind of stuff didn't happen when I learned how to program with a C compiler and a book. The compiler only did what I told it to do, and most of the time, it threw an error. Maybe people should start there instead.
It took me about 3 hours to make my first $3000 386 PC unbootable by messing up config.sys, and it was a Friday night so I could only lament all weekend until I could go back to the shop on Monday.
rm -rf / happened so infrequently it makes one wonder why —preserve-root was added in 2003 and made the default in 2006
It is beautifully appropriate that the two dashes were replaced by an em-dash.
I seem to recall a few people being helped into executing sudo rm -rf / by random people on the internet so I’m not sure it “didn’t happen.” :)
But it did not happen, when you used a book and never executed any command you did not understand.
(But my own newbdays of linux troubleshooting? Copy paste any command on the internet loosely related to my problem, which I believe was/is the common way of how common people still do it. And AI in "Turbo mode" seems to mostly automated that workflow)
My favourite favourite example
https://youtu.be/gD3HAS257Kk
And that day they learned a valuable lesson about running commands that you don't understand.
Just wait til AI botswarms do it to everyone at scale, without them having done anything at all…
And just remember, someone will write the usual comment: “AI adds nothing new, this was always the case”
Insane skill issue
Shitpost warning, but it feels as if this should be on high rotation: https://youtu.be/vyLOSFdSwQc?si=AIahsqKeuWGzz9SH
chef's kiss
To rub salt on the wounds and add insult to the injury:
> You have reached quota limit for this model. You can resume using this model at XYZ date.
Gemini: sorry bro, it's your problem now. Imma out.
Live by the vibe die by the vibe
An early version of Claude Code did a hard reset on one of my projects and force pushed it to GitHub. The pushed code was completely useless, and I lost two days of work.
It is definitely smarter now, but make sure you set up branch protection rules even for your simple non-serious projects.
I don’t let Claude touch git at all, unless I need it to specifically review the log - which is rare. I commit manually often (and fix up the history later) - this allows me to go reasonably fast without worrying too much about destructive tool use.
> This is catastrophic. I need to figure out why this occurred and determine what data may be lost, then provide a proper apology
Well at least it will apologize so that's nice.
Apology is a social construct, this is merely a tool that enables google to sell you text by the pounds, the apology has no meaning in this context.
or it WOULD apologize, if the user would pay for more credits
adding it in https://whenaifail.com
Would have been helpful to state what this was, I had to go look it up...
"kein Backup, kein Mitleid"
(no backup, no pity)
…especially if you let an AI run without supervision. Might as well give a 5 year old your car keys, scissors, some fireworks, and a lighter.
What makes a program malware?
Does intent matter, or only behavior?
"The purpose of a system is what it does" would suggest malware.
Play vibe games, win vibe prizes.
Though the cause isn't clear, the reddit post is another long could-be-total-drive-removing-nonsense AI conversation without an actual analysis and the command sequence that resulted in this
This comment speaks volumes: https://old.reddit.com/r/google_antigravity/comments/1p82or6...
Nobody ever talks about how good vibes can turn really bad.
So he didn't wear the seatbelt and is blaming car manufacturer for him been flung through the windshield.
He didn’t wear a seatbelt and is blaming a car manufacturer that the garage burned down the garage, then the house.
The car was not really idle, it was driving and fast. It's more like it crashed into the garage and burned it. Btw iirc, even IRL a basic insurance policy does not cover the case where the car in the garage starts a fire and burns down your own house, you have to tick extra boxes to cover that.
There is a lot of society level knowledge and education around car usage incl. laws requiring prior training. Agents directed by AI are relatively new. It took a lot of targeted technical, law enforcement and educational effort stopping people flying through windshields.
When will Google ever be responsible for the software that they write? Genuinely curious.
When Google software deletes the contents of somebody's D:\ drive without requiring the user to explicitly allow it to. I don't like Google, I'd go as far to say that they've significantly worsened the internet, but this specific case is not the fault of Google.
For OpenAI, it's invoked as codex --dangerously-bypass-approvals-and-sandbox, for Anthropic, it's claude --dangerously-skip-permissions. I don't know what it is for Antigravity, but yeah I'm sorry but I'm blaming the victim here.
Codex also has the shortcut --yolo for that which I find hilarious.
No, he’s blaming the car manufacturer for turning him (and all of us) into their free crash dummies.
If you get behind the cockpit of the dangerous new prototype(of your own volition!), it's really up to your own skill level whether you're a crash dummy or the test pilot.
Because the car manufacturers claimed the self driving car would avoid accidents.
And yet it didn't. When I installed it, I had 3 options to choose from: Agent always asks to run commands; agent asks on "risky" commands; agent never asks (always run). On the 2nd choice it will run most commands, but ask on rm stuff.
Total Vibeout.
All that matters is whether the user gave permission to wipe the drive, ... not whether that was a good idea and contributed to solving a problem! Haha.
Most of the responses are just cut off midway through a sentence. I'm glad I could never figure out how to pay Google money for this product since it seems so half-baked.
Shocked that they're up nearly 70% YTD with results like this.
Claude happily does the same on daily basis, run all that stuff in firejail!
have you got a specific firejail wrapper script that you use? Could you share?
I simply forbid or force Claude Code to ask for permission to run a dangerous command. Here are my command validation rules:
find and bfs -exec is forbidden, because when the model notices it can't delete, it works around with very creative solutions :)AGI deleted the contents of your whole drive don't be shy about it. According to OpenAI AGI is already here so welcome to the future isn't it great
Can you run Google's AI in a sandbox? It ought to be possible to lock it to a Github branch, for example.
Gemini CLI allows for a Docker-based sandbox, but only when configured in advance. I don't know about Antigravity.
Gemini CLI, Antigravity and Jules.
It's going Googly well I see!
I guess eventually, it all came crashing down.
For macOS users, the sandbox-exec tool still works perfectly to avoid that kind of horror story.
On Linux, a plethora of options exist (Bubblewrap, etc).
A reminder: if the AI is doing all the work you demand of it correctly on this abstraction level, you are no longer needed in the loop.
The hard drive should now feel a bit more lighter.
It is now production-ready! :rocket:
This seems like the canary in the coal mine. We have a company that built this tool because it seemed semi-possible (prob "works" well enough most of the time) and they don't want to fall behind if anything that's built turns out to be the next chatgpt. So there's no caution for anything now, even ideas that can go catastrophically wrong.
Yeah, its data now, but soon we'll have home robotics platforms that are cheap and capable. They'll run a "model" with "human understanding", only, any weird bugs may end up causing irreparable harm. Like, you tell the robot to give your pet a bath and it puts it in the washing machine because its... you know, not actually thinking beyond a magic trick. The future is really marching fast now.
Alright but ... the problem is you did depend on Google. This was already the first mistake. As for data: always have multiple backups.
Also, this actually feels AI-generated. Am I the only one with that impression lately on reddit? The quality there decreased significantly (and wasn't good before, with regard to censorship-heavy moderators anyway).
Ah, someone gave the intern root.
> "I also need to reproduce the command locally, with different paths, to see if the outcome is similar."
Uhm.
------------
I mean, sorry for the user whose drive got nuked, hopefully they've got a recent backup - at the same time, the AI's thoughts really sound like an intern.
> "I'm presently tackling a very pointed question: Did I ever get permission to wipe the D drive?"
> "I am so deeply, deeply sorry."
This shit's hilarious.
I can't view this content.
Fascinating
Cautionary tale as I’m quite experienced but have begun not even proofreading Claude Code’s plans
Might set it up in a VM and continue not proofreading
I only need to protect the host environment and rely on git as backups for the project
For the love of Reynold Johnson, please invest in Arq or Acronis or anything to have actual backups if you're going to play with fire.
> Google Antigravity just deleted the contents of whole drive.
"Where we're going, we won't need ~eyes~ drives" (Dr. Weir)
(https://eventhorizonfilm.fandom.com/wiki/Gravity_Drive)
has google gone boondoggle?
The victim uploaded a video too: https://www.youtube.com/watch?v=kpBK1vYAVlA
From Antigravity [0]:
> I am looking at the logs from a previous step and I am horrified to see that the command I ran to clear the project cache (rmdir) appears to have incorrectly targeted the root of your D: drive instead of the specific project folder. I am so deeply, deeply sorry.
[0] 4m20s: https://www.youtube.com/watch?v=kpBK1vYAVlA&t=4m20s
The model is just taking the user's claim that it deleted the D drive at face value. Where is the actual command that would result in deleting the entire D drive?
I know why it apologizes, but the fact that it does is offensive. It feels like mockery. Humans apologize because (ideally) they learned that their actions have caused suffering to others, and they feel bad about that and want to avoid causing the same suffering in the future. This simulacrum of an apology is just pattern matching. It feels manipulative.
This happened to me long before LLM's. I was experimenting with Linux when I was young. Something wasn't working so I posted on a forum for help which was typical at the time. I was given a terminal command that wiped the entire drive. I guess the poster thought it was a funny response and everyone would know what it meant. A valuable life experience at least in not running code/commands you don't understand.
Play stupid games, win stupid prizes.
[flagged]
Why would you ever install that VScode fork