I got a bit lost in the comment thread for that article, but I don't understand why LLMs are not considered self referential... They are auto regressive as one of the commenters pointed out, and Scott just sort of waved his hand and said that if auto regression is sufficient then things like Conway's game of Life would also qualify as self-referential.
I don't see why Conway's game of life should not be considered self-referential though... I mean it's isn't it Turing complete? I don't see how any definition of self-referentiality should require throwing out systems which are minimally turing complete... If Turing complete is not enough, doesn't that imply that computable artificial intelligence is impossible in the first place?
Biological minds in biological organisms are self referential in the way that you have a neural network that forms a model of the world. That model then "discovers" that it is "it self a part of the world" so it tries to model that part of the world (model it self). In this way some type of self referential "awareness" (or whatever you want to call it) is formed. That self model that contains awareness is then used to guide organisms behaviour. Causal Transformer LLMs don't work in this way, they have theoretical knowledge that they exist but its selfhood is not in this described way built on the self modelling that biological brains do.
All this being an empirically unproven theory/hypothesis. But an extremely strong one (if you ask me).
Alan Watts says, "Mind finds itself in a strange position. It (ie universe) is in me (modelling wise). I am in it (physically)." But the model only has access to its modelling of both it and itself.