The reason endermen don’t like it when you look at them is because they communicate telepathically with one another by locking eyes! Humans are absolutely not designed to do this so when we look at them we are accidentally projecting all of our thoughts into them at the same time and it hurts :(
But like, since the player is not of the Minecraft world, the player is just what the use to explore it, what if it’s like: