He's fundamentally damaged. In one of the later episodes, you see him without his armor. He's covered with old, badly-healed scars. He's pale, and with his helmet on, he's a looming, imposing figure. Without it he looks almost normal. But simply leaving it off doesn't hide who he is; there is fundamentally something
wrong with him, and almost everyone realizes it just by being
near the guy.
What makes the show interesting isn't Goblin Slayer himself per se; while how crazy prepared for random situations he is
is wildly entertaining in and of itself, the emotional core of this show is his dynamic with the rest of the cast. It's him slowly being saved from himself, and in the process, others realizing why the fuck he does what he does, and why it's so goddamned critical. His traveling companions quickly realize that the Goblin Slayer is the sort of person who would be institutionalized in any other setting. He does not
function. He's turned himself into something less than human in order to fight things that aren't human at all. Any other adventurer at the guildhall could stop adventuring tomorrow and live a peaceful life somewhere. Goblin Slayer can't.
That's why the others getting into his life makes for such a compelling thing to watch: His humanity is slowly coming back. The big finale for the season is not the fall of the Demon King (which basically happens off-camera with another party entirely); it's the Goblin Slayer coming to the guildhall to do the fucking unthinkable:
to ask for help killing Goblins. The realization is that this creepy, ominous, murderous figure is just as human as everyone else at the guild, and the ending sequence, featuring Goblin Slayer taking his helmet off around his guildmates, brings the whole thing full circle.