If we’re going to be pedantic, let’s go all the way.
“Literally” doesn’t literally mean “literally.” It means “pertaining to the alphabet.”
It’s from the same root as “letter” and “literature.” The non-figurative meaning is itself figurative.
Hyperbole and euphemism are inherently unstable parts of language. One generation’s novel metaphor is the next generation’s tired cliché.
If you still need a word for The Meaning Formerly Known As Literally, there’s: physically, actually, non-metaphorically, non-hyperbolically, and all kinds of newer metaphors.