You should get familiar with the conecept of policies:
There is not just one global token. Tokens are issued in accordance with policies that should explicitly allow access to secrets, but prohibit the rest.
Furthermore, the token does not have to be stored in the plain text on the server. And if it did, it could be wrapped beforehand with a response token, meaning it could be additionally protected.
And if you land on the server with the user who has the permissions to read the token, you may “deserve” to get it … just kidding. : D