I would be against just using the public key (or 32-byte AccountId). The SS58-encoded address also includes a checksum, which has a few nice user-protection properties.
One, it protects users against missed/altered characters. If you miss or change a character in the address, the whole address becomes “wrong” and UIs would reject it as a bad address (go to a UI and try it).
Second, there was a case a while back where someone pasted in an 20-byte Ethereum address and the UI just padded it with 12 bytes of 0s to make it into a Polkadot AccountId. This was obviously inaccessible.
The encoding with the checksum (besides being a bit shorter and easier on the eyes) also protects users from these mistakes.