The Bug Hiding in Plain Sight
Almost every JavaScript developer has reached for btoa() to Base64-encode a string, and it works fine — right up until someone pastes in an emoji, an accented letter, or a CJK character, and the whole thing throws InvalidCharacterError: character out of range. This isn't a bug in your code; it's a fundamental limitation of how btoa() was specified.
btoa() was designed decades ago to Base64-encode binary strings — strings where every character code point fits in a single byte (0-255, the Latin-1 range). The moment a string contains a character outside that range, like 🚀 (which needs 4 bytes in UTF-8) or é (2 bytes), btoa() either throws an error or, worse in older engines, silently truncates and corrupts the data.
The Correct Fix: Encode to UTF-8 Bytes First
The reliable pattern is to never hand a raw Unicode string to btoa() directly. Instead:
- Use
TextEncoderto convert your string into its actual UTF-8 byte representation (aUint8Array). - Convert those bytes into a Latin-1 "binary string" that
btoa()can safely consume — each byte value (0-255) maps directly to one character code. - Call
btoa()on that binary string to get valid, correct Base64.
Decoding reverses the process: atob() back to a binary string, convert each character back to its byte value, then run those bytes through TextDecoder to reconstruct the original UTF-8 string — emoji, accents, and all.
Hex Encoding Has the Same Trap
The same mistake shows up in hex encoding. A naive implementation that iterates a string's UTF-16 code units and converts each to hex will produce garbage for any character outside the Basic Multilingual Plane, or mishandle surrogate pairs used for emoji. The fix is identical: convert to UTF-8 bytes with TextEncoder first, then map each byte to a two-character hex pair. Decoding reverses it with TextDecoder.
A Quick Test You Can Run Right Now
If you're unsure whether an encoding function in your codebase is UTF-8 safe, try round-tripping a string like "Hello 🚀 café" through it. If the decoded result doesn't exactly match the original — or if encoding throws an error — the function is operating on UTF-16 code units instead of proper UTF-8 bytes, and needs the TextEncoder/TextDecoder treatment described above.
Why This Matters More in 2026
User-generated content increasingly includes emoji, and international products handle non-Latin scripts as a baseline requirement, not an edge case. An encoding utility that only works for plain ASCII text is a liability the moment real user data reaches it — silently corrupting names, messages, or any field a user is free to type into. Testing your encoding functions against genuinely international, emoji-containing input isn't optional polish; it's the actual test of whether the implementation is correct.
Where Each Encoding Actually Belongs
It's common to see developers reach for the wrong encoding simply because the four options solve visually similar-looking problems. Base64 is for safely embedding arbitrary binary data (images, files, tokens) inside text-based formats like JSON or URLs — it's not intended for readability or security, and it's trivially reversible by anyone. URL encoding exists specifically for characters that would otherwise break a URL's structure — spaces, ampersands, question marks — and should be applied to individual query parameter values, not to an entire URL at once, or you'll end up double-encoding the scheme and slashes.
HTML entity encoding solves a completely different problem: preventing user-supplied text from being interpreted as markup when it's inserted into an HTML page. Skipping this step is one of the oldest and still most common sources of cross-site scripting vulnerabilities — a comment field that renders <script> tags verbatim instead of as escaped text is an open door. Hex encoding, meanwhile, shows up most often in lower-level contexts: representing cryptographic hashes, binary protocol payloads, or byte sequences in logs and debugging output where Base64's shorter output isn't as important as hex's direct, one-byte-per-two-characters readability.
Picking the right one comes down to asking what the encoded output needs to survive passing through — a URL, an HTML document, a JSON string, or a binary-safe transport — and matching the encoding to that specific constraint rather than defaulting to whichever one is most familiar.