The Bug Hiding in Plain Sight
Almost every JavaScript developer has reached for btoa() to Base64-encode a string, and it works fine — right up until someone pastes in an emoji, an accented letter, or a CJK character, and the whole thing throws InvalidCharacterError: character out of range. This isn't a bug in your code; it's a fundamental limitation of how btoa() was specified.
btoa() was designed decades ago to Base64-encode binary strings — strings where every character code point fits in a single byte (0-255, the Latin-1 range). The moment a string contains a character outside that range, like 🚀 (which needs 4 bytes in UTF-8) or é (2 bytes), btoa() either throws an error or, worse in older engines, silently truncates and corrupts the data.
The Correct Fix: Encode to UTF-8 Bytes First
The reliable pattern is to never hand a raw Unicode string to btoa() directly. Instead:
- Use
TextEncoderto convert your string into its actual UTF-8 byte representation (aUint8Array). - Convert those bytes into a Latin-1 "binary string" that
btoa()can safely consume — each byte value (0-255) maps directly to one character code. - Call
btoa()on that binary string to get valid, correct Base64.
Decoding reverses the process: atob() back to a binary string, convert each character back to its byte value, then run those bytes through TextDecoder to reconstruct the original UTF-8 string — emoji, accents, and all.
Hex Encoding Has the Same Trap
The same mistake shows up in hex encoding. A naive implementation that iterates a string's UTF-16 code units and converts each to hex will produce garbage for any character outside the Basic Multilingual Plane, or mishandle surrogate pairs used for emoji. The fix is identical: convert to UTF-8 bytes with TextEncoder first, then map each byte to a two-character hex pair. Decoding reverses it with TextDecoder.
A Quick Test You Can Run Right Now
If you're unsure whether an encoding function in your codebase is UTF-8 safe, try round-tripping a string like "Hello 🚀 café" through it. If the decoded result doesn't exactly match the original — or if encoding throws an error — the function is operating on UTF-16 code units instead of proper UTF-8 bytes, and needs the TextEncoder/TextDecoder treatment described above.
Why This Matters More in 2026
User-generated content increasingly includes emoji, and international products handle non-Latin scripts as a baseline requirement, not an edge case. An encoding utility that only works for plain ASCII text is a liability the moment real user data reaches it — silently corrupting names, messages, or any field a user is free to type into. Testing your encoding functions against genuinely international, emoji-containing input isn't optional polish; it's the actual test of whether the implementation is correct.
Where Each Encoding Actually Belongs
It's common to see developers reach for the wrong encoding simply because the four options solve visually similar-looking problems. Base64 is for safely embedding arbitrary binary data (images, files, tokens) inside text-based formats like JSON or URLs — it's not intended for readability or security, and it's trivially reversible by anyone. URL encoding exists specifically for characters that would otherwise break a URL's structure — spaces, ampersands, question marks — and should be applied to individual query parameter values, not to an entire URL at once, or you'll end up double-encoding the scheme and slashes.
HTML entity encoding solves a completely different problem: preventing user-supplied text from being interpreted as markup when it's inserted into an HTML page. Skipping this step is one of the oldest and still most common sources of cross-site scripting vulnerabilities — a comment field that renders <script> tags verbatim instead of as escaped text is an open door. Hex encoding, meanwhile, shows up most often in lower-level contexts: representing cryptographic hashes, binary protocol payloads, or byte sequences in logs and debugging output where Base64's shorter output isn't as important as hex's direct, one-byte-per-two-characters readability.
Picking the right one comes down to asking what the encoded output needs to survive passing through — a URL, an HTML document, a JSON string, or a binary-safe transport — and matching the encoding to that specific constraint rather than defaulting to whichever one is most familiar.
Frequently Asked Questions
btoa() was designed decades ago to Base64-encode binary strings where every character code point fits in a single byte (0-255, the Latin-1 range). An emoji like 🚀 needs 4 bytes in UTF-8 and an accented letter like é needs 2, both outside that range, so btoa() either throws InvalidCharacterError or, in older engines, silently truncates and corrupts the data instead of erroring. This isn't a bug in your code — it's a fundamental limitation of how the function was specified.
Never hand a raw Unicode string to btoa() directly. Instead, use TextEncoder to convert the string into its actual UTF-8 byte representation (a Uint8Array), map those bytes into a Latin-1 'binary string' that btoa() can safely consume (each byte value maps to one character code), and then call btoa() on that binary string. Decoding reverses the process: atob() back to the binary string, convert each character back to its byte value, then run the bytes through TextDecoder to reconstruct the original string, emoji included.
Neither is a universal answer — they solve different problems. Base64 is for safely embedding arbitrary binary data like files, images, or tokens inside text-based formats such as JSON, and it isn't meant for readability or security since it's trivially reversible. URL encoding exists specifically to escape characters that would otherwise break a URL's structure — spaces, ampersands, question marks — and should be applied to individual query parameter values rather than an entire URL at once, or you'll end up double-encoding the scheme and slashes.
Round-trip a string like "Hello 🚀 café" through it — encode, then decode, and compare the result to the original. If the decoded output doesn't exactly match, or encoding throws an error, the function is operating on raw UTF-16 code units instead of proper UTF-8 bytes and needs the TextEncoder/TextDecoder treatment. This matters more than it used to since user-generated content routinely includes emoji and non-Latin scripts as a baseline case, not an edge case.
Yes — a Multi-Format Encoder & Decoder handles Base64, hex, URL, and HTML entity encoding with proper UTF-8 byte handling built in, so emoji and accented characters round-trip correctly instead of throwing the InvalidCharacterError that a plain btoa() call produces. It's also useful for quickly checking which encoding actually fits your use case — Base64 for binary-safe embedding, URL encoding for query parameters, or hex for byte-level debugging output.