SECURITY Signal 516
Python str.lower() in IDNA 2003 implementation deviates from Unicode 3.2.0 spec causing encoding mismatch
↗
Python’s IDNA 2003 codec used str.lower() instead of Unicode 3.2.0 case-folding rules, producing inconsistent domain name encodings.
This vulnerability breaks interoperability with systems expecting RFC 3454-compliant IDNA 2003 encoding. Engineers relying on Python’s built-in idna codec may unknowingly generate non-standard domain names, risking security or compatibility issues in applications handling internationalized domains.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Python’s IDNA 2003 implementation used str.lower() with the interpreter’s Unicode version instead of the required Unicode 3.2.0 rules.
The mismatch caused different encoded outputs for the same input, violating RFC 3454’s StringPrep specification.
A fix was implemented by adding exceptions to align str.lower() behavior with Unicode 3.2.0 for the affected functions.
THE READ
What the cluster adds up to.
Python’s IDNA 2003 support, accessed via str.encode('idna'), relies on the StringPrep algorithm defined in RFC 3454. This algorithm mandates case-folding rules from Unicode 3.2.0 to ensure consistent domain name encoding across systems. However, Python’s implementation used str.lower(), which dynamically applies the Unicode version shipped with the interpreter. This created a divergence from the specification, as newer Unicode versions handle case-folding differently than Unicode 3.2.0.
The vulnerability manifests when encoding domain names containing characters outside the ASCII range. For example, the Cherokee letter 'Ꭰ' (U+13A0) encodes differently under Unicode 3.2.0 and newer versions. Systems expecting RFC 3454-compliant output would reject or misinterpret the encoded result, breaking interoperability. The issue is particularly problematic for applications that must strictly adhere to IDNA 2003, such as legacy systems or protocols that have not migrated to IDNA 2008.
The fix involved patching Python’s stringprep module to override str.lower() behavior for specific codepoints. By comparing the output of str.lower() across Unicode versions, the patch records exceptions where the behavior deviates from Unicode 3.2.0. These exceptions ensure that str.lower() mimics the older Unicode rules only for the IDNA 2003 use case. This approach avoids breaking other parts of Python that rely on the interpreter’s native Unicode handling while restoring compliance with the specification.
Engineers should audit their use of str.encode('idna') for IDNA 2003 encoding. While the fix resolves the immediate issue, the underlying risk highlights the importance of using the idna package for IDNA 2008 support, which is the current standard. For applications requiring IDNA 2003, the patched stringprep module now provides the correct behavior, but reliance on outdated standards should be minimized where possible to avoid similar compatibility pitfalls.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER