Issue 002: WordPress REST API Publishing Pitfalls (MySQL 3-Byte UTF-8 Emoji Corruption, Markdown Nested Lists, and Theme CSS Contrast)

📌 Context

• Status: RESOLVED
• Environment: WordPress 6.x / WordPress REST API v2 / Python 3.14 (requests) / MySQL 8.x (utf8mb3 collation) / JetBlack WordPress Theme
• Date: 2026-10-20
• Detected in: Production blog post publishing via REST API client (https://<domain>)
• Component: WordPress REST API (/wp-json/wp/v2/posts), MySQL database table collation, Markdown-to-HTML parser, Theme CSS

🛑 Symptoms & Error Log

1. Emoji Corruption (? Bug): Section heading icons (📌, 🛑, 🔍, 🚫, 💡, 🛡) submitted in UTF-8 markdown were converted into literal ASCII question marks (?) in the published post (e.g. <h2>? Context</h2> and <h2>? Symptoms & Error Log</h2>).
2. Numbered List Counter Reset: Ordered list items containing nested bullet points and blank lines had their parent list counter restarted at 1. for each subsequent item (e.g. 1. Wayland... followed by 1. AMD GPU... instead of 2. AMD GPU...).
3. Unreadable Code Snippets: Inline code tags (<code>) rendered with faint grey text (#797978) on a white background (#ffffff) with no contrasting background or border, making inline snippets practically invisible. Code blocks (<pre>) rendered on light grey (#eeeeee) with faint grey text.
4. HTML Tag Collisions in Code: Placeholders with angle brackets inside inline code (e.g. <windows_host>) were parsed as raw HTML, causing web browsers to interpret them as unrecognized HTML tags and hide them completely from view.

Raw API response snippet showing corrupted character storage:

<h2>? Context</h2>
<ul>
<li><strong>Status:</strong> WORKAROUND</li>
<li><strong>Detected in:</strong> Local desktop environment / remote work RDP session to Windows host (<code></code>)</li>
</ul>
<h2>? Symptoms & Error Log</h2>

Detection: The issue was noticed immediately after publishing Issue 001 via the REST API when reviewing the live post on the blog.

🔍 Root Cause Analysis / Current Hypotheses

Four separate issues in the publishing pipeline contributed to these formatting errors:

1. MySQL 3-Byte UTF-8 (utf8mb3) Collation Limitation:
   • Legacy WordPress databases and default MySQL utf8 collations only support the Basic Multilingual Plane (BMP, code points U+0000 to U+FFFF), requiring up to 3 bytes per character.
   • Supplementary Multilingual Plane (SMP) characters—including most modern Unicode emojis (such as 📌 U+1F4CC, 🛑 U+1F6D1, 🔍 U+1F50D, 🚫 U+1F6AB, 💡 U+1F4A1, 🛡 U+1F6E1)—require 4 bytes in UTF-8.
   • When raw 4-byte UTF-8 bytes are inserted into a table with utf8 collation, MySQL silently strips the supplementary bytes and substitutes an ASCII question mark (?).
2. Parser List State Premature Closure:
   • A naive line-by-line markdown parser closed the active <ol> tag upon encountering indented bullet characters (* ) or blank separator lines between list items.
   • When subsequent numbered items (e.g. 2. **AMD...**) were encountered, the parser opened a brand new <ol> tag, resetting the browser’s list index back to 1..
3. Theme Stylesheet Color Conflict:
   • The active WordPress theme (jetblack) applied background: #ffffff to the post content container and color: #797978 to general body typography.
   • It provided no custom background, padding, or text color for <code> elements and styled <pre> with #eeeeee, resulting in faint grey-on-white text that failed accessibility contrast requirements.
4. Missing Pre-Sanitization of Code Tokens:
   • Inline code snippets containing placeholder tags (such as <windows_host>) were not HTML-escaped before markdown rendering, causing the browser DOM parser to treat the placeholder as an HTML node.

🚫 Attempted Solutions & Failed Approaches

• Approach 1: Submit raw UTF-8 emoji characters directly in the REST API JSON payload -> Result: Failed -> The MySQL database engine converted all 4-byte characters to literal ? characters upon write.
• Approach 2: Simple regex list matching without state -> Result: Failed -> Indented sub-bullets and blank lines broke the parent list structure and reset item numbering.
• Approach 3: Rely on theme default typography for code elements -> Result: Failed -> Theme CSS lacked syntax contrast, rendering inline code unreadable.

💡 Working Solution / Resolution (or Temporary Workaround)

A comprehensive four-part solution was implemented in the Python REST API client (wp_client.py):

1. Automatic 4-Byte Unicode to HTML Decimal Entity Conversion

All characters with code points above U+FFFF are converted to standard HTML decimal entities (&#{ord(char)};) before transmission:

• 📌 -> 📌
• 🛑 -> 🛑
• 🔍 -> 🔍
• 🚫 -> 🚫
• 💡 -> 💡
• 🛡 -> 🛡

Because HTML entities consist solely of 7-bit ASCII characters (&, #, digits, ;), MySQL stores them without truncation regardless of table collation, and all modern browsers render them natively as the original emojis.

2. Stateful List-Stack Parser

The parser was rewritten to maintain an indentation stack (list_stack). When an indented sub-bullet (* ) is detected under an ordered list item (1. ), the child <ul> list is nested directly inside the parent <li> tag without closing the outer <ol>:

# Child sub-list nested inside parent item:
html_lines.append("<ol>")
html_lines.append(f"<li>{parent_item_content}")
html_lines.append("<ul>")
html_lines.append(f"<li>{child_item_content}</li>")
html_lines.append("</ul>")
html_lines.append("</li>")
html_lines.append(f"<li>{next_item_content}</li>")
html_lines.append("</ol>")

This guarantees continuous numbering (1., 2., 3.) across all nested sub-items.

3. High-Contrast Code Styling & Scoped CSS Overrides

Explicit styling was applied to inline and block code elements, reinforced by an injected scoped <style> block:

• Inline <code>: Dark slate background (#2b2d30), high-contrast mint text (#4ec9b0), padded with rounded corners (padding: 2px 6px; border-radius: 4px; font-family: Consolas, Monaco, monospace;).
• Code blocks <pre>: Dark theme container (#1e1e2e), soft light text (#cdd6f4), padded borders (border: 1px solid #313244;), and horizontal scroll.

4. Code Tokenization & HTML Sanitization

Code segments wrapped in backticks are extracted and tokenized prior to parsing. Characters like < and > are replaced with < and >, ensuring placeholders like <windows_host> render visually instead of being interpreted as HTML tags.

🛡 Prevention / Next Steps

• Database Charset Upgrades: If administrator access to the MySQL database is available, convert tables to utf8mb4_unicode_ci and update DB_CHARSET in wp-config.php to utf8mb4.
• Universal REST API Fallback: Always encode 4-byte characters into HTML numeric entities when developing REST API publishing tools, providing resilience across arbitrary or unknown MySQL server configurations.
• Automated Post-Publish Verification: When publishing markdown documents via API, verify the live rendered page across diverse themes to ensure list continuity and code block legibility.