Bug: cuts off html after 8 kbyte with special chars

We use HTML Purifier to clean up HTML mails from customers before displaying them. Under certain circumstances an ISO-8859-1 HTML string is cut off in the middle. The following scripts reproduces the problem:

require_once "HTMLPurifier.auto.php";

$in = "€".str_repeat(".", 50000);

$cfg = HTMLPurifier_Config::createDefault();
$cfg->set("Core.Encoding", "iso-8859-1");
$purifier = new HTMLPurifier($cfg);
$out = $purifier->purify($in);

echo "in: ".strlen($in)."";
echo "out: ".strlen($out)."";
echo $out;

Output:

in: 50007
out: 8159
................... [...]

Expected Output:

in: 50007
out: 50007
[Euro symbol]............ [...]

The problem does not occur with encoding set to UTF-8. Unfortunately we cannot just convert the encoding as the encoding is also declared in the HTML header of the input string.

This may be a bug in your version of iconv, in which it cuts off your text after an invalid character. I'm not really sure how to work around this. What happens if you pass the HTML through iconv with params ISO-8559-1 and UTF-8//IGNORE?

Thank you for your quick reply! I am not sure what you mean by passing the HTML through iconv. Both $in and $out are pure US-ASCII. There is nothing to convert.

What happens if you set %Core.EscapeNonASCIICharacters to true.

This workaround works fine! Thank you very much for your help!

OK, Iconv bug. We should probably detect and work around this.

Looks like this bug: https://bugs.php.net/bug.php?id=48147

I just submitted two upstream bugs on this issue:

http://sources.redhat.com/bugzilla/show_bug.cgi?id=13518

http://sources.redhat.com/bugzilla/show_bug.cgi?id=13517

I've added a workaround for this bug in master. Unfortunately, this will be broken in PHP 5.4 if a closely related bug in PHP isn't fixed.

I just submitted two upstream bugs on this issue:

http:// sources.redhat <dot> com/bugzilla/show_bug.cgi?id=13518

http:// sources.redhat <dot> com/bugzilla/show_bug.cgi?id=13517

Just want to add to this list since I was just reading through the related bugs and ended up searching for it, myself:

http:// sources.redhat <dot> com/bugzilla/show_bug.cgi?id=13541

That's the follow-up to Bug 13518. Maybe this'll save someone the search! :)

(spaces and <dot>s so aksimet doesn't spam-file me.)

(Edit: Fixed formatting after an HTML encoding bug ravaged the forum ^-^)

Edited 1 time(s). Last edit at 07/30/2012 01:51PM by pinkgothic.