EllisLab text mark
Advanced Search
     
CI can’t display UTF8 data that equivalent PHP/mysql code has no trouble with
Posted: 09 November 2012 02:13 PM   [ Ignore ]
Joined: 2009-01-07
26 posts

I’m trying to use CodeIgniter to display some unicode data stored in MySQL. I am able to successfully query and display the data using PHP/mysql but the unicode data does not display correctly when I query it using CodeIgniter.

Here is the SQL I used to create the table:

CREATE TABLE `languages` (
  `
idchar(2NOT NULL DEFAULT '' COMMENT 'The ISO code for this language.',
  `
namevarchar(40NOT NULL COMMENT 'The name of the language in English.',
  `
local_namevarchar(40NOT NULL DEFAULT '' COMMENT 'The name of the language in that language',
  `
sequenceint(11) DEFAULT '0',
  `
createddatetime NOT NULL DEFAULT '0001-01-01 00:00:00',
  `
modifieddatetime NOT NULL DEFAULT '0001-01-01 00:00:00',
  
PRIMARY KEY (`id`)
ENGINE=InnoDB DEFAULT CHARSET=utf8 

Here is the code I used to successfully query and output data from the table using PHP/mysql:

if($cn mysql_connect('localhost''xxxx''xxxx')) {
 
echo "Connected to db.<br />";
else {
 
die("Could not connect to db.");
}

mysql_select_db
('mydb');
$result mysql_query("SELECT local_name FROM languages;");

while(
$row mysql_fetch_array($result)) {
 
echo $row['local_name'];
 echo 
"<br />";

Here is the code I used to query and output the code using CodeIgniter:

function index() {
        
echo "<!doctype html>\n";
        echo 
"<html><head><title>test</title>\n";
        echo 
"<meta http-equiv=\"Content-Type\" content=\"text/htmlcharset=utf-8\" />\n";
        echo 
"</head>";
        echo 
"<body>";
        
$langs $this->db->get('languages')->result();
        foreach(
$langs as $lang{
                
echo $lang->local_name;
                echo 
"<br />";
        
}
        
echo "</body></html>";

My CodeIgniter config/config.php setting contains this setting:

$config['charset''UTF-8'

My CodeIgniter config/database.php contains these settings:

$db['default']['char_set''utf8';
$db['default']['dbcollat''utf8_general_ci'

I spent literally all day yesterday twiddling with various settings for MySQL, Apache, PHP, and CodeIgniter, but the fact that the output works using PHP/mysql and doesn’t work with the equivalent code in CodeIgniter suggests to me that the problem exists at the CodeIgniter level. (Although I should add that I am only seeing the problem when I run the code on my Ubuntu testing machine—when I run similar code on my OSX development machine the CodeIgniter version of the code does run correctly.)

Any ideas for something else to try to get this to work?

 
Posted: 09 November 2012 02:40 PM   [ Ignore ]   [ # 1 ]   [ Rating: 0 ]
Avatar
Joined: 2012-01-09
117 posts

The answer is pretty simple - don’t echo, use view files. smile

Nowadays, setting the character set via meta tags isn’t really effective - most browsers seem to just ignore it. CodeIgniter takes care of that in a real HTTP Content-Type header, using the value that you’ve set with $config[‘charset’], but when you echo data from your controller - it gets outputted before response headers are sent and that breaks the whole thing.

 
Posted: 09 November 2012 02:45 PM   [ Ignore ]   [ # 2 ]   [ Rating: 0 ]
Joined: 2009-01-07
26 posts

I don’t think the fact that I’m echoing code from a controller function is the problem. The problem happens when I use a view to output the data as well.

Also, the HTTP Content-Type header is actually being sent correctly (according to Chrome, it’s “text/html; charset=utf-8” which is what I expected).

 
Posted: 09 November 2012 03:46 PM   [ Ignore ]   [ # 3 ]   [ Rating: 0 ]
Avatar
Joined: 2012-01-09
117 posts

While headers will be sent (and received by the browser), any output, including echoing from the controller, will cause a problem. If you have the same problem when using a view, then you probably have some white space characters outputted (preceeding a <?php tag maybe?).

Alter your php.ini settings to set display_errors to 1 and error_reporting to E_ALL - if I’m correct, then you’ll see numerous PHP E_WARNING (or E_NOTICE, not sure) messages saying that output is already sent when header() is executed.

 
Posted: 09 November 2012 04:17 PM   [ Ignore ]   [ # 4 ]   [ Rating: 0 ]
Joined: 2009-01-07
26 posts

> any output, including echoing from the controller, will cause a problem

Why? Why is this a problem echoing from CI when it’s not a problem echoing the identical data from PHP/mysql?

> you probably have some white space characters outputted

It’s difficult to see how something like this could be the problem, since it’s happening consistently with several separate files.

I set error_reporting to E_ALL and display_errors to 1. No change and no error messages.

 
Posted: 09 November 2012 04:34 PM   [ Ignore ]   [ # 5 ]   [ Rating: 0 ]
Avatar
Joined: 2012-01-09
117 posts
Jeffrey McManus - 09 November 2012 04:17 PM

> any output, including echoing from the controller, will cause a problem

Why? Why is this a problem echoing from CI when it’s not a problem echoing the identical data from PHP/mysql?

Because your “pure PHP/mysql” script either doesn’t send custom headers or sends them at its very start, while CodeIgniter will send headers after your controller code is executed.

This also answers your other question:

Jeffrey McManus - 09 November 2012 04:17 PM

> you probably have some white space characters outputted

It’s difficult to see how something like this could be the problem, since it’s happening consistently with several separate files.

I don’t know why you’re not seeing any error_messages though, unless you’ve set your environment to ‘production’.

 
Posted: 09 November 2012 04:46 PM   [ Ignore ]   [ # 6 ]   [ Rating: 0 ]
Joined: 2009-01-07
26 posts

My PHP/mysql script does not alter any HTTP headers, that is correct.

So if CI is really attempting to send headers after the controller code is executed (which seems wrong to me), how should I proceed? Remember that I’m seeing this error whether I echo the UTF8 data from a controller function or from within a view.

By “set your environment to production,” what do you mean? Are you referring to configuration groups in database.php?

 
Posted: 09 November 2012 04:56 PM   [ Ignore ]   [ # 7 ]   [ Rating: 0 ]
Avatar
Joined: 2012-01-09
117 posts
Jeffrey McManus - 09 November 2012 04:46 PM

My PHP/mysql script does not alter any HTTP headers, that is correct.

So if CI is really attempting to send headers after the controller code is executed (which seems wrong to me), how should I proceed? Remember that I’m seeing this error whether I echo the UTF8 data from a controller function or from within a view.

Well, at this point unless you show me your full controller and view(s) code, I can’t really say anything more.

Jeffrey McManus - 09 November 2012 04:46 PM

By “set your environment to production,” what do you mean? Are you referring to configuration groups in database.php?

No, there’s an ENVIRONMENT constant that is defined in index.php. If you choose ‘production’ there, you’ll see that it also disables error reporting.

 
Posted: 09 November 2012 05:05 PM   [ Ignore ]   [ # 8 ]   [ Rating: 0 ]
Joined: 2009-01-07
26 posts

Ah, I see. No, my environment is set to development.

Here is the full source of my test controller:

http://files.codelesson.com/beta.codelesson.com/test.php.txt

The output of this controller is here:

http://beta.codelesson.com/test

 
Posted: 09 November 2012 05:20 PM   [ Ignore ]   [ # 9 ]   [ Rating: 0 ]
Avatar
Joined: 2012-01-09
117 posts

You DO have to switch to using views ... It won’t get fixed any way if you’re still echoing from the controller (unless you change your default apache charset).

 
Posted: 09 November 2012 08:46 PM   [ Ignore ]   [ # 10 ]   [ Rating: 0 ]
Avatar
Joined: 2012-01-09
117 posts

Btw, a second look points me to another guess - it is quite possible that the data was erroneously encoded while it was inserted (assuming you didn’t do that via CodeIgniter).

Another thing that you should take care of is to have the database itself created with default character set utf8 (not just the table).

 
Posted: 11 November 2012 09:25 PM   [ Ignore ]   [ # 11 ]   [ Rating: 0 ]
Joined: 2009-01-07
26 posts

The database is not where the problem lies. If the database were the problem, then I would not be able to query and display the database correctly using PHP/mysql. (Right?)

That said, SHOW CREATE DATABASE confirms that the database was created with a default character set of UTF8.

 
Posted: 11 November 2012 09:39 PM   [ Ignore ]   [ # 12 ]   [ Rating: 0 ]
Joined: 2009-01-07
26 posts

I just thought of one more thing to try that pretty much confirms my assertion that this is an issue with CodeIgniter 2.1.x rather than a configuration issue or a problem with the data.

I grabbed a copy of CodeIgniter 3.0-dev from Github, configured it to work against the same database, on one of the same dev machines were I saw the problem last week.

Display of the data works correctly in this scenario. So from this, it’s clear that between 2.1.2 and 3.0-dev, something changed with the way that unicode data is being handled. PHP/mysql does it right, CI 3.0-dev does it right, but CI 2.1 does not do it right.

 
Posted: 12 November 2012 12:36 AM   [ Ignore ]   [ # 13 ]   [ Rating: 0 ]
Joined: 2009-01-07
26 posts

And by trying the same code in a fresh install of CI 2.1.3, I am able to confirm definitively that the unicode problem is a bug in CI 2.1.2.

Thanks for your suggestions for resolving this, Narf.

 
Posted: 12 November 2012 04:18 AM   [ Ignore ]   [ # 14 ]   [ Rating: 0 ]
Avatar
Joined: 2012-01-09
117 posts

You’re welcome.

However, nothing has changed between 2.1.2 and 2.1.3 that affects this.

EDIT: Links somehow get broken, can’t post the compare diff link.