mikeash.com pyblog/unicode-comments-support.html comments

Gaurav sharma - 2009-10-20 12:53:33

Tue, 20 Oct 2009 12:53:33 GMT

MoMolog - 2009-10-16 10:52:45

Fri, 16 Oct 2009 10:52:45 GMT

Это правда. Отлично!

Cy - 2009-09-15 15:44:36

Tue, 15 Sep 2009 15:44:36 GMT

Interesting... so now I can say: España!

Pierre Lebeaupin - 2009-08-28 13:37:37

Fri, 28 Aug 2009 13:37:37 GMT

Voyons voir si ça marche… Désolé, sur un PC, donc pas accès à tout ce que je voudrais. À part ça, pas de problème. Cœur. €. « Pardon ? » lui dit-elle.

Jean-Daniel Dupas - 2009-08-26 20:56:35

Wed, 26 Aug 2009 20:56:35 GMT

Welcome to the wonderful world of Unicode and MySQL.
It's one of many reasons why i never used MySQL, but under menace. I'm usually using Postgres when I need a full features, multi-clients database and SQLite for simpler needs :-)

Augie Fackler - 2009-08-24 15:18:18

Mon, 24 Aug 2009 15:18:18 GMT

I've actually seen lots of problems with MySQL and Unicode - in general, you have to do the right dance when you make the database or else it'll fail miserably on the unicode characters.

Pádraig Brady - 2009-08-24 09:36:53

Mon, 24 Aug 2009 09:36:53 GMT

There seem to be no end of problems with text encodings and MySQL. It's at least hard to configure but never having used MySQL myself I don't know the details. Here's an example of madness coming from a MySQL database configured by someone who knows about this stuff: http://www.pixelbeat.org/docs/unicode_utils/

cheers,
Pádraig.

mikeash - 2009-08-23 12:53:41

Sun, 23 Aug 2009 12:53:41 GMT

Believe me, I tried that. I tried everything. Nothing worked. I believe that it can work, but it would not work for me. I have no idea where the error was (which was a big part of the difficulty; it's hard to fix things when you don't know where the problem is) I just know that I couldn't make it work after a lot of trying.

Plus, SQLite is easier to work with and easier to back up, so, bonus!

nils - 2009-08-23 08:13:02

Sun, 23 Aug 2009 08:13:02 GMT

normally you should only have to supply a charset on connect ( connect(..., charset="utf8")) an then throw unicode objects at it and the whole stuff should be converted automatically

@astrange: Well utf8 takes the same storage space as latin1 unless you use non-ASCII chars.

mikeash - 2009-08-23 05:17:02

Sun, 23 Aug 2009 05:17:02 GMT

Could very well have been in the Python MySQLdb module, rather than in MySQL itself. I really don't know. I know I tried a lot of things on both sides and nothing ended up working.

I don't parse entities. Everything you write gets HTML escaped except for a very small, predefined set of known tags. There's just too much guesswork involved as to whether you want unichar 8238 or whether you want ampersand, hash mark, digit eight, etc.

astrange - 2009-08-23 05:07:45

Sun, 23 Aug 2009 05:07:45 GMT

Hmm, sounds more like a Python-side problem. You can store all the arbitrary bytes you want in MySQL text no matter the collation; I never bother changing from latin1, since it takes half the storage space (unless your script is smaller in UTF16) and I don't need fulltext indexes. But a DB library might try to do charset conversion from that.

Does ‮ work?

mikeash - 2009-08-23 04:58:33

Sun, 23 Aug 2009 04:58:33 GMT

Téstïng 1, 2, 3....

再见。

mikeash - 2009-08-22 23:25:41

Sat, 22 Aug 2009 23:25:41 GMT

Well, found one bug already: my timestamps were being generated in local (Pacific) time but displayed as if they were GMT. Result: posts in the future! Got it fixed now. Oops.