Subject: String encoding

From: Olivier Chapuis <[email-masked]>
Date: Fri, 7 Jun 2002 22:55:11 +0200

Hello,

At the present time with the new locale stuff when you load
a font, fvwm assume that the string which will be displayed
with this font use the font charset as encoding (assuming
for simplicity that UTF8 = iso10646-1, which is not really
the case). 

Two examples (which cover all the situations):

- Window Title name use the locale encoding, so a font for
the window titles should be a font with the locale charset as
charset. In the case of core fonts (i.e., not xft) a font with
the good charset is loaded if no charset is specified in the
font name.
- If you have a file with a menu and this file is encoded
with a given charset, you just have to use a font with
this charset and the menu will be displayed in the good way
(may fail with a "multibyte charset" if this charset is not
the locale charset).

Every things seems ok, but in fact there are some problems,
in particular with xft font, but also with core iso10646-1
fonts and maybe utf8 locales.

First an xft font support only a few encoding. Xft1 support
iso10646-1, ISO8859-1, apple-roman, glyphs-fontspecific (adobe
symbole). Xft2 support USC-4 only, and in a non direct way 
ISO8859-1 (8-bit USC-4), USC-2 (16-bit USC-4) and also UTF-8
(with a special function).
So, for window title the only adequate locale charset are UTF-8
and  ISO8859-1 for Xft1, and UTF8 for Xft2. We should make some
conversion for the others locale. So, there is no particular
problems (just bugs in the fvwm code at present time). However,
we may try to support the ISO8859-1 charset for Xft2 (to avoid
conversion), this can be done automatically by checking the locale
charset. Also, in the Xft1 case if no encoding is given by the user
and the locale charset != ISO8859-1, an iso10646-1 font should be
loaded. 

For menus and xft font one see that the encoding of the config
file should be UTF-8 or ISO8859-1 with xft1 and UTF-8 for Xft2.
This is not so good: I think that, by default, we should assume
that the configuration file uses the locale charset as encoding
(this is totally consistent with windows titles).
However, more and more things are encoded using UTF-8 (e.g., desktop
menu entries), so one may have configuration encoded in UTF-8. 
So we should provide to the user the possibility to ask fvwm
to consider that the strings to display is encoded using UTF-8.
An other reason  come from core X fonts, there are memory problem
with iso10646-1 core font. So the typical case here is: config file
in UTF-8 with only char from a given charset (e.g., KIO8-R) and we
want to use a font with this charset. So again the users should says
to fvwm that the encoding of the config file is UTF-8.

The solution?

at present time a font name is given as follow:

  shadow_stuff:xft_orand_core_font_description[[/A][/B]]

where A is the _real_ charset of the font and B the name
of an iconv converter (from/to UTF-8 to/from the charset of the
font). In a normal situation these charsets hints are not
needed (good font name and standard iconvs).

What we can do is to add an other hint in between ( and ) before
the "/" hints: (C)/A/B. C indicates that the charset/encoding of
the chars which should be displayed. Of course this have sense
only with an unicode font or if C is UTF-8. Moreover, in the case
of an xft font the default for C should be the locale charset as
with core fonts the default is the charset of the loaded font.

Any comments before I start to implement this?

Olivier
--
Visit the official FVWM web page at <URL:http://www.fvwm.org/>.
To unsubscribe from the list, send "unsubscribe fvwm-workers" in the
body of a message to [email-masked].
To report problems, send mail to [email-masked]

---

From: Mikhael Goikhman <[email-masked]>
Date: Fri, 7 Jun 2002 22:12:44 +0000

> Any comments before I start to implement this?

String encoding should better be a property of a string, not a font.
But practically it is good (easier) to specify it together with font(s).

Hovewer string encoding should by definition be the same with all fonts
in the *Font font list, so it should be specified just like shadow=1:.
The default may be different for different font types (if you think so),
but if a user specified a string encoding, map it to any found font.

I think encoding=utf-8: is a good syntax; encoding is applied to strings
and ideally (i.e. hopefully) users will not confuse it with font charset.

Regards,
Mikhael.

---

From: Olivier Chapuis <[email-masked]>
Date: Thu, 13 Jun 2002 06:42:28 +0200

> in the *Font font list, so it should be specified just like shadow=1:.

Agree.

> but if a user specified a string encoding, map it to any found font.

Ok, so the default can be the same for xft font and core font: if an
xft font has encoding=iso8859-1 or encoding=iso10646-1 in its name (and
no string encoding is specified by the user) the corresponding encoding
for string encoding will be used. If no "encoding" hints are given in
an xft name the locale logic will be used for the string encoding as
with FontSet.

> and ideally (i.e. hopefully) users will not confuse it with font charset.
 
Here the problem is that we can have a confusion with xft font name
which use "encoding=*". What about "string_encoding=*" or "str_encoding=*"
or "StrEncoding=*" or "StringEncoding=*" (and change shadow= to Shadow=
in the man page; of course the code will be case insensitive).

BTW, the new code will be ready on Monday (at least half ready as there
are pbs in the implementation: text conversion are done each time a
Flocale text drawing or text width function is called, as conversion
should be done only when the text change; I will come back on this pbs
later). This may cause 2.5.2 release delay, if you plan to release it
next week.

Olivier

---

From: Mikhael Goikhman <[email-masked]>
Date: Thu, 13 Jun 2002 06:18:40 +0000

> in the man page; of course the code will be case insensitive).

StringEncoding=utf-8 seems ok to avoid confusion with encoding=iso10646-1.

> later). This may cause 2.5.2 release delay, if you plan to release it

We are not in harry, I suggested this week, because I have some time for
this, I am not sure yet about the next week.

Regards,
Mikhael.

---

From: Mikhael Goikhman <[email-masked]>
Date: Sun, 23 Jun 2002 06:49:18 +0000

> later). This may cause 2.5.2 release delay, if you plan to release it

Is there something left to do except for non critical optimizations?

Regards,
Mikhael.

---

From: Olivier Chapuis <[email-masked]>
Date: Sun, 23 Jun 2002 18:59:40 +0200

> Is there something left to do except for non critical optimizations?

No, I do not think so. I will come on optimization soon but this
may need discussion.

Olivier