Subject: Re: Composing characters within fvwm

From: Olivier Chapuis <[email-masked]>
Date: Fri, 7 Mar 2003 08:59:36 +0100

On Wed, Feb 12, 2003 at 04:11:48PM -0800, Nadim Shaikli wrote:
> Fvwm doesn't seem to render what Unicode terms as
> "combining" or "composing" characters/marks/ligatures
> (eg. U+0308 or U+064E) correctly.  When the character
> is displayed properly its supposed to be superimposed
> on whatever proceeded it; the composing character should
> also never occupy a space of its own.  I'm attaching two
> small screenshot samples to note how its supposed to look
> versus how fvwm displays it (don't bother with the fact
> that the characters are drastically different looking,
> that's due to the fact that these composing characters
> are not combined with what's proceeding them which
> screws the required arabic joining code).
> 
> Here are some links that kinda talk about these special
> characters,
> 
>   http://www.unicode.org/faq/char_combmark.html
>   http://www.unicode.org/unicode/uni2book/ch08.pdf
> 
> Is this fixable within fvwm ?

Maybe. Does a "composing_char + a_given_char" = "another_given_char"
in UTF-8 or USC-2? If yes and a table can be found somewhere we can add
an other converter (as FBidiJoint.c). If not we should draw string
char by char and take care of these composing characters (I do not like
so much this solution). A last solution is to use Pango, but it is not
the direction we have taken so far.

Can you send me the fvwm config which allows you to produce
composing_fvwm_bad.jpg.

Regards, Olivier
--
Visit the official FVWM web page at <URL:http://www.fvwm.org/>.
To unsubscribe from the list, send "unsubscribe fvwm-workers" in the
body of a message to [email-masked].
To report problems, send mail to [email-masked]

---

From: Marcus Lundblad <[email-masked]>
Date: Fri, 7 Mar 2003 20:15:18 +0100 (CET)

> the direction we have taken so far.

I think composing is a little more complicated than this.
If I understand correctly the spec. several composing chars can follow a
character forming a new character, so that for each composable character
there is forms for all combinations of composing chars, but the order of
composing chars is irrelevant. 
So for example "a" + "umlaut" + "cedilla" would be the same as "a" +
"cedilla" + "umlaut".
So a solution would be to sort all composing characters following a
character and map the resulting "substring" to a precomposed character.

//Marcus

---

From: Nadim Shaikli <[email-masked]>
Date: Fri, 7 Mar 2003 12:11:58 -0800 (PST)

--- Olivier Chapuis <olivier chapuis free fr> wrote:
> in UTF-8 or USC-2? If yes and a table can be found somewhere we can
> add an other converter (as FBidiJoint.c).

Olivier, I don't think its "another_given_char" that needs to be produced.
What needs to happen (per what unicode notes and what I've seen) is
to super-impose the second character on _top_ of the first -- that's
why those characters (including 'umlaut') have a dashed/dotted circle
within their glyph representations in the various code charts.  Those
dashed/dotted circles are there to denote a hypothetical location of
the character that proceeded it.

  http://www.unicode.org/charts/PDF/U0300.pdf

> Can you send me the fvwm config which allows you to produce

Sure, grab the font noted here,

  http://prdownloads.sf.net/arabeyes/ae_fonts.tar.bz2

Include that font in the various "style" and "font" related resources in
your ~/.fvwm/.fvwm2rc file; the font name being --

  -arabeyes-fixed-medium-r-normal--20-200-75-75-c-100-iso10646-1

For simplicity, I'm attaching an archive that includes a minimal fvwm2rc
file as well as a screenshot of what the string(s) are supposed to look
like.  As can be seen from the RC-file, an xterm will start upon "init"
and another example is available upon "restart".

Let me know if there is _anything_ I can do to help.

 - Nadim

__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - forms, calculators, tips, more
http://taxes.yahoo.com/

---

From: Marcus Lundblad <[email-masked]>
Date: Fri, 7 Mar 2003 23:00:51 +0100 (CET)

> the character that proceeded it.

Actually it needs to be replaced by a new glyph.
For example:
"G" + cedilla => "G" with a cedilla _under_
"g" + cedilla => "g" with a cedilla _above_

(U+0122 and U+0123)

//Marcus

---

From: Nadim Shaikli <[email-masked]>
Date: Fri, 7 Mar 2003 15:13:43 -0800 (PST)

> "g" + cedilla => "g" with a cedilla _above_

Marcus, out of curiosity -- is this done visually only or is
what's stored to disk actually changed ?  ie. what is stored to
disk ("U+0047 + U+0327" or "U+0122") ?

I can see that there are a few exceptions to the superimposition
(you note one above with U+0327 when its proceeded with "g" -- looks
like the cedilla oddly enough goes from below to above mysteriously),
but overall it would still visually look OK if things were superimposed
given the exceptions were taken care of, check the various other
characters -> http://www.unicode.org/charts/PDF/U0100.pdf).

I can only conclude one of two things at this point,

 1. All composers are superimposed with a few exceptions in which
    replacements are used.

-or-

 2. There are two flavors of composing characters and their meaning.
    Some that get superimposed while others dictate the need to change
    a previous form.

I would tend to think it'd be #1

In any regard, for various languages this superimposing is the only
option (as there is nothing to substitute with).

Regards,

 - Nadim

---

From: Marcus Lundblad <[email-masked]>
Date: Sat, 8 Mar 2003 00:29:10 +0100 (CET)

> disk ("U+0047 + U+0327" or "U+0122") ?

Actually both ways can be used in cases where a precomposed form is
present in Unicode.
Although it's recommended to used composing characters. The characters in
the Latin 1-part will mostly be stored in precomposed form due to
tradition.

Further, sorting is another interesting thing. Precomposed and composed
variants is equal. And then there are different sorting order for
different languages, but that's another story...

> I would tend to think it'd be #1

I think using a precomposed form, if defined and available and superimpose
otherwise is a good idea.

By the way, how general is the shaping and joining code you wrote? Some
other alphabets (besides Arabic) could make use of this (perhaps in a
modified variant).

//Marcus

---

From: Nadim Shaikli <[email-masked]>
Date: Fri, 7 Mar 2003 16:39:16 -0800 (PST)

--- Marcus Lundblad <ml update uu se> wrote:
[snip]
> different languages, but that's another story...

Sounds very familiar; nothing that concerns Fvwm though :-)

> other alphabets (besides Arabic) could make use of this (perhaps in a

Its generic and includes various other languages as well beyond pure Arabic
(note the header on FBidiJoin.c).

The substitution you note for the example discussed earlier would be a very
simple addition if my memory hasn't failed (add the appropriate encodings
to the 'shaped_comb_table' array).  The only catch there is you need to
compile fvwm with fribidi with the appropriate flags defined, etc.

Its good to see there are other benefits from the code at hand (I knew
there would be :-).  I'm sure the composing character superimposing (once
implemented) will also open a flurry of other much needed and missed
uses/languages as well as entice a number of new users to convert.

Regards,

 - Nadim

---

From: Olivier Chapuis <[email-masked]>
Date: Sat, 8 Mar 2003 07:52:14 +0100

> uses/languages as well as entice a number of new users to convert.

Well,

So if I well understand, FBidiJoint.c should be updated to handle
some "composing char". Nadim can you send a patch? An other issue
is that FBidiJoint has nothing to do with bidi and it should be use
even if libfribidi is not detected?

After that we should take care of the superimpose case. We need a
converter which split an USC-2 or UTF-8 string into two "strings" str
and
 
{
	int position; /* position in the str of c */
	XChar2b c;    /* USC-2 char */
}
superimpose_char *superimpose_str;

Then we can easily change Flocale.c to draw these two strings when
needed. Nadim or Marcus can you write such converter I will take
care of the integration into Flocale.c

Do I well understand?

Olivier

---

From: Marcus Lundblad <[email-masked]>
Date: Sat, 8 Mar 2003 14:34:28 +0100 (CET)

> even if libfribidi is not detected?

I think the way to go is this:
First map all characters to their "combing class", ("normal" characters
have combining class = 0, various combing chars have different combinging
classes != 0).
Then traverse the string, looking at pairs of chars: if
combining-class(c1) > combining-class(c2), swap c1 and c2 (and
corresponding combining classes).
Traverse one character at a time like this.
Do this until no more swapping can be done.

Now traverse the resulting string each pair of characters if there is a
mapping for it to a precomposed form.
For example: "A" + diaresis" => Ä
Repeat this until no replacing is done.
Note that now multiple combination can be done like this:

"A" + "diaresis" + "acute" => "Ä" + "acute" => "A with diaresis and
acute" (don't have this character at hand...)

Since the combinging characters are orderd, a mapping like this would
work.

Last we can take remaining combining charaters and superimpose them as
Olivier suggests.

I'm not sure though wether we need to do a step before all this that
decomposes any eventual precomposed forms to make it correct.

For example suppose we have:

"Ä" + "acute"
and
"Á" + "diaresis"

The first would yield a "A with diaresis and acute"
but the second would give "Á with diaresis", since there will be no
mapping for this order of combinging, thus the superimposition would be
used in this case.
I don't know if this would be a problem.
We might need to first apply a decomposistion:
such that "A with diaresis and acute" will map to "Ä" + "acute" which
will in turn become "A" + "diaresis" + "acute"

//Marcus