Subject: Re: Support for combining characters

From: Olivier Chapuis <[email-masked]>
Date: Mon, 2 Jun 2003 22:18:56 +0200

On Sat, May 31, 2003 at 07:05:19PM +0200, Marcus Lundblad wrote:
> I implemented a first basic version of combining characters.
> What it does is "compress" sequences of characters to the
> "canonical" pre-composed form.
> 
> Additionally a fix for "superimposing" needs to be added.
> Ie. combining characters that could not be composed into a pre-composed
> form should be drawn superimposed.
> 
> Olivier, could you assist me a bit on this?

Yes, but I know nothing on combining, bidi and joint ...

> If I extend my function that does composing to fill in an array mapping
> character position in the in-string to their visual position.
> 
> For example, let's say there is "o" + "some obscure composing char" + "a",
> where there is no pre-composed form for an "o" with "some obscure
> composing char", it would give an array: [0,0,1]
> That is the first 2 characters should get drawn in first "square", the
> last in the second square.
> 
> What could perhaps be tricky is to handle BIDI. The mapping array must be
> rearranged as characters are reorganised by the BIDI algorithm.
>

I do not understand the problem, sorry. But two remarks/questions:
What about merging FCombineChars, FBidi and FBidiJoin?
Which charset (in the sense of the FlocaleCharsetTable from FLocaleCharset.c)
needs Char Combining and which charset needs Joining (I know for
Bidi)?
 
> I hope my code is not crashing on anyone. Please test it.
> I have tested it, and it seems to work. But maybe I missed something
> critical.
>

Can you send some relevant strings (in UTF-8 or/and others encoding)
for adding test in purify.fvwm2rc?
 
> Also Nadim, could you test this with Arabic combining characters?
>

Marcus, you should always cc to Nadim.

Regards, Olivier
--
Visit the official FVWM web page at <URL:http://www.fvwm.org/>.
To unsubscribe from the list, send "unsubscribe fvwm-workers" in the
body of a message to [email-masked].
To report problems, send mail to [email-masked]

---

From: Marcus Lundblad <[email-masked]>
Date: Mon, 2 Jun 2003 23:25:20 +0200 (CEST)

> needs Char Combining and which charset needs Joining (I know for

Sometimes there is no character defined in Unicode for a specific
combination of an ordinary character (for example an "o") and some unusual
combinational character.
In this case, the best thing to do is to draw the combinational character
at the same position as the "o", to get an approximation of what the
combined character would look like.

I think the combination algorithm is a bit different from the contextual
shaping used for Arabic. Also the Arabic joining and shaping is only done
if BIDI is enabled (and using a charset needing BIDI).
I think Arabic joining should only be done if BIDI is done. It seems to me
it wouldn't make sense to do that without BIDI.
Nadim, do you have an opinion? 

I found some case where the algorithm fails, so I will continue to refine
it before making further descitions

> for adding test in purify.fvwm2rc?

Yes, I will take care of this.

//Marcus

---

From: Nadim Shaikli <[email-masked]>
Date: Mon, 2 Jun 2003 16:02:17 -0700 (PDT)

--- Olivier Chapuis <olivier.chapuis free.fr> wrote:
> > ie. combining characters that could not be composed into a
> > pre-composed form should be drawn superimposed.
> > 
> Yes, but I know nothing on combining, bidi and joining ...

It might be worth while to read over the following thread then,

  http://www.hpc.uh.edu/fvwm/archive/0303/msg00080.html

the crux of which is spelled out in (IMHO),

  http://www.hpc.uh.edu/fvwm/archive/0303/msg00199.html

> What about merging FCombineChars, FBidi and FBidiJoin?

I think, as was noted previously, the best approach to take here is to
do the following (order matters - I'm listing in execution order),

 1. Do Bidi
    - if needed of course
 2. Do Shaping (sometimes described/termed joining)
    - Shaping/Joining is the morphing of characters based on their
      location within a word (initial/medial/final/stand-alone)
 3. Do Combining
    - There are some characters out there that will need to be combined
      (meaning you take A+B -> C -- yup, two characters result in one;
       this is a complete replacement act. C is brand new glyph)
 4. Do Composing
    - This is the superimposition of one character on top of one that
      proceeds it (meaning you take E+F -> G -- yup two characters result
      in one; the key here is that G is NOT a new glyph its simply E with
      F superimposed on it).

Everything is currently coded-up (in one state or another) except for #4
above (its still a mystery to me how that is done, Olivier ?).

--- Marcus Lundblad <ml update.uu.se> wrote:
> > Which charset (in the sense of the FlocaleCharsetTable from
> > FLocaleCharset.c) needs Char Combining and which charset needs
> > Joining (I know for Bidi)?
> 
> combinational character.  In this case, the best thing to do is to draw
> the combinational character at the same position as the "o", to get an
> approximation of what the combined character would look like.

Marcus, if you are in essence describing "composing" (or superimposition;
point #4 above) then I agree.

> if BIDI is enabled (and using a charset needing BIDI).

Yup, as noted above shaping/joining and combining are two different things.
In the case of shaping/joining they are specific to those language that
usually require Bidi.

> it wouldn't make sense to do that without BIDI.  Nadim, do you have an
> opinion?

I would tend to agree, but there might be instances where it might be useful
to have it without Bidi (for debug purposes, etc) and/or if others might
require similar functionality for one reason or another.  I'd vote to
simply have all functions be self-contained and not dependent on each other.

To reiterate the end-result (once we know how to code-up #4 - composing)
could look like this in pseudo code,

  if (bidi_enabled)
     str = bidi_run(str);

  if (shaping_enabled && shape_range(str))
     str = shape_n_join(str);

  if (combining_enabled)
     str = combine(str);

  if (composing_enabled)
     str = compose(str);

I'm opting to include all under qualifiers for speed and controllability
aspects.  What this will then mean,

 /* Assume all options FALSE apriori */
 if (Arabic_enabled)
 {
   bidi_enabled      = TRUE;
   shaping_enabled   = TRUE;
   combining_enabled  = TRUE;
   composing_enabled = TRUE;
 }
 elsif (Nordic_enabled)  /* Marcus' concerns */
 {
   combining_enabled  = TRUE;
   composing_enabled = TRUE;
 }

Hope that makes sense.

PS: CC for faster replies (I'm an archive mole :-)

 - Nadim

__________________________________
Do you Yahoo!?
Yahoo! Calendar - Free online calendar with sync to Outlook(TM).
http://calendar.yahoo.com
--
Visit the official FVWM web page at <URL:http://www.fvwm.org/>.
To unsubscribe from the list, send "unsubscribe fvwm-workers" in the
body of a message to [email-masked].
To report problems, send mail to [email-masked]

---

From: Marcus Lundblad <[email-masked]>
Date: Tue, 3 Jun 2003 07:16:38 +0200 (CEST)

> above (its still a mystery to me how that is done, Olivier ?).
I've done 4 also.
But I do 3 and 4 before 1 and 2. maybe this is not so good.
How does the BIDI handle combining characters?
Does it preserve the order?
So that for example "Alef + combining character" is not swapped.

If that's the case, maybe I should do the combing and composing after
BIDI and then extract remaining combining characters to allow for
"superimposing"

//Marcus

---

From: Nadim Shaikli <[email-masked]>
Date: Mon, 2 Jun 2003 22:26:22 -0700 (PDT)

> > above (its still a mystery to me how that is done, Olivier ?).

Ah, cool - I'll need to see how you did it (I'm dying to know).

> So that for example "Alef + combining character" is not swapped.

Bidi knows nothing of combined characters; it would be wrong for
Bidi to actually see them since they are only noted in 'visual' context.
The added wrinkle to Bidi (and alot of the effects noted above) with
regard to Arabic is that they are all done strictly for visual affect.
In other words, if I have a string on disk somewhere - it'll get Bidi'ed
combined/composed/etc (mangled) so that a user can see it, but the bytes
on the disk never change.  To make a long story short, Bidi only knows
discrete (non-combined/non-shaped) characters.

> BIDI and then extract remaining combining characters to allow for

Yeah, I would stick with the order noted above - that would work best.

Regards,

 - Nadim