Subject: Re: Shaping/Joining within fvwm

From: Mikhael Goikhman <[email-masked]>
Date: Sun, 1 Dec 2002 12:50:38 +0000

On 01 Dec 2002 01:04:09 +0100, Olivier Chapuis wrote:
> 
> On Sun, Nov 24, 2002 at 12:49:35PM -0800, Nadim Shaikli wrote:
> > --- Ethan Blanton wrote:
> > > Nadim Shaikli spake unto us the following wisdom:
> > > > Ideally it would be wonderful to simply run fvwm and this code
> > > > under a debugger and single step around for a better feel for
> > > > what is causing it to go into the weeds.
> > > 
> > > How about running your debugger/fvwm in a screen session?  You can
> > > then connect to that screen session once X is started up and work
> > > from there, and if it bombs out you can restart X and connect to
> > > the crashed session for some post-mortem analysis.
> > 
> > **AND**
> > 
> > --- Mikhael Goikhman wrote:
> > > If you don't want your windows to go down, just verify that X does
> > > not go down. This may be achieved if you start fvwm in the background
> > > using "&" and xterm in the foreground in ~/.xinitrc, so that you may
> > > recompile and launch another fvwm from this (or another) xterm.
> > 
> > With both suggestions in-hand, I tried to run a debugger on fvwm while
> > running it (I've never done such a thing) and needless to say it was
> > a major pain with lots of freezing and rebooting, etc.  In short, I
> > didn't get anywhere.
> > 
> > I do have a possible idea of what is happening.  Mikhael/Olivier, this
> > might make sense to you guys more than others :-) so let me know if you
> > want further detail off the list.  I added a continuous assignment to a
> > Form-B glyph right above the first '#if 0' in the FBidiJoin.c file (say
> > to 0xFED3) and that failed.  Which leads me to believe that unicode
> > conversion within fvwm is failing (this would also explain why the same
> > code doesn't fail outside fvwm).  Does this make sense or ring any
> > bells ?  In other words, a glyph value of 0x0641 is fine where as 0xFED3
> > seems to trigger memory stomping - the difference there is how UTF-8 is
> > represnted and converted (are all UTF-8's within fvwm represented as
> > 4-bytes since that could be the max size ? 
> 
> No and it should not. UTF-8 is not UCS-4. Do I miss somthing?
> 
> > The difference in the example
> > noted above it 2-bytes vs. 3-bytes).  Am I barking up the wrong tree :-) ?
> >
> 
> Ok, I've found the problem. FBidiConvert assume that the "in" string
> and the "out" string have the same length. This is the case until you
> joint. With joint what may happen is:
> 
>   UTF-8 src string have length 34 (as char *)
>   --> convert into UCS-4, the string has length 20 (as char32 *)
>   --> apply bidi algo, the string has length 20 (as char32 *)
>   --> apply joint algo, the string has length 20 (as char32 *)
>   --> convert back to UTF-8, the string has length 40 (as char *)
> 
>     ----> core dump
> 
> the length of the returned string (visual_str) should be computed
> after the joint algo is performed. BTW, fribidi should know the length
> of the result of fribidi_unicode_to_charset before you call it, this
> is a very very strange way to implement a converter!!

I think it should only know the length of the input (not sure why this is
needed, maybe to double check or to limit to the substring), not the
length of the output, which it returns.

> So we need the function:
> 
>   USC4UTF8Length(Char32 *str): return the length of an UCS-4 string
>                                considered as an UTF-8 string
> 
> and also the following function will be useful (to allocate
> logical_unicode_str at exact size):
> 
>   UTF8USC4Length(unsigned char *str): return the length of an utf-8 string
>                                       considered as an USC-4 string
> 
> a simple workaround is to replace  
> 
> 	visual_str = (char *)safemalloc((str_len + 1) * sizeof(char));
> 
> by
> 
> 	visual_str = (char *)safemalloc((4*str_len + 1) * sizeof(char));
> 
> I do not have the time to write down the fix in the next 5 days (at
> least two others thing todo before: IconMan and LocalePath).
> 
> Mikhael, Nadim ?

Good work at finding the problem. I will try to enable the new code by
allocating more space, today or tomorrow. But I think I already tried to
allocate 3*str_len and it still core dumped... I am sure though your
analyze is correct.

Regards,
Mikhael.
--
Visit the official FVWM web page at <URL:http://www.fvwm.org/>.
To unsubscribe from the list, send "unsubscribe fvwm-workers" in the
body of a message to [email-masked].
To report problems, send mail to [email-masked]

---

From: Mikhael Goikhman <[email-masked]>
Date: Sun, 1 Dec 2002 15:54:38 +0000

> length of the output, which it returns.

Ah, I see now what you mean. Yes, fribidi assumes the caller allocate the
space for the output, that is just an incorrect thing to do for such
library, because a caller can't know the output length yet.

> allocate 3*str_len and it still core dumped... I am sure though your

Ok, I did some changes. Including removing alloca (not that this helps
with speed and core dumps).

I allocate 4 * str_len. I don't think it worths to write now more exact
calculations when it is the fribidi task to worry about an allocation.
Seems to work, although I can't guarantee it will not core dump.

Here is some result. It is possible my font is bad, because I don't see
spaces between words. But at least I can see the text is more connected.

  http://fvwm.sf.net/tmp/arabic-nonjoined.gif
  http://fvwm.sf.net/tmp/arabic-joined.gif

Regards,
Mikhael.

---

From: Nadim Shaikli <[email-masked]>
Date: Mon, 2 Dec 2002 18:36:08 -0800 (PST)

--- Olivier Chapuis wrote:
[snip snip]
>                                       considered as an USC-4 string

Olivier, here's my crack at it (code sniplets attached).  Feel free to
modify as you see fit & do please check it.  Looking forward to a working
solution.

Regards & thanks for your continued help.

 - Nadim (CC for faster responses)

__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

---

From: Nadim Shaikli <[email-masked]>
Date: Thu, 5 Dec 2002 17:13:01 -0800 (PST)

Mikhael Goikhman wrote:
> Ok, I did some changes. Including removing alloca (not that
> this helps with speed and core dumps).

Thanks Olivier/Mikhael for all your help (it would have taken
me months to debug that myself).  I just fixed the combinational
character problem (that was noted by myself in the code) and
reformatted the code a bit for better readability and for it to
be no more than 80 characters wide (so I'm including the entire
file so that you can simply substitute what's in CVS with what's
noted (FBidiJoin.c) - I've been running with it for awhile now).
I'm also including the arabic .po files along with the minor
patch to include 'ar' as a "lang".

Arabic (& derivatives) are now 99% complete (I will mail later
about the remaining 1% with regards to harakat/tanween (eg. U+064E)
characters).

BTW: Olivier, did those UCS4<->UTF8 functions look ok ?

Much thanks.

 - Nadim (CC for faster replies)