Subject: Re: Shaping/Joining within fvwm

From: Nadim Shaikli <[email-masked]>
Date: Mon, 18 Nov 2002 12:09:16 -0800 (PST)

Mikhael Goikhman wrote:
> On Tue, Nov 12, 2002 at 02:44:15PM -0800, Nadim Shaikli wrote:
> > Here is my dilemma, I'm including a patch which works on the
> > simple strings (1, 2 words in titles), but seems to fail on
> > more complex ones without any indicators to why.  I'm unable
> > to reproduce those problems without running fvwm (ie. I can't
> > get the code to fail outside of its inclusion into fvwm).  I'm
> > also unfamiliar with Xnest and how to go about setting that whole
> > thing up, etc.  Which leads me to my humble request.  As noted
> > I'm including the patch against CVS snapshot 20021111 (and
> > fribidi-0.10.4) and I'm also including a simple xterm call with
> > an Arabic/utf-8 title.  If you were to chmod 777 that xterm call
> > and invoke it, fvwm fails and core dumps - the traceback doesn't
> > really reveal much to me (as I'm not too familiar with the fvwm
> > code).  So I'm asking for help.
> 
> Log message:
> * incorporated a shape-and-join bidi functionality from Nadim Shaikli
> * it is disabled for now (#if 0) since some Arabic chars cause core dump

Mikhael/Olivier, were you able to reproduce the problem ?  In other
words, is it core-dumping for you as well ?  I'm very curious and
anxious to know what it was/is :-)

Regards,

 - Nadim (I'm at the mercy of the daily fvwm archive updates,
          so please CC me)

__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - Let the expert host your site
http://webhosting.yahoo.com
--
Visit the official FVWM web page at <URL:http://www.fvwm.org/>.
To unsubscribe from the list, send "unsubscribe fvwm-workers" in the
body of a message to [email-masked].
To report problems, send mail to [email-masked]

---

From: Mikhael Goikhman <[email-masked]>
Date: Mon, 18 Nov 2002 20:57:19 +0000

> anxious to know what it was/is :-)

Yes, it more or less core dumps on me when a new xterm with some arabic
text that you sent (or any its substring) is launched or when it is moved.
I simply used some random "*-iso10646-1" font, not the one you sent.
I use fribidi-0.10.1 which does not have fribidi_boolean definition.

Although if you take a look at my "#if 0" it would seem like the problem
is in your code, it is possible that the problem is in FBidi.c or fribidi.
Something is not allocated enough. I tried to increase all allocations in
FBidi and replace alloca, but it still core dumps with your code enabled.
Since your code simply replaces some characters, the only reason I may
think about is you use some unicode character that makes fribidi unhappy.
I may take a look at this later. Or maybe you may fix it yourself.

Regards,
Mikhael.

---

From: Nadim Shaikli <[email-masked]>
Date: Mon, 18 Nov 2002 13:20:54 -0800 (PST)

> I use fribidi-0.10.1 which does not have fribidi_boolean definition.

Luckily fribidi (fribidi.sf.net) is rather easy to download and compile :-)
BTW, the patch was against 0.10.4 (current release version).

> I may take a look at this later. Or maybe you may fix it yourself.

As was noted - I've tried the same exact code outside of fvwm and it
worked flawlessly on thousands of examples (entire directories worth
of files) without any issues.  I will certainly stare at it again
but without the ability to generate a controlled failure (ie. not
having my entire session disappear on me :-) and without me knowing
or having Xnest (or equivalent), I'm more likely to keep asking for
help (ouch).  Could something within fvwm's own code be getting rubbed
the wrong way ?

Regards,

 - Nadim

---

From: Mikhael Goikhman <[email-masked]>
Date: Mon, 18 Nov 2002 22:25:57 +0000

> BTW, the patch was against 0.10.4 (current release version).

You underestimate the problem. User1 may have fribidi-0.9.0, User2 may
have fribidi-0.10.1 and User3 may have fribidi-0.10.4. fvwm should be
built successfully for all of them. It is only that User1 should get no
Bidi support while 2 others should get this support. So we can't start to
use any new fribidi features without also changing configure.in to work
correctly for User2 and UserN. In theory "make" should never fail.

> help (ouch).

I launched another X (this is safe with XFree86-4.2) from another virtual
console with fvwm running no config or a minimal config. I don't think you
need the exact commands to do this, but I may send you.

> Could something within fvwm's own code be getting rubbed the wrong way ?

Maybe. But I would try to narrow the "#if 0" code even more so that there
is still no problem.

Regards,
Mikhael.

---

From: Nadim Shaikli <[email-masked]>
Date: Mon, 18 Nov 2002 14:54:13 -0800 (PST)

--- Mikhael Goikhman wrote:
> > Luckily fribidi (fribidi.sf.net) is rather easy to download and
> > compile :-)  BTW, the patch was against 0.10.4 (current release
> > version).
> 

You are absolutely correct; I wasn't thinking of that - my bad.  As a
workaround we/fvwm could require 0.10.4+ (akin to xpm's minimum library
requirements, etc).  Chances are whomever wants Bidi support, he/she
would be more than willing to install a newer version of the library.
I'm sure there are issues with that over simplification, but its a
thought :-)  Mucking with autotools is never fun.

> need the exact commands to do this, but I may send you.

I primarily have access to a Sun running solaris, thus I'm afraid its
not much of an option.

> Maybe. But I would try to narrow the "#if 0" code even more so that there

That would be interesting to see.  Ideally it would be wonderful to
simply run fvwm and this code under a debugger and single step around
for a better feel for what is causing it to go into the weeds.  In any
regard, I'm all ears :-)

Thanks a heap for the help and attention to this issue.

 - Nadim

---

From: Ethan Blanton <[email-masked]>
Date: Mon, 18 Nov 2002 18:11:48 -0500

> for a better feel for what is causing it to go into the weeds.  In any

How about running your debugger/fvwm in a screen session?  You can
then connect to that screen session once X is started up and work from
there, and if it bombs out you can restart X and connect to the
crashed session for some post-mortem analysis.

Ethan

-- 
And if I claim to be a wise man / it surely means that I don't know.
                -- Kansas, "Carry on Wayward Son"

---

From: Mikhael Goikhman <[email-masked]>
Date: Mon, 18 Nov 2002 23:17:47 +0000

> thought :-)  Mucking with autotools is never fun.

It is possible to think here in functions/types, not in versions. I ported
the code to at least fribidi-0.10.1 by replacing fribidi_boolean.
Otherwise I would need to add fribidi_boolean to the test program in
configure.in (instead of requiring 0.10.4, or is it 0.10.3 or 0.10.5?),
Of course when fribidi will provide the shaping support internally, we
will add this new function to the test program. But currently I see no
reason to require 0.10.4.

> I primarily have access to a Sun running solaris, thus I'm afraid its

If you don't want your windows to go down, just verify that X does not go
down. This may be achieved if you start fvwm in the background using "&"
and xterm in the foreground in ~/.xinitrc, so that you may recompile and
launch another fvwm from this (or another) xterm.

Regards,
Mikhael.

---

From: Olivier Chapuis <[email-masked]>
Date: Mon, 18 Nov 2002 22:17:46 +0100

> anxious to know what it was/is :-)

I've not the time now. If Mikhael does not solve the pbs
I will take a look after I've finished my work on Colorset
in TitleStyle and co.

Olivier

---

From: Nadim Shaikli <[email-masked]>
Date: Sun, 24 Nov 2002 12:49:35 -0800 (PST)

--- Ethan Blanton wrote:
> > Ideally it would be wonderful to simply run fvwm and this code
> > under a debugger and single step around for a better feel for
> > what is causing it to go into the weeds.
> 
> then connect to that screen session once X is started up and work
> from there, and if it bombs out you can restart X and connect to
> the crashed session for some post-mortem analysis.

**AND**

--- Mikhael Goikhman wrote:
> If you don't want your windows to go down, just verify that X does
> not go down. This may be achieved if you start fvwm in the background
> using "&" and xterm in the foreground in ~/.xinitrc, so that you may
> recompile and launch another fvwm from this (or another) xterm.

With both suggestions in-hand, I tried to run a debugger on fvwm while
running it (I've never done such a thing) and needless to say it was
a major pain with lots of freezing and rebooting, etc.  In short, I
didn't get anywhere.

I do have a possible idea of what is happening.  Mikhael/Olivier, this
might make sense to you guys more than others :-) so let me know if you
want further detail off the list.  I added a continuous assignment to a
Form-B glyph right above the first '#if 0' in the FBidiJoin.c file (say
to 0xFED3) and that failed.  Which leads me to believe that unicode
conversion within fvwm is failing (this would also explain why the same
code doesn't fail outside fvwm).  Does this make sense or ring any
bells ?  In other words, a glyph value of 0x0641 is fine where as 0xFED3
seems to trigger memory stomping - the difference there is how UTF-8 is
represnted and converted (are all UTF-8's within fvwm represented as
4-bytes since that could be the max size ? The difference in the example
noted above it 2-bytes vs. 3-bytes).  Am I barking up the wrong tree :-) ?

BTW: for those not familiar with UTF-8, here's a bit of light reading.

  http://www.joconner.com/javai18n/articles/UTF8.html

Thanks

 - Nadim (do please CC replies for faster responses)

__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus – Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
Visit the official FVWM web page at <URL:http://www.fvwm.org/>.
To unsubscribe from the list, send "unsubscribe fvwm-workers" in the
body of a message to [email-masked].
To report problems, send mail to [email-masked]

---

From: Olivier Chapuis <[email-masked]>
Date: Mon, 25 Nov 2002 23:09:43 +0100

> noted above it 2-bytes vs. 3-bytes).  Am I barking up the wrong tree :-) ?

Maybe not. fvwm makes some UTF-8 conversions (others are done by
iconv). I am not sure that these conversions are safe in certain
case. I should take a look.

Regards, Olivier

---

From: Nadim Shaikli <[email-masked]>
Date: Fri, 29 Nov 2002 16:20:12 -0800 (PST)

> case. I should take a look.

Olivier, 'FlocaleUtf8ToUnicodeStr2b' in libs/Flocale.c (lines 160-192)
looks rather suspect since it assumes all resultant unicode bytes will
occupy 2 bytes; am I reading that correctly ?  I tried man'ing the
various Xlib functions and got overwhelmed.  You might want to take a
peek at vim-6.1+ (vim.sf.net - note mbyte.c, function = utf_ptr2char).

Let me know if I can help in anyway.

Thanks.

 - Nadim

Yahoo! Mail Plus - Powerful. Affordable. Sign up now.

---

From: Olivier Chapuis <[email-masked]>
Date: Sat, 30 Nov 2002 09:54:02 +0100

> peek at vim-6.1+ (vim.sf.net - note mbyte.c, function = utf_ptr2char).

Yes, this function is suspect. It converts (char *)UTF-8 into
(XChar2b *)USC-2. It is used under this situation only when the
user uses an iso10646-1 font and does not use an UTF-8 locale.

Olivier

---

From: Olivier Chapuis <[email-masked]>
Date: Sun, 1 Dec 2002 01:04:09 +0100

On Sun, Nov 24, 2002 at 12:49:35PM -0800, Nadim Shaikli wrote:
> 4-bytes since that could be the max size ? 

No and it should not. UTF-8 is not UCS-4. Do I miss somthing?

> The difference in the example

Ok, I've found the problem. FBidiConvert assume that the "in" string
and the "out" string have the same length. This is the case until you
joint. With joint what may happen is:

  UTF-8 src string have length 34 (as char *)
  --> convert into UCS-4, the string has length 20 (as char32 *)
  --> apply bidi algo, the string has length 20 (as char32 *)
  --> apply joint algo, the string has length 20 (as char32 *)
  --> convert back to UTF-8, the string has length 40 (as char *)

    ----> core dump

the length of the returned string (visual_str) should be computed
after the joint algo is performed. BTW, fribidi should know the length
of the result of fribidi_unicode_to_charset before you call it, this
is a very very strange way to implement a converter!!

So we need the function:

  USC4UTF8Length(Char32 *str): return the length of an UCS-4 string
                               considered as an UTF-8 string

and also the following function will be useful (to allocate
logical_unicode_str at exact size):

  UTF8USC4Length(unsigned char *str): return the length of an utf-8 string
                                      considered as an USC-4 string

a simple workaround is to replace  

	visual_str = (char *)safemalloc((str_len + 1) * sizeof(char));

by

	visual_str = (char *)safemalloc((4*str_len + 1) * sizeof(char));

I do not have the time to write down the fix in the next 5 days (at
least two others thing todo before: IconMan and LocalePath).

Mikhael, Nadim ?

Regards, Olivier