Subject: Re: Shaping/Joining within fvwm

From: Nadim Shaikli <[email-masked]>
Date: Thu, 12 Dec 2002 09:53:48 -0800 (PST)

Mikhael/Olivier, I had sent an email with regards to the attached
patch to FBidiJoin.c to this list (as well as in private) so as
to include the the required changes/fixes to the shaping/joining
file.  I'm not sure if my email was received as I didn't get any
responses nor did I see the changes committed to CVS.

I'm attaching an archive which include a 'diff -ubB' (patch) of
the changes as well as the entire file itself (it'll include
cosmetic changes along with the fixes).

Apologies if this issue has already been taken care of.

Regards,

 - Nadim

__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

---

From: Mikhael Goikhman <[email-masked]>
Date: Wed, 18 Dec 2002 01:11:31 +0000

> Apologies if this issue has already been taken care of.

Ok, I reformatted the resulting file again to follow the same formatting
as in other our sources. But please never use in the future diff -bB
options, because the patched version will contain a mix of TAB indented
lines and 5 space indented lines at random (very strewn).

I tested the results and seems to work ok with unicode and iso8859-8.
But I can't see iso8859-6 in titles now. I have a lot of iso8859-6
fonts and I was able to see it in the past. I run:

  xterm -name Arabic -title "`env LANG=ar_JO.iso8859-6 date`"

Now, I can see joined Arabic with unicode font:

  Style Arabic Font StringEncoding=iso8859-6:*-arabeyes-*/iso10646-1

(so all these font prefixes and suffixes may be useful after all).
But if I use iso8859-6 font directly I see only reversed question signs:

  Style Arabic Font -lbi-naskhi-medium-r-normal--12-120-75-75-m-70-iso8859-6

This font does work if I use it with xterm -fn '*naskhi-medium*-12*'
and I see Arabic in it after: env LANG=ar_JO.iso8859-6 date

So something does not work. Can anyone see iso8859-6 text and font?

Regards,
Mikhael.
--
Visit the official FVWM web page at <URL:http://www.fvwm.org/>.
To unsubscribe from the list, send "unsubscribe fvwm-workers" in the
body of a message to [email-masked].
To report problems, send mail to [email-masked]

---

From: Nadim Shaikli <[email-masked]>
Date: Wed, 18 Dec 2002 15:11:44 -0800 (PST)

> lines and 5 space indented lines at random (very strewn).

I thought I'd removed all the tabs (untabify), but OK - will keep that
in mind.

> So something does not work. Can anyone see iso8859-6 text and font?

Keep in mind that iso8859-6 doesn't spell out proper Arabic visual
support since it doesn't include all the shaped/joined glyphs and
as such you should always revert to an 10646-1 font (which ought to
include unicode's Form-B glyphs).  Also in passing, the encoding
should be UTF-8.

So the following will work,

  Style * Font StringEncoding=UTF-8:*-arabeyes-*/iso10646-1
or
  Style * Font *-arabeyes-*/iso10646-1

(the UTF-8 is implied with a 10646-1 font indicator, per what
 Olivier had wisely implemented :-)

I would also think that the following ought to work (but I've
been told that it doesn't), shouldn't this work (Olivier) ?

  Style * Font StringEncoding=UTF-8:*-arabeyes-*

With regard to your 'naskhi' font - if it contains the required
Form-B glyphs (U+FE70 - U+FEFF), then the following ought to work,

  Style Arabic Font *-naskhi-medium-*-iso8859-6/iso10646-1

Regards,

 - Nadim (thanks for CC'ing)

---

From: Mikhael Goikhman <[email-masked]>
Date: Thu, 19 Dec 2002 00:59:17 +0000

>  Olivier had wisely implemented :-)

No, this will not work, because the string encoding is not utf-8.

>   Style Arabic Font *-naskhi-medium-*-iso8859-6/iso10646-1

If it is iso8859-6 font (not unicode), it can't be promoted to iso10646-1.

Nadim, you seem to imply that the only valid way to write Arabic is
unicode. But this is not correct. Here is a valid Arabic that is not
unicode: env LANG=ar_JO.iso8859-6 date

We supported all iso encodings. I see no valid reason to stop to support
iso8859-6. I think the problem is that once shaping is applied fribidi
(or is it iconv?) can't go back to iso8859-6 and uses question marks then,
so we should only apply shaping for unicode encoding of original strings.

Just tried this theory. Yes, if I disable shaping, I see iso8859-6 with:

  Exec env LANG=C xterm -name Arabic -title "`env LANG=ar_JO.iso8859-6 date`"
  Style Arabic Font '*-medium-*-12-*-iso8859-6'

You may try these fvwm commands yourself with and without shaping code.
I don't believe you prefer question marks to non-shaped Arabic. :)

Regards,
Mikhael.

---

From: Nadim Shaikli <[email-masked]>
Date: Wed, 18 Dec 2002 18:57:32 -0800 (PST)

> If it is iso8859-6 font (not unicode), it can't be promoted to iso10646-1.

iso8859-6 is a subset of iso10646-1 -- again, iso8859-6 alone is
simply not usable; its visually incorrect without shaping and one
is not able to shape sans Form-B glyphs (there are a plethora of
posts regarding this topic on the 'net - I can certainly send you
the links if you like so as not to go on a tangent on this forum).

> unicode: env LANG=ar_JO.iso8859-6 date

I don't have any Arabic locale on this machine - sorry.  But I do
indeed imply and state that Arabic should be used with UTF-8 and
nothing else (not even CP-1256 :-)  I'm actually curious to why
fvwm doesn't simply default to UTF-8 at all times ?

> so we should only apply shaping for unicode encoding of original strings.

I don't think its a question of support.  Fvwm is doing the right
thing.  I view this as "faulty/missing font" issue.  The font file
you were using simply doesn't have the _required_ Form-B glyphs and
thus Arabic can't be displayed properly.  Its like wanting to display
chinese without having the correct chinese glyphs and getting question
marks instead.

Out of curiosity, how do
'env LANG=ar_JO.iso8859-6 date' and
'env LANG=ar_JO.UTF-8     date' differ ?

iso8859-6 is an code-table representation (ie. an assignment of
integer numbers to characters) where-as UTF-8 is a representation
format (sequence of bytes).  So I'm not sure what you mean above
by "support all iso encodings".  In other words, my ability to do
StringEncoding=iso8859-6 and StringEncoding=UTF-8 seems a bit like
comparing apples to oranges.  I can understand the following encodings
UTF-8, USC-2, USC-4 and UTF-16, but don't quite understand a setting
akin to 'StringEncoding=iso8859-6' (unless fvwm is mapping names to
encodings which is what I thought it did - "convenience magic").

>   Style Arabic Font '*-medium-*-12-*-iso8859-6'

What you see (non-shaped glyphs) is text that most (if not all) Arab
users will dismiss as broken (yup, shaping is that important) and the
reason you see '??' (or other) when shaping is enabled is due to the
fact that you are missing those Form-B (U+FE70 - U+FEFF) glyphs in
your font file (include them and you are fine).

> I don't believe you prefer question marks to non-shaped Arabic. :)

Actually I would personally prefer questions marks as it will tell me I
have a setup problem instead of seeing non-shaped characters (which a user
would incorrectly assume a problem within fvwm).  FYI, this is what most
applications do.

In short, with Arabic UTF-8 and iso10646-1 fonts should be used (at all
times).  If iso10646-1 fonts are unavailable, your font file must include
Form-B glyphs.

Regards,

---

From: Mikhael Goikhman <[email-masked]>
Date: Thu, 19 Dec 2002 10:05:50 +0000

> the links if you like so as not to go on a tangent on this forum).

I know the theory well now. But you miss my point. I want everything to
work, including iso8859-6. You can't deny that there is such charset (and
encoding) and as you said it is not losing for Arabic. So if a user
requested iso8859-6 fonts (without non iso8859-6 characters, of course)
he don't want to see question marks for valid iso8859-6 characters.

Don't worry about this, I may later fix one-byte Arabic charsets myself.
Or not fix, if you are against supporting all existing Arabic fonts. :)

> fvwm doesn't simply default to UTF-8 at all times ?

If you use iso10646-1 fonts, it defaults to utf-8, is not it?

> chinese without having the correct chinese glyphs and getting question

What you say is that all existing CP1256 and iso8859-6 one byte fonts
should show question marks and never Arabic glyphs that they contain.
I don't know, it is not hard to fix this situation.

> 'env LANG=ar_JO.UTF-8     date' differ ?

The first returns regular one-byte Arabic and English characters, totally
40 bytes. The second returns the same text, but in utf-8, 56 bytes.
Only ascii characters (the first 127) are the same in both encodings.
Try: cat Arabic+English-utf8-encoded-file | iconv -f utf-8 -t iso8859-6

By the way, FVWM supports CP-1256 encoding without problems, as far as
I can see, when I set CP-1256 encoded title using:

  env LANG=ar_JO.iso8859-6 date | iconv -f iso8859-6 -t cp1256

I even see it correctly shaped (I think) if I use unicode font like:

  Style Arabic Font StringEncoding=CP1256:*-arabeyes-*/iso10646-1

The reason arabeyes is not recognized as iso10646-1 is bugs in this font.

> by "support all iso encodings".

Actually, here I meant more "support all charsets, i.e. fonts".

> In other words, my ability to do
> comparing apples to oranges.

No, they are not apples and oranges. iso8859-6 is both charset and
encoding, like any other one-byte charset, where one char is one byte.
There are iso8859-6 encoded texts (short) and utf-8 encoded texts (use
more bytes). It is possible to convert such text in one direction, but
not always in another. It is always possible to convert between iso8859-6
and cp1256 texts except for maybe some funky chars, I guess.

> I can understand the following encodings

If you have text stored in some encoding (iso8859-6 or cp1256), you may
find it useful to be able to convert it to something else, like utf-8
to use with unicode fonts. FVWM allows this using StringEncoding.

Out of curiosity, do you have Arabic text files? Are they all in one or
another unicode encoding? I read literature in several languages, but I
should yet encounter utf-8 text. If there are one byte encodings and there
is only one language (except for English) unicode is a waste of space. :)

Of course, to see one byte encoded text, you should replace "set
encoding=utf-8" in your .vimrc that I know you have.

Regards,
Mikhael.

---

From: Nadim Shaikli <[email-masked]>
Date: Thu, 19 Dec 2002 11:05:59 -0800 (PST)

> he don't want to see question marks for valid iso8859-6 characters.

I won't argue with that - but in all honesty I don't think anyone will
do that unless its for shear testing/curiosity.

> Or not fix, if you are against supporting all existing Arabic fonts. :)

I'm not against it and in theory you are correct - it should work as
well so that fvwm doesn't deviate from "normality".  I'm a bit biased
as a great deal of us on arabeyes.org have been advocating UTF-8/ISO10646
since its solves so many of our problems; do go ahead and fix it, it
shan't affect the proper shaping doing in utf-8.

> > > (or is it iconv?) can't go back to iso8859-6 and uses question marks
> > > then, so we should only apply shaping for unicode encoding of original
> > > strings.
> > 

And fixed it should be (I'm recanting my stance in favor of "theoretical
correctness" :-)

>   env LANG=ar_JO.iso8859-6 date | iconv -f iso8859-6 -t cp1256

Again theoretically - cool :-)  but I sure would not advocate anyone to
use M$'s encodings.

>   Style Arabic Font StringEncoding=CP1256:*-arabeyes-*/iso10646-1

So the above is saying take my strings (which are CP1256) and convert
them to UTF-8 (since iso10646-1 is noted) and use the noted font to
display them, right ?  If so it makes total sense - the man page should
note that 'StringEncoding' acts as the '-from' for 'iconv'.

> to use with unicode fonts. FVWM allows this using StringEncoding.

OK, cool - then what I note above ("acts as the '-from'") is correct.

> is only one language (except for English) unicode is a waste of space. :)

Everything that I have is in UTF-8 (diskspace/memory is cheap); I'd
rather be consistent and correct :-)  If you'd like some Arabic UTF-8
samples, let me know.

> encoding=utf-8" in your .vimrc that I know you have.

:-) and the various emacs settings,

 (setq locale-coding-system   'utf-8)
 (set-terminal-coding-system  'utf-8)
 (set-keyboard-coding-system  'utf-8)
 (set-selection-coding-system 'utf-8)
 (prefer-coding-system        'utf-8)

Thanks & Regards,

 - Nadim (Thanks for CC'ing)