man

[Joseph Rosevear]

Hello Slackers,

I've been using Slackware64 15.0 for a while now. Gee, has it been six
months?

Anyway, I noticed some differences when using man and less that have been
troubling me. I didn't post about them sooner, because I suspected that
my troubles were my own fault.

I solved the problem with less just today. (I'll post about man
separately.) I had been using:

export PAGER="less -R"
export METAMAIL_PAGER="less -R"
export LESSCHARSET=latin1

although I don't remember exactly why. I think one or more of these
lines were for the purpose of causing special characters to display
correctly in less. These lines were in a script that I use everyday, but
haven't thought much about for a long time. The script, incidentally
sets some environment variables. I don't think it has any other
relevance to this post. (Let's not go there now. Dragging my script
into this discussion might get messy.)

I took these lines out and now less seems to work OK.

Can anyone explain what happened? I don't fully understand it, and it is
just complex enough to leave me bamboozled.

The problem with less was that some special characters weren't displaying
correctly in Slackware64 15.0 (using my old script). This problem didn't
happen in Slackware 14.2 (using my my old script).

Here are a few lines (made by lsblk) and displayed in less in Slackware64
15.0 using my old script:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 465.8G 0 disk
�<94><9C>�<94><80>sda1 8:1 0 21.4G 0 part
�<94><9C>�<94><80>sda2 8:2 0 953M 0 part
�<94><9C>�<94><80>sda3 8:3 0 11.2G 0 part

Note that the above lines are displayed differently by xterm--boxes
made of dots instead of the "?" symbols that my newsreader (pan)
displays. I'm curious about the boxes made of dots, because that seems
to be new with Slackware 15.0, but that is not the subject of this post.

And here they are in Slackware64 15.0 using my new script:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 465.8G 0 disk
├─sda1 8:1 0 21.4G 0 part
├─sda2 8:2 0 953M 0 part
├─sda3 8:3 0 11.2G 0 part

So what's going on? I suspect that Slackware64 15.0 has something
different in it that caused my old script to have trouble. It seems I
found a solution, but it is a little unsatisfying not understanding what
happened.

-Joe

[Aragorn]

Post by Joseph Rosevear
Anyway, I noticed some differences when using man and less that have
been troubling me. I didn't post about them sooner, because I
suspected that my troubles were my own fault.
I solved the problem with less just today. (I'll post about man
export PAGER="less -R"
export METAMAIL_PAGER="less -R"
export LESSCHARSET=latin1
although I don't remember exactly why. I think one or more of these
lines were for the purpose of causing special characters to display
correctly in less. [...]
I took these lines out and now less seems to work OK.
Can anyone explain what happened? I don't fully understand it, and
it is just complex enough to leave me bamboozled.

I'm not running Slackware, but given how much time has elapsed since
the previous release of Slackware — which is the one I suspect you
would have created your script for — and the one you're running now,
my money would be on Slackware having in the meantime adopted UTF-8 as
the default character encoding [*], whereas the previous release would
still have been using the US-centric ISO-8859-1 character encoding.


[*] As just about every other distribution already did almost two
decades ago. :p

[Joseph Rosevear]

[snip]
Can anyone explain what happened? I don't fully understand it, and it
is just complex enough to leave me bamboozled.

I'm not running Slackware, but given how much time has elapsed since the
previous release of Slackware — which is the one I suspect you would
have created your script for — and the one you're running now,
my money would be on Slackware having in the meantime adopted UTF-8 as
the default character encoding [*], whereas the previous release would
still have been using the US-centric ISO-8859-1 character encoding.
[*] As just about every other distribution already did almost two
decades ago. :p

Well, that's it, then. Er, at least it points me in the right direction.

I found this useful article:

https://stackoverflow.com/questions/7048745/what-is-the-difference-
between-utf-8-and-iso-8859-1

Ha. We went from 128 characters (which personally I think is enough) to
roughly 2 billion characters. It is no wonder that I'm alarmed by the
result. (Little boxes made of dots! I suppose there are at least a few
other obscure characters that I haven't seen yet.) I genuinely think
this is the result of poor judgement and too much coffee.

Still I would welcome explanations that shed light on what happened
regarding my script.

-Joe

[Rich]

Post by Joseph Rosevear

[snip]
Can anyone explain what happened? I don't fully understand it, and it
is just complex enough to leave me bamboozled.

I'm not running Slackware, but given how much time has elapsed since the
previous release of Slackware ? which is the one I suspect you would
have created your script for ? and the one you're running now,
my money would be on Slackware having in the meantime adopted UTF-8 as
the default character encoding [*], whereas the previous release would
still have been using the US-centric ISO-8859-1 character encoding.
[*] As just about every other distribution already did almost two
decades ago. :p

Still I would welcome explanations that shed light on what happened
regarding my script.

I agree with Aragorn, this looks like what happens when UTF-8 encoded
text is viewed using a single byte encoding.

As to what went wrong, I'd guess this line to be the cause:

export LESSCHARSET=latin1

That would likely end up resulting in less using the ISO-8859-1
character set, even if the rest of the environment is utf-8, and if so,
the results would likely be just what you saw.

Try experimenting with and without that setting and see if it is the
cause.

[Henrik Carlqvist]

Post by Aragorn
my money would be on Slackware having in the meantime adopted UTF-8 as
the default character encoding

I would also guess that explains the differences Joseph sees between
Slackware 14.2 and Slackware 15.0.

A quick fix for the scripts used by Joseph might be to revert to
iso-8859-1 in at least those scripts:

export LANG=en_US.iso-8859-1

However, the output charset selected by the LANG variable must also match
the charset that the terminal expects. Some terminal programs like
konsole use a menu setting like "View, Set Encoding" for this. Other
terminal programs like xterm have a special script uxterm to call xterm
with -u8 to select utf-8.

regards Henrik

[Henrik Carlqvist]

Post by Henrik Carlqvist
A quick fix for the scripts used by Joseph might be to revert to
export LANG=en_US.iso-8859-1
However, the output charset selected by the LANG variable must also
match the charset that the terminal expects. Some terminal programs like
konsole use a menu setting like "View, Set Encoding" for this. Other
terminal programs like xterm have a special script uxterm to call xterm
with -u8 to select utf-8.

I just tried on Slackware 15.0 and to get programs correctly outputting
chars with iso-8859-1 charset in xterm it was neccesary to start xterm
after setting LANG, I could not find any other way for xterm in Slackware
15.0 to output those chars correctly.

Example:

(starting xterm)
export LANG=en_US.iso-8859-1
xterm

(in the xterm)
export LANG=sv_SE.iso-8859-1
man chsh

regards Henrik

[Joseph Rosevear]

Post by Joseph Rosevear
Hello Slackers,
I've been using Slackware64 15.0 for a while now. Gee, has it been six
months?
Anyway, I noticed some differences when using man and less that have
been troubling me. I didn't post about them sooner, because I suspected
that my troubles were my own fault.
[snip]
-Joe

Thanks for your replies, everyone. I would love to follow up, but I
have, rather suddenly, some personal matters to attend to. Maybe in a
few weeks I'll return to this.

-Joe