character set problem in 0.22

Help with using MiniMyth (except Booting)

Moderator: Pablo

gusse02
Member
Posts: 17
Joined: Wed Jan 09, 2008 11:26 pm
Location: Stockholm

character set problem in 0.22

Post by gusse02 » Sun Jan 10, 2010 9:50 pm

First, thanks Pablo for the great job with minimyth. It is really awesome!

After upgrading from 0.21 (72) to 0.22 I experience a problem with character sets. Specifically in mythvideo when scanning for changes.

In my video library I have some files with national characters in the names, such as å, ä and ö. These files were no problems before. After upgrading I could not play these files. I then removed all rows from the videometadata table in the backend (debian) and did a rescan in the minimyth client. The files then showed up with incorrect character encoding, minimyth used latin1 (iso-8859-1) instead of utf8. But the files now were playable but the names looks funny, (strange characters).

Using mythbrowser (on the backend) works fine when scanning video files, shows up with utf8. This can also be verified by selecting in the database with the mysql client and using "set names utf8".

Telneting to my minimyth client and doing "ls" on the nfs-mount where the video files live shows them with correct characters.

Browsing to minimyth client filesystem (http://mythbox:8080/) shows the files with incorrect characters (iso-8859-1).

I have also tried a Ubuntu frontend. When scanning video files it is OK.

So, there seems to be some problem here with minimyth.

Anyone else seen this problem?

Regards,
Magnus

stueble
Member
Posts: 22
Joined: Mon Jan 11, 2010 2:00 pm

Re: character set problem in 0.22

Post by stueble » Mon Jan 11, 2010 2:19 pm

Hi,

I have a similar problem with the same (latest stable) version: I also have on my myth backend (mythbuntu) files with german umlauts (öäü). The minimyth frontend (mythvideo and mythmusic) cannot play these files. When I switch to the console, I can _see_ the files with "ls" corrently, but I cannot, e.g., switch into a directory with umlauts. If I enter the "tab" key to complete the name of a directory with umlauts, the input line simply disappears.

Best regards,
Chris

gusse02
Member
Posts: 17
Joined: Wed Jan 09, 2008 11:26 pm
Location: Stockholm

Re: character set problem in 0.22

Post by gusse02 » Mon Jan 11, 2010 2:37 pm

OK, then we are at least 2 with this problem, :)

I have tried to do some more investigation. On my backend I enabled mysql logging. Then I had a look at what happens when I scan the video library from the minimyth frontend. It does a "SET NAMES utf8" which is good! Then the insert statement (into videometadata) shows those erroneous characters. I suspect that it gets them when querying the filesystem. But for me it looks correct when I list my files from the telnet session in the frontend, strange!

I will try to come up with more info when I know how to investigate.

bam
Contributor
Posts: 65
Joined: Sat May 23, 2009 10:02 pm
Location: Russia, Moscow

Re: character set problem in 0.22

Post by bam » Mon Jan 11, 2010 4:19 pm

I have to set LANG variable when I start mythbackend on my desktop system from init script, like this:

Code: Select all

LANG=ru_RU.KOI8-R HOME=/root /usr/bin/mythbackend -l /var/log/mythbackend.log -p /var/run/mythbackend.pid -v schedule,extra -d
Maybe this helps..

Pablo
Site Admin
Posts: 4182
Joined: Tue Dec 14, 2004 2:13 am
Location: La Jolla
Contact:

Re: character set problem in 0.22

Post by Pablo » Tue Jan 12, 2010 4:17 am

There are at least two problems.

First, the reason UTF-8 was not being used when viewing the files using MiniMyth's web interface, is that lighttpd was not configured for any encoding when serving up directories. As a result, it was using the default. I have added the configuration so that it uses UTF-8.

Second, Qt4 was disabling Unicode because it would test for UTF-16 support, which was not included (at least, that is what it appears to being doing from looking at the Qt4 source code file src/corelib/codecs/qiconvcodec.cpp). MiniMyth did not include qlibc's UTF-16.so module (/usr/lib/gconv/UTF-16.so). I have added it. Given that MythTV 0.21 uses Qt3 whereas MythTV 0.22 uses Qt4, this might be why MiniMyth 0.21-72 worked by MiniMyth 0.22-72 did not.

In addition, while debugging this problem, I found problems with how MM_FONT_FILE_TTF_DELETE was being handled and fixed them.

These changes have been committed and will be in the next build (73b3). However, I do not know whether or not these are sufficient.
MiniMyth running on an Acer ApireRevo 3610 and a Zotac ZBOX-ID80-U. Find out more at my MythTV page.

gusse02
Member
Posts: 17
Joined: Wed Jan 09, 2008 11:26 pm
Location: Stockholm

Re: character set problem in 0.22

Post by gusse02 » Tue Jan 12, 2010 5:14 pm

bam: my LANG is properly set both on frontend and backend. But thanks anyway.

Pablo: I will try 73b3 as soon as possible. Thank you so much!

Regards,
Magnus

gusse02
Member
Posts: 17
Joined: Wed Jan 09, 2008 11:26 pm
Location: Stockholm

Re: character set problem in 0.22

Post by gusse02 » Tue Jan 12, 2010 8:11 pm

Unfortunately I cannot run the test build since I need a legacy nVidia driver. :(

Pablo
Site Admin
Posts: 4182
Joined: Tue Dec 14, 2004 2:13 am
Location: La Jolla
Contact:

Re: character set problem in 0.22

Post by Pablo » Wed Jan 13, 2010 1:12 am

gusse02 wrote:Unfortunately I cannot run the test build since I need a legacy nVidia driver. :(
You might be able to get it to run enough to test by using either MM_X_DRIVER='nouveau' or MM_X_DRIVER='nv'. If X starts but does not display, then you still might be able to check the fonts through VNC.

If that does not work, let me know which legacy driver series (96 or 173) and I can upload an additional test build.
MiniMyth running on an Acer ApireRevo 3610 and a Zotac ZBOX-ID80-U. Find out more at my MythTV page.

gusse02
Member
Posts: 17
Joined: Wed Jan 09, 2008 11:26 pm
Location: Stockholm

Re: character set problem in 0.22

Post by gusse02 » Wed Jan 13, 2010 10:40 pm

Thanks for the tip, I got it running.
Regarding character set, the behavior changed. I deleted all rows from videometadata and did a new scan from minimyth. Now only the films without non-latin characters show up, i.e. all films that were presented with funny characters before now don't even show up in list.

I have logging going from minimyth frontend to backend server syslog. Looking in server /var/log/syslog I can see lines like this:
"Jan 13 22:37:43 mythbox mythfrontend: 2010-01-13 22:37:43.885 Adding : : /home/mythtv/MythVideo/Bamse/Den Lilla Åsnan.avi"
In the log it looks OK but not in the database.

What else can I try?

Regards,
Magnus

stueble
Member
Posts: 22
Joined: Mon Jan 11, 2010 2:00 pm

Re: character set problem in 0.22

Post by stueble » Sat Jan 16, 2010 11:12 pm

Hi,

the current test version 73b3 also does not help. Still, 'ls' shows the NFS-mounted filenames correctly, but I cannot enter them on the console due to the wrong keyboard layout. The auto-name-extension on "Tab" does also not work, it completely deletes the line when using with a name including umlauts. However, if I play, e.g., the file "Beispielü.mp3" with mplayer as "mplayer Beispiel*", it works. But the logs show that mythfronend tries to open "Beispiel.mp3" which, of course, fails. Interestingly, the mythfrontend GUI shows all names correctly. I tried to solve that problem the whole day without success. Any ideas where I can start?

Is somebody successfully using the latest testing or stable release together with non-standard characters?

Regards,
Chris

stueble
Member
Posts: 22
Joined: Mon Jan 11, 2010 2:00 pm

Re: character set problem in 0.22

Post by stueble » Sun Jan 24, 2010 10:09 pm

Hi all, hi Pablo,

any news or success stories regarding 0.22 or 73b3 so far? Minimyth is running nearly perfect on two machines here (EPIA SP8000, Zotac IONITX-A-C), but the problem with the charset prevents its real use. In another discussion (viewtopic.php?=&p=5293) I found information about an nls_utf8 kernel module that should be loaded. I notices that this module will not be loaded on my two installations. Can this be the problem? However, loading the module after mythfrontend started does not solve the problem...

I also changed the encoding of the mythtv database to utf8 and back to latin1. But this only made things worse, since then mythfrontend showed wrong names.

Best regards,
Chris

Pablo
Site Admin
Posts: 4182
Joined: Tue Dec 14, 2004 2:13 am
Location: La Jolla
Contact:

Re: character set problem in 0.22

Post by Pablo » Mon Jan 25, 2010 7:15 am

Yes, I believe I have found the problem, but it has taken a little time to figure out how to fix it. MiniMyth needs the /usr/lib/locale/*.UTF-8 directory corresponding to the LANG=*.UTF-8 environment variable. Otherwise, LC_CTYPE and LC_ALL cannot be set and Qt will assume that locale is something other than *.UTF-8.

Generating the /usr/local/lib/*.UTF-8 files turned out to be somewhat tricky as it requires running a the localedef binary that it part of glibc. However, because we need to use the host computer's glibc to run yet we need to run a localedef binary with the MIniMyth's glibc version, I needed to figure out how to get the glibc package to build a statically linked version of localedef.

Anyway, it works for me at this point. So, after a complete rebuild, I will upload a new build (73b4).

Do you change LANG to something other than en_US.UTF-8? The reason that I ask is that I am only including /usr/lib/locale/en_US.UTF-8 at this point. Including all of them is prohibitive. However, I can include others as needed.
MiniMyth running on an Acer ApireRevo 3610 and a Zotac ZBOX-ID80-U. Find out more at my MythTV page.

norpan
Member
Posts: 15
Joined: Thu May 07, 2009 5:47 am
Location: Harnosand, Sweden

Re: character set problem in 0.22

Post by norpan » Thu Jan 28, 2010 9:47 pm

Iv'e got some problems with Swedish characters as well (I don't really know when it stopped working), and tonight I tested 73b4 and it seems to work again.
As always, great work Pablo!

Regards,
Niclas
-Homer: "There's three ways to do things: the right way, the wrong way, and the Max Power way."
-Bart: "Isnt that the wrong way?"
-Homer: "Yeah, but faster!"

stueble
Member
Posts: 22
Joined: Mon Jan 11, 2010 2:00 pm

Re: character set problem in 0.22

Post by stueble » Sat Jan 30, 2010 6:47 pm

Hi Pablo, hi all,

the new beta also solves my problems with German umlauts in mythmusic and mythvideo. Many thanks! One additional note for others with similar problems: I had to delete some DB entries explicitly using the mysql client. Otherwise, mysql could not update the music file list properly. But maybe the reason for this was that i converted the database two times. I am now using minimyth on two clients.

Best regards,
Chris

gusse02
Member
Posts: 17
Joined: Wed Jan 09, 2008 11:26 pm
Location: Stockholm

Re: character set problem in 0.22

Post by gusse02 » Sun Jan 31, 2010 7:48 pm

Great! Just tested 0.22.0-73b4. This is much better, problem solved!
Thank you so much Pablo!

Post Reply