Performance issues with 0.25 NFS

Help with using MiniMyth (except Booting)

Moderator: Pablo

Post Reply
bcromwell
Contributor
Posts: 64
Joined: Thu Nov 22, 2007 5:08 am

Performance issues with 0.25 NFS

Post by bcromwell » Wed Jun 06, 2012 1:17 am

I am having major performance issues with minimyth-0.25.0-80 and minimyth-0.25.0-81b5. Booting is slow, Gui Screen to Screen is slow, Recordings screen is hardly useable on two separate front ends (One intel duo, the other a foxconn ion) . After much network troubleshooting I think I have narrowed this down to an nfs issue. I am seeing about 1800 NFS getattr queries per second (90-95% overall) when just sitting at the main screen. This nfs server was working fine with minimyth-0.23 and minimyth-0.24 but I never looked at the nfsstats to see if it had the same behavior. Anyone have any suggestions where to start? This would have to be a nfsroot issue as I am not mounting any other nfs shares in the frontend and this nfs server is only used by/for minimyth.


19:58:41 up 5 min, load average: 0.77, 0.77, 0.72
root@mythLR:~ #



Centos 5 nfs server (2.6.18-194.el5)

Server nfs v2:
null getattr setattr root lookup readlink
1 0% 552058 92% 0 0% 0 0% 16027 2% 77 0%
read wrcache write create remove rename
28995 4% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 665 0% 3 0%

Sample from tcpdump:
20:16:04.256969 IP 10.201.30.3.2049 > 10.201.30.125.893187618: reply ok 96 getattr DIR 40755 ids 0/0 sz 4096
20:16:04.256974 IP 10.201.30.3.2049 > 10.201.30.125.893187618: reply ok 96 getattr DIR 40755 ids 0/0 sz 4096
20:16:04.257041 IP 10.201.30.125.909964834 > 10.201.30.3.2049: 120 getattr fh Unknown/0100000100FD000001004100B68B430093A99233000000000000000000000000
20:16:04.257123 IP 10.201.30.3.2049 > 10.201.30.125.909964834: reply ok 96 getattr DIR 40755 ids 0/0 sz 4096
20:16:04.257127 IP 10.201.30.3.2049 > 10.201.30.125.909964834: reply ok 96 getattr DIR 40755 ids 0/0 sz 4096
20:16:04.257489 IP 10.201.30.125.926742050 > 10.201.30.3.2049: 120 getattr fh Unknown/0100000100FD000001004100CC1E0800F2B09233000000000000000000000000
20:16:04.257565 IP 10.201.30.3.2049 > 10.201.30.125.926742050: reply ok 96 getattr DIR 40755 ids 0/0 sz 4096
20:16:04.257569 IP 10.201.30.3.2049 > 10.201.30.125.926742050: reply ok 96 getattr DIR 40755 ids 0/0 sz 4096
20:16:04.257883 IP 10.201.30.125.943519266 > 10.201.30.3.2049: 120 getattr fh Unknown/0100000100FD00000100410047800900E7B89233000000000000000000000000
20:16:04.257968 IP 10.201.30.3.2049 > 10.201.30.125.943519266: reply ok 96 getattr DIR 40755 ids 0/0 sz 4096
20:16:04.257973 IP 10.201.30.3.2049 > 10.201.30.125.943519266: reply ok 96 getattr DIR 40755 ids 0/0 sz 4096
20:16:04.258332 IP 10.201.30.125.960296482 > 10.201.30.3.2049: 120 getattr fh Unknown/0100000100FD0000010041008381090023BA9233000000000000000000000000
20:16:04.258427 IP 10.201.30.3.2049 > 10.201.30.125.960296482: reply ok 96 getattr DIR 40755 ids 0/0 sz 4096
20:16:04.258431 IP 10.201.30.3.2049 > 10.201.30.125.960296482: reply ok 96 getattr DIR 40755 ids 0/0 sz 4096
20:16:04.258731 IP 10.201.30.125.977073698 > 10.201.30.3.2049: 120 getattr fh Unknown/0100000100FD0000010041008881090028BA9233000000000000000000000000
20:16:04.258813 IP 10.201.30.3.2049 > 10.201.30.125.977073698: reply ok 96 getattr REG 100644 ids 0/0 sz 3519
20:16:04.258817 IP 10.201.30.3.2049 > 10.201.30.125.977073698: reply ok 96 getattr REG 100644 ids 0/0 sz 3519

bcromwell
Contributor
Posts: 64
Joined: Thu Nov 22, 2007 5:08 am

Re: Performance issues with 0.25 NFS

Post by bcromwell » Wed Jun 06, 2012 5:17 am

Definitely nfs related, changed over to ram rootfs and everything works perfect. Any suggestions on nfs issue would be greatly appreciated

m_theredhead
Contributor
Posts: 170
Joined: Thu Apr 27, 2006 7:00 pm
Location: Arizona

Re: Performance issues with 0.25 NFS

Post by m_theredhead » Thu Jun 07, 2012 12:20 am

I have seen several discussions of NFS problems on the list recently on the older shuttle platforms, etc without resolution.

Personally, I have given up on NFS for / and just use the ram kernel/ramdisk. This seems to also survive reboots of my master backend ( and NFS server ) much better.

Michael

bcromwell
Contributor
Posts: 64
Joined: Thu Nov 22, 2007 5:08 am

Re: Performance issues with 0.25 NFS

Post by bcromwell » Fri Jun 08, 2012 12:03 am

Thanks for the heads up. I am the type of person that wants to hunt down the issue even though ram fs works fine. Has to be something triggering all of those getattr's. :D

kdewaal
Contributor
Posts: 79
Joined: Mon Sep 12, 2005 8:06 am
Location: Netherlands

Re: Performance issues with 0.25 NFS

Post by kdewaal » Sat Jun 09, 2012 1:18 pm

On my NFS-based Minimyth system, performance has improved from "not usable" to "slow but acceptable" by defining the TZ environment variable. This was suggested some time ago on this forum.
On my NFS server I have edited the following file:

/var/minimyth/nfs/minimyth-0.24.2-80b1/rootfs-ro/etc/conf.d/core

In this file I have added my timezone definition:

TZ="Europe/Amsterdam"
export TZ

I have experimented a bit with different NFS versions but this did not lead to anything. I have also tried to find which task/service/program was responsible for all the NFS getattrs but have not been able to find it. I am most interested in the solution for this!
MiniMyth on Gigabyte GA-73PVM-S2H / Intel E8500 / GT520

JoeHenley
Contributor
Posts: 185
Joined: Sun Aug 06, 2006 4:12 pm

Re: Performance issues with 0.25 NFS

Post by JoeHenley » Sat Jun 09, 2012 3:16 pm

Hi kdewaal,

Not that it will affect the NFS slowness you see, but it might make things easier in the future.... The /etc/conf.d/core file is replaced when you upgrade to a new version, so any mods you make to it will be lost at that time.

Pablo provided an easier way to accomplish your objective by letting you set the dhcp time zone via a kernel command line parameter, or via the minimyth.conf file. Both are explained in the documents section on this site.

Joe

JoeHenley
Contributor
Posts: 185
Joined: Sun Aug 06, 2006 4:12 pm

Re: Performance issues with 0.25 NFS

Post by JoeHenley » Wed Sep 12, 2012 7:35 pm

Hi kdewaal,

I'm having the same getattr problem as you, bcromwell, and m_theredhead.
I have experimented a bit with different NFS versions but this did not lead to anything. I have also tried to find which task/service/program was responsible for all the NFS getattrs but have not been able to find it. I am most interested in the solution for this!
From what I can tell, on my system, the problem is coming from the client (minimyth 0.25.2-81). The nfs on the client appears to using a NFS ver 2 call to the server; which is not recognised by my server. I can't tell if the server (running FC16) no longer supports NFS ver 2, or if the call from the client has errors in it.

I have two different remote clients, one with Atom/Nvidia (CPU/GPU), the other with Via/Via. They both exhibit the problem. I have a third client, on the same machine as the server, with Pent4/Matrox, which does not exhibit the problem (nor use NFS to communicate with the server).

Google shows this problem back in the late 1990's and early 2000's, but not much recently. So the fixes given probably aren't too relevant any more.

Hopefully someone will see this and have a brain-storm about a solution.

Joe Henley

JeanePittman
New Member
Posts: 2
Joined: Thu Sep 20, 2012 7:13 am
Location: http://www.audvdsbuy.com/
Contact:

Re: Performance issues with 0.25 NFS

Post by JeanePittman » Sat Sep 22, 2012 3:28 am

I am the type of person that wants to hunt down the issue even though ram fs works fine.
Huge Thing For Movies --- The Office Season 8 DVD,Creat excellence for your life

JoeHenley
Contributor
Posts: 185
Joined: Sun Aug 06, 2006 4:12 pm

Re: Performance issues with 0.25 NFS

Post by JoeHenley » Mon Sep 24, 2012 3:58 am

Well, I've joined the rest of the crowd. I've given up on the NFS version and gone over to the RAM one. Everything I try comes back to a problem in 0.25.2 with NFS. Oh, well ..... at least we do have a choice.

Joe Henley

Pablo
Site Admin
Posts: 4182
Joined: Tue Dec 14, 2004 2:13 am
Location: La Jolla
Contact:

Re: Performance issues with 0.25 NFS

Post by Pablo » Thu Nov 29, 2012 2:16 am

Have you tried adding "nfsvers=3" to the nfsroot argument on the boot line?
MiniMyth running on an Acer ApireRevo 3610 and a Zotac ZBOX-ID80-U. Find out more at my MythTV page.

JoeHenley
Contributor
Posts: 185
Joined: Sun Aug 06, 2006 4:12 pm

Re: Performance issues with 0.25 NFS

Post by JoeHenley » Thu Nov 29, 2012 2:57 am

Pablo,

I don't know about the other folks, but I don't think I have tried that. IIRC, I still have the NFS version "stuff" around. If so I'll give it a try and let you know.

Joe Henley

PS: Great seeing you back!

Post Reply