Channel editing and Schedules Direct

[PhilB]

Thanks Mike.
I have symptoms with spoofing that there are no channels listed but no crashes either on a 4GB RAM system. Is that what you are getting with your live Mythtv backend?
Will investigate, but it looks subtle so please be patient. May be a 0.29 issue.

What it does tell me is that we also need a --verbose option!

Phil

[MikeB2013]

@PhilB

Hi Phil,

Yes it is an interesting issue. I suspect (but could be totally wrong) that it something to do with the way tk and X are interacting with respect to allocation of screen drawing.memory, maybe a lot of refresh/redraw calls from tk and X not reusing memory so it eventually runs out.

All the mythtv systems (3) are functional with Live TV working and showing channels using either local mythfrontends or remote Kodi frontend (uses pvr.mythtv addon). They are running 0.29pre built from source (with my modifications for a VBOX network tuner) and running under mythbuntu 16.04.

The system with the problem has 8GB ram, the other two have 4GB ram each. The only real difference is the number of channels and videosources, the one with the problem having the largest number of channels. For the working systems I can use chedit2.pl either locally or over ssh (using --backend) from a laptop with no problems.

In case it helps I have attached channels.export files from two other systems, the one ending ip20 is my main "production" system which is in constant use. The other ending ip29 is another test system. In both cases chedit2.pl is working great. If you need more information on the hardware/software on each of the systems let me know.

Mike

[MikeB2013]

@PhilB

Hi Phil,

Just done a test on the problem system.
I deleted one of the capture cards and associated videosource, which reduced the number of channels from 1429 to 987 and chedit2.pl is now working. So it is definitely related to number of channels and possibly videosources.

Mike

[PhilB]

Hi Mike,
thanks for the diagnostics - pinned it down exactly.

If I spoof your data on a NUC (i3, 4GB) I get a failure in exactly the same area of code. It's in sub refresh which generates a big string with the channel data to be displayed and which it then asks the tk routine to put in the pane. My failure though is not a crash but just an empty pane when trying to put the 1429 lines with total size 140042 bytes there.

Reducing the byte count (with custom display) to about 35K makes no difference - tk (on my system) seems though to have a line limit - if I limit to 743 lines it works, 744 does not. Alternatively, filtering with BBC reduces the line count and it displays.

So.... where do we go next?
I can start asking in tk forum.
I can have a major rethink (multiple panes etc - ugh!).
I can limit operation to a single source (your biggest is source 4 with 464 channels). That should not blow up but you could simulate that with a change (the only change to a fresh copy of chedit2):
Line 2254 reads
for my $source (sort keys %sources){
change to:
for my $source (4){
that should just scan source 4 and work (but of course, we need something a little more elegant than that!).

A lot of head scratching ahead of me/us for an elegant solution. Views appreciated.

Phil

[PhilB]

Sorry -didn't see your post. So the line limit is possibly imposed by constraints outside the tk environment.
Phil

[MikeB2013]

@PhilB,

Phil,

I suspect "line limit", is more likely to be a combination of various string lengths e.g. channel name, callsign, xmltvid etc number of rows resulting in "insufficient resources" somewhere.

Your result of 743 lines working and 744 not is interesting, my production system has 789 rows in channel table and I have not seen a problem.

I have also done a test on a low memory system (Lbuntu 16.04 x86_64) 2GB ram works fine with 787 rows, fails at 1429.

Just for fun I tried chedit2.pl on a Raspberry Pi3 (1GB ram arm based system) running raspbian jessie. it worked fine for 1429 rows!!

I am totally confused now.


Mike

[PhilB]

Hi Mike,
Interesting points on the failure graph there. The 744 was failing even if I used custom columns to just show channel ID. There may well be a total size limit but in these tests at least it suggests line count.
Will try on other systems when I get a chance but the diary for the next few week is looking fraught.

It may be that we have to look at alternative mechanisms like up, down, page up down home end button presses but first step will be questions in the tk forum.
Thanks again for your testing.
Phil

[PhilB]

Mike,
would you do one final check so we know exactly where it's crashing please?

With a clean chedit2.pl and no other changes; line 1771 is where I think it is crashing (or in my case is failing to display the data).

Code: Select all

 $pane2 -> configure(-text => $showdata, -anchor =>"nw", -justify=>'left');

This should put the data in the big scrollable label. Would you put logging before and after it please:

Code: Select all

mylog(1,'pre-configure');
$pane2 -> configure(-text => $showdata, -anchor =>"nw", -justify=>'left');
mylog(1,'post-configure');

mylog puts text to the log and (if first param is 1) also to the terminal.
Also on line 333

Code: Select all

MainLoop;

This tells tk to go to sleep and await user input. Change to:

Code: Select all

 mylog(1,'pre-mainloop');   MainLoop;

then run it with data (either a real backend or spoofing) which triggers the X crash.
Which logging do you see to the terminal? I expect to see the 'pre-configure' only.

Many thanks. Will check it when I have time.

Regards
Phil

[MikeB2013]

Phil,

I see both :

Code: Select all

mike@myth-dev:~/build/Channel-Editor$ ./chedit2.pl
Opening /home/mike/.mythtv/config.xml
Found host address localhost
pre-configure
post-configure
X Error of failed request:  BadAlloc (insufficient resources for operation)
  Major opcode of failed request:  53 (X_CreatePixmap)
  Serial number of failed request:  326
  Current serial number in output stream:  328

Mike

[MikeB2013]

Phil,

Some more data points:

I ran up Debian Jessie 8.7 in a VM on my laptop and chedit2.pl worked perfectly displaying all 1429 rows using --backend.

I also tried it on latest Ubuntu 17.04 (dev branch) not in VM (so real hardware), failed with 1429 rows, worked to another backend with 789 rows, Did same test but in a VM on my laptop, it worked in a VM with 1429 rows, so go figure!!

Will try later Debian Jessie 8.7 on real hardware and let you know.

Mike

[MikeB2013]

Phil,

Running Debian Jessie 8.7 on real hardware chedit2.pl works using --backend, all 1429 rows are displayed.

Mike

[PhilB]

Hi Mike,

I have not been able to find an easy solution to this by googling. This is the closest I found: https://ubuntuforums.org/showthread.php?t=1762103

Here, my Intel graphics system (i3 NUC) does not crash but simply fails to show the data. My Asus M3N78-EM mobo (AMD CPU + on-board NVidia 8300 graphics) reproduces your X error message which is the good news!
Two different sets of symptoms. I'm inclined to think that chasing graphics drivers / xorg stuff / tk will be a fruitless exercise. Also that the problem is not a pure perl limit.
I'm going to investigate a rewrite of my code to eliminate the scroll bar on the right and instead do own scrolling with up/down/page up /page down. That way I'll be able to limit the stuff I throw at tk to about 40 lines rather than thousands and hopefully that will be stable. I'll also have to artificially generate really big channel line-ups for robust testing.

That's all assuming that I can master keyboard events and resizing.

Phil

[MikeB2013]

Hi Phil,

I think you are correct, chasing drivers/Xorg is likely fruitless - they are always changing.

If there is anything I can do to help with testing of whatever you come with, let me know.

As I have a workaround, i.e. using a VM, or my Raspberry Pi3 via ssh it is not a major issue for me.

Mike

[PhilB]

Thanks Mike - appreciate it - I'll take you up on that offer when I have something to beta test.
Phil

[PhilB]

Hi Mike,
(or anyone else interested)

I've put new code up here: http://pastebin.com/QS7QRpHT and I'd be grateful for any beta testing that anyone can do. Version 2.08(tki02) 2 March 2017.

To recap: I was putting so much data into a scrollable label that it was blowing limits. It seemed to be number of lines (at lease with Intel) rather than number of characters. To be fair, a big ask of tk!

I've now abolished the scroll bars at the side of the display and instead you need to use keyboard up/down, page up/page down, home/end. Shift puts a turbo boost on these and jumps 10X further. That means that the volume of data thrown at tk is much more limited and it seems ok, even if I artificially expand the data to over 9000 channels.

Interception of keystrokes and scrolling was fairly intuitive but that was not true of resizing, hence the delays in giving a solution. It sounded so very easy - intercept a resize event,
ask the new height of the label, work out how many lines are needed to be put in the label, then re-write the lines. Unfortunately, it didn't quite work out like that and a few compromises were necessary.

a) tk gives you hundreds of resize events as the window is being resized and I could not find a nice solution with googling (or in desperation buying the book!). That was ugly (not the book!) so to avoid epileptic incidents I run a 300mS timer and assume resizing is finished if it is inactive between consecutive 300mS ticks. It gives a short delay after resizing stops but that is the compromise.

b) When you ask tk to give you the height of the of the label it seems to give the lower of container height and height of text within it. When you subsequently write the text it then resizes again - a chicken and egg situation which causes multiple data refreshing and again causes flickering as as it generates resize events again. My fix is to pad out with extra newlines - just what we wanted to avoid in the first place. The compromise is 10 lines at a time - you can see the effect if you resize it to one line then 'full screen' it. I don't think it too bad a compromise.

I am left with a funny with displaying the log file - cursor down injects a further 2 lines into the log so cursor down never gets to the end. Still pondering that one!

Views on those compromises appreciated (or other fixes from any tk expert reading this!).

On File > Exit it shows diagnostic max height and max data. Would you feed back those values please?

Regards and thanks
Phil