[Solved] silence.py on Mythtv 31 and kubuntu 20.04 not working

[halucigenia]

I have just updated to Mythtv 31 and kubuntu 20.04 and I cannot get silence.py to run to detect advert breaks.
I have checked the https://www.mythtv.org/wiki/Commercial_ ... h_silences page and see that there is a patch.txt stating "TimP in a mythtv-users post has created a patch to update silence.py to work with Python3"
I have re-compiled and I have tried applying this patch but got an error

Code: Select all

patch < patch.txt  
patching file silence.py
Hunk #1 succeeded at 1 with fuzz 2.
Hunk #2 FAILED at 145.
Hunk #3 FAILED at 183.
2 out of 3 hunks FAILED -- saving rejects to file silence.py.rej

I think that I understand what it is trying to change and have tried applying the patch manually to change the following in the code:-
from

Code: Select all

 + ', "' + rec.title.encode('utf-8') + ' - ' + rec.subtitle.encode('utf-8') + '"')

to

Code: Select all

+ ', "' + rec.title + ' - ' + rec.subtitle + '"')

also
from

Code: Select all

 stdout=subprocess.PIPE)

to

Code: Select all

stdout=subprocess.PIPE, text=True)

But it is still not working.

On running silence.py manually I am getting

Code: Select all

silence.py --chanid 20081 --starttime 20200428035900 --loglevel debug
2020-04-28 13:01:54.038039 D [204535] Python Database Connection Trying database credentials from: /home/jeff/config.xml
2020-04-28 13:01:54.046400 I [204535]  mythcommflag: 
2020-04-28 13:01:54.049993 I [204535]  mythcommflag: Seeking chanid 20081, starttime 2020-04-28 03:59:00+01:00
2020-04-28 13:01:54.053379 E [204535]  mythcommflag: Failed due to: "DBData() could not read from database"

I have also tried running it as the mythtv user with the same result.

Any help with this would be greatly appreciated.

[dnalorernst]

Due to historical reasons, the Mythtv's Python Bindings use the local time
for a recording at the time the recording was made.
In your case, I assume it is the BST Zone.
In contrast, the filename and all time related data stored in the MySQL database
are stored in UTC.

That means. the log output "starttime 2020-04-28 03:59:00+01:00"
belongs to the *local* time "2020-04-28 03:59:00" with offset to UTC "+01:00".

I assume, you have a recording in the database like "20081_20200428035900.ts".
Is this correct ?
If so try, "--starttime 20200428045900" as parameter, because it was made during BST.

[halucigenia]

Thanks for the tip dnalorernst,

I tried that and it seems to get past the error

Code: Select all

"DBData() could not read from database"

but now I am getting the error

Code: Select all

Failed due to: "can only concatenate str (not "bytes") to str"

I am guessing that is to do with the "patch to update silence.py to work with Python3" not working, or that I have not patched the code correctly since I could not get the patch to automatically patch the file?

[dnalorernst]

The patch on the wiki is definitely not against the file 'silence.py' on the same wiki page.

And, Python would tell you, if you did any syntax mistakes.

Please try the following:
Remove any occurrence of ".encode('utf-8')".
For instance, the wiki version for python2 of 'silence.py' has:

Code: Select all

logger.log('Processing: ' + channel.callsign.encode('utf-8') + ', ' + str(rec.starttime)
  + ', "' + rec.title.encode('utf-8') + ' - ' + rec.subtitle.encode('utf-8')+ '"')
  
and

self.logger.log('No preset found for "' + title.encode('utf-8') + '" or "' + callsign.encode('utf-8') + '"')

This should be changed to:

Code: Select all

logger.log('Processing: ' + channel.callsign + ', ' + str(rec.starttime)
  + ', "' + rec.title + ' - ' + rec.subtitle + '"')
  
and

self.logger.log('No preset found for "' + title + '" or "' + callsign + '"')

[halucigenia]

Thanks for the help dnalorernst
I missed a couple of

Code: Select all

.encode('utf-8')

So I changed them like your example but I now get the error:-

Code: Select all

Failed due to: "MySQL error: a bytes-like object is required, not 'str'"

I really should learn Python if I want to understand this stuff.

[dnalorernst]

halucigenia,
I have good news and bad news.

First, I successfully installed on my test environment
the 'Commercial detection with silences' according the wiki page
and patched the 'silence.py' according our discussion.
See patch file below.

A test run shows that it is doing well:

Code: Select all

$ python3 silence.py --chanid 10102 --starttime 20200202191300
2020-05-01 13:08:16.514245 I [2161]  mythcommflag: 
2020-05-01 13:08:16.516066 I [2161]  mythcommflag: Seeking chanid 10102, starttime 2020-02-02 19:13:00+01:00
2020-05-01 13:08:16.523382 I [2161]  mythcommflag: Processing: XXXX Channel , 2020-02-02 19:13:00+01:00, "XXXXX Title"
2020-05-01 13:08:16.550042 I [2161]  mythcommflag:            Start - End    Start - End      Duration         Interval    Level/Count
2020-05-01 13:08:16.550185 I [2161]  mythcommflag:           frame - frame (mmm:ss-mmm:ss) frame (mm:ss.s)  frame (mmm:ss)
2020-05-01 13:08:22.588485 I [2161]  mythcommflag: < Cluster      1-  6898 (  0:00-  4:36), 6898 ( 4:35.9),     0 (  0:00), [      9]
2020-05-01 13:08:22.589080 I [2161]  mythcommflag: =     Cut      1-  6886 (  0:00-  4:35), 6886 ( 4:35.4),     0 (  0:00), [      0]
2020-05-01 13:08:25.481740 I [2161]  mythcommflag: # Cluster  10603- 10619 (  7:04-  7:05),   17 ( 0:00.7),  3706 (  2:28), [      1]
2020-05-01 13:08:57.768592 I [2161]  mythcommflag: > Cluster  13724- 16536 (  9:09- 11:01), 2813 ( 1:52.5),  3106 (  2:04), [      3]
2020-05-01 13:08:57.768797 I [2161]  mythcommflag: =     Cut  13736- 16536 (  9:09- 11:01), 2801 ( 1:52.0),     0 (  0:00), [      0]
2020-05-01 13:08:57.775916 I [2161]  mythcommflag: Detected 2 adverts.

I can import the markers in the Recording Editor of the frontened.
Albeit, as expected, the found cut marks are not meaningful for that channel.

Used patch file: (attached silence_py_patch_01.patch)

Code: Select all

--- silence_org.py	2020-05-01 12:46:46.918617358 +0200
+++ silence.py	2020-05-01 14:19:14.197684571 +0200
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/python3
 # Build a skiplist from silence in the audio track.
 # v1.0 Roger Siddons
 # v2.0 Fix progid for job/player messages
@@ -89,7 +89,7 @@
               self.argdict.update(v for v in validargs if v[1] is not None)
               break
         else:
-          self.logger.log('No preset found for "' + title.encode('utf-8') + '" or "' + callsign.encode('utf-8') + '"')
+          self.logger.log('No preset found for "' + title + '" or "' + callsign + '"')
     except IOError:
       self.logger.log('Presets file "' + filename + '" not found', MYLOG.ERR)
     return self.argdict
@@ -155,8 +155,8 @@
     rec = MythTV.Recorded((chanid, starttime), db)
     channel = MythTV.Channel(chanid, db)
 
-    logger.log('Processing: ' + channel.callsign.encode('utf-8') + ', ' + str(rec.starttime)
-      + ', "' + rec.title.encode('utf-8') + ' - ' + rec.subtitle.encode('utf-8')+ '"')
+    logger.log('Processing: ' + channel.callsign + ', ' + str(rec.starttime)
+      + ', "' + rec.title + ' - ' + rec.subtitle + '"')
 
     sg = MythTV.findfile(rec.basename, rec.storagegroup, db)
     if sg is None:
@@ -182,7 +182,7 @@
                 stdin=p1.stdout, stdout=subprocess.PIPE)
     # Pipe audio stream to C++ silence which will spit out formatted log lines
     p3 = subprocess.Popen([kExe_Silence, "%d" % p1.pid] + param.getValues(), stdin=p2.stdout,
-                stdout=subprocess.PIPE)
+                stdout=subprocess.PIPE, text=True)
 
     # Purge any existing skip list and flag as in-progress
     rec.commflagged = 2

Since it is not working for you,it could be because of a mixed python setup.
I kindly ask you to show the output of

Code: Select all

$ /usr/bin/env python3 --version

Code: Select all

$ mythpython
>>> import sys
>>> sys.version_info

I propose, you start from scratch with a new run:
- create a folder in your home
- pull the silence.py from wiki
- apply the patch by

Code: Select all

patch < silence_py_patch_01.patch 

- run the test from termial as you did before
- report back the output

If this does not work, please apply the patch below
Add the code marked with ' ### XXX' at line 240:

Code: Select all

  except Exception as e:
    raise        ### XXX for debugging only !
    # get exception before we generate another

Run the test in the terminal again and post the full traceback

[halucigenia]

Thanks for all your help dnalorernst
I tried the following:-
Updated Mythtv to 2:31.0+fixes.202005011857.a2b8c26~ubuntu20.04.1
from ppa

checked python versions:-

Code: Select all

python3 --version
Python 3.8.2

Code: Select all

 mythpython
MythTV Python interactive shell. TAB completion available.
>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=8, micro=2, releaselevel='final', serial=0)

re-downloaded the silence files and your patch
applied the patch

Code: Select all

patch < silence_py_patch_01.patch

No error this time

re-compiled slilcene
ran make-install

but I still got the error

Code: Select all

Failed due to: "MySQL error: a bytes-like object is required, not 'str'"

ran silence with the code marked with ' ### XXX' at line 240:

Trace

Code: Select all

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/MythTV/_conn_mysqldb.py", line 97, in executemany
    return super(LoggedCursor, self).executemany(query, args)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 235, in executemany
    return self._do_execute_many(q_prefix, q_values, q_postfix, args,
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 264, in _do_execute_many
    rows += self.execute(sql + postfix)
  File "/usr/lib/python3/dist-packages/MythTV/_conn_mysqldb.py", line 67, in execute
    query = self._sanitize(query)
  File "/usr/lib/python3/dist-packages/MythTV/_conn_mysqldb.py", line 44, in _sanitize
    def _sanitize(self, query): return query.replace('?', '%s')
TypeError: a bytes-like object is required, not 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/silence.py", line 285, in <module>
    main()
  File "/usr/local/bin/silence.py", line 207, in main
    rec.update()
  File "/usr/lib/python3/dist-packages/MythTV/database.py", line 430, in update
    self._push()
  File "/usr/lib/python3/dist-packages/MythTV/dataheap.py", line 366, in _push
    self.markup.commit()
  File "/usr/lib/python3/dist-packages/MythTV/database.py", line 622, in commit
    cursor.executemany("""INSERT INTO %s (%s) VALUES(%s)""" % \
  File "/usr/lib/python3/dist-packages/MythTV/_conn_mysqldb.py", line 99, in executemany
    raise MythDBError(MythDBError.DB_RAW, e.args)
MythTV.exceptions.MythDBError: MySQL error: a bytes-like object is required, not 'str'

Something wrong with my MythTV python3 libraries?

[dnalorernst]

halucigenia,
I do not think that your installation is wrong.
Rather, I suspect a problem in the MythTV python bindings with python3.8 and mysql.
Do you run MariaDB or MySQL on your backend ?
Please post the version as well.
Currently, my setup is python3.7 and 10.3.22-MariaDB, so I need to install your environment as close at is possible.

Thank you for your patience!

[halucigenia]

dnalorernst python3 version as above

Code: Select all

python
>>> import sys
>>> print(sys.version)
2.7.18rc1 (default, Apr  7 2020, 12:05:55) 
[GCC 9.3.0]

Code: Select all

MariaDB [(none)]> select version();
+--------------------------------------+
| version()                            |
+--------------------------------------+
| 10.4.12-MariaDB-1:10.4.12+maria~eoan |

Thanks

[dnalorernst]

This makes me nervous:

Code: Select all

MariaDB-1:10.4.12+maria~eoan

I thought you are running kubuntu 20.04 (as depicted in your first post), but the ubuntu packages search at
https://packages.ubuntu.com/search?keyw ... adb-server returns only

Code: Select all

MariaDB 1:10.3.

for Ubuntu eoan (19.10) and focal (20.04).
Therefore, the mysql-server is running somewhere else, like docker-image, vm,.... ?

Please list the output for the versions of python-lxml and python-MySQLdb, by running on the backend

Code: Select all

$ mythpython --version

And, please, explain your rather complex setup in more detail.

[halucigenia]

OK, I missed that

Code: Select all

[MariaDB-1:10.4.12+maria~eoan

eoan!
That must be a hangover from issues that I had when upgrading to kubuntu 19.10.
I added https://downloads.mariadb.org/mariadb/r ... rsion=10.4
as I was having dificullties uninstalling and reinstalling MariaDB at the time.

I only see https://downloads.mariadb.org/mariadb/r ... rsion=10.5
10.5 Beta vesion there for focal not 10.3 or 10.4

Code: Select all

mythpython --version
MythTV Python Bindings
  local versions
    bindings version:         31.0.-1.0
    ttvdb version:            2.0-dev
    tmdb version:             v0.7.0
  external versions
    lxml version:             4.5.0
    MySQLdb version:          1.4.4.final.0
  protocol versions
    backend:                  91
    schema:                   1361
    music schema:             1024
    netvision schema:         1007

I have a feeling that this is not going to be an easy fix. Mythtv seems to be running OK though.

[dnalorernst]

It is not only the eoan version of MariaDB, which hit my eyes: See
viewtopic.php?f=2&t=3720
Is there any reason for not using the MariaDB package available from the Ubuntu focal distribution?
For now, there are good reasons to stay at MariaDB 10.3 on Ubuntu focal, which is the default.

[halucigenia]

dnalorernst there's no reason for not using the MariaDB package available from the Ubuntu focal distribution. But I don't want to uninstall and reinstall as there's lots of dependencies as far as I recall, unless there would be an easier way to downgrade?

I never watch live TV anymore, I tend to record and watch the recording after it has started if I want to watch something that is on at the time I want to watch it, so I have not come across the bug.

[dnalorernst]

Bad news and good news:
I can reproduce this issue with the default setup from Ubuntu 20.04.
As expected, it is a combination of Python version and used MySQLdb module.

• Python3.8
• MySQLdb 1.4.4
• MariaDB 10.3

Please see #13614 for proposed patches.
With these patches i get the same result as on Ubuntu 19.10 and mythtv/31.
Thank you four your help and patience.

[halucigenia]

Thanks dnalorernst