Friday, May 3, 2024
 Popular · Latest · Hot · Upcoming
2
rated 0 times [  2] [ 0]  / answers: 1 / hits: 506  / 1 Year ago, sun, april 30, 2023, 12:47:15

I have installed Landscape Server 19.10 (quickstart) on a fresh 18.04 LXC container running in Proxmox. There were no issues with the install, the web GUI worked fine, and I was able to connect my client machines to the server.


Unfortunately I noticed in the notifications pane of the web GUI that the script update_security_db.sh was failing repeatedly every time it was run (it does so every hour, as per /etc/cron.d/landscape-server).


enter image description here


Checking update-security-db.log and running the script manually gives me the following error:


Dec 19 16:40:36 update-security-db ERR  Traceback (most recent call last):
Dec 19 16:40:36 update-security-db ERR File "./process-usns", line 7, in <module>
Dec 19 16:40:36 update-security-db ERR canonical.landscape.scripts.usn.run()
Dec 19 16:40:36 update-security-db ERR File "/opt/canonical/landscape/canonical/landscape/scripts/batch.py", line 77, in __call__
Dec 19 16:40:36 update-security-db ERR code = self.run()
Dec 19 16:40:36 update-security-db ERR File "/opt/canonical/landscape/canonical/landscape/scripts/usn.py", line 40, in run
Dec 19 16:40:36 update-security-db ERR changeset = update_from_usn_tool_db(db)
Dec 19 16:40:36 update-security-db ERR File "/opt/canonical/landscape/canonical/landscape/model/package/usn.py", line 237, in update_from_usn_tool_db
Dec 19 16:40:36 update-security-db ERR "WHERE id = %(temp)s.pkg_id" % {"temp": temp_table})
Dec 19 16:40:37 update-security-db ERR File "/usr/lib/python2.7/dist-packages/storm/store.py", line 109, in execute
Dec 19 16:40:37 update-security-db ERR return self._connection.execute(statement, params, noresult)
Dec 19 16:40:37 update-security-db ERR File "/usr/lib/python2.7/dist-packages/storm/databases/postgres.py", line 306, in execute
Dec 19 16:40:37 update-security-db ERR return Connection.execute(self, statement, params, noresult)
Dec 19 16:40:37 update-security-db ERR File "/usr/lib/python2.7/dist-packages/storm/database.py", line 241, in execute
Dec 19 16:40:37 update-security-db ERR raw_cursor = self.raw_execute(statement, params)
Dec 19 16:40:37 update-security-db ERR File "/usr/lib/python2.7/dist-packages/storm/databases/postgres.py", line 316, in raw_execute
Dec 19 16:40:37 update-security-db ERR return Connection.raw_execute(self, statement, params)
Dec 19 16:40:37 update-security-db ERR File "/usr/lib/python2.7/dist-packages/storm/database.py", line 374, in raw_execute
Dec 19 16:40:37 update-security-db ERR self._run_execution(raw_cursor, args, params, statement)
Dec 19 16:40:37 update-security-db ERR File "/usr/lib/python2.7/dist-packages/storm/database.py", line 392, in _run_execution
Dec 19 16:40:37 update-security-db ERR statement, params or (), error)
Dec 19 16:40:37 update-security-db ERR File "/usr/lib/python2.7/dist-packages/storm/database.py", line 454, in _check_disconnect
Dec 19 16:40:37 update-security-db ERR return function(*args, **kwargs)
Dec 19 16:40:37 update-security-db ERR File "/usr/lib/python2.7/dist-packages/storm/tracer.py", line 248, in trace
Dec 19 16:40:37 update-security-db ERR attr(*args, **kwargs)
Dec 19 16:40:37 update-security-db ERR File "/usr/lib/python2.7/dist-packages/storm/databases/postgres.py", line 463, in connection_raw_execute_error
Dec 19 16:40:37 update-security-db ERR statement, params, "SQL server cancelled statement")
Dec 19 16:40:37 update-security-db ERR storm.exceptions.TimeoutError: 'SQL server cancelled statement', 'UPDATE package SET usn_id = new_package_usn_6KqW4Z.usn_id FROM new_package_usn_6KqW4Z WHERE id = new_package_usn_6KqW4Z.pkg_id', ()

This error occurs every time update_security_db.sh is run. When manually running the script, I noticed that it successfully curls the USN file from the Ubuntu servers. It then passes the file to the process_usns script. The script runs for several minutes before it exits with the SQL error shown above.


I basically installed landscape-server-quickstart directly on a fresh container, and trying the process again on a new container gave me the same issue as well. What is odd is that the log seems to imply that there is some issue with the SQL server processing the newly downloaded USN file. CPU and memory utilization was fine while the script was run (my container has 4GB RAM and 2vCPU). This is a concern for me since the primary reason for me setting up Landscape is to perform patch management for multiple computers.


UPDATE: I spun up a KVM virtual machine today, on a fresh image of Ubuntu Server 18.04. Got the exact same error as before (was curious if this issue only occured if run in LXC).


More From » 18.04

 Answers
5

After some debugging, I figured out that this issue was due to the SQL server timing out when usn.py updated the database. Likely that is due to the fact that my LXC host runs at high load, and its system specs are lower than what's recommended by Landscape. The PostgreSQL log clearly references this timeout too when the error is triggered in the security update log.


The following command in /opt/canonical/landscape/canonical/landscape/model/package/usn.py:235 takes longer than usual on my machine and thus triggers the database to timeout.


     store.execute( 
"UPDATE package SET usn_id = %(temp)s.usn_id FROM %(temp)s "
"WHERE id = %(temp)s.pkg_id" % {"temp": temp_table})

I quickly patched around this by adding the following before that command.


     store.execute("SET LOCAL statement_timeout = 10000")

I am not experienced with SQL, but this basically should temporarily increase the timeout to 10000ms, which is more than enough time to process the db transaction. It works when set to lower values too, but I set it higher just in case. I was unable to find out the timeout value originally set by Landscape.


After the patch, the error is gone and USN updates work normally. It's a dirty hack, but as far as I know it works.


[#2194] Monday, May 1, 2023, 1 Year  [reply] [flag answer]
Only authorized users can answer the question. Please sign in first, or register a free account.
uestred

Total Points: 464
Total Questions: 104
Total Answers: 112

Location: Israel
Member since Wed, Apr 14, 2021
3 Years ago
uestred questions
Wed, Mar 2, 22, 04:04, 2 Years ago
Wed, Jan 5, 22, 05:57, 2 Years ago
;