Arch Linux

Please read this before reporting a bug:
https://wiki.archlinux.org/index.php/Reporting_Bug_Guidelines

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#68903 - [tracker3] testsuite breaks with sqlite 3.34.0

Attached to Project: Arch Linux
Opened by Jan Alexander Steffens (heftig) - Wednesday, 09 December 2020, 10:25 GMT
Last edited by Jan Alexander Steffens (heftig) - Saturday, 12 December 2020, 12:52 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Andreas Radke (AndyRTR)
Jan Alexander Steffens (heftig)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

I tried to package tracker3 3.0.2; tracker-fd-test from the test suite fails with:

Tracker-CRITICAL **: 02:39:41.823: SQLite error: database disk image is malformed (errno: Success)

Apparently sqlite3_step returns SQLITE_CORRUPT. When building with sqlite 3.33.0-2, the test suite passes.

To enable debug output, add `--buildtype debug` to arch-meson and run the tests with `TRACKER_DEBUG=sqlite,sql-statements`.

This shows the last statement executed:

Tracker-Message: 10:22:32.985: Executing update: 'INSERT INTO "main".fts5 (fts5, rowid , "nco:phoneNumber", "nfo:fontFamily", "nmm:artistName", "nfo:tableOfContents", "nfo:fileName", "nfo:genre", "nmm:genre", "nco:emailAddress", "nie:keyword", "nmm:category", "nco:nameAdditional", "nco:nameFamily", "nco:nameGiven", "nco:contactGroupName", "nco:fullname", "nco:nickname", "nco:country", "nco:county", "nco:district", "nco:extendedAddress", "nco:locality", "nco:pobox", "nco:postalcode", "nco:region", "nco:streetAddress", "nco:imID", "nco:imNickname", "nie:comment", "nie:description", "nie:plainTextContent", "nie:subject", "nie:title", "nao:description", "nao:prefLabel", "nco:department", "nco:role", "nco:title", "nco:note") SELECT 'delete', rowid , "nco:phoneNumber", "nfo:fontFamily", "nmm:artistName", "nfo:tableOfContents", "nfo:fileName", "nfo:genre", "nmm:genre", "nco:emailAddress", "nie:keyword", "nmm:category", "nco:nameAdditional", "nco:nameFamily", "nco:nameGiven", "nco:contactGroupName", "nco:fullname", "nco:nickname", "nco:country", "nco:county", "nco:district", "nco:extendedAddress", "nco:locality", "nco:pobox", "nco:postalcode", "nco:region", "nco:streetAddress", "nco:imID", "nco:imNickname", "nie:comment", "nie:description", "nie:plainTextContent", "nie:subject", "nie:title", "nao:description", "nao:prefLabel", "nco:department", "nco:role", "nco:title", "nco:note" FROM "main".fts_view WHERE rowid = 100002'
This task depends upon

Closed by  Jan Alexander Steffens (heftig)
Saturday, 12 December 2020, 12:52 GMT
Reason for closing:  Fixed
Additional comments about closing:  tracker3 3.0.2-1
Comment by Andreas Radke (AndyRTR) - Wednesday, 09 December 2020, 11:05 GMT
Better report it to the upstream tracker. There's not much going on lately and no commits/bugfix release planned it seems:

https://www.sqlite.org/src/timeline?advm=0&udc=1&y=t or https://www.sqlite.org/src/timeline?udc=1&ss=j&n=100&y=ci&advm=0
https://www.sqlite.org/src/timeline?udc=1&ss=j&n=100&y=ci&advm=0
Comment by Jan Alexander Steffens (heftig) - Wednesday, 09 December 2020, 12:20 GMT Comment by Richard Hipp (drhsqlite) - Wednesday, 09 December 2020, 13:06 GMT
Can you please bisect SQLite between 3.33.0 and 3.34.0 to see where the failure occurs?

The complete list of check-ins going from 3.33.0 to 3.34.0 can be seen here: https://www.sqlite.org/src/timeline?from=version-3.33.0&to=version-3.34.0&y=ci

If you are uncomfortable using the canonical Fossil repository for SQLite, you can pull the sources from the GitHub mirror: https://github.com/sqlite/sqlite

For each version of SQLite that you pull to test, you can build the "sqlite3.c" and "sqlite3.h" source files by running "./configure && make sqlite3.c".
Comment by Jan Alexander Steffens (heftig) - Wednesday, 09 December 2020, 16:27 GMT
A Tracker developer says "the fts5 table is an external content one, set on that fts_view in the "select" side" and "fts_view contains exactly what was inserted".

86f477edaa17767b39c7bae5b67cac8580f7a8c1 is the first bad commit https://www.sqlite.org/src/info/b79f19edfd33c2a7
commit 86f477edaa17767b39c7bae5b67cac8580f7a8c1
Author: dan <dan@noemail.net>
Date: Fri Sep 11 15:01:49 2020 +0000

Catch fts5 index corruption caused by issuing 'delete' commands with incorrect data earlier in some cases. Also fix a couple of test script problems.

FossilOrigin-Name: b79f19edfd33c2a75f936c352668e14e81f35acf4f07edc27a21f941a7304b38


I guess the problem is that inserting a 'delete' command with all possible columns but where values that weren't actually inserted are NULL isn't handled correctly?
Comment by Dan Kennedy (dan_at_sqlite) - Wednesday, 09 December 2020, 16:50 GMT
Is there a way I can get the source for tracker3 and run this test locally?

Dan Kennedy.
Comment by Jan Alexander Steffens (heftig) - Wednesday, 09 December 2020, 16:57 GMT
The project is at https://gitlab.gnome.org/GNOME/tracker . I've been building the 3.0.2 tag.
Comment by Dan Kennedy (dan_at_sqlite) - Wednesday, 09 December 2020, 17:33 GMT
Is there a way to get the database before this SQL statement is run?

I can't seem to run the test suite locally. Following the instructions here:

https://gitlab.gnome.org/GNOME/tracker

has the "meson ./build --prefix=$HOME/opt/tracker -Dtracker_core=subproject" line fail with:

The Meson build system
Version: 0.45.1
Source dir: /home/dan/tmp/tracker3/tracker-miners
Build dir: /home/dan/tmp/tracker3/tracker-miners/build
Build type: native build

meson.build:126:18: ERROR: Expecting eol got id.
if optimization in ['0', 'g']
^

A full log can be found at /home/dan/tmp/tracker3/tracker-miners/build/meson-logs/meson-log.txt
Comment by Eli Schwartz (eschwartz) - Wednesday, 09 December 2020, 17:47 GMT
meson_version: '>=0.50'

You'll need to get an updated version of meson. Actually I'm a bit surprised you could even configure this as I thought meson should raise an error saying you need a new version of meson...

Note that meson can be run directly from the source tree as meson.py, or installed via pip
Comment by Carlos Garnacho (garnacho) - Wednesday, 09 December 2020, 21:07 GMT
Hello, Tracker developer here. Thanks Dan/Richard for investigating on the issue!

To ease things a bit, I've left a dump of all SQL queries issued in that test at https://people.gnome.org/~carlosg/tracker-fd-test-sql-commands.txt, with some minor editions so it doesn't rely on functions, tokenizers, collation, virtual tables and other code defined by Tracker.

That file does still reproduce the issue with `cat tracker-fd-test-sql-commands.txt | sqlite3` here.
Comment by Dan Kennedy (dan_at_sqlite) - Wednesday, 09 December 2020, 21:26 GMT
Thanks for this.

If I grep the tracker-fd-test-sql-commands.txt for "fts5", I can see the table being created, the rank function being set, one row being inserted and then two separate 'delete' commands - neither of which look to match the insert. So the index is corrupt. Prior to 3.34.0, this was silently ignored, or reported on when the index was queried. But now the SQLITE_CORRUPT is returned for the second delete.

Dan.
Comment by Carlos Garnacho (garnacho) - Thursday, 10 December 2020, 10:53 GMT
Thank you Dan. I've made https://gitlab.gnome.org/GNOME/tracker/-/merge_requests/353 to address this situation on the Tracker side.

I FTR think this was silently ignored. Tracker had in the past other FTS table consistency errors, and I remember FTS5 integrity-check command coming up clean after fixing.
Comment by Jan Alexander Steffens (heftig) - Thursday, 10 December 2020, 12:12 GMT
Thank you. The patch makes the test suite pass. It's applied in our tracker3 3.0.2-1 package.

Loading...