FS#64648 - file:// sources not extracted automatically anymore

Attached to Project: Pacman
Opened by Sami B. (spider-mario) - Monday, 25 November 2019, 22:15 GMT
Last edited by Andrew Gregory (andrewgregory) - Thursday, 02 July 2020, 20:08 GMT
Task Type Bug Report
Category makepkg
Status Closed
Assigned To Eli Schwartz (eschwartz)
Architecture All
Severity Low
Priority Normal
Reported Version 5.2.1
Due in Version 5.2.2
Due Date Undecided
Percent Complete 100%
Votes 2
Private No

Details

makepkg used to automatically extract 7z archives in the source array, but it does not anymore. Looking at the git repository, it is unclear to me which change could have caused this, but it seems rather recent, since I was able to build https://aur.archlinux.org/packages/pianoteq-stage/ (where this was reported to me) on the 16th of October with pacman 5.1.3-1 and libarchive 3.4.0-2.
This task depends upon

Closed by  Andrew Gregory (andrewgregory)
Thursday, 02 July 2020, 20:08 GMT
Reason for closing:  Fixed
Additional comments about closing:  Commit 349c22d043290ccd0cce9f30981f5415e295442a
Comment by Eli Schwartz (eschwartz) - Monday, 25 November 2019, 22:23 GMT
makepkg successfully extracts a sample test.7z file for me, when added to a test PKGBUILD.

==> Extracting sources...
-> Extracting test.7z with bsdtar

I'm unsure how this could *not* work, since in any case where no previous extraction tool was discovered, we fall back to seeing if bsdtar recognizes it as a file bsdtar knows how to handle.
Comment by Sami B. (spider-mario) - Monday, 25 November 2019, 23:28 GMT
Oh, interesting. Apparently it doesn’t work for net147, and it stopped working for me too (even though bsdtar -x on the same file still succeeds). What could I do to help diagnose the issue?

On the package in question, I get:

==> Retrieving sources...
-> Found pianoteq_stage_linux_v660.7z
-> Found pianoteq_icon_128.png
==> Validating source files with sha512sums...
pianoteq_stage_linux_v660.7z ... Passed
pianoteq_icon_128.png ... Passed
==> Extracting sources...
==> Starting prepare()...

So the archive is not even listed under “Extracting sources”.
Comment by Eli Schwartz (eschwartz) - Monday, 25 November 2019, 23:35 GMT
One thing you could do would be 'sudoedit /usr/share/makepkg/source/file.sh' and, in the beginning of the extract_file() function, add:

local -; set -x

This would tell you what is being run here, and you could verify what branches are being run. I'm not able to debug this case myself, because the relevant file download requires registration and possibly payment.
Comment by Sami B. (spider-mario) - Monday, 25 November 2019, 23:40 GMT
I have just attempted to bisect the issue and here is the outcome:

9c817b654996249b8022e189ee7e2692f4668431 is the first bad commit
commit 9c817b654996249b8022e189ee7e2692f4668431
Author: Eli Schwartz <eschwartz@archlinux.org>
Date: Mon May 28 23:30:29 2018 -0400

libmakepkg: implement extendable source protocols

Lookup the existence of matching functions for each protocol, and
fallback on the generic file handler. New source protocols can then be
added via thirdparty libmakepkg drop-ins without requiring modifications
to source.sh

Fixes  FS#49076 

Signed-off-by: Allan McRae <allan@archlinux.org>

scripts/libmakepkg/source.sh.in | 47 ++++++++-----------------------------
scripts/libmakepkg/source/bzr.sh.in | 5 ++++
scripts/libmakepkg/source/git.sh.in | 5 ++++
scripts/libmakepkg/source/hg.sh.in | 5 ++++
scripts/libmakepkg/source/svn.sh.in | 5 ++++
5 files changed, 30 insertions(+), 37 deletions(-)

Is it because it’s a file:// URI?
Comment by Sami B. (spider-mario) - Monday, 25 November 2019, 23:41 GMT
Here is the output of the set -x trick:

+ local file=file://pianoteq_stage_linux_v660.7z
++ get_filepath file://pianoteq_stage_linux_v660.7z
+++ get_filename file://pianoteq_stage_linux_v660.7z
+++ local netfile=file://pianoteq_stage_linux_v660.7z
+++ [[ file://pianoteq_stage_linux_v660.7z = *::* ]]
++++ get_protocol file://pianoteq_stage_linux_v660.7z
++++ [[ file://pianoteq_stage_linux_v660.7z = *://* ]]
++++ local proto=file://pianoteq_stage_linux_v660.7z
++++ proto=file
++++ printf '%s\n' file
+++ local proto=file
+++ case $proto in
+++ filename=pianoteq_stage_linux_v660.7z
+++ printf '%s\n' pianoteq_stage_linux_v660.7z
++ local file=pianoteq_stage_linux_v660.7z
+++ get_protocol file://pianoteq_stage_linux_v660.7z
+++ [[ file://pianoteq_stage_linux_v660.7z = *://* ]]
+++ local proto=file://pianoteq_stage_linux_v660.7z
+++ proto=file
+++ printf '%s\n' file
++ local proto=file
++ case $proto in
++ [[ -f /home/sami/Programmes/aur4/pianoteq-stage/pianoteq_stage_linux_v660.7z ]]
++ file=/home/sami/Programmes/aur4/pianoteq-stage/pianoteq_stage_linux_v660.7z
++ printf '%s\n' /home/sami/Programmes/aur4/pianoteq-stage/pianoteq_stage_linux_v660.7z
+ local filepath=/home/sami/Programmes/aur4/pianoteq-stage/pianoteq_stage_linux_v660.7z
+ rm -f /tmp/makepkg/pianoteq-stage/src/file://pianoteq_stage_linux_v660.7z
+ ln -s /home/sami/Programmes/aur4/pianoteq-stage/pianoteq_stage_linux_v660.7z /tmp/makepkg/pianoteq-stage/src/
+ in_array file://pianoteq_stage_linux_v660.7z
+ local needle=file://pianoteq_stage_linux_v660.7z
+ shift
+ local item
+ return 1
++ file -bizL -- file://pianoteq_stage_linux_v660.7z
+ local 'file_type=cannot open `file://pianoteq_stage_linux_v660.7z'\'' (No such file or directory)'
+ local ext=7z
+ local cmd=
+ case "$file_type" in
+ bsdtar -tf file://pianoteq_stage_linux_v660.7z -q '*'
+ return 0
+ local file=pianoteq_icon_128.png
++ get_filepath pianoteq_icon_128.png
+++ get_filename pianoteq_icon_128.png
+++ local netfile=pianoteq_icon_128.png
+++ [[ pianoteq_icon_128.png = *::* ]]
++++ get_protocol pianoteq_icon_128.png
++++ [[ pianoteq_icon_128.png = *://* ]]
++++ [[ pianoteq_icon_128.png = *lp:* ]]
++++ printf '%s\n' local
+++ local proto=local
+++ case $proto in
+++ filename=pianoteq_icon_128.png
+++ printf '%s\n' pianoteq_icon_128.png
++ local file=pianoteq_icon_128.png
+++ get_protocol pianoteq_icon_128.png
+++ [[ pianoteq_icon_128.png = *://* ]]
+++ [[ pianoteq_icon_128.png = *lp:* ]]
+++ printf '%s\n' local
++ local proto=local
++ case $proto in
++ [[ -f /home/sami/Programmes/aur4/pianoteq-stage/pianoteq_icon_128.png ]]
++ file=/home/sami/Programmes/aur4/pianoteq-stage/pianoteq_icon_128.png
++ printf '%s\n' /home/sami/Programmes/aur4/pianoteq-stage/pianoteq_icon_128.png
+ local filepath=/home/sami/Programmes/aur4/pianoteq-stage/pianoteq_icon_128.png
+ rm -f /tmp/makepkg/pianoteq-stage/src/pianoteq_icon_128.png
+ ln -s /home/sami/Programmes/aur4/pianoteq-stage/pianoteq_icon_128.png /tmp/makepkg/pianoteq-stage/src/
+ in_array pianoteq_icon_128.png
+ local needle=pianoteq_icon_128.png
+ shift
+ local item
+ return 1
++ file -bizL -- pianoteq_icon_128.png
+ local 'file_type=image/png; charset=binary'
+ local ext=png
+ local cmd=
+ case "$file_type" in
+ bsdtar -tf pianoteq_icon_128.png -q '*'
+ return 0

Looks like it is indeed because of the file:// URI.
Comment by Eli Schwartz (eschwartz) - Monday, 25 November 2019, 23:47 GMT
++ file -bizL -- file://test.7z
+ local 'file_type=cannot open `file://test.7z'\'' (No such file or directory)'
Comment by Eli Schwartz (eschwartz) - Monday, 25 November 2019, 23:53 GMT
Right, because extract_$proto == extract_file, but we special-cased that in the arguments. :(

My opinion is we should make extract_file behave like the other extract_* functions, and expect a netfile then resolve that inside the function itself.
Comment by Eli Schwartz (eschwartz) - Tuesday, 26 November 2019, 00:27 GMT Comment by Sami B. (spider-mario) - Tuesday, 26 November 2019, 08:00 GMT
Great, thanks!
Comment by Eli Schwartz (eschwartz) - Monday, 27 January 2020, 03:30 GMT
  • Field changed: Due in Version (Undecided → 5.2.2)
  • Field changed: Percent Complete (0% → 50%)
Fixed in https://git.archlinux.org/pacman.git/commit/?id=349c22d043290ccd0cce9f30981f5415e295442a but should get backported to the next maint release.
Comment by Eli Schwartz (eschwartz) - Monday, 27 January 2020, 03:36 GMT
  • Field changed: Summary (7z sources not extracted automatically anymore → file:// sources not extracted automatically anymore)
And retitling for visibility...

Loading...