FS#76711 - [mkinitcpio] Fails to add nested symlinks correctly.

Attached to Project: Arch Linux
Opened by Oid Maps (dreieck) - Thursday, 01 December 2022, 22:30 GMT
Last edited by Toolybird (Toolybird) - Wednesday, 07 December 2022, 21:40 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To No-one
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

With a recent update, `mkinitcpio` does *not* copy anymore the target file correctly if it encounters a symlink, resulting in a stale link be embedded into the initial ramdisk.

Specifically, if a symlink points to another symlink, the intermediate symlink, which is the target of the first symbolic link, is omitted:
If in the file system the link structure is
`file_a -> file_b -> file_c`
and `file_a` should be added to the image, only `file_a` and `file_c` are added but `file_b` is omitted.

This is contrary to the previous behaviour, and contrary to the manpage.

`man mkinitcpio` states:

> `add_file path [ destination ] [ mode ]`
> `Adds a file and any needed parent directories to the image. **If it is a symlink, both the symlink and the target file will be added**. Optionally, a destination within the initramfs image as well as a file mode can be specified. By default, the destination and mode will be taken from the source and mode of the file specified by the path.`

*(bold accentuation by myself)*

However, I have a `/lib/initcpio/install/dmcrypt_and_resume` which contains, amongst others,
```
add_file /etc/suspend.conf
```
and on my system `/etc/suspend.conf` is a symbolic link pointing to `/usr/local/etc/suspend.conf`, which in turn points to `/usr/local/platform/GPD_Pocket/etc/suspend.conf` which is the actual file.

For many years, it has worked correctly, but now I find `/etc/suspend.conf` to be a dangling symlink in the image since the intermediate `/usr/local/etc/suspend.conf` is missing, while `/usr/local/platform/GPD_Pocket/etc/suspend.conf` is present.

I see this problem has appeared also for other files which are nested symlinks.

`mkinitcpio`'s package version on my system is 33-1.
This task depends upon

Closed by  Toolybird (Toolybird)
Wednesday, 07 December 2022, 21:40 GMT
Reason for closing:  Fixed
Additional comments about closing:  mkinitcpio 34-1
Comment by Toolybird (Toolybird) - Friday, 02 December 2022, 04:27 GMT
There's been some fixes applied after the 33-1 relase which might be related. You can check them out at the new home of mkinitcpio development [1]. See this issue [2] in particular. Can you confirm?

[1] https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio
[2] https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/issues/140
Comment by nl6720 (nl6720) - Friday, 02 December 2022, 09:58 GMT Comment by Oid Maps (dreieck) - Sunday, 04 December 2022, 17:09 GMT
> There's been some fixes applied after the 33-1 relase which might be related. [...] Can you confirm?

Build from git repository checkout from https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio, resulting in package `mkinitcpio-git` version 33.24.g10368eb-1, does not fix the issue.

> Please test: https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/merge_requests/159

Build from git repository https://gitlab.archlinux.org/nl6720/mkinitcpio, switched to branch `nested-symlinks`, (which seems to be what is referenced by https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/merge_requests/159), does not build -- `makepkg` complains with `fatal: No names found, cannot describe anything.`. (I am not familiar with how to work with git branches, with merge requests and so one -- can you provide a standalone tarball of what I should test?)

Comment by Oid Maps (dreieck) - Sunday, 04 December 2022, 17:14 GMT
> `makepkg` complains with `fatal: No names found, cannot describe anything.`.

OK, I found a way: Checking out https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio, downloading https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/merge_requests/159/diffs.patch as patch file, applying that patch, and then building with `makepkg`.

Nested symlinks are still broken in the reported way.
Comment by nl6720 (nl6720) - Sunday, 04 December 2022, 17:23 GMT
Are you sure you applied the patch correctly?
Try the attached PKGBUILD.
Comment by Oid Maps (dreieck) - Sunday, 04 December 2022, 18:51 GMT
> Are you sure you applied the patch correctly?

I was downloading https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/merge_requests/159/diffs.patch and applying with `patch -p1`. Then executing `makepkg`.

> Try the attached PKGBUILD.

This is indeed different from the `PKGBUILD` I get after applying above patch.

This `PKGBUILD` gives a checksum error:

```
159.patch ... FAILED
```

Interestingly, the `159.patch` downloaded by the `PKGBUILD` also differs from the https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/merge_requests/159/diffs.patch I have downloaded:

Anyway, just ignoring all this and building with the `PKGBUILD` you have provided here explicitly, it actually works.

Also, hooks by themselves can now be (two-staged) symlinks again.
Comment by Oid Maps (dreieck) - Sunday, 04 December 2022, 18:56 GMT
> Also, hooks by themselves can now be (two-staged) symlinks again.

This latter statement was wrong; a hook is still copied as a dangling symlink if the hook is a symlink.

Comment by nl6720 (nl6720) - Monday, 05 December 2022, 13:26 GMT
I think I got it this time. I updated the merge request, please test it again.
Make sure to first delete the downloaded patch file and note that the checksum of it has changed.
Comment by Oid Maps (dreieck) - Tuesday, 06 December 2022, 11:08 GMT
> I updated the merge request, please test it again.

I am really unsure how to make sure that I use the latest update (I am not familiar with git branches/ trees and with merge requests). Can you link to a standalone git repository or tarball which self-contain your latest work as a whole?
Comment by nl6720 (nl6720) - Tuesday, 06 December 2022, 11:11 GMT
It's the "nested-symlinks" branch on my fork: https://gitlab.archlinux.org/nl6720/mkinitcpio/-/tree/nested-symlinks
There's a download button on the top right, next "Clone", where you can download tarballs.

Edit:
https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/merge_requests/159.patch gets updated each time the MR is updated, so you just needed to re-download the file.
Comment by Oid Maps (dreieck) - Tuesday, 06 December 2022, 12:07 GMT
> I think I got it this time. I updated the merge request, please test it again.

I think I got it.

Now it _almost_ works for symlinks in /hooks.

I still have a dangling symlink, because I also have a symlink-abstraction-layer in my machine-specific directory tree (that way I can easily switch my hard disk to another computer, by just changing _one_ symlink):

On my machine, I have (for example):

`/lib/initcpio/hooks` -> `/usr/local/lib/initcpio/hooks/i915` -> `/usr/local/platform/current/lib/initcpio/hooks/i915`

and

`/usr/local/platform/current` -> `GPD_Pocket` (relative symlink, so it points to `/usr/local/platform/GPD_Pocket`)


The generated initcpio is attached. It fails the symlink chain at one point.

Present is:

`/hooks/i915` -> `/usr/lib/initcpio/hooks/i915` -> `/usr/local/lib/initcpio/hooks/i915` which points to `/usr/local/platform/current/lib/initcpio/hooks/i915`

But `/usr/local/platform/current` is not present in the image. Also `/usr/local/platform/GPD_Pocket` is not present in the image.
Comment by Oid Maps (dreieck) - Tuesday, 06 December 2022, 12:13 GMT
> The generated initcpio is attached.

Too big, so I cannot share.

Anyway, what was the reason to change the symlink handling in the first place? Before, the targets of symlinks were directly included, which did not create all those issues.

Regards!
Comment by Oid Maps (dreieck) - Tuesday, 06 December 2022, 12:14 GMT
> On my machine, I have (for example):
>
> `/lib/initcpio/hooks` -> `/usr/local/lib/initcpio/hooks/i915` -> `/usr/local/platform/current/lib/initcpio/hooks/i915`

There was a typo. I forget the first `i915`. It must be:

`/lib/initcpio/hooks/i915` -> `/usr/local/lib/initcpio/hooks/i915` -> `/usr/local/platform/current/lib/initcpio/hooks/i915`

The breakage in the resulting initcpio image occurs where the relative directory symlink
`/usr/local/platform/current` -> `GPD_Pocket`
is not resolved.
Comment by nl6720 (nl6720) - Tuesday, 06 December 2022, 12:16 GMT
It all started with  FS#73439  (for  FS#55535  ). Later I wanted to add support for relative symlinks, so that e.g. https://github.com/archlinux/svntogit-packages/blob/3cfa78fe78d335ddb406b388a7f85e2939b776c4/trunk/install-sd-encrypt#L40-L49 would be relative just like on the real system. It all went downhill from there.
Comment by nl6720 (nl6720) - Tuesday, 06 December 2022, 12:18 GMT
How are you adding these files? Please share your mkinitcpio.conf and custom hooks.
Comment by Oid Maps (dreieck) - Tuesday, 06 December 2022, 12:34 GMT
> How are you adding these files? Please share your mkinitcpio.conf and custom hooks.

Here I attach a tarball which contains the `/etc/mkinitcpio.conf` and as example the `i915` hook, with all the symlink nesting present. I hope I got it complete.
Comment by nl6720 (nl6720) - Tuesday, 06 December 2022, 13:56 GMT
I gave up on the idea of full nested symlink support. Please try now.
Comment by Oid Maps (dreieck) - Tuesday, 06 December 2022, 16:00 GMT
I still have the issue of the relative directory symlink
`/usr/local/platform/current` -> `GPD_Pocket`
is not resolved, but `/hook/i915` is inculded as a symlink-chain (so, still nested symlinks.)

I have uploaded the resulting image here, for you to look at the result:
https://file.io/DiGIDxqSclbJ

Note that also /etc/hostname breaks somewhere in the middle (I have overlooked that before), also where a (relative) directory symlink is involved.

Can you set up a test environment by yourself so you can test the results yourself? That I think would be a more efficient feedback loop.
Comment by Oid Maps (dreieck) - Tuesday, 06 December 2022, 16:03 GMT
> I have uploaded the resulting image here, for you to look at the result:
> https://file.io/DiGIDxqSclbJ

Seems the file self-destructs after first download, here again for you:
https://file.io/Ay42XDxs5QVX
Comment by Oid Maps (dreieck) - Tuesday, 06 December 2022, 16:33 GMT
> It all started with  FS#73439  (for  FS#55535 )

If nested symlinks make problems, and if symlinks are only needed to make systemd happy:

Why not, if you encounter a symlink, create some 'scratch' directory, put there the content of the symlink, and create a symlink to there?

E.g.:

If `/etc/some-file.conf` is a symlink to `/usr/lib/some/file.conf`, then create `/symlinks/usr/lib/some/file.conf` where the content of `/etc/some-file.conf` is written into, and link that to `/etc/some-file.conf`. All maybe intermediate symlinks omitted.

Directory symlinks ignore completely but just copy the target directory in place.

This might lead to data duplication and bigger filesize, though, but might solve the troubles you have.
Comment by nl6720 (nl6720) - Tuesday, 06 December 2022, 17:06 GMT
I don't understand why it doesn't work for you now.

The behaviour of the MR at commit 1eb600af15d4fe6493ed113da75c04efaabf232d is that nested symlinks are not created and the symlink points to the end target. The only special behaviour is that if the first-level-resolved target is not a symlink and resides in the same directory as the symlink, then a relative symlink should be created (by reading the link object with find).

A simple test build hook shows that it works:

build() {
rm -rf /tmp/linktest
install -d /tmp/linktest/testdir1 /tmp/linktest/testdir3
echo test1 > /tmp/linktest/testdir1/test1
ln -sfT /tmp/linktest/testdir3 /tmp/linktest/testdir2
ln -sfT test1 /tmp/linktest/testdir1/test2
ln -sfT /tmp/linktest/testdir1/test2 /tmp/linktest/testdir3/test3
ln -sfT test1 /tmp/linktest/testdir1/test4

echo 'START:'
add_file /tmp/linktest/testdir2/test3
add_file /tmp/linktest/testdir1/test4
}

The verbose output shows (ignore the out of order "adding dir"):

START:
adding file: /tmp/linktest/testdir1/test1
adding dir: /tmp/linktest/testdir1
adding dir: /tmp/linktest/testdir2
adding symlink: /tmp/linktest/testdir2/test3 -> /tmp/linktest/testdir1/test1
overwriting file: /tmp/linktest/testdir1/test1
adding symlink: /tmp/linktest/testdir1/test4 -> test1


I have attached a PKGBUILD that explicitly builds the https://gitlab.archlinux.org/nl6720/mkinitcpio/-/tree/nested-symlinks branch.
   PKGBUILD (1.8 KiB)
Comment by Oid Maps (dreieck) - Tuesday, 06 December 2022, 19:58 GMT
> I don't understand why it doesn't work for you now.

This time it was my fault: I forgot to delete the patch so that it get's downloaded again by `makepkg` (I was using my own `PKGBUILD` in the meantime).

Yes, it works now.

Nested symlinks are created as one-level symlinks.

`/hooks/...` are not symlinks.

`add_dir /some/directory` creates the directory directly, even if it is a symlink.

So from my side you can make this a new release, and then I just need to wait that Artix keeps up with the new version.

Thanks and regards!

Loading...