FS#12708 - Pacman: A suggestion for handling optdepends.
Attached to Project:
Pacman
Opened by Xyne (Xyne) - Thursday, 08 January 2009, 02:41 GMT
Last edited by Allan McRae (Allan) - Sunday, 04 December 2022, 07:32 GMT
Opened by Xyne (Xyne) - Thursday, 08 January 2009, 02:41 GMT
Last edited by Allan McRae (Allan) - Sunday, 04 December 2022, 07:32 GMT
|
Details
I've posted a thread about this here:
http://bbs.archlinux.org/viewtopic.php?pid=476658
Both the sync and local databases contain depends file to keep track of their dependencies with the following simple format: Code: %DEPENDS% foo bar this that %PROVIDES% whatever It should be relatively easy to add another file named "optdepends" which follows the same format, minus the %PROVIDES% section. Each file in the %DEPENDS% section would be treated as a genuine dependency. The "optdepends" file would be created either by pacman or some other tool. As an example of usage, after (or during) installation of a package with optional dependencies, the user would be able to add some of the package's optional dependencies to "optdepends" so that they can be handled transparently as if they were true dependencies. In order for this to be truly useful, optdepends would need to persist across database updates. If they were included in the sync or local databases, they would need to be saved and restored with each update. That shouldn't require too much overhead because very few packages would have them, and it would only affect users who were using them. The one issue that I see with that approach is checking for the files during an update (but even that should have a negligible impact, especially considering how relatively infrequent database updates are). To get around that, rather than search the entire database, the files could either be indexed somewhere or maybe given their own location (e.g. /var/lib/pacman/optdepends). Checking for a file in /var/lib/pacman/optdepends would change little from checking for a file in the same dir as the "depends" file. A further advantage of this would be the facility of creating metapackages that could be easily updated by the user. These would be empty packages with no "depends" that the user could use as a holder for optdepends. This could finally provide a true alternative to groups as well (I can't find it now, but there was a big discussion on Flyspray about getting rid of groups which seemed to have moderate dev support). The advantage of using metapackages over groups (given this way of handling optdepends) is that packages could depend on other metapackages. Users could create custom DE packages that contained their favorite DE along with some other apps thrown in without needing to specify every package in their DE, for example. From all the discussions that I've followed on this, I think this follows the KISS philosophy very well, I think it would involve a minimal change to pacman, and I think it would make the development of metapackaging and optdeps manangers trivial. This could also finally turn "optdepends" into truly optional dependencies, instead of just installation messages. If I knew C or had the time to dive into it right now, I would be writing a patch already. |
This task depends upon
Closed by Allan McRae (Allan)
Sunday, 04 December 2022, 07:32 GMT
Reason for closing: Implemented
Additional comments about closing: Many things in this bug got implemented. Anything that still not handled should be a subject of a new bug report.
Sunday, 04 December 2022, 07:32 GMT
Reason for closing: Implemented
Additional comments about closing: Many things in this bug got implemented. Anything that still not handled should be a subject of a new bug report.
It seems that this is exactly how dependencies are tracked already on the local system ("pacman -S pkg" -> *checks /var/lib/pacman/sync/$repo/pkg/depends* -> etc). Having it check /var/lib/pacman/optdepends/pkg/optdepends or whatever shouldn't be a problem and it could be outside of the current local and sync database.
The only possible issues with this would be changes in dependencies on the packagers side or disappearance of optdepend packages. Pacman could display a message when it discovers redundant dependencies (e.g. optdep->dep, optdep no longer needed) and it could also continue installation if an optdep isn't found (just an idea).
The more I think about this, the more I believe that this is the ideal solution. Admittedly, I don't know what the code looks like, but all that's needed to do this should be in place already.
Sorry, I should have said that "most of" instead of "all of" in that last line. We would need a way to handle user interaction for adding/removing optdepends, either in pacman or with a separate tool. Using the existing pacman file selection dialogue, it should be possible to add a "would you like to include the optdepends?" prompt followed by the option of including all of them or being prompted for each one. The biggest change would be the addition of an argument to enable reconfiguration of the optdepends between sync operations.
The advantage of doing it all in pacman is that during a sync operation, all optdeps included in the package can be selected and installed without having to re-run the installation.
So... let's say I install openssl on my machine, and then install super-ssl-package which depends on openssl. The openssl DB entry was opened and rewritten to add:
%REQUIREDBY%
super-ssl-package
- a package has an optdepend
- you want to install the optdepend and have pacman recognise you installed as a (opt)dep of that package.
Have I got that right or have I over simplified things?
Each package has a plain-text file named "depends" which lists its dependencies.
Pacman already checks these files to figure out the package's dependencies.
We add a second file named "optdepends" which lists user-specified "optional" dependencies (same format as "depends").
When pacman checks for dependencies, it adds any dependencies in "optdepends" to it's internal dependency array that it built from "depends".
Pacman thus handles these optional dependencies exactly as though they were normal dependencies.
The dependencies listed in "optdepends" are managed by the user (through pacman or something else).
Examples:
I install gimp.
pacman says:
Optional dependencies for gimp...
gutenprint: for sophisticated printing only as gimp has built-in cups print support
libwebkit: for the help browser
poppler-glib: for pdf support
hal: for Linux input event controller module
alsa-lib: for MIDI event controller module
I want libwebkit functionality but couldn't care lessa bout the others.
I add (through a tool/manually/whatever) "libwebkit" to the dependencies in /var/lib/pacman/optdeps/gimp/optdepends
libwebkit is now transparently treated as a dependency of gimp
The idea of having /var/lib/pacman/optdeps/<pkgname>/optdepends is that this keeps the files out of the sync and local repos so there's no need to worry about persistence between updates.
Pacman already reads "<pkg>/depends" to check deps, adding a second read shouldn't be problematic.
This would work completely differently than REQUIREDBY. This comes from the "depender", not the "dependee" and uses pacman's in-place dependency resolution.
First, forget the new file. Our database back-end would be even slower ;-)
Basically it is not a bad idea imho and it would be easy to implement, and if we want to do sophisticated optdepend handling we probably need something like this. (I imagine here an interactive GUI asking "Which optional dependencies do you want to resolve?".) But this may conflict with KISS, I dunno. And I don't know what we should do with "user defined" dependencies during a package update. (Keep them? Probably, because any other answer would be worse here.)
If the user can't select them, then they're not optional.
The only reason I suggest using a separate file is to avoid persistence issues during database updates. If they're not kept across database updates and handles as normal dependencies during sync operations, they're pointless. As for the GUI, it would be simple for the community to make tools to handle this. I would hope for a simple dialogue in pacman to begin with. Shaman would probably add this later.
As it stands now, there is no handling of optdepends. The optdepends array of PKGBUILDs are just installation messages. You could write whatever you want there, it makes no difference. This would definitely turn optdepends into something useful.
As for KISS, if there is a dialogue during installation that prompts for optdepends the same way pacman prompts for packages when installing a group (after all, this would enable groups to be replaced with metapackages containing only optdepends, as I mentioned in the OP), then there is no loss of simplicity whatsoever. In fact, simplicity is gained from removing "groups" from pacman, which seems to be a kludge originally.
http://bbs.archlinux.org/viewtopic.php?pid=478808#p478808
I don't even think we need all this at the front end. e.g. I install tk using --asdep as an optdep for python. We can't keep track of this in the tk file (too much like the old REQUIREDBY so no...) so it must be in the python file. What about just recording the OPTDEPENDS for a package in the "depends" file in the local db and taking these into account when assessing what packages are orphans and displaying package information.
e.g.
> pacman -Qi tk
Name : tk
<blah>
Depends On : tcl=8.5.6 libxss libxft
Optional Deps : None
Required By : None
Optional For : python
<blah>
A package would not be listed with "pacman -Qtd" unless it was not a dep or an optdep. I think we would need a method so that optdep status could be ignored.
Maybe there is scope for adding a flag to query the install of optdepends during and install/update but _never_ by default.
That said, I like your idea of adding the "Optional For" field and having pacman exclude packages from the orphans list if they're listed as optional dependencies for something else. The limitation with this approach is that it would not be possible for the user to assign optional dependencies arbitrarily. Let me give an example.
I decide to create a custom desktop environment metapackage that contains my WM and several other packages which I find useful as optdepends (no depends). I distribute this package in my public repo and anyone who installs it is able to pick and choose which optdepends they install. So far, your suggestion, with or without the frontend part, works. Now Bob looks through the list and realizes that his text editor of choice isn't there, but he really wants his text editor as part of the DE metapackage. With my method, he can easily add it to the local optdepends of the package and have it persist across updates. He could even migrate his optdepends database across machines as needed.
As for modifying the PKGBUILD, that would require Bob to modify it each and every database update. That seems unnecessary to me.
What about an "--asoptdep <pkg>" option instead, coupled with your suggestion? Changing optdepends locally would still be a problem with this method though because pacman has no "modify" operation... you would need to re-install the package to pass it new "asoptdep <pkg>" options.
So far I still think that my original idea is the cleanest and most adaptable. It basically enables optdepends to be selectively converted to depends on the fly for all practical purposes and provides the user with the ability to fully manage them without requiring it. On the frontend pacman would only really need to handle the metapackage dialogue if they're to replace groups. Other tools *could* be supplied to handle this, although it really seems like it's pacman's job.
Updated suggestion:
Add an optdepends database in /var/lib/pacman/optdepends/
Add a function to pacman to check if /var/lib/pacman/optdepends/pkgname/optdepends exists and to append packages in it to the internal depends array for the given package so that they are handles transparently as depends
Add a "--getoptdeps" flag to pacman to enable an optdepends dialogue similar to the current group dialogue, where the user is prompted on what to install
Benefits:
This leaves the default behavior unchanged.
The only extra IO for all users is a quick check if a file exists for each package during a sync operation, which should be negligible (How much does [ -e /path/to/file ] in C really cost?).
If a user has added optdepends, then the file is read with the same function that currently reads the depends files, so no need for new code to handle parsing the file.
Appending the packages in the optdepends file to the internal depends array should be trivial, and yet this resolves all future dependency problems using the existing dependency handlers... elegant and simple.
The user has complete control over local optdepends, which equals more control over pacman and package management. This opens the doors for quite a few things and makes Arch itself more versatile.
Pacman already contains dialogues for installing packages from a group. Adapting this to handle optdepends shouldn't require more than a few changes.
Current default frontend behavior is not changed.
The only thing that might require some completely new code would be a way to determine if a package is just a metapackage (no depends, no content, just optdepends) and to automatically invoke the optdepends dialogue. Even in that case, adding a simple flag to a package to designate it as a metapackage would not be more complex than the way that groups currently work.
We only added optdepends in order to have an uniform / standardized way to display optdependencies, and to avoid the need of scriptlets for that.
Before, everyone had to use scriptlets and print the optional dependencies in a different way every time.
Also there were only displayed on install/upgrade, and everything was always displayed.
Now we can also have a smarter handling, like only displaying the new optdepends on upgrade. And all optdepends can be queried at any time with -Q.
That is why I liked optdepends : very simple code and very low complexity thanks to the simple/basic handling, with all the benefits I just described.
I don't understand how my suggestion impinges on any of the benefits that you mentioned.
Honestly, what seems more "complex" to you? The current scenario:
I want libwebkit with gimp.
I install gimp and libwebkit "pacman -S gimp && pacman -S libwebkit --asdeps" or "pacman -S gimp libwebkit".
I have to manually remove libwebkit if I remove gnome.
As the number of installed optdepends increases, I have to manage more and more dependencies manually and I have to deal with more and more packages cluttering "pacman -Qet" or "pacman -Qdt" and denying me a clear overview of my system
or with my suggestion:
I want libwebkit with gimp.
I install gimp with the optdepends dialogue: "pacman -S gimp --getoptdeps".
Pacman now handles libwebkit.
As the number of isntalled optdepends increases, I don't have to do any extra work nor deal with clutter in my package lists.
The actual changes needed to implement this are very small and it doesn't effect users who don't want optdepends.
At the same time, it makes pacman very versatile and opens the doors to metapackages to replace groups as well, which has a very clear benefit when it comes to updating groups/metapackages.
sorry
Let's talk about the user-friendly side now. From my user POV, I absolutely *hate* interactivity.
The interactivity you find amazing when pacman asks you which packages to install from a group is my worst nightmare. I would never ever use that, and I am totally for removing this completely. It would remove unneeded cruft, and reduce the code complexity as well.
If I want to install a group partially, depending on the number I want to install, I would either select a few targets manually, or ignore a few targets manually.
And I much prefer to do the same for managing optdepends.
And actually, if you think about how this could be implemented in a gui, it would be more similar to what we have now. A logical gui would let you select with the mouse which packages to install from a group, or which optdepends to install with a package, very much like when you currently select with pacman, using manual copy/pasting or typing. In both cases, there is no stupid annoying questions asked to the user :)
However, there is a big difference between the gui and the console frontend here. In the gui case, there is an easy link made between the package and its optdepends selected by the user. But in the console case, you do not have this link. Unless we add some crazy flags like : "pacman -S libwebkit --asdepsof gnome"
And since I don't like any ways of installing optdepends with pacman while keeping track of the parent package, I would just give up with this :)
As for the dialogue, you wouldn't need to use it. With this suggestion it becomes trivial to write tools to handle the optdepends. It would even be very simple to do it manually. The dialogue should remain for those users who want it (I would expect that most people use it for groups and would as well for optdepends (which metapackages would use if they replaced groups)) but you would be free to manage them as you wish.
My suggestion empowers the user without changing default behavior for anyone who doesn't want to bother with optdepends.
But, as always, you're welcome to prove me/us wrong and send some patches. I just don't personally feel like it's a good idea.
The fact that an optdep is neither an explicit target nor a pure dependency (causing issues with -Qet or -Qtd and orphan removals)
And the fact that groups are not properly tracked.
So I wouldn't say your proposal is crap and I wouldn't bury it, but I wouldn't push for it either.
I just believe these issues are not so important and that there are other priorities, and that I am fine at least for now with a simple (and possibly too basic) optdepends handling, which is still enough for solving the problems we originally had.
Reuse the "depends" file loader.
Check if a file exists.
Append one list to another in pacman's memory.
Reuse the group dialogue for optdepends.
Add a command-line flag.
I honestly don't think that would involve much complexity, plus it would eliminate any complexity from checking for "groups".
I accept that this suggestion has been rejected now though. I will say that I believe most people have approached this idea with preconceptions that have clouded its understanding, but that's to be expected with the load of suggestions that you must deal with. I do find it a shame though.
Btw, is there any documentation to alpm other than reading pacman's source code?
[1] https://wiki.archlinux.org/index.php/User:Allan/Pacman_OptDepends
[2] https://bbs.archlinux.org/viewtopic.php?pid=478808
1. If you install package A and then install its optdepend B, and another installed package C also optdepends on B, B won't get removed when A is removed with -Rs.
2. If you install package A (to use it standalone) and later install B which optdepends on A, when you remove B with -Rs, A will get uninstalled.
However, explictly tracking the optdepends relationship would hassle the end-user (and these corner cases won't come up very frequently), so I think the optdepends branch is good enough.
Also, point #1 is wrong given the wiki page does not reflect the actual state of optdep handling at all. The only patches that are in line for the 4.1 release are at [1], because the full patchset that was submitted and has the features described on that wiki page had issues that were never fixed. I have addressed the issues in the parts that provide increased notifications about optdepends.
[1] https://projects.archlinux.org/users/allan/pacman.git/log/?h=working-split/optdep
So... "Not all the issues here are addressed yet. My branch improves the situation, but there is still more to be done."