Pacman

Welcome to the Pacman bug tracker. Please search the current bugs and feature requests before filing a new one! Use advanced search and select "Search in Comments".

* Please select the correct category and version.
* Write a descriptive summary, background info, and provide a reproducible test case whenever possible.
Tasklist

FS#12708 - Pacman: A suggestion for handling optdepends.

Attached to Project: Pacman
Opened by Xyne (Xyne) - Thursday, 08 January 2009, 02:41 GMT
Last edited by Aaron Griffin (phrakture) - Thursday, 08 January 2009, 04:08 GMT
Task Type Feature Request
Category Backend/Core
Status Unconfirmed
Assigned To No-one
Architecture All
Severity Low
Priority Normal
Reported Version None
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 14
Private No

Details

I've posted a thread about this here: http://bbs.archlinux.org/viewtopic.php?pid=476658

Both the sync and local databases contain depends file to keep track of their dependencies with the following simple format:
Code:

%DEPENDS%
foo
bar
this
that

%PROVIDES%
whatever

It should be relatively easy to add another file named "optdepends" which follows the same format, minus the %PROVIDES% section. Each file in the %DEPENDS% section would be treated as a genuine dependency. The "optdepends" file would be created either by pacman or some other tool. As an example of usage, after (or during) installation of a package with optional dependencies, the user would be able to add some of the package's optional dependencies to "optdepends" so that they can be handled transparently as if they were true dependencies.

In order for this to be truly useful, optdepends would need to persist across database updates. If they were included in the sync or local databases, they would need to be saved and restored with each update. That shouldn't require too much overhead because very few packages would have them, and it would only affect users who were using them. The one issue that I see with that approach is checking for the files during an update (but even that should have a negligible impact, especially considering how relatively infrequent database updates are). To get around that, rather than search the entire database, the files could either be indexed somewhere or maybe given their own location (e.g. /var/lib/pacman/optdepends). Checking for a file in /var/lib/pacman/optdepends would change little from checking for a file in the same dir as the "depends" file.

A further advantage of this would be the facility of creating metapackages that could be easily updated by the user. These would be empty packages with no "depends" that the user could use as a holder for optdepends. This could finally provide a true alternative to groups as well (I can't find it now, but there was a big discussion on Flyspray about getting rid of groups which seemed to have moderate dev support). The advantage of using metapackages over groups (given this way of handling optdepends) is that packages could depend on other metapackages. Users could create custom DE packages that contained their favorite DE along with some other apps thrown in without needing to specify every package in their DE, for example.

From all the discussions that I've followed on this, I think this follows the KISS philosophy very well, I think it would involve a minimal change to pacman, and I think it would make the development of metapackaging and optdeps manangers trivial. This could also finally turn "optdepends" into truly optional dependencies, instead of just installation messages.

If I knew C or had the time to dive into it right now, I would be writing a patch already.
This task depends upon

Comment by Xavier (shining) - Thursday, 08 January 2009, 08:53 GMT
Doing some kind of dependency tracking in the local db reminds me of the REQUIREDBY field which caused us many problems and that we finally got rid of.
Comment by Xyne (Xyne) - Thursday, 08 January 2009, 09:54 GMT
Can you be more specific about how REQUIREDBY worked and what problems it caused? What do you consider the "local" db? Everything in /var/lib/pacman/, or just /var/lib/pacman/local/?

It seems that this is exactly how dependencies are tracked already on the local system ("pacman -S pkg" -> *checks /var/lib/pacman/sync/$repo/pkg/depends* -> etc). Having it check /var/lib/pacman/optdepends/pkg/optdepends or whatever shouldn't be a problem and it could be outside of the current local and sync database.

The only possible issues with this would be changes in dependencies on the packagers side or disappearance of optdepend packages. Pacman could display a message when it discovers redundant dependencies (e.g. optdep->dep, optdep no longer needed) and it could also continue installation if an optdep isn't found (just an idea).

The more I think about this, the more I believe that this is the ideal solution. Admittedly, I don't know what the code looks like, but all that's needed to do this should be in place already.
Comment by Xyne (Xyne) - Thursday, 08 January 2009, 10:01 GMT
*edit*
Sorry, I should have said that "most of" instead of "all of" in that last line. We would need a way to handle user interaction for adding/removing optdepends, either in pacman or with a separate tool. Using the existing pacman file selection dialogue, it should be possible to add a "would you like to include the optdepends?" prompt followed by the option of including all of them or being prompted for each one. The biggest change would be the addition of an argument to enable reconfiguration of the optdepends between sync operations.

The advantage of doing it all in pacman is that during a sync operation, all optdeps included in the package can be selected and installed without having to re-run the installation.
Comment by Aaron Griffin (phrakture) - Thursday, 08 January 2009, 20:49 GMT
REQUIREDBY used to keep track of, well what the "Required By" lists in a pacman -Qi now.

So... let's say I install openssl on my machine, and then install super-ssl-package which depends on openssl. The openssl DB entry was opened and rewritten to add:
%REQUIREDBY%
super-ssl-package
Comment by Allan McRae (Allan) - Friday, 09 January 2009, 03:15 GMT
So lets see if I can reduce this to a very small summary.

- a package has an optdepend
- you want to install the optdepend and have pacman recognise you installed as a (opt)dep of that package.

Have I got that right or have I over simplified things?
Comment by Xyne (Xyne) - Friday, 09 January 2009, 04:52 GMT
That's somewhat over-simplified imo.

Each package has a plain-text file named "depends" which lists its dependencies.
Pacman already checks these files to figure out the package's dependencies.
We add a second file named "optdepends" which lists user-specified "optional" dependencies (same format as "depends").
When pacman checks for dependencies, it adds any dependencies in "optdepends" to it's internal dependency array that it built from "depends".
Pacman thus handles these optional dependencies exactly as though they were normal dependencies.

The dependencies listed in "optdepends" are managed by the user (through pacman or something else).

Examples:
I install gimp.
pacman says:
Optional dependencies for gimp...
gutenprint: for sophisticated printing only as gimp has built-in cups print support
libwebkit: for the help browser
poppler-glib: for pdf support
hal: for Linux input event controller module
alsa-lib: for MIDI event controller module

I want libwebkit functionality but couldn't care lessa bout the others.
I add (through a tool/manually/whatever) "libwebkit" to the dependencies in /var/lib/pacman/optdeps/gimp/optdepends
libwebkit is now transparently treated as a dependency of gimp


The idea of having /var/lib/pacman/optdeps/<pkgname>/optdepends is that this keeps the files out of the sync and local repos so there's no need to worry about persistence between updates.
Pacman already reads "<pkg>/depends" to check deps, adding a second read shouldn't be problematic.


This would work completely differently than REQUIREDBY. This comes from the "depender", not the "dependee" and uses pacman's in-place dependency resolution.

Comment by Nagy Gabor (combo) - Saturday, 10 January 2009, 16:17 GMT
This is a request for "user defined dependencies". Right?

First, forget the new file. Our database back-end would be even slower ;-)

Basically it is not a bad idea imho and it would be easy to implement, and if we want to do sophisticated optdepend handling we probably need something like this. (I imagine here an interactive GUI asking "Which optional dependencies do you want to resolve?".) But this may conflict with KISS, I dunno. And I don't know what we should do with "user defined" dependencies during a package update. (Keep them? Probably, because any other answer would be worse here.)
Comment by Xyne (Xyne) - Sunday, 11 January 2009, 11:33 GMT
"user-defined dependencies" = "optdepends"
If the user can't select them, then they're not optional.

The only reason I suggest using a separate file is to avoid persistence issues during database updates. If they're not kept across database updates and handles as normal dependencies during sync operations, they're pointless. As for the GUI, it would be simple for the community to make tools to handle this. I would hope for a simple dialogue in pacman to begin with. Shaman would probably add this later.

As it stands now, there is no handling of optdepends. The optdepends array of PKGBUILDs are just installation messages. You could write whatever you want there, it makes no difference. This would definitely turn optdepends into something useful.

As for KISS, if there is a dialogue during installation that prompts for optdepends the same way pacman prompts for packages when installing a group (after all, this would enable groups to be replaced with metapackages containing only optdepends, as I mentioned in the OP), then there is no loss of simplicity whatsoever. In fact, simplicity is gained from removing "groups" from pacman, which seems to be a kludge originally.
Comment by Xyne (Xyne) - Sunday, 11 January 2009, 18:04 GMT
I've added a lengthy post to the forum thread which clarifies this a bit more and includes intended dialogue examples.
http://bbs.archlinux.org/viewtopic.php?pid=478808#p478808
Comment by Borromini (Borromini) - Sunday, 11 January 2009, 23:59 GMT
I think this is an excellent idea Xyne :).
Comment by Allan McRae (Allan) - Monday, 12 January 2009, 16:22 GMT
I have had a look at your forum thread and the example dialogue you give and my main comment is that is going to make updating so much more interactive. As more packages get optdepends, this will be really annoying and I am very much against it. With groups it is OK because these are normally one of installs but I get optdepends presented most updates.

I don't even think we need all this at the front end. e.g. I install tk using --asdep as an optdep for python. We can't keep track of this in the tk file (too much like the old REQUIREDBY so no...) so it must be in the python file. What about just recording the OPTDEPENDS for a package in the "depends" file in the local db and taking these into account when assessing what packages are orphans and displaying package information.

e.g.
> pacman -Qi tk
Name : tk
<blah>
Depends On : tcl=8.5.6 libxss libxft
Optional Deps : None
Required By : None
Optional For : python
<blah>

A package would not be listed with "pacman -Qtd" unless it was not a dep or an optdep. I think we would need a method so that optdep status could be ignored.

Maybe there is scope for adding a flag to query the install of optdepends during and install/update but _never_ by default.
Comment by Xyne (Xyne) - Monday, 12 January 2009, 17:37 GMT
What about a command-line option to enable an interactive optdepends dialogue (e.g. --optdiag), along with something in the config file, and default it to pacman's current behavior. The dialogue would still be necessary for metapackages though (the new "groups").

That said, I like your idea of adding the "Optional For" field and having pacman exclude packages from the orphans list if they're listed as optional dependencies for something else. The limitation with this approach is that it would not be possible for the user to assign optional dependencies arbitrarily. Let me give an example.

I decide to create a custom desktop environment metapackage that contains my WM and several other packages which I find useful as optdepends (no depends). I distribute this package in my public repo and anyone who installs it is able to pick and choose which optdepends they install. So far, your suggestion, with or without the frontend part, works. Now Bob looks through the list and realizes that his text editor of choice isn't there, but he really wants his text editor as part of the DE metapackage. With my method, he can easily add it to the local optdepends of the package and have it persist across updates. He could even migrate his optdepends database across machines as needed.

Comment by Allan McRae (Allan) - Tuesday, 13 January 2009, 03:16 GMT
Would "Bob" not be better just adding the other package to the PKGBUILD? That seems a solution to a non-problem.
Comment by Xyne (Xyne) - Tuesday, 13 January 2009, 09:26 GMT
Dieter@be posted a link to a message that shows the problem with your suggestion of checking for optdepends when determining orphan status: http://www.nabble.com/Re%3A-%22explicit-dependencies%22%2C-a-compromise-between%09explicit-and-deps-p19971183.html

As for modifying the PKGBUILD, that would require Bob to modify it each and every database update. That seems unnecessary to me.


Comment by Allan McRae (Allan) - Tuesday, 13 January 2009, 09:44 GMT
That is why I said we would need a flag to ignore optdepends in things like "pacman -Qtd" and "pacman -Rs".
Comment by Xyne (Xyne) - Tuesday, 13 January 2009, 10:45 GMT
If foo is an optdepend for both bar and baz and we install it for bar, how would using a flag to ignore optdepends enable pacman to know that foo isn't needed/wanted for baz when we later remove bar?

What about an "--asoptdep <pkg>" option instead, coupled with your suggestion? Changing optdepends locally would still be a problem with this method though because pacman has no "modify" operation... you would need to re-install the package to pass it new "asoptdep <pkg>" options.


So far I still think that my original idea is the cleanest and most adaptable. It basically enables optdepends to be selectively converted to depends on the fly for all practical purposes and provides the user with the ability to fully manage them without requiring it. On the frontend pacman would only really need to handle the metapackage dialogue if they're to replace groups. Other tools *could* be supplied to handle this, although it really seems like it's pacman's job.


Updated suggestion:
Add an optdepends database in /var/lib/pacman/optdepends/
Add a function to pacman to check if /var/lib/pacman/optdepends/pkgname/optdepends exists and to append packages in it to the internal depends array for the given package so that they are handles transparently as depends
Add a "--getoptdeps" flag to pacman to enable an optdepends dialogue similar to the current group dialogue, where the user is prompted on what to install

Benefits:
This leaves the default behavior unchanged.
The only extra IO for all users is a quick check if a file exists for each package during a sync operation, which should be negligible (How much does [ -e /path/to/file ] in C really cost?).
If a user has added optdepends, then the file is read with the same function that currently reads the depends files, so no need for new code to handle parsing the file.
Appending the packages in the optdepends file to the internal depends array should be trivial, and yet this resolves all future dependency problems using the existing dependency handlers... elegant and simple.
The user has complete control over local optdepends, which equals more control over pacman and package management. This opens the doors for quite a few things and makes Arch itself more versatile.
Pacman already contains dialogues for installing packages from a group. Adapting this to handle optdepends shouldn't require more than a few changes.
Current default frontend behavior is not changed.

The only thing that might require some completely new code would be a way to determine if a package is just a metapackage (no depends, no content, just optdepends) and to automatically invoke the optdepends dialogue. Even in that case, adding a simple flag to a package to designate it as a metapackage would not be more complex than the way that groups currently work.
Comment by Xavier (shining) - Tuesday, 13 January 2009, 13:53 GMT
Why do we need all this complexity?

We only added optdepends in order to have an uniform / standardized way to display optdependencies, and to avoid the need of scriptlets for that.
Before, everyone had to use scriptlets and print the optional dependencies in a different way every time.
Also there were only displayed on install/upgrade, and everything was always displayed.
Now we can also have a smarter handling, like only displaying the new optdepends on upgrade. And all optdepends can be queried at any time with -Q.

That is why I liked optdepends : very simple code and very low complexity thanks to the simple/basic handling, with all the benefits I just described.
Comment by Xyne (Xyne) - Tuesday, 13 January 2009, 15:29 GMT
What I'm proposing is not complex. Maybe it appears that way from the volume of writing, but that's because I try to be thorough in explaining exactly what I mean. I also have a tendency to delve into tangents.

I don't understand how my suggestion impinges on any of the benefits that you mentioned.

Honestly, what seems more "complex" to you? The current scenario:
I want libwebkit with gimp.
I install gimp and libwebkit "pacman -S gimp && pacman -S libwebkit --asdeps" or "pacman -S gimp libwebkit".
I have to manually remove libwebkit if I remove gnome.
As the number of installed optdepends increases, I have to manage more and more dependencies manually and I have to deal with more and more packages cluttering "pacman -Qet" or "pacman -Qdt" and denying me a clear overview of my system

or with my suggestion:
I want libwebkit with gimp.
I install gimp with the optdepends dialogue: "pacman -S gimp --getoptdeps".
Pacman now handles libwebkit.
As the number of isntalled optdepends increases, I don't have to do any extra work nor deal with clutter in my package lists.

The actual changes needed to implement this are very small and it doesn't effect users who don't want optdepends.
At the same time, it makes pacman very versatile and opens the doors to metapackages to replace groups as well, which has a very clear benefit when it comes to updating groups/metapackages.
Comment by Xyne (Xyne) - Tuesday, 13 January 2009, 15:30 GMT
s/gnome/gimp/g
sorry
Comment by Xavier (shining) - Tuesday, 13 January 2009, 16:20 GMT
I am sorry my post was not clear enough. When I am talking about complexity here, I always mean the code complexity, never the user (un)friendliness.

Let's talk about the user-friendly side now. From my user POV, I absolutely *hate* interactivity.
The interactivity you find amazing when pacman asks you which packages to install from a group is my worst nightmare. I would never ever use that, and I am totally for removing this completely. It would remove unneeded cruft, and reduce the code complexity as well.
If I want to install a group partially, depending on the number I want to install, I would either select a few targets manually, or ignore a few targets manually.
And I much prefer to do the same for managing optdepends.

And actually, if you think about how this could be implemented in a gui, it would be more similar to what we have now. A logical gui would let you select with the mouse which packages to install from a group, or which optdepends to install with a package, very much like when you currently select with pacman, using manual copy/pasting or typing. In both cases, there is no stupid annoying questions asked to the user :)
However, there is a big difference between the gui and the console frontend here. In the gui case, there is an easy link made between the package and its optdepends selected by the user. But in the console case, you do not have this link. Unless we add some crazy flags like : "pacman -S libwebkit --asdepsof gnome"
And since I don't like any ways of installing optdepends with pacman while keeping track of the parent package, I would just give up with this :)
Comment by Xyne (Xyne) - Tuesday, 13 January 2009, 16:29 GMT
I really don't think that this requires an increase in the code complexity as most of what's needed is already there. I've rewritten the forum post and I address this directly in the "My Suggestion" section: http://bbs.archlinux.org/viewtopic.php?pid=476658#p476658

As for the dialogue, you wouldn't need to use it. With this suggestion it becomes trivial to write tools to handle the optdepends. It would even be very simple to do it manually. The dialogue should remain for those users who want it (I would expect that most people use it for groups and would as well for optdepends (which metapackages would use if they replaced groups)) but you would be free to manage them as you wish.

My suggestion empowers the user without changing default behavior for anyone who doesn't want to bother with optdepends.
Comment by Aaron Griffin (phrakture) - Tuesday, 13 January 2009, 16:52 GMT
For the record, I'm on Xavier's side here. I feel like this is way too much code complexity for something like this.

But, as always, you're welcome to prove me/us wrong and send some patches. I just don't personally feel like it's a good idea.
Comment by Xavier (shining) - Tuesday, 13 January 2009, 17:38 GMT
However, I do recognize there are some issues that your proposal address, and it is all indeed well presented in your forum post.
The fact that an optdep is neither an explicit target nor a pure dependency (causing issues with -Qet or -Qtd and orphan removals)
And the fact that groups are not properly tracked.

So I wouldn't say your proposal is crap and I wouldn't bury it, but I wouldn't push for it either.
I just believe these issues are not so important and that there are other priorities, and that I am fine at least for now with a simple (and possibly too basic) optdepends handling, which is still enough for solving the problems we originally had.
Comment by Xyne (Xyne) - Tuesday, 13 January 2009, 17:39 GMT
Add a database,
Reuse the "depends" file loader.
Check if a file exists.
Append one list to another in pacman's memory.
Reuse the group dialogue for optdepends.
Add a command-line flag.

I honestly don't think that would involve much complexity, plus it would eliminate any complexity from checking for "groups".

I accept that this suggestion has been rejected now though. I will say that I believe most people have approached this idea with preconceptions that have clouded its understanding, but that's to be expected with the load of suggestions that you must deal with. I do find it a shame though.

Btw, is there any documentation to alpm other than reading pacman's source code?
Comment by Rémy Oudompheng (remyoudompheng) - Tuesday, 24 May 2011, 06:19 GMT
The database format has changed since then. Is this request still valid?
Comment by Kerrick Staley (KerrickStaley) - Sunday, 18 November 2012, 03:35 GMT
Allan's optdepends branch [1] improves optdepends handling, and addresses the issue described in the forum post [2]. Should this bug be closed?

[1] https://wiki.archlinux.org/index.php/User:Allan/Pacman_OptDepends
[2] https://bbs.archlinux.org/viewtopic.php?pid=478808
Comment by Allan McRae (Allan) - Sunday, 18 November 2012, 03:39 GMT
Not all the issues here are addressed yet. My branch improves the situation, but there is still more to be done.
Comment by Kerrick Staley (KerrickStaley) - Sunday, 18 November 2012, 04:16 GMT
The main issue is that your optdepends branch doesn't explictly track which packages were installed as optdepends (and which packages optdepend on them), meaning:

1. If you install package A and then install its optdepend B, and another installed package C also optdepends on B, B won't get removed when A is removed with -Rs.
2. If you install package A (to use it standalone) and later install B which optdepends on A, when you remove B with -Rs, A will get uninstalled.

However, explictly tracking the optdepends relationship would hassle the end-user (and these corner cases won't come up very frequently), so I think the optdepends branch is good enough.
Comment by Allan McRae (Allan) - Sunday, 18 November 2012, 04:32 GMT
Point #2 is completely wrong. It is even wrong if you change "optdepends" to "depends" in the description.

Also, point #1 is wrong given the wiki page does not reflect the actual state of optdep handling at all. The only patches that are in line for the 4.1 release are at [1], because the full patchset that was submitted and has the features described on that wiki page had issues that were never fixed. I have addressed the issues in the parts that provide increased notifications about optdepends.

[1] https://projects.archlinux.org/users/allan/pacman.git/log/?h=working-split/optdep

So... "Not all the issues here are addressed yet. My branch improves the situation, but there is still more to be done."
Comment by Kerrick Staley (KerrickStaley) - Sunday, 18 November 2012, 05:05 GMT
Ah, OK. You're right, I misread the wiki regarding #2.

Loading...