FS#68757 - [ruby] set GEM_HOME on the environment to avoid inconsistencies

Attached to Project: Arch Linux
Opened by Felipe Contreras (felipec) - Friday, 27 November 2020, 02:38 GMT
Last edited by Buggy McBugFace (bugbot) - Saturday, 25 November 2023, 20:24 GMT
Task Type Feature Request
Category Packages: Extra
Status Closed
Assigned To Anatol Pomozov (anatolik)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

The instructions in the wiki to allow user installations of gems mention the --user-install option, but this is ignored by the bundler program.

First, the instructions say to do this to disable building the documentation:

~/.gemrc:
gem: --no-document

But this would override the system's /etc/gemrc, which has --user-install, so the user actually has to do:

~/.gemrc:
gem: --user-install --no-document


Then the instructions proceed to explain --user-install, and that the gems end up in `~/.gems/ruby`.

Then the instructions for bundler are mentioned, and the following command is suggested:

bundle config path ~/.gem

However, that's not the same path gem uses (`~/.gem/ruby`).

The actual path can be found running the command `ruby -e 'puts Gem.user_dir'` (e.g. `~/.gem/ruby/2.7.0`).

So the instructions are wrong.

I and others have contacted the ruby/bundler developers multiple times over the years, and they have refused to fix the issues:

* https://github.com/rubygems/rubygems/issues/4027
* https://github.com/rubygems/bundler/issues/710
* https://github.com/rubygems/bundler/issues/2565
* https://github.com/rubygems/bundler/issues/2667

There is a very simple solution to get around all these problems, just distribute a script for profile.d:

/etc/profile.d/ruby.sh
export GEM_HOME="$(ruby -e 'puts Gem.user_dir')"

This way no --user-install is needed: `/etc/gemrc` can be dropped, and all the instructions for `~/.gemrc` and the bundle path config (which is wrong), since bundle uses $GEM_HOME by default.

Additionally, we can modify the PATH so the user doesn't have to:

export PATH="$PATH:$GEM_HOME/bin"

So the instructions about the setup can be removed.

So this fixes everything:

/etc/profile.d/ruby.sh
export GEM_HOME="$(ruby -e 'puts Gem.user_dir')"
export PATH="$PATH:$GEM_HOME/bin"
This task depends upon

Closed by  Buggy McBugFace (bugbot)
Saturday, 25 November 2023, 20:24 GMT
Reason for closing:  Moved
Additional comments about closing:  https://gitlab.archlinux.org/archlinux/p ackaging/packages/ruby/issues/1
Comment by Eli Schwartz (eschwartz) - Sunday, 06 December 2020, 15:02 GMT
Are you just doing this a lot now? Same behavior in  FS#68868 

In linked issue:
> I will take this ticket as evidence and suggest a hacky solution downstream to my distribution maintainers, since you leave us no other choice.

Here:
> they have refused to fix the issues:

Upstream is actually just annoyed that you spend more time arguing over whether gem should have implemented something bundler later doesn't respect. So they locked your issue. But that does NOT constitute refusing to fix the issue, since they opened https://github.com/rubygems/rubygems/issues/4031 instead to discuss it with fewer personal attacks.

Since it seems logical that upstream is in fact going to fix the issue, and you know it because you even participated in the new issue, I propose you document your workaround on the wiki while we wait for a builtin method that upstream is actively working on. There's really no reason your user configuration for bundler *needs* to be directly packaged in the rubygems package.
Comment by Eli Schwartz (eschwartz) - Sunday, 06 December 2020, 15:07 GMT
Note that if you're going to make a regular habit of getting into bad faith fights with upstream projects with the sole goal of not solving anything, then threatening to get Arch to downstream patch the behavior in strict violation of our patch policy, we might just decide your bug reports here are no longer worth it, and disable your account.

Please stop speaking on behalf of Arch when you say we "will" do whatever you are threatening them with.
Comment by Felipe Contreras (felipec) - Tuesday, 04 May 2021, 20:05 GMT
  • Field changed: Percent Complete (100% → 0%)
It is not my issue, the first bug report was **TEN** years ago: https://github.com/rubygems/bundler/issues/710.

And it would not be fixed. If instead of censoring discussion you allowed for at least one reply before closing I could explain why.

But do whatever you want. It's only the Arch Linux users that suffer.

https://wiki.archlinux.org/index.php/Talk:Ruby
Comment by Andreas Radke (AndyRTR) - Tuesday, 04 May 2021, 20:06 GMT
May our distribution package maintainer be informed and decide if we care at distribution level.
Comment by Felipe Contreras (felipec) - Tuesday, 04 May 2021, 21:06 GMT
Note as I predicted the "fix" hasn't happened (as it hasn't for the last ten years), and there's no update on the ticket since five months ago: https://github.com/rubygems/rubygems/issues/4031.

I would not hold my breath on it getting fixed.

We shouldn't ask **all** our users to manually workaround their issues.
Comment by Eli Schwartz (eschwartz) - Tuesday, 04 May 2021, 21:59 GMT
We also shouldn't slowly run arbitrary ruby programs in login scripts in order to set environment variables that aren't needed by most people simply installing ruby-based software via pacman.

Imagine if every program thought this was a good idea...
Comment by Felipe Contreras (felipec) - Tuesday, 04 May 2021, 22:38 GMT
Nobody is suggesting running arbitrary ruby programs.

The command has been carefully studied by multiple parties (including me) and is obviously the best solution (giving the limitations of RubyGems).

If you have something against Ruby code, I can replicate the exact same functionality using shell.
Comment by Eli Schwartz (eschwartz) - Tuesday, 04 May 2021, 22:59 GMT
I don't have a problem with "ruby code".

I do think it's a bit weak to have the instinctual reaction "this program does something I don't like! Let's solve it by forking to external disk utilities during shell initialization".

It's not quite so bad as programs that do `eval "$(fooprog init)" in bashrc, because profile scripts are only run during shell login, not every interactive shell init. But it's still not great, and I think people should think twice before doing it once, rather than getting in the habit of doing it many times.

If it's a known static location then yes, implementing this in pure sh would be preferable as that runs in-process and much faster, at a point in time (e.g. tty login) where you want to optimize for startup responsiveness.

There is also the fact that it's kind of meh to add OS wide environment variables to every process running ever, without a really excellent reason; the Environmental Protection Agency isn't happy with your suggestion. But that isn't quite as big a deal.

The larger issue is that as far as I can tell the only thing this solves is bundle, not gem, and only in the sense that it stops bundle from running sudo stuff for you which would normally get stopped by a password prompt and could be patched out anyway but would be best discussed with upstream as they were open to fixing this!

...

Aside for that, I have implementation comments: there is an append_path function that is supposed to be used for profile scripts to avoid duplicate entries, and given this directory often will not exist it would be nice to add it to PATH only if it does.
Comment by Felipe Contreras (felipec) - Tuesday, 04 May 2021, 23:44 GMT
Nobody is saying "this program does something I don't like". What I am saying is that this program does something demonstrably wrong.

And people have thought way more than twice about solutions, for **TEN** years.

I have worked on this issue extensively, and I've sent patches to the rubygems projects to no avail, if you have a better solution let me know. And if you do, why haven't you updated the Ruby wiki page explaining that solution?

We do have a good reason to add the environment variable, as I have already explained in the long details above.

The problem you describe with bundle is not the actual problem. Read again my details above.

And it is not a static location, it depends, I'm attaching a shell script that does basically the same.

Comparing how much each solution takes:

ruby -e 'puts Gem.user_dir' 0.05s user 0.01s system 98% cpu 0.060 total

sh gem_home 0.00s user 0.00s system 94% cpu 0.009 total
   gem_home (0.2 KiB)
Comment by Eli Schwartz (eschwartz) - Wednesday, 05 May 2021, 00:17 GMT
> The problem you describe with bundle is not the actual problem. Read again my details above.

You provided four links, 3 of which complain that 'bundle install' runs sudo, and one which complains that bundle doesn't respect --user-install and therefore targets /usr/ instead of $HOME/

The fact that bundle uses /usr/ is of course only a problem because it succeeds, and it only succeeds because it uses sudo, right?

> I've sent patches to the rubygems projects to no avail

From 2013 at least you already created your reputation in the bundler developer community, which definitively involved you claiming you were morally unwilling to submit patches:

Project member:
> If you'd like to add a --user-install option to Bundler that sets the installed path to Gem.user_dir, I would be happy to merge it.

you:
> I would do it if I had any trust in the project. The first time I use Bundler it doesn't work. I report a problem that clearly would affect all Arch Linux users and it gets immediately closed. Why would I even use Bundler? I can just grep the Gemfile and use gem install directly. So much for Bundler's convenience.


me:
> Well gee, I wonder why felipec is so mad that bundler doesn't want to tie itself to the gem implementation and config file, and responds by "losing all trust in the project" and refusing to take up their offer of a patch to add a similar option to bundler, which would be read from e.g. $BUNDLE_USER_INSTALL (settable in /etc/profile.d/bundler.sh) or $HOME/.bundle/config

But as we all know, you did not submit this Arch ticket out of an interest in making ruby or gem or bundler better, but out of an interest in "sticking it to the rubygems team". I guess if you really cared, you'd have submitted the patch for bundler in 2013 and the solution here would have been infinitely easier, and probably present for the better part of a decade to boot.

But of course you insist you've tried so hard and sent patches in trying to be helpful, Mr. I Would Submit Patches If I Had Any Trust In The Project
Comment by Felipe Contreras (felipec) - Wednesday, 05 May 2021, 05:33 GMT
> The fact that bundle uses /usr/ is of course only a problem because it succeeds, and it only succeeds because it uses sudo, right?

Wrong.

> From 2013 at least you already created your reputation in the bundler developer community, which definitively involved you claiming you were morally unwilling to submit patches:

I have contributed at least 2700 commits to 28 open source projects, including Ruby MRI itself (which didn't track my contributions).

https://www.openhub.net/accounts/felipec

Stop the ad hominem attacks.

> Well gee, I wonder why felipec is so mad that bundler doesn't want to tie itself to the gem implementation and config file

You don't know what I want.

> refusing to take up their offer of a patch to add a similar option to bundler

That offer does not help me, nor the Arch Linux community. Adding a --user-install option to Bundler does **NOT** fix this issue.

> But as we all know, you did not submit this Arch ticket out of an interest in making ruby or gem or bundler better

Now you are accusing me of bad faith. This is objectively the worst way to reply to a contributor. I am done replying to you.

Anyone who actually wants to fix the actual problem (which Eli isn't even interested in understanding) let me know.
Comment by Eli Schwartz (eschwartz) - Wednesday, 05 May 2021, 05:44 GMT
> Now you are accusing me of bad faith. This is objectively the worst way to reply to a contributor. I am done replying to you.

Incorrect, I am not now accusing you of bad faith.

I accused you of bad faith on December 06, 2020 and haven't seen any reason to change my mind.

> I have contributed at least 2700 commits to 28 open source projects

What does this have to do with the bundler project, which you refused to contribute to with the message:

> I would do it if I had any trust in the project. The first time I use Bundler it doesn't work. I report a problem that clearly would affect all Arch Linux users and it gets immediately closed. Why would I even use Bundler? I can just grep the Gemfile and use gem install directly. So much for Bundler's convenience.
Comment by Eli Schwartz (eschwartz) - Wednesday, 05 May 2021, 05:56 GMT
If you disagree with my analysis of the problem, you will need to do so using more words than just "Wrong".

If all you can say is "no, you're wrong", but you don't specify what is "right" instead, then all we can possibly judge this ticket based on, is the analysis that analyzed, i.e. mine. And you may find this ticket being closed again.

Protip: talking all about how the other person engages in ad hominem, then engaging in it yourself ("which Eli isn't even interested in understanding" as an excuse to not explain your issue) doesn't really work out so great. But you could always try to NOT be the person that submitted:

 FS#68868 
https://gitlab.gnome.org/GNOME/vte/-/issues/319

and then did the exact same shenanigans here with your attitude toward the rubygems/bundler developers.
Comment by Felipe Contreras (felipec) - Thursday, 20 May 2021, 21:02 GMT
To reiterate what has already been explained very clearly:

This task will be resolved when:

1. gem does user installs by default (not system install)
2. bundler does user installs by default (not system install)
3. both gem and bundler use the same location

As of today only #1 is fulfilled.

The proposed solution:

export GEM_HOME="$(ruby -e 'puts Gem.user_dir')"

In /etc/profile.d/ruby.sh does all of the above.

No other solution has been suggested to fix all of the above.

Originally this task was closed with the rationale "upstream", presumably suggesting upstream will fix this, which is clearly not true. Supposedly the upstream task that is going to fix this is https://github.com/rubygems/rubygems/issues/4031, but that is not true, because it would not necessarily fix all the 3 points above. There is no patch, and no comment since December 2020.

Even if rubygems issue 4031 is fixed, we would still need to set an environment variable like BUNDLE_NO_SUDO, so we still need /etc/profile.d/ruby.sh. And of course the fact that bundler doesn't do sudo doesn't fix *any* of the points above.

Nobody ever suggested that bundler succeeding in installing gems in /usr was the problem.

Now, I have proposed 3 solutions:

1. Set GEM_HOME in /etc/profile.d
2. Apply a patch like the one I sent to rubygems: https://github.com/rubygems/rubygems/pull/4028 (closed with no reason given)
3. Override Gem.default_dir by the functionality rubygems gives packagers: operating_system.rb (see  FS#70961 )

*Nobody* else has provided another patch, or another solution, not even upstream rubygems developers.
Comment by Eli Schwartz (eschwartz) - Thursday, 20 May 2021, 22:43 GMT
1) this is still not an easy or safe fix as it results in incorrect behavior after upgrading ruby but before you log out and back in

2) I don't see the utility of this, it merely gives another way of specifying the cli option from gemrc.

3) seems plausible. As a matter of curiosity how would you then specify the system install dir (e.g. when packaging in a PKGBUILD I guess?)

> Nobody ever suggested that bundler succeeding in installing gems in /usr was the problem.

I certainly suggested it. If the default is for gem to install to the user dir, and bundler to abort with a permission error unless manually directed to the user dir, then that's a valid and reasonable solution to the problem as far as a distro is concerned, and if people want bundler to work out of the box then that's an upstream bug we don't care about.

Consistency is maintained, and bundler then acts the same way python-pip has since time immemorial until relatively recently when it learned to log a warning and switch to the python user dir upon seeing a permission error in the system dir.
Comment by Felipe Contreras (felipec) - Friday, 21 May 2021, 11:52 GMT
And now there's three more:

4. A whole branch with patches, tests, documentation, an update to `gem env`, and bug fixes completely ready for upstream to pick up as-is

https://github.com/felipec/rubygems/tree/fc/user-install

Then all Arch Linux packagers have to do is:

def user_install
true
end

Or even better: `Process.uid != 0` to turn this off automatically for the root user.

The whole thing is:

lib/rubygems/commands/environment_command.rb | 5 +++++
lib/rubygems/commands/uninstall_command.rb | 2 +-
lib/rubygems/defaults.rb | 8 ++++++++
lib/rubygems/installer.rb | 20 ++++++++++++--------
lib/rubygems/installer_test_case.rb | 12 ++++--------
lib/rubygems/path_support.rb | 4 +++-
lib/rubygems/test_case.rb | 2 +-
lib/rubygems/uninstaller.rb | 17 ++++++++++++-----
test/rubygems/test_gem.rb | 10 ++++++++++
test/rubygems/test_gem_commands_uninstall_command.rb | 4 ++--
test/rubygems/test_gem_installer.rb | 18 +++++++++---------
test/rubygems/test_gem_uninstaller.rb | 21 ++++++++++++++++++++-
12 files changed, 87 insertions(+), 36 deletions(-)

5. A simple patch that only adds `Gem.user_install` so that distributions can easily pick it up and enable in their operating_system.rb

https://github.com/felipec/rubygems/commit/1dfb46a174e06cb940cc052206eb10ea10bcea1a

6. An even simpler patch that does the same as above without any need for operating_system.rb:

- @home = env["GEM_HOME"] || Gem.default_dir
+ @home = env["GEM_HOME"] || Gem.user_dir

All these solutions work perfectly fine, and all other comments are wrong, because 1) gemrc is ignored by bundler, and 2) to install gems in the system directory all a user has to do is --no-user-install.

Since rubygems maintainers have blocked me I sent a mail with a couple of patches to the ruby-talk mailing list, and a link the the branch in my GitHub repository.

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/441137

Nobody else has proposed any solution, much less a single patch.


And to demonstrate how the rubygems handling of user installs is so bad, here's a list of all the *huge* modifications all distributions do, because they have given up on rubygems doing something sensible:

Debian: 110 lines:

https://salsa.debian.org/ruby-team/rubygems-integration/-/blob/master/lib/rubygems/defaults/operating_system.rb

Fedora: 148 lines:

https://src.fedoraproject.org/rpms/rubygems/raw/rawhide/f/operating_system.rb

Gentoo: 63 lines:

https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-ruby/rubygems/files/gentoo-defaults-4.rb

But that's only one of the default files, Gentoo has *three*.

This is the kind quality commits that have led us here:

https://github.com/rubygems/rubygems/commit/6d33ec4a91

That's when they introduced the user installs: if Gem.dir isn't writable, just install to ~/.gem.
Comment by Felipe Contreras (felipec) - Friday, 02 September 2022, 19:16 GMT
It's been more than a year, are you ready to accept upstream isn't going to fix this problem any time soon, just like they haven't done in the past 12 years?

I wrote a blog post explaining my solution:

https://felipec.wordpress.com/2022/08/25/fixing-ruby-gems-installation/

My new proposal is even simpler now:

https://github.com/felipec/rubygems/commit/39f6721bce

All Arch Linux would need to do is distribute an `operating_system.rb` file overriding Gem.default_install so it returns Gem.user_dir. All distributions already provide an `operating_system.rb` file, which was created by RubyGems developers precisely for this purpose: to override the defaults.

Another option would be to patch the code to change Gem.default_dir to Gem.user_dir in Gem::PathSupport.initialize which achieves the same purpose. It wouldn't be a clean solution, but certainly not any less clean than the current patch to Gem.install_extension_in_lib, which BTW is the wrong way to do it, the correct way is overriding the method in `operating_system.rb` just like Fedora does.

Right now if users don't configure GEM_HOME `bundle install` will try to install gems in system directories using sudo.

How much longer are you going to keep bothering Arch Linux users in the hopes that upstream fixes this?

There is no need to wait, you are already patching rubygems, just add another patch to fix the problem in the meantime. If and when upstream fixes the problem, then you remove the patch.

If you are interested in fixing this issue I can provide you with a solution that deals with all the problems in a clean way.
Comment by Felipe Contreras (felipec) - Monday, 27 February 2023, 16:56 GMT
@eschwartz Still think they are going to fix it? It's 2023 now, **thirteen** years after the first bug report.
Comment by Buggy McBugFace (bugbot) - Tuesday, 08 August 2023, 19:11 GMT
This is an automated comment as this bug is open for more then 2 years. Please reply if you still experience this bug otherwise this issue will be closed after 1 month.
Comment by Alexandre ZANNI (noraj) - Monday, 04 September 2023, 10:56 GMT
If it's still something considered, I suggest export GEM_HOME="$(gem env user_gemhome)" rather than export GEM_HOME="$(ruby -e 'puts Gem.user_dir')".

PS: the current discussion: https://github.com/rubygems/rubygems/issues/5682

Loading...