FS#70306 - [puppet] incompatiblity with ruby 3.x due to URI.escape

Attached to Project: Community Packages
Opened by Simon Peeters (SimonPe) - Monday, 05 April 2021, 19:29 GMT
Last edited by Tim (bastelfreak) - Friday, 21 May 2021, 10:10 GMT
Task Type Bug Report
Category Packages: Testing
Status Closed
Assigned To Thore Bödecker (foxxx0)
Tim (bastelfreak)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 5
Private No

Details

Description:

puppet upstream (even 7.5) is currently incompatible with ruby 3.x due to usage of `URI.escape`.
It seems like upstream has put this on the backburner: https://tickets.puppetlabs.com/browse/PUP-10535

Steps to reproduce:

run any puppet command, like `puppet apply -e 'notify {"test":}'`.
the result is:
```
/usr/lib/ruby/vendor_ruby/3.0.0/puppet/util.rb:476:in `uri_encode': undefined method `escape' for URI:Module (NoMethodError)
from /usr/lib/ruby/vendor_ruby/3.0.0/puppet/util.rb:344:in `path_to_uri'
from /usr/lib/ruby/vendor_ruby/3.0.0/puppet/pops/model/ast.rb:4957:in `register_pcore_types'
from /usr/lib/ruby/vendor_ruby/3.0.0/puppet/pops.rb:120:in `<module:Puppet>'
from /usr/lib/ruby/vendor_ruby/3.0.0/puppet/pops.rb:1:in `<top (required)>'
from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from /usr/lib/ruby/vendor_ruby/3.0.0/puppet/parser/compiler.rb:8:in `<top (required)>'
from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from /usr/lib/ruby/vendor_ruby/3.0.0/puppet/parser.rb:6:in `<top (required)>'
from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from /usr/lib/ruby/vendor_ruby/3.0.0/puppet.rb:360:in `<top (required)>'
from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from /usr/lib/ruby/vendor_ruby/3.0.0/puppet/util/command_line.rb:12:in `<top (required)>'
from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from /usr/bin/puppet:4:in `<main>'
```

possible fixes:
- either we package puppet against community-testing/ruby2.7 (so we're on a similar ruby version as the official AIO packages.)
- or we patch around the issue here:
```
diff --git a/lib/puppet/util.rb b/lib/puppet/util.rb
index db1aa64006..11b153978d 100644
--- a/lib/puppet/util.rb
+++ b/lib/puppet/util.rb
@@ -473,7 +473,7 @@ module Util
orig_verbose = $VERBOSE
$VERBOSE = nil unless Puppet::Util::Platform.jruby?
begin
- encoded += URI.escape(parts[:path]) unless parts[:path].nil?
+ encoded += URI::DEFAULT_PARSER.escape(parts[:path]) unless parts[:path].nil?
ensure
$VERBOSE = orig_verbose unless Puppet::Util::Platform.jruby?
end
```
This task depends upon

This task blocks these from closing
 FS#70675 - [puppet] Does not work with ruby 3.0 
Closed by  Tim (bastelfreak)
Friday, 21 May 2021, 10:10 GMT
Reason for closing:  Fixed
Comment by M (sirffuzzylogik) - Monday, 19 April 2021, 10:29 GMT
Is there a possible workaround in the meantime? ruby2.7 is in the community repo and can be installed alongside ruby (version 3 from extra), the missing part now is making Puppet using 2.7 rather than ruby 3.
Comment by Sebastian Jaekel (bsjaekel) - Tuesday, 20 April 2021, 09:15 GMT
I have patched the file `/usr/lib/ruby/vendor_ruby/3.0.0/puppet/util.rb`, Line 476, i have replaced

```
encoded += URI.escape(parts[:path]) unless parts[:path].nil?
```

with

```
encoded += URI::DEFAULT_PARSER.escape(parts[:path]) unless parts[:path].nil?
```

Comment by Christopher Gurley (Xarin) - Tuesday, 20 April 2021, 21:52 GMT
After applying that patch to you still get this as an error, or is it just me?

Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': undefined method `unescape' for URI:Module
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: undefined method `unescape' for URI:Module
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': undefined method `unescape' for URI:Module
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: undefined method `unescape' for URI:Module
Info: Retrieving locales
Error: /File[/var/lib/puppet/locales]: Failed to generate additional resources using 'eval_generate': undefined method `unescape' for URI:Module
Error: /File[/var/lib/puppet/locales]: Could not evaluate: Could not retrieve file metadata for puppet:///locales: undefined method `unescape' for URI:Module
Comment by Sebastian Jaekel (bsjaekel) - Wednesday, 21 April 2021, 09:16 GMT
No, i get lot's of error message too, fiddled around 2 hours with it and gave up afterwards (too much work to do). I have deployed a Debian10 virtual machine, added official puppet repos, installed Puppet/PDK and use it for remote development with VSCode via SSH. Works perfectly and keeps local machine clean from broken package management...
Comment by DeLord (DeLord) - Wednesday, 21 April 2021, 23:46 GMT
Got the same issue, worked around it until there is a proper fix by downgrading ruby & puppet packages:
```
pacman -U /var/cache/pacman/pkg/puppet-6.21.1-1-any.pkg.tar.zst /var/cache/pacman/pkg/rubygems-3.2.7-1-any.pkg.tar.zst /var/cache/pacman/pkg/ruby-reline-0.2.2-1-any.pkg.tar.zst /var/cache/pacman/pkg/ruby-irb-1.3.0-1-any.pkg.tar.zst /var/cache/pacman/pkg/ruby-2.7.2-1-x86_64.pkg.tar.zst /var/cache/pacman/pkg/ruby-semantic_puppet-1.0.3-1-any.pkg.tar.zst /var/cache/pacman/pkg/ruby-augeas-0.5.0-4-x86_64.pkg.tar.xz /var/cache/pacman/pkg/ruby-shadow-2.5.0-5-x86_64.pkg.tar.xz /var/cache/pacman/pkg/ruby-concurrent-1.1.8-1-any.pkg.tar.zst /var/cache/pacman/pkg/ruby-multi_json-1.15.0-1-any.pkg.tar.zst /var/cache/pacman/pkg/ruby-hocon-1.3.1-1-any.pkg.tar.zst /var/cache/pacman/pkg/ruby-puppet-resource_api-1.8.13-1-any.pkg.tar.zst /var/cache/pacman/pkg/ruby-httpclient-2.8.3-7-any.pkg.tar.zst /var/cache/pacman/pkg/ruby-deep_merge-1.2.1-2-any.pkg.tar.zst /var/cache/pacman/pkg/ruby-sync-0.5.0-3-any.pkg.tar.zst
```
Comment by Joel Jensen (yobert) - Friday, 23 April 2021, 04:22 GMT
Making a similar change on line puppet/util.rb line 501 for URI.unescape fixed the errors for me. Changed that line to:

return URI::DEFAULT_PARSER.unescape(path)

I think after that you might be home free.
Comment by Christopher Gurley (Xarin) - Friday, 23 April 2021, 05:59 GMT
That did the trick, Thanks yobert, SimonPe!
Comment by Trysdyn Black (Trysdyn) - Wednesday, 28 April 2021, 13:34 GMT
The edit mentioned upthread is not a sufficient mitigation. Multiple core modules in puppet forge rely on the existence of URI::escape. An environment will fail or not depending on what forge modules are installed with the proposed patch in place.

Puppet development at the moment, by definition, presumes the underlying Ruby engine is Ruby 2.7. This needs to be adjusted or massive swathes of ruby code in commonly-accepted-working modules will fail.
Comment by Tim (bastelfreak) - Sunday, 02 May 2021, 18:20 GMT
Hi,
as soon as I've some free time I will rebuild the package to use Ruby 2.7. There are multiple upstream issues with Ruby 3.0 and it doesn't look like they will be fixed soonish.
Comment by Obs (obspm) - Friday, 07 May 2021, 06:17 GMT
Hi everyone,

Just a remark about puppet-agent (the «agent» part), the official (meaning from puppetlabs) pkg for Debian/CentOS/etc are install in /opt and come with everything need by the puppet (ruby, facter, etc..) I know it's a bulky pkg and not very efficient (in disk usage) way to manage pkg. But on every day using it's very convenient, because puppet-agent can use it's own version of puppet and the OS his version without any conflict.

Comment by Tim (bastelfreak) - Sunday, 16 May 2021, 20:36 GMT
Hi,
I pushed a new puppet package to community-testing thats build against Ruby 2.7. I'm still testing it. If one of you wants to test it as well that would be nice.
Comment by Ivan (fizmat) - Monday, 17 May 2021, 12:52 GMT
Thanks! I enabled the community-testing repo, updated everything (-Suy). Now the puppet agent seems to work for me.

Admittedly, I didn't do any in-depth testing. I just ran my usual `sudo puppet agent --test` to apply my config changes, and there were no errors.
Comment by Tim (bastelfreak) - Monday, 17 May 2021, 13:19 GMT
thanks for the feedback! can you also run `sudo facter` and `sudo facter -p` and check if you miss any facts or if any errors occour?
Comment by Ivan (fizmat) - Monday, 17 May 2021, 13:47 GMT
There are no errors after `sudo facter` and `sudo facter -p`
I didn't notice any missing facts, results looked normal. I don't have a json of facts from before the ruby update to just diff and be sure, so no guarantees.
Comment by Tim (bastelfreak) - Friday, 21 May 2021, 10:09 GMT
Hi,
I moved all the packages to community. Please raise a new issue if you run into issues. In addition we will remove the puppet5 package. I think people had enough time to transition to Puppet 6 now and puppet 5 is EoL and doesn't receive updates anymore.

Loading...