FS#72298 - [gitlab-shell] Cannot push using SSH

Attached to Project: Community Packages
Opened by simonzack (simonzack) - Thursday, 30 September 2021, 09:09 GMT
Last edited by Anatol Pomozov (anatolik) - Wednesday, 16 February 2022, 17:03 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Anatol Pomozov (anatolik)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 5
Private No

Details

Description:

Since updating to *GitLab 14.3.0-1*, I can no longer push.

Additional info:
* package version(s) *GitLab 14.3.0-1*

Steps to reproduce:

```
$ git push
remote:
remote: ========================================================================
remote:
remote: The project you were looking for could not be found or you don't have permission to view it.
remote:
remote: ========================================================================
remote:
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
$ ssh -T arch_gitlab
remote:
remote: ========================================================================
remote:
remote: Failed to get username: who='' is invalid
remote:
remote: ========================================================================
remote:
```
This task depends upon

Closed by  Anatol Pomozov (anatolik)
Wednesday, 16 February 2022, 17:03 GMT
Reason for closing:  Fixed
Additional comments about closing:  gitlab-shell-3:13.22.0-1-x86_64
Comment by Antonin Fischer (raven2cz) - Thursday, 30 September 2021, 11:21 GMT
I have same issue. I already write the bug to gitlab, but no response yet.

I checked similar ssh -Tvvvvv gitlab@ip

and remote: Failed to get username: who='' is invalid.

I already check the /var/lib/gitlab/.ssh/authorized_keys.
And my previous generated keys are commented, instead of it there are new generated keys key-2 and key-3 which doesn't work.
I tried to return my previous generated key to the file key-1, but it doesn't work too anymore.

I read this:
https://docs.gitlab.com/ee/administration/operations/fast_ssh_key_lookup.html

I tried to rebuild authorized keys:
sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake gitlab:shell:setup

But the authorized_keys files includes previous key-2 and key-3 again only. Which DOESN'T WORK...
I tried to clear my SSH key, and add it again, but NOTHING ROW was added to the authorized_keys in gitlab, in previous version, it was added, this is difference.

How is bind "gitlab" user (arch installation) with my SSH keys which are inserted from my user?

In any cases, there is critical bug, because SSH access is everywhere. HTTPS works, but it is not solution.

Do you have some workaround first? We have many tasks which have to be solved today!

Comment by Torben Peichl (parapilot) - Thursday, 30 September 2021, 18:08 GMT
I had the same problem. While I didn't get rid of the pty allocation error, I was able to get back git functionality by editing /etc/passwd and setting regular login shell for gitlab user.
gitlab:x:105:105::/var/lib/gitlab:/bin/bash
I'm not quite sure why but command restriction on ssh is still enforced even there's only the fast ssh key lookup in sshd_config:
Match User gitlab
AuthorizedKeysCommand /var/lib/gitlab/gitlab-shell/bin/gitlab-shell-authorized-keys-check gitlab %u %k
AuthorizedKeysCommandUser gitlab
Match all
-> definitely only workaround reason of behavior not clear
Comment by Antonin Fischer (raven2cz) - Thursday, 30 September 2021, 20:14 GMT
Parapilot, thank you very much for this workaround. I know that there will be some problem with AuthorizedKeysCommand and very bad solution with hacked shell.

I hope that gitlab developers or arch best maintainers will find some clear solution.

Thanks again, our work can continue now...
Comment by Torben Peichl (parapilot) - Friday, 01 October 2021, 05:21 GMT
Minor update: I hesitated downdrading gitlab because of the complex database dependencies. However gitlab-shell like gitlab-runner is somehow an quite independent component.
Thus the better solution for now is to just downgrade gitlab-shell only with the previous version here: https://archive.archlinux.org/packages/g/gitlab-shell/gitlab-shell-2%3A13.20.0-1-x86_64.pkg.tar.zst
And keep this package in pacman.conf IngnorePkg until problem is resolved.
I also think that the pty allocation is intended and only caused by directly interacting with ssh with the shell. Because the shell and git usage ins non-interactive.
Comment by Antonin Fischer (raven2cz) - Friday, 01 October 2021, 05:31 GMT
I'm not sure if the problem will be resolved from the gitlab side. Omnibus hasn't error and arch has.
This is not good situation now.
Parapilot, please provides your results to this thread
https://gitlab.com/gitlab-org/gitlab/-/issues/342028#note_691875503

There is very important to finish this discussion, if gitlab has to make some fixes. Thx.
Comment by Anatol Pomozov (anatolik) - Sunday, 10 October 2021, 23:52 GMT
I was able to get a response from gitlab shell after I change its $SHELL with `sudo usermod -s /bin/sh gitlab`.
Comment by Antonin Fischer (raven2cz) - Monday, 11 October 2021, 06:06 GMT
Anatolik. This was already discussed by @parapilot in the post Comment by Torben Peichl (parapilot) - Thursday, 30 September 2021, 18:08 GMT.
Next discussion see in opened issue gitlab-shell, link and moved issue in the previous comment.
The shell change of gitlab is not solution. I mean not enough solution. The gitlab-shell-authorized-keys-check has to be correctly handled if you do not want to open additional security bug.
Comment by simonzack (simonzack) - Monday, 11 October 2021, 08:25 GMT
@raven2cz Are you sure we have to run `gitlab-shell-authorized-keys-check`? Per my reading of https://docs.gitlab.com/ee/administration/operations/fast_ssh_key_lookup.html, it appears that the point of this is of efficiency, so we can check users by using a "fingerprint of the SSH key" instead of checking `/var/lib/gitlab/.ssh/authorized_keys`.
Comment by Antonin Fischer (raven2cz) - Monday, 11 October 2021, 09:03 GMT
My side is not important. Important is correct analysis by developers from gitlab-shell.

This issue https://gitlab.com/gitlab-org/gitlab-shell/-/issues/530 still doesn't include correct analysis. Just trying, nothing more. If the previous version works and after replacing just gitlab-shell stops working. There has to be detailed analysis "what happened". If you make some workaround without explanation, there can be security bug. Because analysis of this new implementation is important before change of arch configuration.

For example fast lookup in the database can be disabled by gitlab configuration, after that the /var/opt/gitlab/.ssh/authorized_keys will be used. I just know that I tried to add my key, it was added to this file again and validated according it by gitlab-shell-authorized-keys-check.

But again, I'm not person to provide some result. It has to be provided by developers of gitlab-shell after analysis. I expect that the change of the shell is not enough. But it depends on the changes of inside gilab-shell project.
Comment by Anatol Pomozov (anatolik) - Monday, 11 October 2021, 19:11 GMT
Ok, let's wait for more info from upstream.

Meanwhile, to fix the breakage, I am going to downgrade gitlab-shell package back to 13.20.0.
Comment by Anatol Pomozov (anatolik) - Monday, 11 October 2021, 19:25 GMT
gitlab-shell-3:13.20.0-1 just got landed community repo. Please try it and let me know if you still see the issue.
Comment by simonzack (simonzack) - Tuesday, 12 October 2021, 01:56 GMT
By the comment at https://gitlab.com/gitlab-org/gitlab-shell/-/issues/530#note_699928327:

> In any case, I think it makes sense not to specify `gitlab-shell` for the user's shell.

won't the shell need to be changed anyway?
Comment by Anatol Pomozov (anatolik) - Tuesday, 09 November 2021, 16:49 GMT
Hi folks I have a question at https://gitlab.com/gitlab-org/gitlab-shell/-/issues/530 if you have a chance could you please look at it? Or better help with debugging/clarifying this issue?

I would love to see it resolved and moving forward with updating gitlab packages.
Comment by Anatol Pomozov (anatolik) - Tuesday, 23 November 2021, 16:13 GMT
Upstream fixed this issue. I pushed gitlab-shell-3:13.22.0-1-x86_64.pkg.tar.zst to [community-testing]. Please verify that it works as expected.
Comment by simonzack (simonzack) - Thursday, 25 November 2021, 02:50 GMT
I tested it and it works fine:

$ sudo usermod --shell /usr/share/webapps/gitlab-shell/bin/gitlab-shell gitlab
$ ssh -T arch_gitlab
Welcome to GitLab, @root!

Although I think it's probably better to just switch to using `/bin/sh` as the shell for `gitlab` per https://gitlab.com/gitlab-org/gitlab-shell/-/issues/530#note_728598338.

Loading...