Arch Linux

Please read this before reporting a bug:
https://wiki.archlinux.org/title/Bug_reporting_guidelines

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#73231 - [linux] Installer doesn't detect the internal eMMC drive

Attached to Project: Arch Linux
Opened by trapanator (trapanator) - Sunday, 02 January 2022, 15:16 GMT
Last edited by David Runge (dvzrv) - Wednesday, 02 February 2022, 15:19 GMT
Task Type Bug Report
Category Kernel
Status Assigned
Assigned To Jan Alexander Steffens (heftig)
David Runge (dvzrv)
Architecture All
Severity Critical
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 0
Private No

Details

Description:

I have a baytrail device (Minix Z64).
I can boot it with Archlinux installer (adding `acpi=off` on the grub menu to be able to boot it).
But when I am in the shell, I cannot see the internal eMMC drive.
I tried to load these kernel modules (with modprobe):

* mmc_block
* sdhci
* sdhci_acpi

Still I cannot see the internal eMMC. Maybe I forgot other module?

Additional info:
* Installer, december version

This task depends upon

Comment by trapanator (trapanator) - Monday, 03 January 2022, 13:53 GMT
Using Fedora 35 and Ubuntu 20.04 media installers, they can see internal eMMC drive.

I think that there's something missing...
Comment by AK (Andreaskem) - Monday, 03 January 2022, 14:50 GMT
What exactly do you mean by "see"? Where do you expect it? Is it not listed by lsblk? Is it missing in /dev?
Comment by trapanator (trapanator) - Monday, 03 January 2022, 14:51 GMT
yes, it is not listed by lsblk and neither there is /dev/mmc*
Comment by trapanator (trapanator) - Monday, 03 January 2022, 14:59 GMT
I'm attaching dmesg and lsblk outputs
Comment by Morten Linderud (Foxboron) - Monday, 03 January 2022, 15:09 GMT
Probably missing this backport from efivar. It was pointed out in #archlinux-releng a day or two ago.

https://github.com/rhboot/efivar/commit/f0d3ed17ef3b2bbdfdff4dde12ec0a82d1ccdd33
Comment by AK (Andreaskem) - Monday, 03 January 2022, 15:12 GMT
edit: (the previous comment seems to offer a likely explanation)

ACPI is used to discover and configure hardware components. I would not be surprised if setting acpi=off is the reason for your issue. Is there some other option you could try to get the system to boot?
Comment by trapanator (trapanator) - Monday, 03 January 2022, 15:13 GMT
I have to turn off the acpi because I cannot see anything on the screen.

Are there another alternatives?
Comment by trapanator (trapanator) - Monday, 03 January 2022, 15:14 GMT
Note that the system is an Intel Atom Baytrail Z3735F
Comment by AK (Andreaskem) - Monday, 03 January 2022, 15:24 GMT
To rule out acpi=off as the culprit, you might get lucky with the options described here (although the commit mentioned above seems more likely to be required)

https://wiki.ubuntu.com/DebuggingACPI#Debugging_procedure

If "acpi=off" allows the system to boot, try to isolate the ACPI issue with the following boot parameters
Try booting with "acpi=ht"
This disables all of ACPI except just enough to enable Hyper Threading. If acpi=off works and acpi=ht fails, then the issue is in the ACPI table parsing code itself, or perhaps the SMP code.
Try booting with "pci=noacpi"
This disables ACPI for IRQ routing and PCI scanning.
Try booting with "acpi=noirq"
This disables ACPI for IRQ routing.
Try booting with "pnpacpi=off"
This disables the ACPI component of the Linux Plug and Play code.
Try booting with "noapic"
Disables the IO-APIC for IRQ routing or PCI scanning.
Try booting with "nolapic"
Disables the local APIC.
Comment by trapanator (trapanator) - Monday, 03 January 2022, 15:55 GMT
Bingo! The 'nolapic' did the trick.
But lsblk doesn't show the eMMC device;
But (again) lsmod now lists "mmc_block", "sdhci" and "sdhci_acpi" (before it didn't with acpi=off!)

And now? I should wait for the patch?
Comment by David Runge (dvzrv) - Tuesday, 04 January 2022, 19:16 GMT
@trapanator: Thanks for the report.

I am currently evaluating what can be done here, as there has not been a release for efivar since 2018 and upstream does not respond (https://github.com/rhboot/efivar/issues/137). Backporting seems a bit risky as there are three years of commits and the commit in question does not apply cleanly. Another (also potentlially risky) option would be to bump to some commit in the list of recent commits.

My assumption is, that other downstream distributions use some specific commit or heavily patch the sources to make this work.
Comment by trapanator (trapanator) - Tuesday, 04 January 2022, 19:28 GMT
Should I test with an older ISO version?
Comment by BTD Master (btdmaster) - Saturday, 15 January 2022, 20:46 GMT Comment by trapanator (trapanator) - Sunday, 16 January 2022, 08:12 GMT
Great! Waiting for next version of installer. Or can I grab a testing iso, now?
Comment by trapanator (trapanator) - Wednesday, 02 February 2022, 07:45 GMT
I've tried the archiso 2022.02.01, but it still doesn't see the eMMC.

Tried with OpenSuse Leap 15.3, it can install on it.
Comment by Kristian (klausenbusk) - Wednesday, 02 February 2022, 08:35 GMT
Can you provide dmesg from the Fedora installer?
Comment by trapanator (trapanator) - Wednesday, 02 February 2022, 14:45 GMT
attaching dmesg and lsmod of current running Opensuse Leap 15.3
Comment by trapanator (trapanator) - Wednesday, 02 February 2022, 14:46 GMT
Running this command on OpenSuse:

# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk1 179:0 0 29,1G 0 disk
├─mmcblk1p1 179:1 0 512M 0 part /boot/efi
├─mmcblk1p2 179:2 0 26,6G 0 part /
└─mmcblk1p3 179:3 0 2G 0 part [SWAP]
mmcblk1boot0 179:8 0 4M 1 disk
mmcblk1boot1 179:16 0 4M 1 disk
Comment by trapanator (trapanator) - Wednesday, 02 February 2022, 15:04 GMT
attaching same logs from archiso 2022.02.02:

Comment by trapanator (trapanator) - Wednesday, 02 February 2022, 15:05 GMT
lsblk output from archiso:

root@archiso ~ # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 657.4M 1 loop /run/archiso/airootfs
sda 8:0 1 14.4G 0 disk
├─sda1 8:1 1 735M 0 part /run/archiso/bootmnt
└─sda2 8:2 1 77M 0 part
Comment by David Runge (dvzrv) - Wednesday, 02 February 2022, 15:13 GMT
@trapanator: What jumps out initially is, that the opensuse system also loads mmc_block (but archiso didn't).

Can you see whether loading that module helps?
Comment by David Runge (dvzrv) - Wednesday, 02 February 2022, 15:19 GMT
Another thing that is odd is the stack trace around xhci_pci in the archiso system. I will add heftig to this ticket so he may have a look at https://bugs.archlinux.org/task/73231?getfile=21126 as well.
Comment by Tobias Powalowski (tpowa) - Wednesday, 02 February 2022, 15:30 GMT
you could try add_efi_memmap boot parameter
Comment by trapanator (trapanator) - Wednesday, 02 February 2022, 15:45 GMT
I tried either with 'modprobe mmc_block' and 'add_efi_memmap' as boot parameter, and they do not fix the problem.
Comment by David Runge (dvzrv) - Wednesday, 02 February 2022, 16:05 GMT
@trapanator: Maybe this is related to missing firmware for the controller (and xhci_pci crapping out subsequently).

Could you provide some more details on the controller (e.g. lspci?).

Hm, it's a bit hard to compare the opensuse kernel (5.3.x) to the one used in archiso (5.16.4).
There can be many reasons for why they behave differently (e.g. buggy driver).
Comment by trapanator (trapanator) - Wednesday, 02 February 2022, 16:12 GMT
lscpi and lscpi -vvv logs


If you want, I can try with a recent linux distro. Fedora 35 is ok?
Comment by Kristian (klausenbusk) - Wednesday, 02 February 2022, 17:07 GMT
Yes, please try Fedora 35. If it works, please also try archiso 2021.12.01 (shipped with the 5.15 kernel like Fedora 35) to rule out a regression in the 5.16 kernel.
Comment by trapanator (trapanator) - Wednesday, 02 February 2022, 18:54 GMT
I tried with Fedora Rawhide, that has a more recent kernel:

Linux version 5.17.0-0.rc0.20220112gitdaadb3bd0e8d.63.fc36.x86_64 (mockbuild@bkernel01.iad2.fedoraproject.org) (gcc (GCC) 11.2.1 20211203 (Red Hat 11.2.1-7), GNU ld version 2.37-22.fc36) #1 SMP PREEMPT Wed Jan 12 18:54:57 UTC 2022

I'm attaching all logs.

The live of Fedora Rawhide sees the eMMC drive, as you can see in the lsblk-fedora-rawhide.log
Comment by Kristian (klausenbusk) - Wednesday, 02 February 2022, 19:16 GMT
One thing sticking out in the archlinux dmesg log is:
> WARNING: CPU: 0 PID: 132 at include/linux/msi.h:262 __pci_enable_msi_range+0x422/0x570

If you are feeling brave you could try the Fedora kernel with Arch Linux (and rebuild the initramfs), that would rule out user spaces issues.

Could you try booting Fedora with `nolapic`? Maybe that is causing the breakage.
Comment by trapanator (trapanator) - Thursday, 03 February 2022, 10:47 GMT
Very interesting... booting the Fedora live with *nolapic* option, it cannot see the eMMC !!!!

The problem with Archlinux iso is that during boot the screen is black and I cannot see anything, furthermore it reboots at any point during booting...
Comment by Tobias Powalowski (tpowa) - Thursday, 03 February 2022, 12:25 GMT
Have you tried the add_efi_memmap parameter on boot line?
Comment by trapanator (trapanator) - Thursday, 03 February 2022, 13:37 GMT
@tpowa: add add_efi_memmap to which distribution? archiso or fedora rawhide?
Comment by Tobias Powalowski (tpowa) - Thursday, 03 February 2022, 13:48 GMT
Please archiso boot parameter, you can also try archboot iso: https://pkgbuild.com/~tpowa/archboot-images/x86_64/2022.01/archlinux-archboot-2022.01.30-06.01-x86_64.iso
it uses grub as bootloader.
Comment by trapanator (trapanator) - Thursday, 03 February 2022, 17:37 GMT
see the result with add_efi_memmap:

https://drive.google.com/file/d/1OUh8xe9rMDaVae7kn3hcSNO7RU-L-LYc/view?usp=sharing


blank screen then it reboots.
Comment by Tobias Powalowski (tpowa) - Thursday, 03 February 2022, 17:41 GMT
On your screen you used add_efi_memmap_ the correct parameter is add_efi_memmap without _ at the end.
Comment by trapanator (trapanator) - Thursday, 03 February 2022, 17:43 GMT
that was the cursor :-)
Comment by trapanator (trapanator) - Sunday, 06 February 2022, 16:56 GMT
Today update: I tried the newly released Slackware 15, that has the kernel 5.15.

No problem at all, it installs on eMMC correctly.
Comment by trapanator (trapanator) - Sunday, 06 February 2022, 17:04 GMT
Further note: all distributions I can see booting text after the grub boot menu. Archlinux is the only distro that the screen is black after the grub boot menu (see the above video).

Is there any way to *disable* framebuffer, KMS, whatever graphic mode after grub boot screen?
Comment by Tobias Powalowski (tpowa) - Sunday, 06 February 2022, 20:33 GMT
Some hardware does not like the KMS activation, use radeon.modeset=0, i915.modeset=0 or nouveau.modeset=0 on boot prompt.
Comment by Tobias Powalowski (tpowa) - Sunday, 06 February 2022, 20:34 GMT
I think the kernel config is the problem, somewhere it differs on the other distributions from Arch Linux.
Comment by Tobias Powalowski (tpowa) - Sunday, 06 February 2022, 20:36 GMT
zcat /proc/config.gz | grep BAY
on the other distributions as start would help.
Comment by trapanator (trapanator) - Monday, 07 February 2022, 07:53 GMT
On archiso boot menu, setting i915.modeset=0 (since mine has an intel GPU) turns the black screen... into an entire white screen! :-) :-)

On slackware 15 I'm attaching the config.gz. These are rows with "grep BAY":

CONFIG_BAYCOM_SER_FDX=m
CONFIG_BAYCOM_SER_HDX=m
CONFIG_BAYCOM_PAR=m
CONFIG_I2C_DESIGNWARE_BAYTRAIL=y
CONFIG_PINCTRL_BAYTRAIL=y
CONFIG_SND_SOC_SOF_BAYTRAIL=m
(application/x-gzip)    config.gz (56.5 KiB)
Comment by Tobias Powalowski (tpowa) - Monday, 07 February 2022, 08:01 GMT
Could you please attach a complete config from working distro:
zcat /proc/config.gz > config.txt
Comment by trapanator (trapanator) - Monday, 07 February 2022, 08:11 GMT
@tpowa: the last attached "config.gz" (see my previous comment) is from a working Slackware 15 with kernel 15. Hope it is sufficient
Comment by Tobias Powalowski (tpowa) - Monday, 07 February 2022, 08:26 GMT
Please disable the following modules on kernel cmdline:
module_blacklist=pinctrl-cherryview,pinctrl-lynxpoint,pinctrl-alderlake,pinctrl-broxton,pinctrl-cannonlake,pinctrl-cedarfork,pinctrl-denverton,pinctrl-elkhartlake,pinctrl-emmitsburg,pinctrl-geminilake,pinctrl-icelake,pinctrl-jasperlake,pinctrl-lakefield,pinctrl-lewisburg,pinctrl-sunrisepoint,pinctrl-tigerlake
Then only the basic pinctrl-intel.ko is used.
Comment by trapanator (trapanator) - Wednesday, 16 February 2022, 12:37 GMT
Sorry for delay. How can I insert that long line into the grub menu of usb disk without typing charater by character?

Loading...