Reducing dependence on BIOS for allocating memory at PCI root port?

[Rajat Jain]

Hello,

I have huge hot-plug systems that can have hundreds of hot plug PCI slots that may not be populated at the boot time. Since BIOS doesn't see all devices could be plugged in, so we ask it to hard code and allocate a fixed chunk of memory space (say 4 GB) for PCI at the root port.

Now as we keep on developing new add-on cards with more PCI memory requirements - we have a problem because we need to increase that root port allocation. While asking the BIOS vendor to increase the memory for new releases is an option, I am in general thinking if there is a way to do this without depending on the BIOS allocation at all. We don't want to upgrade BIOS for just this. I'm hopeful this is a common problem - so I wanted to ask if there are there existing mechanisms that allow / instruct the kernel to ignore the BIOS allocations, and reprogram the PCI host controllers, (by maybe a kernel configuration or command line parameters etc for example).

My question is whether some thing exists (or if it makes sense)?

Thanks,

Rajat

PS: The immediate thought of device tree comes to mind - but what about architectures that don't support it yet. (FWIW - I'm using x86 - and I do not know if its host controller driver supports / will support any device tree options to specify PCI windows)

[Bjorn Helgaas]

[+cc linux-acpi]

Post by Rajat Jain
Hello,
I have huge hot-plug systems that can have hundreds of hot plug PCI slots that may not be populated at the boot time. Since BIOS doesn't see all devices could be plugged in, so we ask it to hard code and allocate a fixed chunk of memory space (say 4 GB) for PCI at the root port.
Now as we keep on developing new add-on cards with more PCI memory requirements - we have a problem because we need to increase that root port allocation. While asking the BIOS vendor to increase the memory for new releases is an option, I am in general thinking if there is a way to do this without depending on the BIOS allocation at all. We don't want to upgrade BIOS for just this. I'm hopeful this is a common problem - so I wanted to ask if there are there existing mechanisms that allow / instruct the kernel to ignore the BIOS allocations, and reprogram the PCI host controllers, (by maybe a kernel configuration or command line parameters etc for example).

As you know, the transition from CPU address space to PCI address
space is in the host bridge. For PCIe, this is logically part of the
Root Complex and is logically upstream from the Root Ports. If you
want to reallocate space between Root Ports, i.e., remove space from
one and add it to another port below the same Root Complex, that's
strictly a PCI thing and is at least conceptually possible today with
no new firmware or DT support.

But you're probably asking about increasing the space assigned to the
Root Complex itself. The ACPI model supports that -- the PNP0A08 host
bridge device could advertise a _PRS that shows how much space could
potentially be assigned to the bridge, and we could use _SRS to assign
more space (possibly after using _SRS on a different bridge to make
space available). There is no Linux code to do this today, so at this
point it's just a theoretical possibility. And, of course, it
requires BIOS support.

The ACPI driver (drivers/acpi/pci_root.c) is generic, so it doesn't
know how to reprogram the host controller except by using _PRS and
_SRS (and even that isn't implemented yet).

Post by Rajat Jain
PS: The immediate thought of device tree comes to mind - but what about architectures that don't support it yet. (FWIW - I'm using x86 - and I do not know if its host controller driver supports / will support any device tree options to specify PCI windows)

If you have DT and a native host controller driver, the driver may
know how to reprogram the host controller. Then you would just have
to work out how to manage the address space you want to assign to the
controller.

If you're using the generic DT driver
(drivers/pci/host/pci-host-generic.c), it's in the same situation as
the ACPI driver: it doesn't know how to reprogram the windows. And as
far as I know there's no DT mechanism similar to _PRS/_SRS.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Rajat Jain]

Hello,

As you know, the transition from CPU address space to PCI address space is
in the host bridge. For PCIe, this is logically part of the Root Complex and is
logically upstream from the Root Ports. If you want to reallocate space
between Root Ports, i.e., remove space from one and add it to another port
below the same Root Complex, that's strictly a PCI thing and is at least
conceptually possible today with no new firmware or DT support.

Yes, I understand.

But you're probably asking about increasing the space assigned to the Root
Complex itself.

Bingo! That's the problem I am bothered about.

The ACPI model supports that -- the PNP0A08 host bridge
device could advertise a _PRS that shows how much space could potentially
be assigned to the bridge, and we could use _SRS to assign more space
(possibly after using _SRS on a different bridge to make space available).
There is no Linux code to do this today, so at this point it's just a theoretical
possibility. And, of course, it requires BIOS support.

Thanks a lot for the pointers, I'll look it up.

The ACPI driver (drivers/acpi/pci_root.c) is generic, so it doesn't know how to
reprogram the host controller except by using _PRS and _SRS (and even that
isn't implemented yet).

Post by Rajat Jain
PS: The immediate thought of device tree comes to mind - but what
about architectures that don't support it yet. (FWIW - I'm using x86 -
and I do not know if its host controller driver supports / will
support any device tree options to specify PCI windows)

If you have DT and a native host controller driver, the driver may know how
to reprogram the host controller. Then you would just have to work out how
to manage the address space you want to assign to the controller.
If you're using the generic DT driver
(drivers/pci/host/pci-host-generic.c), it's in the same situation as the ACPI
driver: it doesn't know how to reprogram the windows. And as far as I know
there's no DT mechanism similar to _PRS/_SRS.

OK, understand. I guess there is work that needs to be done there. Yes, We are in the process of adopting device trees for x86. I'll dig into if we can use that.

Thanks again,