SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

Discussion:

Bjorn Helgaas

2012-11-09 15:26:59 UTC

[+ linux-pci, Yinghai]

The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.

very thanks Greg,my server Dell R710 with latest BIOS version and
option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused
that Does R710 provide full support for SR-IOV, kernel or ixgbe
driver's bug? but I'm not sure where the problem lies,anyone has any
experience about this? .

Linux normally uses the resource assignments done by the BIOS, but it
is possible for the kernel to reassign those. We don't have good
automatic support for that yet, but on a recent upstream kernel, you
can try "pci=realloc". I doubt this option is in CentOS 6.3, though.

If an upstream kernel with "pci=realloc" still doesn't work, please
post the entire dmesg log.

Yinghai Lu

2012-11-09 16:56:56 UTC

Permalink

Post by Bjorn Helgaas
[+ linux-pci, Yinghai]

The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.

Linux normally uses the resource assignments done by the BIOS, but it
is possible for the kernel to reassign those. We don't have good
automatic support for that yet, but on a recent upstream kernel, you
can try "pci=realloc". I doubt this option is in CentOS 6.3, though.
If an upstream kernel with "pci=realloc" still doesn't work, please
post the entire dmesg log.

Even better, upstream kernel will enable realloc automatically if it
finds some SRIOV BAR is not assigned.
related logic is in pci_realloc_detect().

BTW, we may need another patch about realloc.

http://git.kernel.org/?p=linux/kernel/git/yinghai/linux-yinghai.git;a=patch;h=afb9975bddd92b17c2f9859676ce521e2e014997

Post by Bjorn Helgaas
From afb9975bddd92b17c2f9859676ce521e2e014997 Mon Sep 17 00:00:00 2001

From: Yinghai Lu <***@kernel.org>
Date: Wed, 3 Oct 2012 19:15:57 -0700
Subject: [PATCH] PCI: Add root bus children dev's res to fail list

We can stop trying according to try number now and do not need to use
root_bus checking as stop sign anymore.
In extreme case we could need to reallocate resource for device just
under root bus.

Signed-off-by: Yinghai Lu <***@kernel.org>
---
drivers/pci/setup-bus.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 1e808ca..9e625d84 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -283,7 +283,7 @@ static void
assign_requested_resources_sorted(struct list_head *head,
idx = res - &dev_res->dev->resource[0];
if (resource_size(res) &&
pci_assign_resource(dev_res->dev, idx)) {
- if (fail_head && !pci_is_root_bus(dev_res->dev->bus)) {
+ if (fail_head) {
/*
* if the failed res is for ROM BAR, and it will
* be enabled later, don't add it to the list

--
1.7.7.6

Don Dutile

2012-11-12 19:46:59 UTC

Permalink

Post by Bjorn Helgaas
[+ linux-pci, Yinghai]

The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.

Try moving the device into a different slot.
You may be trying it in a non-ARI slot in the 710; that is a problem
in RHEL6 (needing to realloc bus &/or mem-space).
A non-ARI slot will want to use one bus number per VF.... which will
be problematic.
I know I've seen ixgbe's (& their vfs's) working on a dell 710;
but they may also be one of those systems that has a slot off the ich10
with no ARI support.

Post by Bjorn Helgaas
If an upstream kernel with "pci=realloc" still doesn't work, please
post the entire dmesg log.
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

Jason Gao

2012-11-13 13:38:05 UTC

Permalink

I'm very sorry for delayed reply.now SR-IOV works for me in Centos
6.3,thank all of you.

Post by Bjorn Helgaas
Linux normally uses the resource assignments done by the BIOS, but it
is possible for the kernel to reassign those. We don't have good
automatic support for that yet, but on a recent upstream kernel, you
can try "pci=realloc". I doubt this option is in CentOS 6.3, though

Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.

Post by Bjorn Helgaas
DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is v.6.3.0 and release date is 07/24/2012
Please check if you configured intel_iommu=on in the grub.conf file.
If you did, check your kernel .config file under Device Drivers-> IOMMU Hardware support->enable Support for Intel IOMMU using DMA remapping Devices, enable Intel DMA Remapping Devices by Default, enable Support for Interrupt Remapping.

thank you Sibai,Our server "Dell R710",its BIOS version is just
v.6.3.0 and release date is 07/24/2012,and I also configured
intel_iommu=on in the grub.conf file,but I can't find these IOMMU
options in "Device Drivers" in my kernel(2.6.32-279) .config file ,
btw my os is Centos 6.3(RHEL6.3),although the problem solved,I'd like
to know what's your os version ,kernel version?

Li, Sibai

2012-11-13 16:04:53 UTC

Permalink

-----Original Message-----
Sent: Tuesday, November 13, 2012 5:38 AM
Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources
for SR-IOV)
I'm very sorry for delayed reply.now SR-IOV works for me in Centos 6.3,thank all
of you.

Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.

Post by Bjorn Helgaas
DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is
v.6.3.0 and release date is 07/24/2012 Please check if you configured

intel_iommu=on in the grub.conf file.

Post by Bjorn Helgaas
If you did, check your kernel .config file under Device Drivers-> IOMMU

Hardware support->enable Support for Intel IOMMU using DMA remapping
Devices, enable Intel DMA Remapping Devices by Default, enable Support for
Interrupt Remapping.
thank you Sibai,Our server "Dell R710",its BIOS version is just
v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in
the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my
kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the
problem solved,I'd like to know what's your os version ,kernel version?

I am using RHEL6.3 with unstable kernel 3.7.0-rc

Don Dutile

2012-11-13 16:40:04 UTC

Permalink

Post by Li, Sibai

Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.

Post by Bjorn Helgaas
DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is
v.6.3.0 and release date is 07/24/2012 Please check if you configured

intel_iommu=on in the grub.conf file.

Post by Bjorn Helgaas
If you did, check your kernel .config file under Device Drivers-> IOMMU

Sibai is referring to kernel config options. RHEL6.3 has the IOMMU options built into
the kernel, but not enabled by default -- have to add 'intel_iommu=on' to the kernel
cmdline to enable IOMMU. SRIOV support (CONFIG_IOV) is built into the RHEL6.3 kernel as well.

Post by Li, Sibai

kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the
problem solved,I'd like to know what's your os version ,kernel version?

I am using RHEL6.3 with unstable kernel 3.7.0-rc
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

Yinghai Lu

2012-11-13 18:16:57 UTC

Permalink

Post by Li, Sibai

Post by Jason Gao
Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.
thank you Sibai,Our server "Dell R710",its BIOS version is just
v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in
the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my
kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the
problem solved,I'd like to know what's your os version ,kernel version?

I am using RHEL6.3 with unstable kernel 3.7.0-rc

that means that config has
CONFIG_PCI_REALLOC_ENABLE_AUTO=y

So you don't need to append "pci=realloc"

Yinghai

Li, Sibai

2012-11-13 18:25:58 UTC

Permalink

-----Original Message-----
Yinghai Lu
Sent: Tuesday, November 13, 2012 10:17 AM
To: Li, Sibai
Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources
for SR-IOV)

Post by Li, Sibai

Post by Jason Gao
Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for

me.

Post by Li, Sibai

Post by Jason Gao
thank you Sibai,Our server "Dell R710",its BIOS version is just
v.6.3.0 and release date is 07/24/2012,and I also configured
intel_iommu=on in the grub.conf file,but I can't find these IOMMU
options in "Device Drivers" in my
kernel(2.6.32-279) .config file , btw my os is Centos
6.3(RHEL6.3),although the problem solved,I'd like to know what's your os

version ,kernel version?

Post by Li, Sibai
I am using RHEL6.3 with unstable kernel 3.7.0-rc

that means that config has
CONFIG_PCI_REALLOC_ENABLE_AUTO=y
So you don't need to append "pci=realloc"
Yinghai

Never append "pci=realloc" for both kernel 2.6.32.279 and kernel 3.5.0 above.

Yinghai Lu

2012-11-13 18:29:28 UTC

Permalink

Post by Li, Sibai
Never append "pci=realloc" for both kernel 2.6.32.279 and kernel 3.5.0 above.

well, can you both post boot log with "debug ignore_loglevel" ?