[PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.

[Marcel Apfelbaum]

Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.

Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.

The solution leverages driver_override functionality introduced by

commit: 782a985d7af26db39e86070d28f987cad21313c0
Author: Alex Williamson <***@redhat.com>
Date: Tue May 20 08:53:21 2014 -0600

PCI: Introduce new device binding path using pci_dev.driver_override

In order to bind PCI slots to specific drivers use:
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...

Signed-off-by: Marcel Apfelbaum <***@redhat.com>
---
v3 -> v4:
- Addressed Alex Williamson's comments:
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
v2 -> v3:
- Corrected subject line
v1 -> v2:
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Addressed Alex Williamson's comments:
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.

Notes:
- I have further ideas on top of this patch based on your reviews.
I thought of:
- Use wildcards to specify entire buses/devices, something like:
driver[0001:02:*.*]=pci-stub
- Use comma to separate several devices:
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
- Make domain optional:
driver[00:03.0]=pci-stub

Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 5ae8608..c1cbb4c 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2631,6 +2631,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
pcie_scan_all Scan all possible PCIe devices. Otherwise we
only look for one device below a PCIe downstream
port.
+ driver Provide an override to the devid<->driver mapping
+ for a specific slot.
+ Bind PCI slot 0001:02:03.4 to pci-stub by:
+ driver[0001:02:03.4]=pci-stub

pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power
Management.
diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 73aef51..b49f5cc 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -15,6 +15,8 @@
#include <linux/proc_fs.h>
#include <linux/slab.h>

+#include <asm/setup.h>
+
#include "pci.h"

void pci_add_resource_offset(struct list_head *resources, struct resource *res,
@@ -230,6 +232,114 @@ EXPORT_SYMBOL(pci_bus_alloc_resource);

void __weak pcibios_resource_survey_bus(struct pci_bus *bus) { }

+struct driver_override_entry {
+ u16 domain;
+ u8 bus;
+ u8 devfn;
+ char *driver_name;
+ struct list_head list;
+};
+
+static LIST_HEAD(driver_override_entries);
+
+static int pci_device_parse_driver_override(char *param, char *val,
+ const char *unused)
+{
+ unsigned int domain, bus, dev, fn;
+ char *buf;
+ struct driver_override_entry *entry;
+ int ret;
+
+ buf = kmalloc(COMMAND_LINE_SIZE, GFP_KERNEL);
+ if (!buf)
+ goto err_buf;
+
+ while (val) {
+ char *k = strchr(val, ',');
+
+ if (k)
+ *k++ = 0;
+
+ if (strncmp(val, "driver", 6)) {
+ val = k;
+ continue;
+ }
+
+ memset(buf, 0, COMMAND_LINE_SIZE);
+ ret = sscanf(val + 6, "[%4x:%2x:%2x.%2x]=%s",
+ &domain, &bus, &dev, &fn, buf);
+ if (ret != 5) {
+ pr_warn("PCI: Invalid command line: %s\n", val);
+ val = k;
+ continue;
+ }
+
+ entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ goto err_entry;
+
+ INIT_LIST_HEAD(&entry->list);
+ entry->domain = domain;
+ entry->bus = bus;
+ entry->devfn = PCI_DEVFN(dev, fn);
+ entry->driver_name = kstrdup(buf, GFP_KERNEL);
+ if (!entry->driver_name)
+ goto err_driver_name;
+
+ list_add_tail(&entry->list, &driver_override_entries);
+ val = k;
+ }
+
+ kfree(buf);
+ return 0;
+
+err_driver_name:
+ kfree(entry);
+
+err_entry:
+ kfree(buf);
+
+err_buf:
+ pr_err("PCI: Out of memory while parsing command line: %s\n", val);
+ return -ENOMEM;
+}
+
+static void pci_device_setup_driver_override(struct pci_dev *dev)
+{
+ static int parse_done;
+ struct driver_override_entry *entry;
+
+ if (!parse_done) {
+ char *cmdline = kstrdup(saved_command_line, GFP_KERNEL);
+
+ if (!cmdline)
+ goto err_out_of_mem;
+
+ parse_args("pci", cmdline, NULL,
+ 0, 0, 0, &pci_device_parse_driver_override);
+ kfree(cmdline);
+ parse_done = 1;
+ }
+
+ list_for_each_entry(entry, &driver_override_entries, list) {
+ if (pci_domain_nr(dev->bus) != entry->domain ||
+ dev->bus->number != entry->bus ||
+ dev->devfn != entry->devfn)
+ continue;
+
+ dev->driver_override = kstrdup(entry->driver_name, GFP_KERNEL);
+ if (!dev->driver_override)
+ goto err_out_of_mem;
+
+ break;
+ }
+
+ return;
+
+err_out_of_mem:
+ pr_err("PCI: Out of memory while setting up driver override\n");
+}
+
/**
* pci_bus_add_device - start driver for a single device
* @dev: device to add
@@ -245,6 +355,7 @@ void pci_bus_add_device(struct pci_dev *dev)
* are not assigned yet for some devices.
*/
pci_fixup_device(pci_fixup_final, dev);
+ pci_device_setup_driver_override(dev);
pci_create_sysfs_dev_files(dev);
pci_proc_attach_device(dev);

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 625a4ac..37809d4 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4508,6 +4508,8 @@ static int __init pci_setup(char *str)
pcie_bus_config = PCIE_BUS_PEER2PEER;
} else if (!strncmp(str, "pcie_scan_all", 13)) {
pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
+ } else if (!strncmp(str, "driver", 6)) {
+ /* lazy evaluation by the pci subsystem */
} else {
printk(KERN_ERR "PCI: Unknown option `%s'\n",
str);

[Alex Williamson]

[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.
Perhaps:

driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform

Finding delimiters that don't conflict may be challenging. Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

It also seems like there's a question of how long should this override
last and how does the user disable it? I think with pci-stub.ids=
$VENDOR:$DEVICE a user can echo the IDs to the pci-stub/remove_id sysfs
entry to cancel the effect. The only option here seems to be a reboot.
Do we need a /sys/bus/pci/driver_overrides/{add_name,remove_name} for
this interface? Thanks,

Alex

Post by Marcel Apfelbaum
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 5ae8608..c1cbb4c 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2631,6 +2631,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
pcie_scan_all Scan all possible PCIe devices. Otherwise we
only look for one device below a PCIe downstream
port.
+ driver Provide an override to the devid<->driver mapping
+ for a specific slot.
+ driver[0001:02:03.4]=pci-stub
pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power
Management.
diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 73aef51..b49f5cc 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -15,6 +15,8 @@
#include <linux/proc_fs.h>
#include <linux/slab.h>
+#include <asm/setup.h>
+
#include "pci.h"
void pci_add_resource_offset(struct list_head *resources, struct resource *res,
@@ -230,6 +232,114 @@ EXPORT_SYMBOL(pci_bus_alloc_resource);
void __weak pcibios_resource_survey_bus(struct pci_bus *bus) { }
+struct driver_override_entry {
+ u16 domain;
+ u8 bus;
+ u8 devfn;
+ char *driver_name;
+ struct list_head list;
+};
+
+static LIST_HEAD(driver_override_entries);
+
+static int pci_device_parse_driver_override(char *param, char *val,
+ const char *unused)
+{
+ unsigned int domain, bus, dev, fn;
+ char *buf;
+ struct driver_override_entry *entry;
+ int ret;
+
+ buf = kmalloc(COMMAND_LINE_SIZE, GFP_KERNEL);
+ if (!buf)
+ goto err_buf;
+
+ while (val) {
+ char *k = strchr(val, ',');
+
+ if (k)
+ *k++ = 0;
+
+ if (strncmp(val, "driver", 6)) {
+ val = k;
+ continue;
+ }
+
+ memset(buf, 0, COMMAND_LINE_SIZE);
+ ret = sscanf(val + 6, "[%4x:%2x:%2x.%2x]=%s",
+ &domain, &bus, &dev, &fn, buf);
+ if (ret != 5) {
+ pr_warn("PCI: Invalid command line: %s\n", val);
+ val = k;
+ continue;
+ }
+
+ entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ goto err_entry;
+
+ INIT_LIST_HEAD(&entry->list);
+ entry->domain = domain;
+ entry->bus = bus;
+ entry->devfn = PCI_DEVFN(dev, fn);
+ entry->driver_name = kstrdup(buf, GFP_KERNEL);
+ if (!entry->driver_name)
+ goto err_driver_name;
+
+ list_add_tail(&entry->list, &driver_override_entries);
+ val = k;
+ }
+
+ kfree(buf);
+ return 0;
+
+ kfree(entry);
+
+ kfree(buf);
+
+ pr_err("PCI: Out of memory while parsing command line: %s\n", val);
+ return -ENOMEM;
+}
+
+static void pci_device_setup_driver_override(struct pci_dev *dev)
+{
+ static int parse_done;
+ struct driver_override_entry *entry;
+
+ if (!parse_done) {
+ char *cmdline = kstrdup(saved_command_line, GFP_KERNEL);
+
+ if (!cmdline)
+ goto err_out_of_mem;
+
+ parse_args("pci", cmdline, NULL,
+ 0, 0, 0, &pci_device_parse_driver_override);
+ kfree(cmdline);
+ parse_done = 1;
+ }
+
+ list_for_each_entry(entry, &driver_override_entries, list) {
+ if (pci_domain_nr(dev->bus) != entry->domain ||
+ dev->bus->number != entry->bus ||
+ dev->devfn != entry->devfn)
+ continue;
+
+ dev->driver_override = kstrdup(entry->driver_name, GFP_KERNEL);
+ if (!dev->driver_override)
+ goto err_out_of_mem;
+
+ break;
+ }
+
+ return;
+
+ pr_err("PCI: Out of memory while setting up driver override\n");
+}
+
/**
* pci_bus_add_device - start driver for a single device
@@ -245,6 +355,7 @@ void pci_bus_add_device(struct pci_dev *dev)
* are not assigned yet for some devices.
*/
pci_fixup_device(pci_fixup_final, dev);
+ pci_device_setup_driver_override(dev);
pci_create_sysfs_dev_files(dev);
pci_proc_attach_device(dev);
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 625a4ac..37809d4 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4508,6 +4508,8 @@ static int __init pci_setup(char *str)
pcie_bus_config = PCIE_BUS_PEER2PEER;
} else if (!strncmp(str, "pcie_scan_all", 13)) {
pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
+ } else if (!strncmp(str, "driver", 6)) {
+ /* lazy evaluation by the pci subsystem */
} else {
printk(KERN_ERR "PCI: Unknown option `%s'\n",
str);

[Marcel Apfelbaum]

Post by Alex Williamson
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.

The real question here if those bus types/devices would benefit from this
feature, and I also must confess that I have no knowledge of the other buses.
Can anyone confirm that it does make sense for them?

Post by Alex Williamson
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging. Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

For PCI, sure the domain:bus:dev.func is unique, for platform I have no idea,
can anyone that knows "platform" confirm or deny?

Post by Alex Williamson
It also seems like there's a question of how long should this override
last and how does the user disable it?
I think with pci-stub.ids=
$VENDOR:$DEVICE a user can echo the IDs to the pci-stub/remove_id sysfs
entry to cancel the effect.

The way I see it is simple, the override specified in kernel command line
last as long as the user does not specifically remove it using
echo "" > /sys/.../driver_override
and then unbind and bind the device again.

Post by Alex Williamson
The only option here seems to be a reboot.

Please see above

Post by Alex Williamson
Do we need a /sys/bus/pci/driver_overrides/{add_name,remove_name} for
this interface? Thanks,

While it does not hurt, I see it as optional since a simple removal of
driver_override and rebind does the same

Thanks,
Marcel

Post by Alex Williamson
Alex

Post by Marcel Apfelbaum
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 5ae8608..c1cbb4c 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2631,6 +2631,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
pcie_scan_all Scan all possible PCIe devices. Otherwise we
only look for one device below a PCIe downstream
port.
+ driver Provide an override to the devid<->driver mapping
+ for a specific slot.
+ driver[0001:02:03.4]=pci-stub
pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power
Management.
diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 73aef51..b49f5cc 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -15,6 +15,8 @@
#include <linux/proc_fs.h>
#include <linux/slab.h>
+#include <asm/setup.h>
+
#include "pci.h"
void pci_add_resource_offset(struct list_head *resources, struct resource *res,
@@ -230,6 +232,114 @@ EXPORT_SYMBOL(pci_bus_alloc_resource);
void __weak pcibios_resource_survey_bus(struct pci_bus *bus) { }
+struct driver_override_entry {
+ u16 domain;
+ u8 bus;
+ u8 devfn;
+ char *driver_name;
+ struct list_head list;
+};
+
+static LIST_HEAD(driver_override_entries);
+
+static int pci_device_parse_driver_override(char *param, char *val,
+ const char *unused)
+{
+ unsigned int domain, bus, dev, fn;
+ char *buf;
+ struct driver_override_entry *entry;
+ int ret;
+
+ buf = kmalloc(COMMAND_LINE_SIZE, GFP_KERNEL);
+ if (!buf)
+ goto err_buf;
+
+ while (val) {
+ char *k = strchr(val, ',');
+
+ if (k)
+ *k++ = 0;
+
+ if (strncmp(val, "driver", 6)) {
+ val = k;
+ continue;
+ }
+
+ memset(buf, 0, COMMAND_LINE_SIZE);
+ ret = sscanf(val + 6, "[%4x:%2x:%2x.%2x]=%s",
+ &domain, &bus, &dev, &fn, buf);
+ if (ret != 5) {
+ pr_warn("PCI: Invalid command line: %s\n", val);
+ val = k;
+ continue;
+ }
+
+ entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ goto err_entry;
+
+ INIT_LIST_HEAD(&entry->list);
+ entry->domain = domain;
+ entry->bus = bus;
+ entry->devfn = PCI_DEVFN(dev, fn);
+ entry->driver_name = kstrdup(buf, GFP_KERNEL);
+ if (!entry->driver_name)
+ goto err_driver_name;
+
+ list_add_tail(&entry->list, &driver_override_entries);
+ val = k;
+ }
+
+ kfree(buf);
+ return 0;
+
+ kfree(entry);
+
+ kfree(buf);
+
+ pr_err("PCI: Out of memory while parsing command line: %s\n", val);
+ return -ENOMEM;
+}
+
+static void pci_device_setup_driver_override(struct pci_dev *dev)
+{
+ static int parse_done;
+ struct driver_override_entry *entry;
+
+ if (!parse_done) {
+ char *cmdline = kstrdup(saved_command_line, GFP_KERNEL);
+
+ if (!cmdline)
+ goto err_out_of_mem;
+
+ parse_args("pci", cmdline, NULL,
+ 0, 0, 0, &pci_device_parse_driver_override);
+ kfree(cmdline);
+ parse_done = 1;
+ }
+
+ list_for_each_entry(entry, &driver_override_entries, list) {
+ if (pci_domain_nr(dev->bus) != entry->domain ||
+ dev->bus->number != entry->bus ||
+ dev->devfn != entry->devfn)
+ continue;
+
+ dev->driver_override = kstrdup(entry->driver_name, GFP_KERNEL);
+ if (!dev->driver_override)
+ goto err_out_of_mem;
+
+ break;
+ }
+
+ return;
+
+ pr_err("PCI: Out of memory while setting up driver override\n");
+}
+
/**
* pci_bus_add_device - start driver for a single device
@@ -245,6 +355,7 @@ void pci_bus_add_device(struct pci_dev *dev)
* are not assigned yet for some devices.
*/
pci_fixup_device(pci_fixup_final, dev);
+ pci_device_setup_driver_override(dev);
pci_create_sysfs_dev_files(dev);
pci_proc_attach_device(dev);
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 625a4ac..37809d4 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4508,6 +4508,8 @@ static int __init pci_setup(char *str)
pcie_bus_config = PCIE_BUS_PEER2PEER;
} else if (!strncmp(str, "pcie_scan_all", 13)) {
pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
+ } else if (!strncmp(str, "driver", 6)) {
+ /* lazy evaluation by the pci subsystem */
} else {
printk(KERN_ERR "PCI: Unknown option `%s'\n",
str);

[Alex Williamson]

Post by Marcel Apfelbaum

Post by Alex Williamson
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.

The real question here if those bus types/devices would benefit from this
feature, and I also must confess that I have no knowledge of the other buses.
Can anyone confirm that it does make sense for them?

Platform devices are adding vfio support, so I expect the next logical
question will be how to reserve devices for use by vfio at boot.

Post by Marcel Apfelbaum

Post by Alex Williamson
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging. Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

For PCI, sure the domain:bus:dev.func is unique, for platform I have no idea,
can anyone that knows "platform" confirm or deny?

Post by Alex Williamson
It also seems like there's a question of how long should this override
last and how does the user disable it?
I think with pci-stub.ids=
$VENDOR:$DEVICE a user can echo the IDs to the pci-stub/remove_id sysfs
entry to cancel the effect.

The way I see it is simple, the override specified in kernel command line
last as long as the user does not specifically remove it using
echo "" > /sys/.../driver_override
and then unbind and bind the device again.

Post by Alex Williamson
The only option here seems to be a reboot.

Please see above

That's only a temporary removal though, if the device is removed and
re-added, either via physical hotplug or sysfs, the override is
re-applied. Thanks,

Alex

Post by Marcel Apfelbaum

Post by Alex Williamson
Do we need a /sys/bus/pci/driver_overrides/{add_name,remove_name} for
this interface? Thanks,

While it does not hurt, I see it as optional since a simple removal of
driver_override and rebind does the same
Thanks,
Marcel

Post by Alex Williamson
Alex

Post by Marcel Apfelbaum
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 5ae8608..c1cbb4c 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2631,6 +2631,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
pcie_scan_all Scan all possible PCIe devices. Otherwise we
only look for one device below a PCIe downstream
port.
+ driver Provide an override to the devid<->driver mapping
+ for a specific slot.
+ driver[0001:02:03.4]=pci-stub
pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power
Management.
diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 73aef51..b49f5cc 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -15,6 +15,8 @@
#include <linux/proc_fs.h>
#include <linux/slab.h>
+#include <asm/setup.h>
+
#include "pci.h"
void pci_add_resource_offset(struct list_head *resources, struct resource *res,
@@ -230,6 +232,114 @@ EXPORT_SYMBOL(pci_bus_alloc_resource);
void __weak pcibios_resource_survey_bus(struct pci_bus *bus) { }
+struct driver_override_entry {
+ u16 domain;
+ u8 bus;
+ u8 devfn;
+ char *driver_name;
+ struct list_head list;
+};
+
+static LIST_HEAD(driver_override_entries);
+
+static int pci_device_parse_driver_override(char *param, char *val,
+ const char *unused)
+{
+ unsigned int domain, bus, dev, fn;
+ char *buf;
+ struct driver_override_entry *entry;
+ int ret;
+
+ buf = kmalloc(COMMAND_LINE_SIZE, GFP_KERNEL);
+ if (!buf)
+ goto err_buf;
+
+ while (val) {
+ char *k = strchr(val, ',');
+
+ if (k)
+ *k++ = 0;
+
+ if (strncmp(val, "driver", 6)) {
+ val = k;
+ continue;
+ }
+
+ memset(buf, 0, COMMAND_LINE_SIZE);
+ ret = sscanf(val + 6, "[%4x:%2x:%2x.%2x]=%s",
+ &domain, &bus, &dev, &fn, buf);
+ if (ret != 5) {
+ pr_warn("PCI: Invalid command line: %s\n", val);
+ val = k;
+ continue;
+ }
+
+ entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ goto err_entry;
+
+ INIT_LIST_HEAD(&entry->list);
+ entry->domain = domain;
+ entry->bus = bus;
+ entry->devfn = PCI_DEVFN(dev, fn);
+ entry->driver_name = kstrdup(buf, GFP_KERNEL);
+ if (!entry->driver_name)
+ goto err_driver_name;
+
+ list_add_tail(&entry->list, &driver_override_entries);
+ val = k;
+ }
+
+ kfree(buf);
+ return 0;
+
+ kfree(entry);
+
+ kfree(buf);
+
+ pr_err("PCI: Out of memory while parsing command line: %s\n", val);
+ return -ENOMEM;
+}
+
+static void pci_device_setup_driver_override(struct pci_dev *dev)
+{
+ static int parse_done;
+ struct driver_override_entry *entry;
+
+ if (!parse_done) {
+ char *cmdline = kstrdup(saved_command_line, GFP_KERNEL);
+
+ if (!cmdline)
+ goto err_out_of_mem;
+
+ parse_args("pci", cmdline, NULL,
+ 0, 0, 0, &pci_device_parse_driver_override);
+ kfree(cmdline);
+ parse_done = 1;
+ }
+
+ list_for_each_entry(entry, &driver_override_entries, list) {
+ if (pci_domain_nr(dev->bus) != entry->domain ||
+ dev->bus->number != entry->bus ||
+ dev->devfn != entry->devfn)
+ continue;
+
+ dev->driver_override = kstrdup(entry->driver_name, GFP_KERNEL);
+ if (!dev->driver_override)
+ goto err_out_of_mem;
+
+ break;
+ }
+
+ return;
+
+ pr_err("PCI: Out of memory while setting up driver override\n");
+}
+
/**
* pci_bus_add_device - start driver for a single device
@@ -245,6 +355,7 @@ void pci_bus_add_device(struct pci_dev *dev)
* are not assigned yet for some devices.
*/
pci_fixup_device(pci_fixup_final, dev);
+ pci_device_setup_driver_override(dev);
pci_create_sysfs_dev_files(dev);
pci_proc_attach_device(dev);
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 625a4ac..37809d4 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4508,6 +4508,8 @@ static int __init pci_setup(char *str)
pcie_bus_config = PCIE_BUS_PEER2PEER;
} else if (!strncmp(str, "pcie_scan_all", 13)) {
pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
+ } else if (!strncmp(str, "driver", 6)) {
+ /* lazy evaluation by the pci subsystem */
} else {
printk(KERN_ERR "PCI: Unknown option `%s'\n",
str);

[Marcel Apfelbaum]

Post by Alex Williamson

Post by Marcel Apfelbaum

Post by Alex Williamson
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.

The real question here if those bus types/devices would benefit from this
feature, and I also must confess that I have no knowledge of the other buses.
Can anyone confirm that it does make sense for them?

Platform devices are adding vfio support, so I expect the next logical
question will be how to reserve devices for use by vfio at boot.

Well, I'll have to learn more about vfio before saying anything,
but my question is if it can be deferred or it has to be part of
this patch. If the platform devices do not have a slot like hw address,
maybe it can be configured separately.

I saw this patch as a PCI patch, and not "driver_override" expansion;
meaning that I only leveraged an existing feature, not trying to
extend it.
If I am wrong, please point me to a more robust solution.

Post by Alex Williamson

Post by Marcel Apfelbaum

Post by Alex Williamson
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging. Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

For PCI, sure the domain:bus:dev.func is unique, for platform I have no idea,
can anyone that knows "platform" confirm or deny?

Post by Alex Williamson
It also seems like there's a question of how long should this override
last and how does the user disable it?
I think with pci-stub.ids=
$VENDOR:$DEVICE a user can echo the IDs to the pci-stub/remove_id sysfs
entry to cancel the effect.

The way I see it is simple, the override specified in kernel command line
last as long as the user does not specifically remove it using
echo "" > /sys/.../driver_override
and then unbind and bind the device again.

Post by Alex Williamson
The only option here seems to be a reboot.

Please see above

That's only a temporary removal though, if the device is removed and
re-added, either via physical hotplug or sysfs, the override is
re-applied. Thanks,

Bear with me please,

If we empty the driver_override file (cat "" /sys/bus/.../slot/driver_override)
as suggested above, the override will not be reapplied.

So, it is a consistent model:
1. The end-user specified the driver in command-line, so he wants it there
even after unbinding/binding or hotplug operations.
2. If the end-user "changes his mind", he doesn't need to reboot the system,
only to clear the driver_override sysfs file and from this moment on
the system behaves like usual. (at next unbind/bind or hotplug)

I hope I got it right,
Thanks,
Marcel

Post by Alex Williamson
Alex

Post by Marcel Apfelbaum

Post by Alex Williamson
Do we need a /sys/bus/pci/driver_overrides/{add_name,remove_name} for
this interface? Thanks,

While it does not hurt, I see it as optional since a simple removal of
driver_override and rebind does the same
Thanks,
Marcel

Post by Alex Williamson
Alex

Post by Marcel Apfelbaum
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 5ae8608..c1cbb4c 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2631,6 +2631,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
pcie_scan_all Scan all possible PCIe devices. Otherwise we
only look for one device below a PCIe downstream
port.
+ driver Provide an override to the devid<->driver mapping
+ for a specific slot.
+ driver[0001:02:03.4]=pci-stub
pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power
Management.
diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 73aef51..b49f5cc 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -15,6 +15,8 @@
#include <linux/proc_fs.h>
#include <linux/slab.h>
+#include <asm/setup.h>
+
#include "pci.h"
void pci_add_resource_offset(struct list_head *resources, struct resource *res,
@@ -230,6 +232,114 @@ EXPORT_SYMBOL(pci_bus_alloc_resource);
void __weak pcibios_resource_survey_bus(struct pci_bus *bus) { }
+struct driver_override_entry {
+ u16 domain;
+ u8 bus;
+ u8 devfn;
+ char *driver_name;
+ struct list_head list;
+};
+
+static LIST_HEAD(driver_override_entries);
+
+static int pci_device_parse_driver_override(char *param, char *val,
+ const char *unused)
+{
+ unsigned int domain, bus, dev, fn;
+ char *buf;
+ struct driver_override_entry *entry;
+ int ret;
+
+ buf = kmalloc(COMMAND_LINE_SIZE, GFP_KERNEL);
+ if (!buf)
+ goto err_buf;
+
+ while (val) {
+ char *k = strchr(val, ',');
+
+ if (k)
+ *k++ = 0;
+
+ if (strncmp(val, "driver", 6)) {
+ val = k;
+ continue;
+ }
+
+ memset(buf, 0, COMMAND_LINE_SIZE);
+ ret = sscanf(val + 6, "[%4x:%2x:%2x.%2x]=%s",
+ &domain, &bus, &dev, &fn, buf);
+ if (ret != 5) {
+ pr_warn("PCI: Invalid command line: %s\n", val);
+ val = k;
+ continue;
+ }
+
+ entry = kzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ goto err_entry;
+
+ INIT_LIST_HEAD(&entry->list);
+ entry->domain = domain;
+ entry->bus = bus;
+ entry->devfn = PCI_DEVFN(dev, fn);
+ entry->driver_name = kstrdup(buf, GFP_KERNEL);
+ if (!entry->driver_name)
+ goto err_driver_name;
+
+ list_add_tail(&entry->list, &driver_override_entries);
+ val = k;
+ }
+
+ kfree(buf);
+ return 0;
+
+ kfree(entry);
+
+ kfree(buf);
+
+ pr_err("PCI: Out of memory while parsing command line: %s\n", val);
+ return -ENOMEM;
+}
+
+static void pci_device_setup_driver_override(struct pci_dev *dev)
+{
+ static int parse_done;
+ struct driver_override_entry *entry;
+
+ if (!parse_done) {
+ char *cmdline = kstrdup(saved_command_line, GFP_KERNEL);
+
+ if (!cmdline)
+ goto err_out_of_mem;
+
+ parse_args("pci", cmdline, NULL,
+ 0, 0, 0, &pci_device_parse_driver_override);
+ kfree(cmdline);
+ parse_done = 1;
+ }
+
+ list_for_each_entry(entry, &driver_override_entries, list) {
+ if (pci_domain_nr(dev->bus) != entry->domain ||
+ dev->bus->number != entry->bus ||
+ dev->devfn != entry->devfn)
+ continue;
+
+ dev->driver_override = kstrdup(entry->driver_name, GFP_KERNEL);
+ if (!dev->driver_override)
+ goto err_out_of_mem;
+
+ break;
+ }
+
+ return;
+
+ pr_err("PCI: Out of memory while setting up driver override\n");
+}
+
/**
* pci_bus_add_device - start driver for a single device
@@ -245,6 +355,7 @@ void pci_bus_add_device(struct pci_dev *dev)
* are not assigned yet for some devices.
*/
pci_fixup_device(pci_fixup_final, dev);
+ pci_device_setup_driver_override(dev);
pci_create_sysfs_dev_files(dev);
pci_proc_attach_device(dev);
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 625a4ac..37809d4 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4508,6 +4508,8 @@ static int __init pci_setup(char *str)
pcie_bus_config = PCIE_BUS_PEER2PEER;
} else if (!strncmp(str, "pcie_scan_all", 13)) {
pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
+ } else if (!strncmp(str, "driver", 6)) {
+ /* lazy evaluation by the pci subsystem */
} else {
printk(KERN_ERR "PCI: Unknown option `%s'\n",
str);

[Stuart Yoder]

-----Original Message-----
Sent: Thursday, October 23, 2014 8:37 AM
To: Alex Williamson
Subject: Re: [PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.

Post by Alex Williamson

Post by Marcel Apfelbaum

Post by Alex Williamson
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.

The real question here if those bus types/devices would benefit from this
feature, and I also must confess that I have no knowledge of the other buses.
Can anyone confirm that it does make sense for them?

Platform devices are adding vfio support, so I expect the next logical
question will be how to reserve devices for use by vfio at boot.

Well, I'll have to learn more about vfio before saying anything,
but my question is if it can be deferred or it has to be part of
this patch. If the platform devices do not have a slot like hw address,
maybe it can be configured separately.
I saw this patch as a PCI patch, and not "driver_override" expansion;
meaning that I only leveraged an existing feature, not trying to
extend it.

I think other buses may want to use the same mechanism to specify driver
bindings, so I think the main thing is to not define a syntax that makes
that problematic.

Looking at your original syntax, I think it could work for different
bus types. Could have something like:

platform=driver[xxxx]=foo,driver[xxxx]=vfio-platform
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar

So no strong opinion on that vs Alex's proposal:

driver_override=pci,0000:02:00.0=pci-stub;platform,xxxx=vfio-platform

It seems to me th

[Marcel Apfelbaum]

-----Original Message-----
Sent: Thursday, October 23, 2014 7:32 AM
To: Alex Williamson
Subject: Re: [PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.

Post by Alex Williamson
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.

The real question here if those bus types/devices would benefit from this
feature, and I also must confess that I have no knowledge of the other buses.
Can anyone confirm that it does make sense for them?

We don't have vfio-platform in use yet, so not much actual real world user
experience yet. But, I think it makes sense. Especially, given that we are
inventing a kernel parameter I think we should design the syntax so that it
can work buses can implement support for this. The driver_override mechanism
is not bus specific, so let's not make the kernel parameter bus specific.

Make sense, point taken.
I'll come up with something.

Thank you Stuart,
Marcel

Post by Alex Williamson
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging. Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

For PCI, sure the domain:bus:dev.func is unique, for platform I have no idea,
can anyone that knows "platform" confirm or deny?

Yes, dev-name will be unique. All platform devices are under a single
platform bus.
Stuart

[Stuart Yoder]

-----Original Message-----
Sent: Thursday, October 23, 2014 7:32 AM
To: Alex Williamson
Subject: Re: [PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.

Post by Alex Williamson
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.

The real question here if those bus types/devices would benefit from this
feature, and I also must confess that I have no knowledge of the other buses.
Can anyone confirm that it does make sense for them?

We don't have vfio-platform in use yet, so not much actual real world user
experience yet. But, I think it makes sense. Especially, given that we are
inventing a kernel parameter I think we should design the syntax so that it
can work buses can implement support for this. The driver_override mechanism
is not bus specific, so let's not make the kernel parameter bus specific.

Post by Alex Williamson
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging. Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

For PCI, sure the domain:bus:dev.func is unique, for platform I have no idea,
can anyone that knows "platform" confirm or deny?

Yes, dev-name will be unique. All platform devices

[Stuart Yoder]

-----Original Message-----
Sent: Wednesday, October 22, 2014 1:33 PM
To: Marcel Apfelbaum
Subject: Re: [PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging.

I think what you proposed works-- <bus-name>,<bus-dev>=<driver>;

Think that will work for PCI, platform, and the new fsl-mc bus we are working
on.

Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

I think that has to be the case.

It also seems like there's a question of how long should this override
last and how does the user disable it?

Isn't that a general question for the "driver_overrride" mechanism?
I'm forgetting if the mechanism in the kernel now has a way to disable
it-- e.g. echo /dev/null > /sys/pci/devices/.../driver_override ??

So, it would last until explicitly disabled through sysfs.

I think with pci-stub.ids=
$VENDOR:$DEVICE a user can echo the IDs to the pci-stub/remove_id sysfs
entry to cancel the effect. The only option here seems to be a reboot.
Do we need a /sys/bus/pci/driver_overrides/{add_name,remove_name} for
this interface? Thanks,

Thanks,
Stu

[Alex Williamson]

Post by Stuart Yoder

-----Original Message-----
Sent: Wednesday, October 22, 2014 1:33 PM
To: Marcel Apfelbaum
Subject: Re: [PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging.

I think what you proposed works-- <bus-name>,<bus-dev>=<driver>;
Think that will work for PCI, platform, and the new fsl-mc bus we are working
on.

Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

I think that has to be the case.

It also seems like there's a question of how long should this override
last and how does the user disable it?

Isn't that a general question for the "driver_overrride" mechanism?
I'm forgetting if the mechanism in the kernel now has a way to disable
it-- e.g. echo /dev/null > /sys/pci/devices/.../driver_override ??
So, it would last until explicitly disabled through sysfs.

Yes, when you set a driver_override on a device you cancel it by writing
a NULL string to the same interface. The problem is that here we have a
new entity in the driver scan that keeps re-applying the driver_override
as devices are scanned with no way to stop it. So you can certainly
undo the immediate effect and bind the device to another driver, but if
the device is removed and re-scanned there's no way to stop if from
re-applying the override. Thanks,

Alex

Post by Stuart Yoder

I think with pci-stub.ids=
$VENDOR:$DEVICE a user can echo the IDs to the pci-stub/remove_id sysfs
entry to cancel the effect. The only option here seems to be a reboot.
Do we need a /sys/bus/pci/driver_overrides/{add_name,remove_name} for
this interface? Thanks,

Thanks,
Stuart

[Marcel Apfelbaum]

Post by Alex Williamson

Post by Stuart Yoder

-----Original Message-----
Sent: Wednesday, October 22, 2014 1:33 PM
To: Marcel Apfelbaum
Subject: Re: [PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging.

I think what you proposed works-- <bus-name>,<bus-dev>=<driver>;
Think that will work for PCI, platform, and the new fsl-mc bus we are working
on.

Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

I think that has to be the case.

It also seems like there's a question of how long should this override
last and how does the user disable it?

Isn't that a general question for the "driver_overrride" mechanism?
I'm forgetting if the mechanism in the kernel now has a way to disable
it-- e.g. echo /dev/null > /sys/pci/devices/.../driver_override ??
So, it would last until explicitly disabled through sysfs.

Yes, when you set a driver_override on a device you cancel it by writing
a NULL string to the same interface. The problem is that here we have a
new entity in the driver scan that keeps re-applying the driver_override
as devices are scanned with no way to stop it. So you can certainly
undo the immediate effect and bind the device to another driver, but if
the device is removed and re-scanned there's no way to stop if from
re-applying the override. Thanks,

Hi Alex,

I checked the above scenario and after driver_override is cleared
an we do bind/unbind, the mapping defined in the command line
does not apply anymore.

My steps were:
1. define the override in command-line -> the mapped driver is used instead of the native one
2. unbind the device from the slot -> no driver for device
3. remove the driver_override mapping form the slot -> no mapping defined
3, bind the device again -> native driver in use.

Thanks,
Marcel

Post by Alex Williamson
Alex

Post by Stuart Yoder

I think with pci-stub.ids=
$VENDOR:$DEVICE a user can echo the IDs to the pci-stub/remove_id sysfs
entry to cancel the effect. The only option here seems to be a reboot.
Do we need a /sys/bus/pci/driver_overrides/{add_name,remove_name} for
this interface? Thanks,

Thanks,
Stuart

[Alex Williamson]

Post by Marcel Apfelbaum

Post by Alex Williamson

Post by Stuart Yoder

-----Original Message-----
Sent: Wednesday, October 22, 2014 1:33 PM
To: Marcel Apfelbaum
Subject: Re: [PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging.

I think what you proposed works-- <bus-name>,<bus-dev>=<driver>;
Think that will work for PCI, platform, and the new fsl-mc bus we are working
on.

Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

I think that has to be the case.

It also seems like there's a question of how long should this override
last and how does the user disable it?

Isn't that a general question for the "driver_overrride" mechanism?
I'm forgetting if the mechanism in the kernel now has a way to disable
it-- e.g. echo /dev/null > /sys/pci/devices/.../driver_override ??
So, it would last until explicitly disabled through sysfs.

Yes, when you set a driver_override on a device you cancel it by writing
a NULL string to the same interface. The problem is that here we have a
new entity in the driver scan that keeps re-applying the driver_override
as devices are scanned with no way to stop it. So you can certainly
undo the immediate effect and bind the device to another driver, but if
the device is removed and re-scanned there's no way to stop if from
re-applying the override. Thanks,

Hi Alex,
I checked the above scenario and after driver_override is cleared
an we do bind/unbind, the mapping defined in the command line
does not apply anymore.
1. define the override in command-line -> the mapped driver is used instead of the native one
2. unbind the device from the slot -> no driver for device
3. remove the driver_override mapping form the slot -> no mapping defined
3, bind the device again -> native driver in use.

That's not the scenario I'm describing. Use the remove and rescan sysfs
attributes to do a software hotplug and you'll see that the
driver_override will always be re-applied to the device. For example:

# echo "" > /sys/bus/pci/devices/0000:02:00.0/driver_override
# echo 1 > /sys/bus/pci/devices/0000:02:00.0/remove
# echo 1 > /sys/bus/pci/rescan
# cat /sys/bus/pci/devices/0000:02:00.0/driver_override

I expect the last step will show the original override re-applied.
Thanks,

Alex

[Marcel Apfelbaum]

Post by Alex Williamson

Post by Marcel Apfelbaum

Post by Alex Williamson

Post by Stuart Yoder

-----Original Message-----
Sent: Wednesday, October 22, 2014 1:33 PM
To: Marcel Apfelbaum
Subject: Re: [PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging.

I think what you proposed works-- <bus-name>,<bus-dev>=<driver>;
Think that will work for PCI, platform, and the new fsl-mc bus we are working
on.

Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

I think that has to be the case.

It also seems like there's a question of how long should this override
last and how does the user disable it?

Isn't that a general question for the "driver_overrride" mechanism?
I'm forgetting if the mechanism in the kernel now has a way to disable
it-- e.g. echo /dev/null > /sys/pci/devices/.../driver_override ??
So, it would last until explicitly disabled through sysfs.

Yes, when you set a driver_override on a device you cancel it by writing
a NULL string to the same interface. The problem is that here we have a
new entity in the driver scan that keeps re-applying the driver_override
as devices are scanned with no way to stop it. So you can certainly
undo the immediate effect and bind the device to another driver, but if
the device is removed and re-scanned there's no way to stop if from
re-applying the override. Thanks,

Hi Alex,
I checked the above scenario and after driver_override is cleared
an we do bind/unbind, the mapping defined in the command line
does not apply anymore.
1. define the override in command-line -> the mapped driver is used instead of the native one
2. unbind the device from the slot -> no driver for device
3. remove the driver_override mapping form the slot -> no mapping defined
3, bind the device again -> native driver in use.

That's not the scenario I'm describing. Use the remove and rescan sysfs
attributes to do a software hotplug and you'll see that the
# echo "" > /sys/bus/pci/devices/0000:02:00.0/driver_override
# echo 1 > /sys/bus/pci/devices/0000:02:00.0/remove
# echo 1 > /sys/bus/pci/rescan
# cat /sys/bus/pci/devices/0000:02:00.0/driver_override
I expect the last step will show the original override re-applied.

I can check this, but I am sure you are right.
We need to think about this, if is acceptable or not.

Thanks,
Marcel

Post by Alex Williamson
Thanks,
Alex

[Alex Williamson]

Post by Marcel Apfelbaum

Post by Alex Williamson

Post by Marcel Apfelbaum

Post by Alex Williamson

Post by Stuart Yoder

-----Original Message-----
Sent: Wednesday, October 22, 2014 1:33 PM
To: Marcel Apfelbaum
Subject: Re: [PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging.

I think what you proposed works-- <bus-name>,<bus-dev>=<driver>;
Think that will work for PCI, platform, and the new fsl-mc bus we are working
on.

Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

I think that has to be the case.

It also seems like there's a question of how long should this override
last and how does the user disable it?

Isn't that a general question for the "driver_overrride" mechanism?
I'm forgetting if the mechanism in the kernel now has a way to disable
it-- e.g. echo /dev/null > /sys/pci/devices/.../driver_override ??
So, it would last until explicitly disabled through sysfs.

Yes, when you set a driver_override on a device you cancel it by writing
a NULL string to the same interface. The problem is that here we have a
new entity in the driver scan that keeps re-applying the driver_override
as devices are scanned with no way to stop it. So you can certainly
undo the immediate effect and bind the device to another driver, but if
the device is removed and re-scanned there's no way to stop if from
re-applying the override. Thanks,

Hi Alex,
I checked the above scenario and after driver_override is cleared
an we do bind/unbind, the mapping defined in the command line
does not apply anymore.
1. define the override in command-line -> the mapped driver is used instead of the native one
2. unbind the device from the slot -> no driver for device
3. remove the driver_override mapping form the slot -> no mapping defined
3, bind the device again -> native driver in use.

That's not the scenario I'm describing. Use the remove and rescan sysfs
attributes to do a software hotplug and you'll see that the
# echo "" > /sys/bus/pci/devices/0000:02:00.0/driver_override
# echo 1 > /sys/bus/pci/devices/0000:02:00.0/remove
# echo 1 > /sys/bus/pci/rescan
# cat /sys/bus/pci/devices/0000:02:00.0/driver_override
I expect the last step will show the original override re-applied.

I can check this, but I am sure you are right.
We need to think about this, if is acceptable or not.

I think we're going to need a sysfs way to manipulate the set of active
overrides. I was thinking about whether a one-shot implementation might
be acceptable, ie. discard the override after use, but I think that
would look just as unpredictable to users as the current approach.
Thanks,

Alex

[Marcel Apfelbaum]

Post by Alex Williamson

Post by Marcel Apfelbaum

Post by Alex Williamson

Post by Marcel Apfelbaum

Post by Alex Williamson

Post by Stuart Yoder

-----Original Message-----
Sent: Wednesday, October 22, 2014 1:33 PM
To: Marcel Apfelbaum
Subject: Re: [PATCH v4] PCI: add kernel parameter to override devid<->driver mapping.
[cc+ stuart]

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.
On other scenarios there is a need to bind a driver to a specific slot.
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type. Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...
---
- Modified the type of driver_override_entry's fields
- Used PCI_DEVFN when appropriated
- Removed redundant checks
- Replaced BUG_ON with pr_err messages
- Simpler command line parsing
- Addressed Michael S. Tsirkin comments
- removed DRIVER_OVERRIDE_NAME_LENGTH limitation
- Corrected subject line
- Addressed Michael S. Tsirkin comments
- Removed 32 slots limitation
- Better handling of memory allocation failures
(preferred BUG_ON over error messages)
- Modified commit message to show parameter usage more clear.
- I preferred to re-use parse_args instead of manually using
strstr in order to better comply with command line parsing
rules.
- I didn't use any locking when parsing the command line args
(see parse_done usage) assuming that first call will be
early in system boot and no race can occur. Please correct
me if I am wrong.
- I have further ideas on top of this patch based on your reviews.
driver[0001:02:*.*]=pci-stub
driver[0001:02:03.4,0001:02:04.0,...]=pci-stub
driver[00:03.0]=pci-stub
Comments will be appreciated,
Thanks,
Marcel
Documentation/kernel-parameters.txt | 4 ++
drivers/pci/bus.c | 111 ++++++++++++++++++++++++++++++++++++
drivers/pci/pci.c | 2 +
3 files changed, 117 insertions(+)

The driver_override feature that we're making use of here is also going
to be supported by platform devices and potentially more bustypes in the
future, so I'm concerned that making a pci specific kernel parameter is
too shortsighted. Instead we could hook on to BUS_NOTIFY_ADD_DEVICE for
bustypes that support driver_override so we can have a common interface.
driver_override=pci,0000:02:00.0=pci-stub;platform,fakename=vfio-platform
Finding delimiters that don't conflict may be challenging.

I think what you proposed works-- <bus-name>,<bus-dev>=<driver>;
Think that will work for PCI, platform, and the new fsl-mc bus we are working
on.

Also, can we
assume that bus-name:dev-name is unique for every bustype? It is for
pci, platform?

I think that has to be the case.

It also seems like there's a question of how long should this override
last and how does the user disable it?

Isn't that a general question for the "driver_overrride" mechanism?
I'm forgetting if the mechanism in the kernel now has a way to disable
it-- e.g. echo /dev/null > /sys/pci/devices/.../driver_override ??
So, it would last until explicitly disabled through sysfs.

Yes, when you set a driver_override on a device you cancel it by writing
a NULL string to the same interface. The problem is that here we have a
new entity in the driver scan that keeps re-applying the driver_override
as devices are scanned with no way to stop it. So you can certainly
undo the immediate effect and bind the device to another driver, but if
the device is removed and re-scanned there's no way to stop if from
re-applying the override. Thanks,

Hi Alex,
I checked the above scenario and after driver_override is cleared
an we do bind/unbind, the mapping defined in the command line
does not apply anymore.
1. define the override in command-line -> the mapped driver is used instead of the native one
2. unbind the device from the slot -> no driver for device
3. remove the driver_override mapping form the slot -> no mapping defined
3, bind the device again -> native driver in use.

That's not the scenario I'm describing. Use the remove and rescan sysfs
attributes to do a software hotplug and you'll see that the
# echo "" > /sys/bus/pci/devices/0000:02:00.0/driver_override
# echo 1 > /sys/bus/pci/devices/0000:02:00.0/remove
# echo 1 > /sys/bus/pci/rescan
# cat /sys/bus/pci/devices/0000:02:00.0/driver_override
I expect the last step will show the original override re-applied.

I can check this, but I am sure you are right.
We need to think about this, if is acceptable or not.

I think we're going to need a sysfs way to manipulate the set of active
overrides. I was thinking about whether a one-shot implementation might
be acceptable, ie. discard the override after use, but I think that
would look just as unpredictable to users as the current approach.
Thanks,

Sure, maybe a /sys/bus/<bus type>/driver_overrides file that has
the same format as the kernel parameter (and can be edited by the end user)
or something based on operations like you suggested:
/sys/bus/pci/driver_overrides/{add_name,remove_name}

Thanks for the ideas,
Marcel

Post by Alex Williamson
Alex

[Bjorn Helgaas]

Hi Marcel,

I'm not quite clear on what the objective is here, so I apologize for
some questions that probably seem silly.

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.

I think what takes a lot of time is the .probe() method for some
drivers, right? I first thought you meant that it took a long time
for the PCI core to enumerate a lot of devices, but you're not
changing anything there.

If the intent is to reduce boot time, I don't think this is a general
solution. Drivers should be able to schedule asynchronous things in
their .probe() methods if necessary.

Post by Marcel Apfelbaum
On other scenarios there is a need to bind a driver to a specific slot.

A short example here would be good. Are you talking about something
like binding a NIC driver to one device while leaving others unbound
for use by guests?

Post by Marcel Apfelbaum
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type.

I assume you mean booting with "pci-stub.ids=$VENDOR:$DEVICE" will
make pci-stub bind to *all* matching devices, and you only want it to
bind to some. Maybe pci-stub could be extended to pay attention to
PCI bus addresses in addition to vendor/device IDs.

Post by Marcel Apfelbaum
Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...

If/when you address Alex's comments about other bus types, can you
also update the changelog to use the canonical commit reference
format, i.e., 782a985d7af2 ("PCI: Introduce new device binding path
using pci_dev.driver_override") in this case?

PCI bus numbers are mutable, e.g., they can change with hotplug or
other configuration changes. But I don't have any better suggestion,
so I guess all we can do is be aware of this.

Speaking of hotplug, this is only a boot-time kernel parameter, with
no opportunity to use this, e.g., to add slot/driver pairs, after
boot. Do you not need that because of Alex's driver_override thing?
How can we integrate this all together into a coherent whole? I'm a
little confused as to how this would all be documented in a form
usable by end-users.

Bjorn

[Marcel Apfelbaum]

Post by Bjorn Helgaas
Hi Marcel,

Hi Bjorn,
Thank you for the review!

Post by Bjorn Helgaas
I'm not quite clear on what the objective is here, so I apologize for
some questions that probably seem silly.

I appreciate you took your time to go over it.

Post by Bjorn Helgaas

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.

I think what takes a lot of time is the .probe() method for some
drivers, right? I first thought you meant that it took a long time
for the PCI core to enumerate a lot of devices, but you're not
changing anything there.

Yes indeed.

Post by Bjorn Helgaas
If the intent is to reduce boot time, I don't think this is a general
solution. Drivers should be able to schedule asynchronous things in
their .probe() methods if necessary.

I agree, but sadly we cannot go over *all* existing drivers and fix,
we can of course do the best effort :)
By the way this was not the only reason as you also thought, see bellow

Post by Bjorn Helgaas

Post by Marcel Apfelbaum
On other scenarios there is a need to bind a driver to a specific slot.

A short example here would be good. Are you talking about something
like binding a NIC driver to one device while leaving others unbound
for use by guests?

Exactly! This is the "perfect" example, thanks!

Post by Bjorn Helgaas

Post by Marcel Apfelbaum
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type.

I assume you mean booting with "pci-stub.ids=$VENDOR:$DEVICE" will
make pci-stub bind to *all* matching devices, and you only want it to
bind to some.

You are right again.

Post by Bjorn Helgaas
Maybe pci-stub could be extended to pay attention to
PCI bus addresses in addition to vendor/device IDs.

A few thoughts here:
- We will have a race here between the "native" driver and pci-stub, right?
- Why not leverage the existing driver_override feature that is already
there and gives us exactly what we want: slot<->driver mapping?
- Maybe there are other scenarios that can benefit from slot<->driver mapping,
not only pci-stub.

Post by Bjorn Helgaas

Post by Marcel Apfelbaum
Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...

If/when you address Alex's comments about other bus types, can you
also update the changelog to use the canonical commit reference
format, i.e., 782a985d7af2 ("PCI: Introduce new device binding path
using pci_dev.driver_override") in this case?

Sure, thanks for the tip.

Post by Bjorn Helgaas
PCI bus numbers are mutable, e.g., they can change with hotplug or
other configuration changes. But I don't have any better suggestion,
so I guess all we can do is be aware of this.

Well, indeed, there is so much that can be done. (We can listen to an event and remap...)

Post by Bjorn Helgaas
Speaking of hotplug, this is only a boot-time kernel parameter, with
no opportunity to use this, e.g., to add slot/driver pairs, after
boot. Do you not need that because of Alex's driver_override thing?

Well actually Alex's "driver_override" feature does that for runtime
(adds slot/driver pair), the only thing missing is the boot time
mapping.

Post by Bjorn Helgaas
How can we integrate this all together into a coherent whole? I'm a
little confused as to how this would all be documented in a form
usable by end-users.

For end-users it will be like this:
They want to create a slot/driver mapping.
In order to do that they will use the "driver_override" feature:
1. Run-time use:
- Use sysfs to edit driver_override file associated with the slot.
2. Boot-time use:
- Use the pci's driver_override parameter.

Thanks,
Marcel

Post by Bjorn Helgaas
Bjorn

[Michael S. Tsirkin]

Post by Bjorn Helgaas
Hi Marcel,
I'm not quite clear on what the objective is here, so I apologize for
some questions that probably seem silly.

Post by Marcel Apfelbaum
Scanning a lot of devices during boot requires a lot of time.

I think what takes a lot of time is the .probe() method for some
drivers, right? I first thought you meant that it took a long time
for the PCI core to enumerate a lot of devices, but you're not
changing anything there.
If the intent is to reduce boot time, I don't think this is a general
solution. Drivers should be able to schedule asynchronous things in
their .probe() methods if necessary.

If this worked for all devices, we could just make probe
asynchronous in PCI core.
Unfortunately this doesn't work esp for storage devices
since people expect disks to be available for mount immediately.

If the point of the patch is to speed up boot, we could
try to probe everything in parallel?
Probe is serialized now, right?

Post by Bjorn Helgaas

Post by Marcel Apfelbaum
On other scenarios there is a need to bind a driver to a specific slot.

A short example here would be good. Are you talking about something
like binding a NIC driver to one device while leaving others unbound
for use by guests?

Post by Marcel Apfelbaum
Binding devices to pci-stub driver does not work,
as it will not differentiate between devices of the
same type.

I assume you mean booting with "pci-stub.ids=$VENDOR:$DEVICE" will
make pci-stub bind to *all* matching devices, and you only want it to
bind to some. Maybe pci-stub could be extended to pay attention to
PCI bus addresses in addition to vendor/device IDs.

Post by Marcel Apfelbaum
Using some start scripts is error prone.
The solution leverages driver_override functionality introduced by
commit: 782a985d7af26db39e86070d28f987cad21313c0
Date: Tue May 20 08:53:21 2014 -0600
PCI: Introduce new device binding path using pci_dev.driver_override
pci=driver[xxxx:xx:xx.x]=foo,driver[xxxx:xx:xx.x]=bar,...

If/when you address Alex's comments about other bus types, can you
also update the changelog to use the canonical commit reference
format, i.e., 782a985d7af2 ("PCI: Introduce new device binding path
using pci_dev.driver_override") in this case?
PCI bus numbers are mutable, e.g., they can change with hotplug or
other configuration changes. But I don't have any better suggestion,
so I guess all we can do is be aware of this.

We could use slot capability for addressing if that's available.

Post by Bjorn Helgaas
Speaking of hotplug, this is only a boot-time kernel parameter, with
no opportunity to use this, e.g., to add slot/driver pairs, after
boot. Do you not need that because of Alex's driver_override thing?
How can we integrate this all together into a coherent whole? I'm a
little confused as to how this would all be documented in a form
usable by end-users.
Bjorn