IRQ11 shared by a number of devices on a ThinkPad notebook (IO-APIC disabled?)

Dear everyone,

where I work, a colleague of mine's got a ThinkPad R51 1829-R6G. A
while ago he's paid a visit, complaining about the sluggishness of his
notebook, and that some PC Cards didn't work.

I've discovered that maybe 10 or 12 devices shared IRQ 11. Never seen
this before, certainly not on an ACPI-capable system with an IO-APIC
on board. This degree of IRQ sharing inherently means awful CPU load
consisting of calls to interrupt service routines.

Fumbled through the BIOS setup, found all PCI INT lines configured for
IRQ 11 (eight of them). Toggled them all to "auto-select", but still
the BIOS would assign IRQ 11 to all PCI devices.
Flashed the BIOS to the most recent version available (1.35, I
believe), to no avail - still the same, BIOS defaults mean "set
everything to IRQ 11", auto-select does just the same, there's no way
to select specific APIC IRQ's above IRQ 15.

Is this possibly a BIOS bug of some sort? The machine's got an ICH4-M
south bridge, this one has an IO-APIC with 24 IRQ lines. IRQ's 16
through 23 should be available!
Is the IO-APIC disabled? If yes, how do I re-enable it? I couldn't
find a relevant entry in the BIOS setup.

Found this out in Windows XP, verified using a Linux live CD - it's
definitely a BIOS issue.

Not a word about this in the online tech support information. I've
seen ACPI BIOS / IRQ routing issues before, mostly on no-name PC
hardware.

IBM/Lenovo did respond swiftly to my tech support request and were
definitely very kind in dealing with me, and I'm grateful for that
sort of treatment Yet in the end they haven't quite acknowledged a
mere existence of the issue. A country-wide authorized service center
(external partner company) didn't help, either. That's why I'm
resorting to asking this in a public newsgroup, perhaps someone out
there has more experience, perhaps it's a problem on my part
The IBM/Lenovo notebooks are otherwise an incredibly robust hardware
platform.

Any ideas are welcome

Frank Rysanek

 

Actually no. It means that each time IRQ 11 is called an awful lot of
devices have to be polled to find out whether they were the source.
Polling can take really long.

No idea what the issue actually is. But the sliggishness is indeed
likely caused by it. Can you manually addisgn different IRQs?

Arno

 

Dear Arno,

thanks for responding in the first place

Right. The number of ISR calls corresponding to a shared IRQ line
grows with a square of the number of devices on that shared IRQ line.
(That is, unless the ISR's return a trustworthy result code, and the
growth is only somewhat less if they do - this is OS-dependent.)
If those same devices each had their very own IRQ line, the growth
would be linear.

The PC card that ultimately got this particular system to its knees
was a Gb Ethernet adaptor with a Cardbus attachment to the host

The IBM BIOS setup actually does contain eight relevant entries in the
"PCI/PnP configuration" screen (or whatever it's called). The original
intention was probably nice, but practically those configurable
entries are completely useless.

The IO-APIC included on-chip in the ICH4-M south bridge used in the
Thinkpad has 8 dedicated PCI interrupt inputs (apart from the standard
XT/AT-PIC IRQ0 through 15). Even in the package pinout, the ICH4-M
chip's got the standard set of PCI IRQ inputs called INTA through INTD
(for a single PCI bus segment), along with four more optional inputs
called INTE throught INTH or something like that (GPIO pins with an
alternative function). In an OS that can use the APIC, these eight
input pins are normally routed to IRQ numbers from 16 to 23, or some
other arbitrary high numbers (OS-dependent behavior), so that the
legacy IRQ's 0 through 15 remain available to legacy ISA devices
(internally present in the chipset of every IBM-compatible PC to date)
without IRQ sharing.
The 8 additional PCI interrupt inputs of the IO-APIC can be used for
both ICH on-chip and other off-chip PCI devices. On-chip and off-chip
PCI devices can share individual PCI interrupt lines.

The IBM BIOS clearly contains configuration entries for these 8 PCI
interrupt inputs, and the labels on those entries suggest that they
were intended to allow you to "hard-wire" a particular INT input to a
particular IRQ number. (Silly as it seeems, in an APIC-based system,
where the IRQ numbers as seen by the software running on the host CPU
are more or less arbitrary.)
In other words, I was quite curious when I first saw them - I was
wondering if these would offer the legacy IRQ lines (0 through 15) or
some expected APIC based IRQ numbers that the OS was expected to come
up with, or what the hell.
It has turned out that the BIOS default is to set all these entries as
"hard-wired to IRQ11". Found out by reverting to BIOS defaults on
purpose, to see what happens. (Originally I thought this setup was the
result of someone's past human configuration attempts.) Anyway so I
tried changing the hard-wired IRQ11 to something different, and the
only other option is "auto-select". Which practically settles on IRQ11
anyway :-(

As for trying to change this in the operating systems, I believe that
is pointless.
Both Linux and modern Windows (such as XP) deliberately prevent you
from fiddling with IRQ numbers on PCI devices. This makes perfect
sense - the answer to "how do I change the IRQ" is that "you don't,
the BIOS + OS + PCI bus do that for you." A human administrator
doesn't have a chance to cope with the complexities of APIC-based IRQ
routing in the system hardware, which has chipset-specific basis and
board-specific custom configurations. The only correct way to
implement this is "automagically", via a set of standards-based HW and
SW interfaces such as APIC's, ACPI or the older MPS. Consequently, the
IRQ numbers become largely arbitrary - just an allocated slot in the
CPU's local APIC, wired by the BIOS+OS-driven black magic to some IO-
APIC input, to an actual PCI device...

Windows XP does run with an ACPI HAL on the culprit machine, and I'm
happy with that. There's nothing else you can do (ACPI can't be turned
off in the ThinkPad BIOS, so I couldn't even try to reinstall Windows
with MPS-based HAL).

Linux does offer a few bootloader options (kernel command line
options) to modify the methods of IRQ routing setup: pci=routeirq,
pci=noacpi, acpi=off. None of this has any effect, even in Linux all
the PCI devices share IRQ 11.

I believe it's up to the BIOS to get the ACPI/MPS tables right, or
maybe to enable the IO-APIC in the first place.
I was just wondering if someone knew about a BIOS update, or some IBM
configuration utility, that might be able to achieve that - or perhaps
if I'm just missing something obvious...

Thanks for everybody's attention, have a nice weekend

Frank Rysanek

 

I've seen it mentioned that turning off PnP in the bios then assinging IRQ's
manually could work.

 

FYI ...

The Importance of Implementing APIC-Based Interrupt Subsystems on
Uniprocessor PCs:

http://www.microsoft.com/whdc/system/sysperf/apic.mspx

"In 1996, Microsoft was told by Intel to expect all its machines to
have I/O APICs by 1998. So, the IRQ arbiter was designed to cause
every PCI device to share its IRQ with ACPI, when ACPI was enabled.
This solved most of the interrupt constraint problems, but caused a
huge amount of IRQ sharing."

"On the machine being used to write this paper, there are 11 devices
sharing IRQ 11. If any one of them interrupts, the operating system
begins at the head of the ISR chain and examines each device until it
finds the one that interrupted. Each hardware access can take several
microseconds, and each ISR may access its device several (or even
many) times. The result is that hardware accesses cause delays in the
processing of all other tasks on the machine. Each interrupt can
potentially cause a delay of most of a millisecond during which no
other work can be done. This makes it very difficult to guarantee that
time-sensitive devices, such as audio devices, function correctly.
(This machine actually glitches its audio whenever there is a large
amount of other activity in the machine.)"

- Franc Zabkar

 

I've seen it mentioned that turning off PnP in the bios then assinging IRQ's

I'm afraid that advice is very generic, and doesn't seem to apply to
the Thinkpad. There's no way to disable PnP support in the BIOS, and
assigning IRQ's manually likely wouldn't work for the APIC-based
IRQ's, even if it was possible.

I am aware that specifically Award BIOSes typically allow you to
select, for each IRQ (of a selected set where this is supported), one
of the following two options:
A) "reserved" (for legacy ISA devices) - route this IRQ to the
corresponding physical ISA IRQ line only
B) "PCI/ISA PnP" - means leave assignment/usage of this IRQ number up
to the PnP subsystem in the BIOS. The PnP BIOS will likely route the
IRQ to one of the on-chip/on-board devices, either ISA/LPC-based or
PCI-based. Or if you still happen to have an ISAPnP compatible ISA
addon board, that will be involved, too.

I believe this generic Award functionality is what you are referring
to...

Frank Rysanek

 

Just a few more comments: this is the latest BIOS for the machine,
i.e. 3.21.
If I boot Linux:
- it does detect and use ACPI, including for interrupt routing
- it says local APIC detected but disabled by BIOS.
Tried re-enabling it, to no effect.
- it says "no IO APIC detected".
- it says "no SMP motherboard detected", i.e. no MPS, no MPtable

As if the machine indeed had no APIC at all. Yet the ICH4-M does
contain one.
Without ACPI support and without a valid MPtable, it's no use anyway,
even if I could enable it in the ICH4.

Frank Rysanek

 

I found this thread after the Ethernet port on a customer's Lenovo Thinkpad R61 running WinXP Professional stopped working. I enabled the "Intel Virtualization Technology" option in the bios because I was just curious to see how it would affect the performance of laptop. To my surprise the Ethernet port stopped working and checking in the Device Manager of Windows showed the yellow exclamation mark on the device and stated the device couldn't start. Even when I changed back the settings in the bios and then updated the bios the Ethernet port still didn't work. I finally realized that the change that I made in the bios had bumped the Ethernet controller off the IRQ it had. So to get back the IRQ the Ethernet controller needed I disabled all the IRQs in the bios then booted into Windows. After POST an error message came up stating that no IRQs were assigned. I just hit Esc key to skip this and continued the boot process. This forced Windows to assign the IRQs for all the devices instead of the Bios. And presto the Ethernet port was back working. I rebooted the laptop and went back into the bios to change the IRQ settings to "Auto Select" because the error message would continue to appear during the boot process if I hadn't. Before the first time that I changed the IRQ settings they were all set to 11 like stated in the above posts. Hope this helps someone in the future.

 

Spreading Thinkpad PCI IRQ mappings

Through trial and error - and insight into how IBM architects "THINK" (I was an IBMer for 20+ years) - I found out how to get the older Thinkpads to spread out PCI bus interrupt mappings to more than just IRQ 11. (This effort was done while getting ESXi to run on some surplus laptops at my employer so that I could deploy a "small model" virtual appliance for some of our small sites. Laptops are being used because they have a built-in UPS commonly called a "battery" )

FYI, this process was done on a Thinkpad T400, but I suspect it applies to all Thinkpads which have a Config -> PCI interface. (My new T430s Thinkpad does not - I haven't yet looked to see how it maps PCI interrupts... since it has a more modern chipset, I would hope it allows operating systems to re-assign interrupts more easily.)

The "trick" is that "Auto Select" really should be called something like more like "Default". What Auto Select really does is:

1. Determine the list of "available" low IRQs for assignment. (IRQ11 is always in this list.)
2. For each listed PCI IRQ mapping (INTA-INTH on my T400):
   • If there is no previous assignment, assign a low IRQ "round robin" from the list in step 1.
   • If there is a previous assignment but that IRQ is no longer in the list, assign a new IRQ "round robin" from the list. (An example of this case would be if the vector was previously mapped to IRQ4 with the motherboard/docking station serial port disabled, then the serial port is enabled for COM 1 (which uses IRQ 4).)
   • Otherwise (default case): assign the same IRQ as was used previously.

But the problem is - IBM/Lenovo shipped these with all of the PCI interrupts mapped to IRQ 11 - how do we get out of that as the "default" IRQ? One answer (which worked for me) is:

• Go into BIOS and set ALL of the PCI IRQ mappings to Disabled.
• F10 to exist and reboot. The reboot will fail miserably - because there is no boot device available (the hard drive and CD ROM aren't available).
• Power-cycle, go back into BIOS, set all of the PCI IRQs to Auto Select. F10 to save/reboot.
• Presto! The old IRQ11 mappings are gone... the PCI IRQs will now be nicely "spread" across all of the available IRQs (including, but not exclusively, IRQ 11).

I confirmed this behavior with both Linux and ESXi (ESXi does a really nice job showing all of the PCI device/interrupt mappings in the vmkernel.log file). As noted previously, this would likely help out significantly with Windows performance on these devices.

Probably somewhat late to be of a lot of use - but thought I'd share it anyways.

Garrett