DragonFly kernel List (threaded) for 2005-11
network01.patch - MPSAFE network interrupt
Ok, here is my first go at making the network interrupt MP SAFE. The
patch can be fetched here:
In order to test the patch you need to:
(1) Apply the patch
(2) ON SMP systems sysctl kern.intr_mpsafe=1 (or assign it in
/boot/loader.conf). Note: you can boot with intr_mpsafe's
0 default and set it to 1 at any time.
(3) ON SMP systems Make sure there are no other drivers sharing
the network interrupt. vmstat -i will tell you. For testing
purposes, remove any other non-network drivers that share the
(4) Make sure that both the INVARIANTS and INVARIANT_SUPPORT options
(5) Do a full kernel recompile.
On SMP systems, to verify that the network interrupt is running MPSAFE,
ping or exercise the network a little, then do a 'ps axl' and grep out
the ithread for the network interrupt irq number. The thread should
be marked 'M' (which means 'MP'... not holding the big giant lock).
It is quite typical for firewire to share the same irq. If this is
the case, you will have to compile a kernel without firewire to test
IF YOU DONT HAVE A SMP BOX, the patch also needs serious testing on UP
boxes. The changes to the interrupt interlock infrastructure applies
to both UP and SMP. On UP systems we are now relying on the serializer
instead of a critical section to protect the networking hardware.
This patch needs serious testing. I intend to commit it some time next
week. Possibly as early as monday but it will depend heavily on what
kind of feedback I get.
Making the network interface MP safe involves protecting portions of
the IFNET structure, in particular the flags and altq. Since the
code paths are fairly long I could not use the spinlock implementation,
but it turned out to be a good thing because using the LWKT serialization
subsystem (which is just as efficient for the non-blocking case) allowed
me to integrate the network interrupt serialization with the IFNET
serialization but using the same serialization structure for both.
This means that only a single serialization lock is acquired for the
entire interrupt path.
The rest of the work involved protecting all entry points into the
network device driver, mainly if_start, if_init, if_ioctl, and so forth.
It turned out to be fairly minimally invasive to the actual network
drivers themselves which means I was able to make ALL of them MPSAFE
rather then just a few test cases.
After we release (middle of December) I am going to make a big push to
get Jeff's parallel routing implementation in and to make the TCP and
UDP network protocol threads MP safe. The current patch was designed
to dovetail into those efforts.