DragonFly users List (threaded) for 2006-11
[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index][Thread Index]
At 11:42 AM 11/26/2006, Matthew Dillon wrote:
First one stream, then a second... Rate is about the same.
Then I load the kld, and add some poorly written rules
Same throughput
Whats odd however, is
Rule 1000 should have WAY more hits
And sure enough
But when I use all, it seems to work
Something seems busted.
Yes, if I set the polling hz to 5000, I am getting 596,231 on the single stream, and when I bring online the stream in the opposite direction, I get 335,006 in one direction and 160,262 in the other.
doesnt seem to have much impact statistically.
Even at 5000Hz, throughput is pretty steady... what comes in, goes out
Top shows
load averages: 0.18, 0.05, 0.01 up 0+05:17:52 15:58:38
29 processes: 1 running, 28 sleeping
CPU0 states: 0.0% user, 0.0% nice, 88.7% system, 0.0% interrupt, 11.3% idle
CPU1 states: 0.0% user, 0.0% nice, 0.0% system, 1.6% interrupt, 98.4% idle
Mem: 9432K Active, 10M Inact, 58M Wired, 9216K Buf, 1929M Free
One thing I found on FreeBSD that helps with polling is setting kern.polling.idle_poll=1
By comparison, FreeBSD 7 with polling (no firewall) and NP (non polling, normal HZ)
[r2-current]# sysctl -a kern.clockrate
kern.clockrate: { hz = 4000, tick = 250, profhz = 2666, stathz = 533 }
[r2-current]#
4000 582,652 288,532 290,964
NP 590,068 312,334 320,538
---Mike
[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index][Thread Index]
[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index][Thread Index]
Re: dual port EM nic wedging under load
| From: | Mike Tancsa <mike@xxxxxxxxxx> |
| Date: | Sun, 26 Nov 2006 17:49:15 -0500 |
At 11:42 AM 11/26/2006, Matthew Dillon wrote:
:Using polling and fastfwd on, I am able to get about 300Kpps in a :unidirectional blast and still see that rate even with 10 poorly :written ipfw rules !?!
Are you sure the ipfw ruleset is being executed? :-) ... whenever something looks to be too good to be true....
It does look odd, but this is what I was doing. Here is without ipfw [r2-dragonfly]# kldunload ipfw IP firewall unloaded [r2-dragonfly]# ifstat -b bge0 em0 em1 Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out 0.00 0.00 0.00 0.00 0.00 0.00 2.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.46 0.00 0.00 0.00 0.00 0.00 0.00 0.00 78943.98 0.46 0.46 78872.37 0.46 0.00 140623.2 0.00 0.00 140623.2 0.00 0.00 140622.8 0.00 0.00 140622.8 0.46 0.00 140610.4 0.00 0.00 140610.4 0.00 0.00 140607.5 0.00 0.00 140607.5 2.91 0.00 140501.6 47328.70 47337.99 140501.6 0.00 0.00 140607.2 140607.2 140607.2 140607.2 0.46 0.00 140607.9 140607.9 140607.9 140607.9 0.00 0.00 140607.0 140607.0 140607.0 140607.0 0.46 0.00 140599.7 140599.7 140599.7 140599.7 0.00 0.00 140615.4 140615.4 140615.4 140615.4 0.46 0.00 140606.8 140606.8 140606.8 140606.8 0.00 0.00 140606.2 140606.2 140606.2 140606.2 0.46 0.00 140539.4 140539.4 140609.0 140609.0 0.00 0.00 140596.2 140596.2 140526.5 140526.5 0.00 0.00 140624.2 99093.16 99093.16 140624.2 0.46 0.00 140615.0 0.00 0.00 140615.0
First one stream, then a second... Rate is about the same.
Then I load the kld, and add some poorly written rules
[r2-dragonfly]# ipfw show 00010 0 0 deny log ip from 10.0.0.1 to any 00020 0 0 deny log ip from 172.0.0.1 to any 00030 0 0 deny log tcp from 172.0.1.1 to any 00040 0 0 deny log udp from 172.2.1.1 to any 00050 0 0 deny log tcp from any to 10.90.1.1 135-139 00060 0 0 unreach port tcp from 192.168.43.98 to me 25 00070 0 0 deny log tcp from any to 10.66.2.2 445 00080 0 0 deny log tcp from any to 10.61.2.2 138 00090 0 0 deny log ip from any to 219.0.0.0/16 00100 0 0 deny log esp from 218.0.0.0/8 to any 01000 1 242 allow ip from any to any 65535 0 0 deny ip from any to any [r2-dragonfly]#
Same throughput
[r2-dragonfly]# ifstat -b
bge0 em0 em1
Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out
0.46 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
0.93 0.00 97387.37 0.00 0.00 97378.09
0.00 0.00 140628.8 0.00 0.00 140628.8
0.87 0.00 140607.9 0.00 0.00 140607.9
0.93 0.00 140541.2 0.00 0.00 140541.2
0.00 0.00 140647.7 27050.50 27055.88 140647.7
0.46 0.00 140607.2 140607.2 140607.2 140607.2
0.00 0.00 140606.5 140606.5 140606.5 140606.5
0.46 0.00 140607.2 140607.2 140607.2 140607.2
0.46 0.00 140603.3 140603.3 140603.3 140603.3
0.46 0.00 140611.8 140611.8 140611.8 140611.8
0.00 0.00 140606.9 140606.9 140606.9 140606.9
0.46 0.00 140525.3 140525.3 140525.3 140525.3
0.00 0.00 140598.5 140598.5 140598.5 140598.5
0.46 0.00 140617.8 140617.8 140617.8 140617.8
0.00 0.00 140606.9 140606.9 140606.9 140606.9Whats odd however, is
[r2-dragonfly]# ipfw show 00010 0 0 deny log ip from 10.0.0.1 to any 00020 0 0 deny log ip from 172.0.0.1 to any 00030 0 0 deny log tcp from 172.0.1.1 to any 00040 0 0 deny log udp from 172.2.1.1 to any 00050 0 0 deny log tcp from any to 10.90.1.1 135-139 00060 0 0 unreach port tcp from 192.168.43.98 to me 25 00070 0 0 deny log tcp from any to 10.66.2.2 445 00080 0 0 deny log tcp from any to 10.61.2.2 138 00090 0 0 deny log ip from any to 219.0.0.0/16 00100 0 0 deny log esp from 218.0.0.0/8 to any 01000 482 19162 allow ip from any to any 65535 0 0 deny ip from any to any [r2-dragonfly]#
Rule 1000 should have WAY more hits
And sure enough
[r2-dragonfly]# ipfw show 00005 0 0 deny ip from any to 192.168.44.1 00010 0 0 deny log ip from 10.0.0.1 to any 00020 0 0 deny log ip from 172.0.0.1 to any 00030 0 0 deny log tcp from 172.0.1.1 to any 00040 0 0 deny log udp from 172.2.1.1 to any 00050 0 0 deny log tcp from any to 10.90.1.1 135-139 00060 0 0 unreach port tcp from 192.168.43.98 to me 25 00070 0 0 deny log tcp from any to 10.66.2.2 445 00080 0 0 deny log tcp from any to 10.61.2.2 138 00090 0 0 deny log ip from any to 219.0.0.0/16 00100 0 0 deny log esp from 218.0.0.0/8 to any 00200 0 0 count ip from any to 192.168.44.1 00210 0 0 deny ip from any to 192.168.44.1 01000 489 20536 allow ip from any to any 01010 0 0 count ip from any to 192.168.44.1 65535 0 0 deny ip from any to any [r2-dragonfly]#
[r2-dragonfly]# ipfw add 210 deny ip from any to 192.168.44.1 00210 deny ip from any to 192.168.44.1 [r2-dragonfly]# ipfw show 00010 0 0 deny log ip from 10.0.0.1 to any 00020 0 0 deny log ip from 172.0.0.1 to any 00030 0 0 deny log tcp from 172.0.1.1 to any 00040 0 0 deny log udp from 172.2.1.1 to any 00050 0 0 deny log tcp from any to 10.90.1.1 135-139 00060 0 0 unreach port tcp from 192.168.43.98 to me 25 00070 0 0 deny log tcp from any to 10.66.2.2 445 00080 0 0 deny log tcp from any to 10.61.2.2 138 00090 0 0 deny log ip from any to 219.0.0.0/16 00100 0 0 deny log esp from 218.0.0.0/8 to any 00200 0 0 count ip from any to 192.168.44.1 00210 0 0 deny ip from any to 192.168.44.1 01000 488 20294 allow ip from any to any 01010 0 0 count ip from any to 192.168.44.1 65535 0 0 deny ip from any to any [r2-dragonfly]# ipfw add 5 deny ip from any to 192.168.44.1 00005 deny ip from any to 192.168.44.1 [r2-dragonfly]# ipfw show 00005 0 0 deny ip from any to 192.168.44.1 00010 0 0 deny log ip from 10.0.0.1 to any 00020 0 0 deny log ip from 172.0.0.1 to any 00030 0 0 deny log tcp from 172.0.1.1 to any 00040 0 0 deny log udp from 172.2.1.1 to any 00050 0 0 deny log tcp from any to 10.90.1.1 135-139 00060 0 0 unreach port tcp from 192.168.43.98 to me 25 00070 0 0 deny log tcp from any to 10.66.2.2 445 00080 0 0 deny log tcp from any to 10.61.2.2 138 00090 0 0 deny log ip from any to 219.0.0.0/16 00100 0 0 deny log esp from 218.0.0.0/8 to any 00200 0 0 count ip from any to 192.168.44.1 00210 0 0 deny ip from any to 192.168.44.1 01000 489 20536 allow ip from any to any 01010 0 0 count ip from any to 192.168.44.1 65535 0 0 deny ip from any to any
But when I use all, it seems to work
[r2-dragonfly]# ipfw add 4 deny all from any to 192.168.44.1 00004 deny ip from any to 192.168.44.1 [r2-dragonfly]# ipfw show 00004 238982 9081316 deny ip from any to 192.168.44.1 00005 0 0 deny ip from any to 192.168.44.1 00010 0 0 deny log ip from 10.0.0.1 to any 00020 0 0 deny log ip from 172.0.0.1 to any 00030 0 0 deny log tcp from 172.0.1.1 to any 00040 0 0 deny log udp from 172.2.1.1 to any 00050 0 0 deny log tcp from any to 10.90.1.1 135-139 00060 0 0 unreach port tcp from 192.168.43.98 to me 25 00070 0 0 deny log tcp from any to 10.66.2.2 445 00080 0 0 deny log tcp from any to 10.61.2.2 138 00090 0 0 deny log ip from any to 219.0.0.0/16 00100 0 0 deny log esp from 218.0.0.0/8 to any 00200 0 0 count ip from any to 192.168.44.1 00210 0 0 deny ip from any to 192.168.44.1 01000 793 32521 allow ip from any to any 01010 0 0 count ip from any to 192.168.44.1 65535 0 0 deny ip from any to any [r2-dragonfly]#
Something seems busted.
:Also it handles the load quite smoothly for the bi-directional test : :Here is the output of ifstat -b as seen from the box acting as router : :You can see the first stream starting up, and then the second on the :opposite stream. Rates remain constant throughout, which is quite :different from FreeBSD. ipfw on Dragonfly has no ill effect for some reason. :...
I'm not sure if you are overloading the system intentionally or not. There seem to be an excessive number of ENOBUF issues during packet processing, assuming the count start at 0 when the test began.
:em0: Missed Packets = 221709682 <<<<<< :em0: Receive No Buffers = 53994896 <<<<<< This seems excessive to me :em0: Receive length errors = 0 :em0: Receive errors = 0 :em0: Crc errors = 0 :em0: Alignment errors = 0 :em0: Carrier extension errors = 0 :em0: RX overruns = 55192 <<<<<< :... :em0: Good Packets Rcvd = 187109273 :em0: Good Packets Xmtd = 30553884
:em1: Missed Packets = 17508539 <<<<<< :em1: Receive No Buffers = 14148594 <<<<<< :em1: Receive length errors = 0 :em1: Receive errors = 0 :em1: Crc errors = 0 :em1: Alignment errors = 0 :em1: Carrier extension errors = 0 :em1: RX overruns = 0 <<<<<< :em1: Good Packets Rcvd = 30553871 :em1: Good Packets Xmtd = 187031079
One really nice thing about DragonFly's NETIF polling code is that you can control the polling frequency (kern.polling.pollhz) and other parameters on the fly with sysctl's. You can actually change the polling frequency on a live system!
Yes, if I set the polling hz to 5000, I am getting 596,231 on the single stream, and when I bring online the stream in the opposite direction, I get 335,006 in one direction and 160,262 in the other.
kern.polling.each_burst: 5 HZ 1 blast blast A and blast B 3000 450,072 349,696 349,814 4000 588,514 391,615 391,753 5000 592,655 344,472 360,468 6000 593,966 353,047 352,177
Changing kern.polling.each_burst: 20,40
doesnt seem to have much impact statistically.
Even at 5000Hz, throughput is pretty steady... what comes in, goes out
[r2-dragonfly]# ifstat -b
bge0 em0 em1
Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out
0.47 0.00 158176.3 158176.3 158219.2 158219.2
0.00 0.00 167004.1 167004.1 166937.3 166937.3
0.46 0.00 167158.7 167158.7 167158.7 167158.7
0.00 0.00 167161.0 167161.0 167194.9 167194.9
0.46 0.00 167110.8 167110.8 167076.9 167076.9
0.00 0.00 155460.8 155460.8 155460.8 155460.8
0.46 0.00 167033.7 167033.7 167033.7 167033.7
^C
[r2-dragonfly]#Top shows
load averages: 0.18, 0.05, 0.01 up 0+05:17:52 15:58:38
29 processes: 1 running, 28 sleeping
CPU0 states: 0.0% user, 0.0% nice, 88.7% system, 0.0% interrupt, 11.3% idle
CPU1 states: 0.0% user, 0.0% nice, 0.0% system, 1.6% interrupt, 98.4% idle
Mem: 9432K Active, 10M Inact, 58M Wired, 9216K Buf, 1929M Free
[r2-dragonfly]# sysctl -a kern.polling kern.polling.burst: 119 kern.polling.each_burst: 30 kern.polling.burst_max: 150 kern.polling.user_frac: 50 kern.polling.reg_frac: 20 kern.polling.short_ticks: 39675403 kern.polling.lost_polls: 679638 kern.polling.pending_polls: 1 kern.polling.residual_burst: 0 kern.polling.handlers: 2 kern.polling.enable: 1 kern.polling.phase: 2 kern.polling.suspect: 12710 kern.polling.stalled: 2 kern.polling.pollhz: 4000 [r2-dragonfly]#
One thing I found on FreeBSD that helps with polling is setting kern.polling.idle_poll=1
By comparison, FreeBSD 7 with polling (no firewall) and NP (non polling, normal HZ)
[r2-current]# sysctl -a kern.clockrate
kern.clockrate: { hz = 4000, tick = 250, profhz = 2666, stathz = 533 }
[r2-current]#
4000 582,652 288,532 290,964
NP 590,068 312,334 320,538
---Mike
- References:
- dual port EM nic wedging under load
- From: Mike Tancsa <mike@sentex.net>
- Re: dual port EM nic wedging under load
- From: "Sepherosa Ziehau" <sepherosa@gmail.com>
- Re: dual port EM nic wedging under load
- From: Mike Tancsa <mike@sentex.net>
- Re: dual port EM nic wedging under load
- From: "Sepherosa Ziehau" <sepherosa@gmail.com>
- Re: dual port EM nic wedging under load
- From: Mike Tancsa <mike@sentex.net>
- Re: dual port EM nic wedging under load
- From: "Sepherosa Ziehau" <sepherosa@gmail.com>
- Re: dual port EM nic wedging under load
- From: Mike Tancsa <mike@sentex.net>
- Re: dual port EM nic wedging under load
- From: Matthew Dillon <dillon@apollo.backplane.com>
- dual port EM nic wedging under load
[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index][Thread Index]