I've encountered sort of a performance "regression" in using cfq (and the cfq-based bfq) in comparison with the other io-schedulers:
1) interactivity during load is much better compared to the others (thanks a lot for that, that made me appreciate this scheduler) BUT 2) everything seems to take somewhat longer to load (big applications like firefox, etc. ) 3) hdparm shows the same behavior
since I've started using cfq only for a few days (approx. 1-2 weeks now) I didn't really notice it until I tested "performance" via hdparm:
/dev/sdd: Timing buffered disk reads: 308 MB in 3.01 seconds = 102.22 MB/sec
/dev/sdd: Timing buffered disk reads: 306 MB in 3.01 seconds = 101.66 MB/sec
/dev/sdd: Timing buffered disk reads: 304 MB in 3.02 seconds = 100.77 MB/sec noop [anticipatory] deadline cfq
deadline & noop are similar, the test of noop finishes pretty fast ...
/dev/sdd: Timing buffered disk reads: 170 MB in 3.02 seconds = 56.27 MB/sec
/dev/sdd: Timing buffered disk reads: 176 MB in 3.02 seconds = 58.21 MB/sec
/dev/sdd: Timing buffered disk reads: 176 MB in 3.02 seconds = 58.22 MB/sec noop anticipatory deadline [cfq]
this behavior occurs on an jmicron sata-controller (JMB363/361) and the probably the 4th port of the Intel ICH7R but only with cfq selected, the first (?) and second (?) port of the Intel ICH7R are fine performance-wise, don't know why it's that picky with the other schedulers it's fine
I've tested: 2.6.24-gentoo-r7 (+ 2.6.24.7), 2.6.24-gentoo-r3, 2.6.25, 2.6.25.2 (+ 2.6.25-zen1), 2.6.25-rc8, the kernel of the ubuntu desktop-livecd amd64 (ubuntu 8.04) (cfq enabled) all show this worse "performance" compared to the other schedulers all kernels are amd64 on gentoo ~amd64, glibc-2.7.1, gcc-4.2.3 hardened
hardware: Asus P5W DH Deluxe
I unfortuantely can't test earlier kernel-versions due to the fact that I'm using reiser4 for /(root) and the earlier kernels + reiser4 aren't that stable in terms of data safety
hopefully this is reproducable & you guys can explain if this is something to "worry" about (performance) and/or a real regression or just some kind of placebo effect
Many thanks in advance & thanks a lot for this great scheduler (cfq; I'm looking forward to bfq in mainline which seems to work even better under load)
> Hi, I'm experiencing some cfq/bfq performance issues too, but I'm > still not able to track down the reasons, so before posting on this > topic on the mailing list I'd ask you a couple of questions.
> 1) Are you running the hdparm performance test under some cpu load? > (Even two hdparm instances ran in parallel could do.)
> 2) Does using a bigger value of slice_idle increase the throughput?
Hi,
1) no it was always in (almost) complete idle
2) a bigger value even made it worse, setting it to "0" however seemingly "fixed" it, I however don't know how the overall effect/impact is, this will need some more real-world testing ;)
cat /sys/block/sdd/queue/iosched/slice_idle
0
hdparm -t /dev/sdd
/dev/sdd: Timing buffered disk reads: 314 MB in 3.01 seconds = 104.32 MB/sec
hdparm -t /dev/sdd
/dev/sdd: Timing buffered disk reads: 312 MB in 3.00 seconds = 103.86 MB/sec
hdparm -t /dev/sdd
/dev/sdd: Timing buffered disk reads: 314 MB in 3.01 seconds = 104.24 MB/sec
one side-node / question:
will this cause more wakeups on the cpu and/or decrease battery runtime on, e.g. laptops ?
> Thank you very much, I'll try hdparm on my test boxes and come back > to the list if I find something on that.
> As a sidenote, Ingo is not the author/maintainer of cfq, maybe the > next time CC: Jens Axboe for that.
oops, didn't know that, thanks - didn't want to give the wrong person the "credits" hi & kudos to Jens ;)
here's a nice site which explains all of the settings:
> From: Matthew <jackdac...@gmail.com> > Date: Sat, May 10, 2008 10:39:50PM +0200
> > 2) Does using a bigger value of slice_idle increase the throughput?
> Hi,
> 2) a bigger value even made it worse, setting it to "0" however > seemingly "fixed" it, I however don't know how the overall > effect/impact is, this will need some more real-world testing ;)
Well, it's not a fix... the overall effect is that you should end up with more seeks (and so reduced throughput) on loads consisting of more than one process, and at least one of those processes is a synchronous sequential reader.
> one side-node / question:
> will this cause more wakeups on the cpu and/or decrease battery > runtime on, e.g. laptops ?
I don't know the overall effect on battery life, btw with no idling you have one less timer active in the system (that however, depending on the load, does not fire frequently) and more continuous disk activity. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Matthew wrote: >> 2) Does using a bigger value of slice_idle increase the throughput?
> [..]
> 2) a bigger value even made it worse, setting it to "0" however > seemingly "fixed" it, I however don't know how the overall > effect/impact is, this will need some more real-world testing ;)
As Fabio said, you may lose throughput if you have multiple processes with at least one sync. seq. reader. However, for other workloads, you should see a large global throughput improvement. This is because CFQ tends to idle without too much regard to thinktime or seekiness, often wasting a few ms. The trade-off is that your slow sync. processes may suffer a little.
I've been experiencing this for a while also; an almost 50% regression is seen for single-process reads (ie sync) if slice_idle is 1ms or more (eg default of 8) [1], which seems phenomenal.
Jens, is this the expected price to pay for optimal busy-spindle scheduling, a design issue, bug or am I missing something totally?
Thanks, Daniel
--- [1]
# cat /sys/block/sda/queue/iosched/slice_idle 8 # echo 1 >/proc/sys/vm/drop_caches # dd if=/dev/sda of=/dev/null bs=64k count=5000 5000+0 records in 5000+0 records out 327680000 bytes (328 MB) copied, 4.92922 s, 66.5 MB/s
# echo 0 >/sys/block/sda/queue/iosched/slice_idle # echo 1 >/proc/sys/vm/drop_caches # dd if=/dev/sda of=/dev/null bs=64k count=5000 5000+0 records in 5000+0 records out 327680000 bytes (328 MB) copied, 2.74098 s, 120 MB/s
# hdparm -Tt /dev/sda
/dev/sda: Timing cached reads: 15464 MB in 2.00 seconds = 7741.05 MB/sec Timing buffered disk reads: 342 MB in 3.01 seconds = 113.70 MB/sec
[120MB/s is known platter-rate for this disc, so expected] -- Daniel J Blueman -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > I've been experiencing this for a while also; an almost 50% regression > is seen for single-process reads (ie sync) if slice_idle is 1ms or > more (eg default of 8) [1], which seems phenomenal.
> Jens, is this the expected price to pay for optimal busy-spindle > scheduling, a design issue, bug or am I missing something totally?
> Thanks, > Daniel
> --- [1]
> # cat /sys/block/sda/queue/iosched/slice_idle > 8 > # echo 1 >/proc/sys/vm/drop_caches > # dd if=/dev/sda of=/dev/null bs=64k count=5000 > 5000+0 records in > 5000+0 records out > 327680000 bytes (328 MB) copied, 4.92922 s, 66.5 MB/s
> # echo 0 >/sys/block/sda/queue/iosched/slice_idle > # echo 1 >/proc/sys/vm/drop_caches > # dd if=/dev/sda of=/dev/null bs=64k count=5000 > 5000+0 records in > 5000+0 records out > 327680000 bytes (328 MB) copied, 2.74098 s, 120 MB/s
> # hdparm -Tt /dev/sda
> /dev/sda: > Timing cached reads: 15464 MB in 2.00 seconds = 7741.05 MB/sec > Timing buffered disk reads: 342 MB in 3.01 seconds = 113.70 MB/sec
> [120MB/s is known platter-rate for this disc, so expected]
This appears to be what i get aswell..
root@quadstation # dd if=/dev/sda of=/dev/null bs=64k count=5000 5000+0 records in 5000+0 records out 327680000 bytes (328 MB) copied, 5.48209 s, 59.8 MB/s root@quadstation # echo 0 >/sys/block/sda/queue/iosched/slice_idle root@quadstation # echo 1 >/proc/sys/vm/drop_caches root@quadstation # dd if=/dev/sda of=/dev/null bs=64k count=5000 5000+0 records in 5000+0 records out 327680000 bytes (328 MB) copied, 2.93932 s, 111 MB/s root@quadstation # hdparm -Tt /dev/sda Timing cached reads: 7264 MB in 2.00 seconds = 3633.82 MB/sec Timing buffered disk reads: 322 MB in 3.01 seconds = 107.00 MB/se root@quadstation # echo 0 >/sys/block/sda/queue/iosched/slice_idle root@quadstation # echo 1 >/proc/sys/vm/drop_caches root@quadstation # hdparm -Tt /dev/sda Timing cached reads: 15268 MB in 2.00 seconds = 7643.54 MB/sec Timing buffered disk reads: 328 MB in 3.01 seconds = 108.85 MB/sec
To be sure, i did it all again: noop: root@quadstation # echo 1 >/proc/sys/vm/drop_caches root@quadstation # dd if=/dev/sda of=/dev/null bs=64k count=5000 5000+0 records in 5000+0 records out 327680000 bytes (328 MB) copied, 2.85503 s, 115 MB/s root@quadstation # echo 1 >/proc/sys/vm/drop_caches root@quadstation # hdparm -tT /dev/sda Timing cached reads: 14076 MB in 2.00 seconds = 7045.78 MB/sec Timing buffered disk reads: 328 MB in 3.01 seconds = 109.12 MB/sec
anticipatory: root@quadstation # echo 1 >/proc/sys/vm/drop_caches root@quadstation # dd if=/dev/sda of=/dev/null bs=64k count=5000 5000+0 records in 5000+0 records out 327680000 bytes (328 MB) copied, 2.96948 s, 110 MB/s root@quadstation # echo 1 >/proc/sys/vm/drop_caches root@quadstation # hdparm -tT /dev/sda Timing cached reads: 13424 MB in 2.00 seconds = 6719.29 MB/sec Timing buffered disk reads: 328 MB in 3.01 seconds = 109.13 MB/sec
cfq: root@quadstation # echo 1 >/proc/sys/vm/drop_caches root@quadstation # dd if=/dev/sda of=/dev/null bs=64k count=5000 5000+0 records in 5000+0 records out 327680000 bytes (328 MB) copied, 5.25252 s, 62.4 MB/s root@quadstation # echo 1 >/proc/sys/vm/drop_caches root@quadstation # hdparm -tT /dev/sda Timing cached reads: 13434 MB in 2.00 seconds = 6723.59 MB/sec Timing buffered disk reads: 188 MB in 3.00 seconds = 62.57 MB/sec
Thisd would appear to be quite a considerable performance difference.
On Sun, May 11 2008, Kasper Sandberg wrote: > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > I've been experiencing this for a while also; an almost 50% regression > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > more (eg default of 8) [1], which seems phenomenal.
> > Jens, is this the expected price to pay for optimal busy-spindle > > scheduling, a design issue, bug or am I missing something totally?
> Thisd would appear to be quite a considerable performance difference.
Indeed, that is of course a bug. The initial mail here mentions this as a regression - which kernel was the last that worked ok?
If someone would send me a blktrace of such a slow run, that would be nice. Basically just do a blktrace /dev/sda (or whatever device) while doing the hdparm, preferably storing output files on a difference device. Then send the raw sda.blktrace.* files to me. Thanks!
On Tue, May 13, 2008 at 2:20 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> On Sun, May 11 2008, Kasper Sandberg wrote: > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > > I've been experiencing this for a while also; an almost 50% regression > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > > more (eg default of 8) [1], which seems phenomenal.
> > > Jens, is this the expected price to pay for optimal busy-spindle > > > scheduling, a design issue, bug or am I missing something totally?
> > > Thanks, > > > Daniel
[snip] ... [snip]
> > Thisd would appear to be quite a considerable performance difference.
> Indeed, that is of course a bug. The initial mail here mentions this as > a regression - which kernel was the last that worked ok?
> If someone would send me a blktrace of such a slow run, that would be > nice. Basically just do a blktrace /dev/sda (or whatever device) while > doing the hdparm, preferably storing output files on a difference > device. Then send the raw sda.blktrace.* files to me. Thanks!
> -- > Jens Axboe
Hi Jens,
I called this a "regression" since I wasn't sure if this is a real bug or just something introduced recently, I just started to use cfq as main io-scheduler so I can't tell ...
testing 2.6.17 unfortunately is somewhat impossible for me (reiser4; too new hardware - problems with jmicron)
On Tue, May 13 2008, Matthew wrote: > On Tue, May 13, 2008 at 2:20 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> > On Sun, May 11 2008, Kasper Sandberg wrote: > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > > > I've been experiencing this for a while also; an almost 50% regression > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > > > more (eg default of 8) [1], which seems phenomenal.
> > > > Jens, is this the expected price to pay for optimal busy-spindle > > > > scheduling, a design issue, bug or am I missing something totally?
> > > Thisd would appear to be quite a considerable performance difference.
> > Indeed, that is of course a bug. The initial mail here mentions this as > > a regression - which kernel was the last that worked ok?
> > If someone would send me a blktrace of such a slow run, that would be > > nice. Basically just do a blktrace /dev/sda (or whatever device) while > > doing the hdparm, preferably storing output files on a difference > > device. Then send the raw sda.blktrace.* files to me. Thanks!
> > -- > > Jens Axboe
> Hi Jens,
> I called this a "regression" since I wasn't sure if this is a real bug > or just something introduced recently, I just started to use cfq as > main io-scheduler so I can't tell ...
> testing 2.6.17 unfortunately is somewhat impossible for me (reiser4; > too new hardware - problems with jmicron)
On Tue, 2008-05-13 at 14:20 +0200, Jens Axboe wrote: > On Sun, May 11 2008, Kasper Sandberg wrote: > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > > I've been experiencing this for a while also; an almost 50% regression > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > > more (eg default of 8) [1], which seems phenomenal. <snip>
> > Thisd would appear to be quite a considerable performance difference.
> Indeed, that is of course a bug. The initial mail here mentions this as > a regression - which kernel was the last that worked ok?
I am afraid i cannot exactly tell you..
But i do have some additional information for you.
I have a server running with identical disk to mine, however, with an older intel ahci controller..
This one gets 80mb/s with cfq, and 100mb/s with anticipatory/deadline/noop with hdparm..
This server is running debian stable with a .18 kernel. I am sad to say however, that i will be unable to do any testing on this box, since it is a production server, and i can not shut it down.
haltek:~/blktrace# ./blktrace /dev/sda BLKTRACESETUP: Inappropriate ioctl for device Failed to start trace on /dev/sda
However, on the box where you saw the previous numbers, i sure will be able to provide you with the data you need.
i expect to get around to doing this this afternoon, or tonight at ~02:00 (im GMT+1).
> If someone would send me a blktrace of such a slow run, that would be > nice. Basically just do a blktrace /dev/sda (or whatever device) while > doing the hdparm, preferably storing output files on a difference > device. Then send the raw sda.blktrace.* files to me. Thanks!
On Tue, May 13 2008, Matthew wrote: > On Tue, May 13, 2008 at 3:05 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> > On Tue, May 13 2008, Matthew wrote: > > > On Tue, May 13, 2008 at 2:20 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> > > > On Sun, May 11 2008, Kasper Sandberg wrote: > > > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > > > > > I've been experiencing this for a while also; an almost 50% regression > > > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > > > > > more (eg default of 8) [1], which seems phenomenal.
> > > > > > Jens, is this the expected price to pay for optimal busy-spindle > > > > > > scheduling, a design issue, bug or am I missing something totally?
> unfortunately that snapshot wouldn't compile for me because of an error, > I used the in-tree provided snapshot from portage: 0.0.20071210202527 > I hope that's ok, too;
That's fine, it doesn't really matter. But I'd appreciate if you sent me the compile error in private, so that I can fix it :-)
> attached you'll fine the btrace (2 files) as a tar.bz2 package
> from cfq
> here the corresponding hdparm-output: > hdparm -t /dev/sdd
> /dev/sdd: > Timing buffered disk reads: 152 MB in 3.02 seconds = 50.38 MB/sec
> blktrace /dev/sdd > Device: /dev/sdd > CPU 0: 0 events, 4136 KiB data > CPU 1: 0 events, 11 KiB data > Total: 0 events (dropped 0), 4146 KiB data
> and the corresponding output of anticipatory and attached the btrace of it:
> hdparm -t /dev/sdd
> /dev/sdd: > Timing buffered disk reads: 310 MB in 3.02 seconds = 102.76 MB/sec
> blktrace /dev/sdd > Device: /dev/sdd > CPU 0: 0 events, 7831 KiB data > CPU 1: 0 events, 132 KiB data > Total: 0 events (dropped 0), 7962 KiB data
They seem to start out the same, but then CFQ gets interrupted by a timer unplug (which is also odd) and after that the request size drops. On most devices you don't notice, but some are fairly picky about request sizes. The end result is that CFQ has an average dispatch request size of 142kb, where AS is more than double that at 306kb. I'll need to analyze the data and look at the code a bit more to see WHY this happens.
On Tue, May 13 2008, Jens Axboe wrote: > On Tue, May 13 2008, Matthew wrote: > > On Tue, May 13, 2008 at 3:05 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> > > On Tue, May 13 2008, Matthew wrote: > > > > On Tue, May 13, 2008 at 2:20 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> > > > > On Sun, May 11 2008, Kasper Sandberg wrote: > > > > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > > > > > > I've been experiencing this for a while also; an almost 50% regression > > > > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > > > > > > more (eg default of 8) [1], which seems phenomenal.
> > > > > > > Jens, is this the expected price to pay for optimal busy-spindle > > > > > > > scheduling, a design issue, bug or am I missing something totally?
> > unfortunately that snapshot wouldn't compile for me because of an error, > > I used the in-tree provided snapshot from portage: 0.0.20071210202527 > > I hope that's ok, too;
> That's fine, it doesn't really matter. But I'd appreciate if you sent me > the compile error in private, so that I can fix it :-)
> > attached you'll fine the btrace (2 files) as a tar.bz2 package
> > from cfq
> > here the corresponding hdparm-output: > > hdparm -t /dev/sdd
> > /dev/sdd: > > Timing buffered disk reads: 152 MB in 3.02 seconds = 50.38 MB/sec
> > blktrace /dev/sdd > > Device: /dev/sdd > > CPU 0: 0 events, 4136 KiB data > > CPU 1: 0 events, 11 KiB data > > Total: 0 events (dropped 0), 4146 KiB data
> > and the corresponding output of anticipatory and attached the btrace of it:
> > hdparm -t /dev/sdd
> > /dev/sdd: > > Timing buffered disk reads: 310 MB in 3.02 seconds = 102.76 MB/sec
> > blktrace /dev/sdd > > Device: /dev/sdd > > CPU 0: 0 events, 7831 KiB data > > CPU 1: 0 events, 132 KiB data > > Total: 0 events (dropped 0), 7962 KiB data
> They seem to start out the same, but then CFQ gets interrupted by a > timer unplug (which is also odd) and after that the request size drops. > On most devices you don't notice, but some are fairly picky about > request sizes. The end result is that CFQ has an average dispatch > request size of 142kb, where AS is more than double that at 306kb. I'll > need to analyze the data and look at the code a bit more to see WHY this > happens.
Here's a test patch, I think we get into this situation due to CFQ being a bit too eager to start queuing again. Not tested, I'll need to spend some testing time on this. But I'd appreciate some feedback on whether this changes the situation! The final patch will be a little more involved.
- if (cfqq == cfqd->active_queue) { - /* - * if we are waiting for a request for this queue, let it rip - * immediately and flag that we must not expire this queue - * just now - */ - if (cfq_cfqq_wait_request(cfqq)) { - cfq_mark_cfqq_must_dispatch(cfqq); - del_timer(&cfqd->idle_slice_timer); - blk_start_queueing(cfqd->queue); - } - } else if (cfq_should_preempt(cfqd, cfqq, rq)) { + if ((cfqq != cfqd->active_queue) && + cfq_should_preempt(cfqd, cfqq, rq)) { /* * not the active queue - expire current slice if it is * idle and has expired it's mean thinktime or this new queue
On Tue, May 13, 2008 at 8:40 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> On Tue, May 13 2008, Jens Axboe wrote: > > On Tue, May 13 2008, Matthew wrote: > > > On Tue, May 13, 2008 at 3:05 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> > > > On Tue, May 13 2008, Matthew wrote: > > > > > On Tue, May 13, 2008 at 2:20 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> > > > > > On Sun, May 11 2008, Kasper Sandberg wrote: > > > > > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > > > > > > > I've been experiencing this for a while also; an almost 50% regression > > > > > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > > > > > > > more (eg default of 8) [1], which seems phenomenal.
> > > > > > > > Jens, is this the expected price to pay for optimal busy-spindle > > > > > > > > scheduling, a design issue, bug or am I missing something totally?
> > They seem to start out the same, but then CFQ gets interrupted by a > > timer unplug (which is also odd) and after that the request size drops. > > On most devices you don't notice, but some are fairly picky about > > request sizes. The end result is that CFQ has an average dispatch > > request size of 142kb, where AS is more than double that at 306kb. I'll > > need to analyze the data and look at the code a bit more to see WHY this > > happens.
> Here's a test patch, I think we get into this situation due to CFQ being > a bit too eager to start queuing again. Not tested, I'll need to spend > some testing time on this. But I'd appreciate some feedback on whether > this changes the situation! The final patch will be a little more > involved.
[snip] ... [snip]
> -- > Jens Axboe
unfortunately that patch didn't help:
hdparm -t /dev/sde
/dev/sde: Timing buffered disk reads: 178 MB in 3.03 seconds = 58.67 MB/sec
hdparm -t /dev/sdd
/dev/sdd: Timing buffered disk reads: 164 MB in 3.00 seconds = 54.61 MB/sec
-> the first should be around 74 MB/sec, the second around 102 MB/sec
On Tue, May 13 2008, Matthew wrote: > On Tue, May 13, 2008 at 8:40 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> > On Tue, May 13 2008, Jens Axboe wrote: > > > On Tue, May 13 2008, Matthew wrote: > > > > On Tue, May 13, 2008 at 3:05 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> > > > > On Tue, May 13 2008, Matthew wrote: > > > > > > On Tue, May 13, 2008 at 2:20 PM, Jens Axboe <jens.ax...@oracle.com> wrote:
> > > > > > > On Sun, May 11 2008, Kasper Sandberg wrote: > > > > > > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > > > > > > > > I've been experiencing this for a while also; an almost 50% regression > > > > > > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > > > > > > > > more (eg default of 8) [1], which seems phenomenal.
> > > > > > > > > Jens, is this the expected price to pay for optimal busy-spindle > > > > > > > > > scheduling, a design issue, bug or am I missing something totally?
> > > They seem to start out the same, but then CFQ gets interrupted by a > > > timer unplug (which is also odd) and after that the request size drops. > > > On most devices you don't notice, but some are fairly picky about > > > request sizes. The end result is that CFQ has an average dispatch > > > request size of 142kb, where AS is more than double that at 306kb. I'll > > > need to analyze the data and look at the code a bit more to see WHY this > > > happens.
> > Here's a test patch, I think we get into this situation due to CFQ being > > a bit too eager to start queuing again. Not tested, I'll need to spend > > some testing time on this. But I'd appreciate some feedback on whether > > this changes the situation! The final patch will be a little more > > involved. > [snip] > ... > [snip]
> > -- > > Jens Axboe
> unfortunately that patch didn't help:
> hdparm -t /dev/sde
> /dev/sde: > Timing buffered disk reads: 178 MB in 3.03 seconds = 58.67 MB/sec
> hdparm -t /dev/sdd
> /dev/sdd: > Timing buffered disk reads: 164 MB in 3.00 seconds = 54.61 MB/sec
> -> the first should be around 74 MB/sec, the second around 102 MB/sec
Can you capture blktrace for that run as well, please? Just to have something to compare with.
On Tue, 2008-05-13 at 15:51 +0200, Kasper Sandberg wrote: > On Tue, 2008-05-13 at 14:20 +0200, Jens Axboe wrote: > > On Sun, May 11 2008, Kasper Sandberg wrote: > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > > > I've been experiencing this for a while also; an almost 50% regression > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > > > more (eg default of 8) [1], which seems phenomenal. > <snip>
<snip>
> i expect to get around to doing this this afternoon, or tonight at > ~02:00 > (im GMT+1).
it contains the blktrace with cfq, noop, anticipatory and deadline, along with the output of blktrace and hdparm.
Hope it helps.
> > If someone would send me a blktrace of such a slow run, that would be > > nice. Basically just do a blktrace /dev/sda (or whatever device) while > > doing the hdparm, preferably storing output files on a difference > > device. Then send the raw sda.blktrace.* files to me. Thanks!
> > > > > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > > > > > > > I've been experiencing this for a while also; an almost 50% regression > > > > > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > > > > > > > more (eg default of 8) [1], which seems phenomenal.
> > > > > > > > Jens, is this the expected price to pay for optimal busy-spindle > > > > > > > > scheduling, a design issue, bug or am I missing something totally? [snip] > > They seem to start out the same, but then CFQ gets interrupted by a > > timer unplug (which is also odd) and after that the request size drops. > > On most devices you don't notice, but some are fairly picky about > > request sizes. The end result is that CFQ has an average dispatch > > request size of 142kb, where AS is more than double that at 306kb. I'll > > need to analyze the data and look at the code a bit more to see WHY this > > happens.
> Here's a test patch, I think we get into this situation due to CFQ being > a bit too eager to start queuing again. Not tested, I'll need to spend > some testing time on this. But I'd appreciate some feedback on whether > this changes the situation! The final patch will be a little more > involved.
> - if (cfqq == cfqd->active_queue) { > - /* > - * if we are waiting for a request for this queue, let it rip > - * immediately and flag that we must not expire this queue > - * just now > - */ > - if (cfq_cfqq_wait_request(cfqq)) { > - cfq_mark_cfqq_must_dispatch(cfqq); > - del_timer(&cfqd->idle_slice_timer); > - blk_start_queueing(cfqd->queue); > - } > - } else if (cfq_should_preempt(cfqd, cfqq, rq)) { > + if ((cfqq != cfqd->active_queue) && > + cfq_should_preempt(cfqd, cfqq, rq)) { > /* > * not the active queue - expire current slice if it is > * idle and has expired it's mean thinktime or this new queue
I find this does address the issue (both with 64KB stride dd and hdparm -t; presumably the requests getting merged). Tested on 2.6.26-rc2 on Ubuntu HH 804 x86-64, with slice_idle defaulting to 8 and AHCI on ICH9; disk is ST3320613AS.
Blktrace profiles from 'dd if=/dev/sda of=/dev/null bs=64k count=1000' are at:
Let me know when you have an updated patch to test, Daniel -- Daniel J Blueman -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
On Wed, May 14 2008, Daniel J Blueman wrote: > > > > > > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > > > > > > > > > I've been experiencing this for a while also; an almost 50% regression > > > > > > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > > > > > > > > > more (eg default of 8) [1], which seems phenomenal.
> > > > > > > > > Jens, is this the expected price to pay for optimal busy-spindle > > > > > > > > > scheduling, a design issue, bug or am I missing something totally? > [snip] > > > They seem to start out the same, but then CFQ gets interrupted by a > > > timer unplug (which is also odd) and after that the request size drops. > > > On most devices you don't notice, but some are fairly picky about > > > request sizes. The end result is that CFQ has an average dispatch > > > request size of 142kb, where AS is more than double that at 306kb. I'll > > > need to analyze the data and look at the code a bit more to see WHY this > > > happens.
> > Here's a test patch, I think we get into this situation due to CFQ being > > a bit too eager to start queuing again. Not tested, I'll need to spend > > some testing time on this. But I'd appreciate some feedback on whether > > this changes the situation! The final patch will be a little more > > involved.
> > - if (cfqq == cfqd->active_queue) { > > - /* > > - * if we are waiting for a request for this queue, let it rip > > - * immediately and flag that we must not expire this queue > > - * just now > > - */ > > - if (cfq_cfqq_wait_request(cfqq)) { > > - cfq_mark_cfqq_must_dispatch(cfqq); > > - del_timer(&cfqd->idle_slice_timer); > > - blk_start_queueing(cfqd->queue); > > - } > > - } else if (cfq_should_preempt(cfqd, cfqq, rq)) { > > + if ((cfqq != cfqd->active_queue) && > > + cfq_should_preempt(cfqd, cfqq, rq)) { > > /* > > * not the active queue - expire current slice if it is > > * idle and has expired it's mean thinktime or this new queue
> I find this does address the issue (both with 64KB stride dd and > hdparm -t; presumably the requests getting merged). Tested on > 2.6.26-rc2 on Ubuntu HH 804 x86-64, with slice_idle defaulting to 8 > and AHCI on ICH9; disk is ST3320613AS.
> Blktrace profiles from 'dd if=/dev/sda of=/dev/null bs=64k count=1000' are at:
Goodie! I think the below patch is better - we do want to schedule the queue immediately, but we do not want to interrupt the queuer. So just kick the workqueue handling of the queue instead of entering the dispatcher directly. Can you test this one as well? Thanks!
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c index f4e1006..e8c1941 100644 --- a/block/cfq-iosched.c +++ b/block/cfq-iosched.c @@ -1107,7 +1107,6 @@ static int cfq_dispatch_requests(struct request_queue *q, int force)
if (cfqq == cfqd->active_queue) { - /* - * if we are waiting for a request for this queue, let it rip - * immediately and flag that we must not expire this queue - * just now - */ if (cfq_cfqq_wait_request(cfqq)) { - cfq_mark_cfqq_must_dispatch(cfqq); del_timer(&cfqd->idle_slice_timer); - blk_start_queueing(cfqd->queue); + kblockd_schedule_work(&cfqd->unplug_work); } } else if (cfq_should_preempt(cfqd, cfqq, rq)) { /* @@ -1787,7 +1780,7 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq, */ cfq_preempt_queue(cfqd, cfqq); cfq_mark_cfqq_must_dispatch(cfqq); - blk_start_queueing(cfqd->queue); + kblockd_schedule_work(&cfqd->unplug_work); } }
On Wed, May 14, 2008 at 9:26 AM, Jens Axboe <jens.ax...@oracle.com> wrote: > On Wed, May 14 2008, Daniel J Blueman wrote: >> > > > > > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: >> > > > > > > > > I've been experiencing this for a while also; an almost 50% regression >> > > > > > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or >> > > > > > > > > more (eg default of 8) [1], which seems phenomenal.
>> > > > > > > > > Jens, is this the expected price to pay for optimal busy-spindle >> > > > > > > > > scheduling, a design issue, bug or am I missing something totally? >> [snip] >> > > They seem to start out the same, but then CFQ gets interrupted by a >> > > timer unplug (which is also odd) and after that the request size drops. >> > > On most devices you don't notice, but some are fairly picky about >> > > request sizes. The end result is that CFQ has an average dispatch >> > > request size of 142kb, where AS is more than double that at 306kb. I'll >> > > need to analyze the data and look at the code a bit more to see WHY this >> > > happens.
>> > Here's a test patch, I think we get into this situation due to CFQ being >> > a bit too eager to start queuing again. Not tested, I'll need to spend >> > some testing time on this. But I'd appreciate some feedback on whether >> > this changes the situation! The final patch will be a little more >> > involved.
>> > - if (cfqq == cfqd->active_queue) { >> > - /* >> > - * if we are waiting for a request for this queue, let it rip >> > - * immediately and flag that we must not expire this queue >> > - * just now >> > - */ >> > - if (cfq_cfqq_wait_request(cfqq)) { >> > - cfq_mark_cfqq_must_dispatch(cfqq); >> > - del_timer(&cfqd->idle_slice_timer); >> > - blk_start_queueing(cfqd->queue); >> > - } >> > - } else if (cfq_should_preempt(cfqd, cfqq, rq)) { >> > + if ((cfqq != cfqd->active_queue) && >> > + cfq_should_preempt(cfqd, cfqq, rq)) { >> > /* >> > * not the active queue - expire current slice if it is >> > * idle and has expired it's mean thinktime or this new queue
>> I find this does address the issue (both with 64KB stride dd and >> hdparm -t; presumably the requests getting merged). Tested on >> 2.6.26-rc2 on Ubuntu HH 804 x86-64, with slice_idle defaulting to 8 >> and AHCI on ICH9; disk is ST3320613AS.
>> Blktrace profiles from 'dd if=/dev/sda of=/dev/null bs=64k count=1000' are at:
> Goodie! I think the below patch is better - we do want to schedule the > queue immediately, but we do not want to interrupt the queuer. So just > kick the workqueue handling of the queue instead of entering the > dispatcher directly. Can you test this one as well? Thanks!
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c > index f4e1006..e8c1941 100644 > --- a/block/cfq-iosched.c > +++ b/block/cfq-iosched.c > @@ -1107,7 +1107,6 @@ static int cfq_dispatch_requests(struct request_queue *q, int force)
> if (cfqq == cfqd->active_queue) { > - /* > - * if we are waiting for a request for this queue, let it rip > - * immediately and flag that we must not expire this queue > - * just now > - */ > if (cfq_cfqq_wait_request(cfqq)) { > - cfq_mark_cfqq_must_dispatch(cfqq); > del_timer(&cfqd->idle_slice_timer); > - blk_start_queueing(cfqd->queue); > + kblockd_schedule_work(&cfqd->unplug_work); > } > } else if (cfq_should_preempt(cfqd, cfqq, rq)) { > /* > @@ -1787,7 +1780,7 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq, > */ > cfq_preempt_queue(cfqd, cfqq); > cfq_mark_cfqq_must_dispatch(cfqq); > - blk_start_queueing(cfqd->queue); > + kblockd_schedule_work(&cfqd->unplug_work); > } > }
Applied on top of 2.6.26-rc2, I get platter-speed (118MB/s) with 'dd if=/dev/sda of=/dev/null bs=64k' and 'hdparm -t', so looks good. Identical testing without the patch (ie pure mainline) consistently yields 65MB/s.
<daniel.blue...@gmail.com> wrote: > On Wed, May 14, 2008 at 9:26 AM, Jens Axboe <jens.ax...@oracle.com> wrote: >> On Wed, May 14 2008, Daniel J Blueman wrote: >>> > > > > > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: >>> > > > > > > > > I've been experiencing this for a while also; an almost 50% regression >>> > > > > > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or >>> > > > > > > > > more (eg default of 8) [1], which seems phenomenal.
>>> > > > > > > > > Jens, is this the expected price to pay for optimal busy-spindle >>> > > > > > > > > scheduling, a design issue, bug or am I missing something totally? >>> [snip] >>> > > They seem to start out the same, but then CFQ gets interrupted by a >>> > > timer unplug (which is also odd) and after that the request size drops. >>> > > On most devices you don't notice, but some are fairly picky about >>> > > request sizes. The end result is that CFQ has an average dispatch >>> > > request size of 142kb, where AS is more than double that at 306kb. I'll >>> > > need to analyze the data and look at the code a bit more to see WHY this >>> > > happens.
>>> > Here's a test patch, I think we get into this situation due to CFQ being >>> > a bit too eager to start queuing again. Not tested, I'll need to spend >>> > some testing time on this. But I'd appreciate some feedback on whether >>> > this changes the situation! The final patch will be a little more >>> > involved.
>>> > - if (cfqq == cfqd->active_queue) { >>> > - /* >>> > - * if we are waiting for a request for this queue, let it rip >>> > - * immediately and flag that we must not expire this queue >>> > - * just now >>> > - */ >>> > - if (cfq_cfqq_wait_request(cfqq)) { >>> > - cfq_mark_cfqq_must_dispatch(cfqq); >>> > - del_timer(&cfqd->idle_slice_timer); >>> > - blk_start_queueing(cfqd->queue); >>> > - } >>> > - } else if (cfq_should_preempt(cfqd, cfqq, rq)) { >>> > + if ((cfqq != cfqd->active_queue) && >>> > + cfq_should_preempt(cfqd, cfqq, rq)) { >>> > /* >>> > * not the active queue - expire current slice if it is >>> > * idle and has expired it's mean thinktime or this new queue
>>> I find this does address the issue (both with 64KB stride dd and >>> hdparm -t; presumably the requests getting merged). Tested on >>> 2.6.26-rc2 on Ubuntu HH 804 x86-64, with slice_idle defaulting to 8 >>> and AHCI on ICH9; disk is ST3320613AS.
>>> Blktrace profiles from 'dd if=/dev/sda of=/dev/null bs=64k count=1000' are at:
>> Goodie! I think the below patch is better - we do want to schedule the >> queue immediately, but we do not want to interrupt the queuer. So just >> kick the workqueue handling of the queue instead of entering the >> dispatcher directly. Can you test this one as well? Thanks!
>> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c >> index f4e1006..e8c1941 100644 >> --- a/block/cfq-iosched.c >> +++ b/block/cfq-iosched.c >> @@ -1107,7 +1107,6 @@ static int cfq_dispatch_requests(struct request_queue *q, int force)
>> if (cfqq == cfqd->active_queue) { >> - /* >> - * if we are waiting for a request for this queue, let it rip >> - * immediately and flag that we must not expire this queue >> - * just now >> - */ >> if (cfq_cfqq_wait_request(cfqq)) { >> - cfq_mark_cfqq_must_dispatch(cfqq); >> del_timer(&cfqd->idle_slice_timer); >> - blk_start_queueing(cfqd->queue); >> + kblockd_schedule_work(&cfqd->unplug_work); >> } >> } else if (cfq_should_preempt(cfqd, cfqq, rq)) { >> /* >> @@ -1787,7 +1780,7 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq, >> */ >> cfq_preempt_queue(cfqd, cfqq); >> cfq_mark_cfqq_must_dispatch(cfqq); >> - blk_start_queueing(cfqd->queue); >> + kblockd_schedule_work(&cfqd->unplug_work); >> } >> }
> Applied on top of 2.6.26-rc2, I get platter-speed (118MB/s) with 'dd > if=/dev/sda of=/dev/null bs=64k' and 'hdparm -t', so looks good. > Identical testing without the patch (ie pure mainline) consistently > yields 65MB/s.
On Wed, May 14 2008, Matthew wrote: > On Wed, May 14, 2008 at 10:52 PM, Daniel J Blueman > <daniel.blue...@gmail.com> wrote: > > On Wed, May 14, 2008 at 9:26 AM, Jens Axboe <jens.ax...@oracle.com> wrote: > >> On Wed, May 14 2008, Daniel J Blueman wrote: > >>> > > > > > > > On Sun, 2008-05-11 at 14:14 +0100, Daniel J Blueman wrote: > >>> > > > > > > > > I've been experiencing this for a while also; an almost 50% regression > >>> > > > > > > > > is seen for single-process reads (ie sync) if slice_idle is 1ms or > >>> > > > > > > > > more (eg default of 8) [1], which seems phenomenal.
> >>> > > > > > > > > Jens, is this the expected price to pay for optimal busy-spindle > >>> > > > > > > > > scheduling, a design issue, bug or am I missing something totally? > >>> [snip] > >>> > > They seem to start out the same, but then CFQ gets interrupted by a > >>> > > timer unplug (which is also odd) and after that the request size drops. > >>> > > On most devices you don't notice, but some are fairly picky about > >>> > > request sizes. The end result is that CFQ has an average dispatch > >>> > > request size of 142kb, where AS is more than double that at 306kb. I'll > >>> > > need to analyze the data and look at the code a bit more to see WHY this > >>> > > happens.
> >>> > Here's a test patch, I think we get into this situation due to CFQ being > >>> > a bit too eager to start queuing again. Not tested, I'll need to spend > >>> > some testing time on this. But I'd appreciate some feedback on whether > >>> > this changes the situation! The final patch will be a little more > >>> > involved.
> >>> > - if (cfqq == cfqd->active_queue) { > >>> > - /* > >>> > - * if we are waiting for a request for this queue, let it rip > >>> > - * immediately and flag that we must not expire this queue > >>> > - * just now > >>> > - */ > >>> > - if (cfq_cfqq_wait_request(cfqq)) { > >>> > - cfq_mark_cfqq_must_dispatch(cfqq); > >>> > - del_timer(&cfqd->idle_slice_timer); > >>> > - blk_start_queueing(cfqd->queue); > >>> > - } > >>> > - } else if (cfq_should_preempt(cfqd, cfqq, rq)) { > >>> > + if ((cfqq != cfqd->active_queue) && > >>> > + cfq_should_preempt(cfqd, cfqq, rq)) { > >>> > /* > >>> > * not the active queue - expire current slice if it is > >>> > * idle and has expired it's mean thinktime or this new queue
> >>> I find this does address the issue (both with 64KB stride dd and > >>> hdparm -t; presumably the requests getting merged). Tested on > >>> 2.6.26-rc2 on Ubuntu HH 804 x86-64, with slice_idle defaulting to 8 > >>> and AHCI on ICH9; disk is ST3320613AS.
> >>> Blktrace profiles from 'dd if=/dev/sda of=/dev/null bs=64k count=1000' are at:
> >> Goodie! I think the below patch is better - we do want to schedule the > >> queue immediately, but we do not want to interrupt the queuer. So just > >> kick the workqueue handling of the queue instead of entering the > >> dispatcher directly. Can you test this one as well? Thanks!
> >> if (cfqq == cfqd->active_queue) { > >> - /* > >> - * if we are waiting for a request for this queue, let it rip > >> - * immediately and flag that we must not expire this queue > >> - * just now > >> - */ > >> if (cfq_cfqq_wait_request(cfqq)) { > >> - cfq_mark_cfqq_must_dispatch(cfqq); > >> del_timer(&cfqd->idle_slice_timer); > >> - blk_start_queueing(cfqd->queue); > >> + kblockd_schedule_work(&cfqd->unplug_work); > >> } > >> } else if (cfq_should_preempt(cfqd, cfqq, rq)) { > >> /* > >> @@ -1787,7 +1780,7 @@ cfq_rq_enqueued(struct cfq_data *cfqd, struct cfq_queue *cfqq, > >> */ > >> cfq_preempt_queue(cfqd, cfqq); > >> cfq_mark_cfqq_must_dispatch(cfqq); > >> - blk_start_queueing(cfqd->queue); > >> + kblockd_schedule_work(&cfqd->unplug_work); > >> } > >> }
> > Applied on top of 2.6.26-rc2, I get platter-speed (118MB/s) with 'dd > > if=/dev/sda of=/dev/null bs=64k' and 'hdparm -t', so looks good. > > Identical testing without the patch (ie pure mainline) consistently > > yields 65MB/s.
> > I'll check for performance regressions with postmark on XFS; anything > > else worth running while I've got this in hand?
> > Daniel > > -- > > Daniel J Blueman
> so it seems something specific to >=2.6.26-rc2 + the patch fixed it for you ?
> were there any notable changes from 2.6.25 -> 2.6.26 in cfq or the VFS > in general ?
> I'm curious if this also works with 2.6.25, could you please test that too ?
> I'll give .26-rc2 a test-ride later
I don't think it's 2.6.25 vs 2.6.26-rc2, I can still reproduce some request size offsets with the patch. So still fumbling around with this, I'll be sending out another test patch when I'm confident it's solved the size issue.
> I don't think it's 2.6.25 vs 2.6.26-rc2, I can still reproduce some > request size offsets with the patch. So still fumbling around with this, > I'll be sending out another test patch when I'm confident it's solved > the size issue.
IMO an interesting thing is how/why anticipatory doesn't show the issue. The device is not put into ANTIC_WAIT_NEXT if there is no dispatch returning no requests while the queue is not empty. This seems to be enough in the reported workloads.
I don't think this behavior is the correct one (it is still racy WRT merges after breaking anticipation) anyway it should make things a little bit better. I fear that a complete solution would not involve only the scheduler.
Introducing the very same behavior in cfq seems to be not so easy (i.e., start idling only if there was a dispatch round while the last request was being served) but an approximated version can be introduced quite easily. The patch below should do that, rescheduling the dispatch only if necessary; it is not tested at all, just posted for discussion.
--- diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c index b399c62..41f1e0e 100644 --- a/block/cfq-iosched.c +++ b/block/cfq-iosched.c @@ -169,6 +169,7 @@ enum cfqq_state_flags { CFQ_CFQQ_FLAG_queue_new, /* queue never been serviced */ CFQ_CFQQ_FLAG_slice_new, /* no requests dispatched in slice */ CFQ_CFQQ_FLAG_sync, /* synchronous queue */ + CFQ_CFQQ_FLAG_dispatched, /* empty dispatch while idling */ };
On Thu, May 15 2008, Fabio Checconi wrote: > > From: Jens Axboe <jens.ax...@oracle.com> > > Date: Thu, May 15, 2008 09:01:28AM +0200
> > I don't think it's 2.6.25 vs 2.6.26-rc2, I can still reproduce some > > request size offsets with the patch. So still fumbling around with this, > > I'll be sending out another test patch when I'm confident it's solved > > the size issue.
> IMO an interesting thing is how/why anticipatory doesn't show the > issue. The device is not put into ANTIC_WAIT_NEXT if there is no > dispatch returning no requests while the queue is not empty. This > seems to be enough in the reported workloads.
> I don't think this behavior is the correct one (it is still racy > WRT merges after breaking anticipation) anyway it should make things > a little bit better. I fear that a complete solution would not > involve only the scheduler.
> Introducing the very same behavior in cfq seems to be not so easy > (i.e., start idling only if there was a dispatch round while the > last request was being served) but an approximated version can be > introduced quite easily. The patch below should do that, rescheduling > the dispatch only if necessary; it is not tested at all, just posted > for discussion.
Daniel (and others in this thread), can you give this a shot as well? It looks promising, it'll allow greater buildup of the request. From my testing, instead of getting nicely aligned 128k or 256k requests, we'd end up in a nasty 4k+124k stream. Delaying the first queue kick should fix that, since we wont dispatch that first 4k request until it has been merged.
I think we can improve this further without getting too involved. If a 2nd request is seen in cfq_rq_enqueued(), then DO schedule a dispatch since this likely means that we wont be doing more merges on the first one.
On Fri, May 16 2008, Fabio Checconi wrote: > > From: Jens Axboe <jens.ax...@oracle.com> > > Date: Fri, May 16, 2008 08:40:03AM +0200
> ... > > I think we can improve this further without getting too involved. If a > > 2nd request is seen in cfq_rq_enqueued(), then DO schedule a dispatch > > since this likely means that we wont be doing more merges on the first > > one.
> But isn't there the risk that even the second request would be > dispatched, while it still could have grown?
Certainly, you'd only want to dispatch the first request. Ideally we'd just get rid of this logic of 'did empty dispatch round' and only dispatch requests once merging is done, it's basically the wrong thing to do to make it visible to the io scheduler so soon. Well of course even more ideally we'd always get big requests submitted, but unfortunately many producers aren't that nice.
The per-process plugging actually solves this nicely, since we do the merging outside of the io scheduler. Perhaps just not dispatch on a plugged queue would help a bit. I'm somewhat against this principle of messing too much with dispatch logic in the schedulers, it'd be nicer to solve this higher up.
... > I think we can improve this further without getting too involved. If a > 2nd request is seen in cfq_rq_enqueued(), then DO schedule a dispatch > since this likely means that we wont be doing more merges on the first > one.
But isn't there the risk that even the second request would be dispatched, while it still could have grown?
Moreover I am still unsure about how to handle (and if it's worth handling) the case in which we restart queueing after an empty dispatch round due to idling, as it would still have the same problem.
(Also anticipatory doesn't handle this case too well.) -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
On Fri, May 16 2008, Jens Axboe wrote: > On Fri, May 16 2008, Fabio Checconi wrote: > > > From: Jens Axboe <jens.ax...@oracle.com> > > > Date: Fri, May 16, 2008 08:40:03AM +0200
> > ... > > > I think we can improve this further without getting too involved. If a > > > 2nd request is seen in cfq_rq_enqueued(), then DO schedule a dispatch > > > since this likely means that we wont be doing more merges on the first > > > one.
> > But isn't there the risk that even the second request would be > > dispatched, while it still could have grown?
> Certainly, you'd only want to dispatch the first request. Ideally we'd > just get rid of this logic of 'did empty dispatch round' and only > dispatch requests once merging is done, it's basically the wrong thing > to do to make it visible to the io scheduler so soon. Well of course > even more ideally we'd always get big requests submitted, but > unfortunately many producers aren't that nice.
> The per-process plugging actually solves this nicely, since we do the > merging outside of the io scheduler. Perhaps just not dispatch on a > plugged queue would help a bit. I'm somewhat against this principle of > messing too much with dispatch logic in the schedulers, it'd be nicer to > solve this higher up.
|
