Commit graph

44 commits

Author SHA1 Message Date
Jens Axboe
dd67d05152 [PATCH] rbtree: support functions used by the io schedulers
They all duplicate macros to check for empty root and/or node, and
clearing a node. So put those in rbtree.h.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-23 17:10:39 +02:00
Jens Axboe
fd61af0384 [PATCH] cfq-iosched: rq update fixes
- Remember to set ->last_sector so that the cfq_choose_req() logic
  works correctly.

- Remove redundant call to cfq_choose_req()

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-23 17:10:39 +02:00
Jens Axboe
caaa5f9f0a [PATCH] cfq-iosched: many performance fixes
This is a collection of patches that greatly improve CFQ performance
in some circumstances.

- Change the idling logic to only kick in after a request is done and we
  are deciding what to do. Before the idling included the request service
  time, so it was hard to adjust. Now it's true think/idle time.

- Take advantage of TCQ/NCQ/queueing for seeky sync workloads, but keep
  it in control for sync and sequential (or close to) workloads.

- Expire queues immediately and move on to other busy queues, if we are
  not going to idle after the current one finishes.

- Don't rearm idle timer if there are no busy queues. Just leave the
  system idle.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-23 17:10:39 +02:00
Jens Axboe
35e6077cb1 [PATCH] cfq-iosched: correctly set ioprio on both targets
Patch originally from Vasily Tarasov <vtaras@sw.ru>

If you set io-priority of process 1 using sys_ioprio_set system call by
another process 2 (like ionice do), then cfq_init_prio_data() function
sets priority of process 2 (current) on queue of process 1 and clears
the flag, that designates change of ioprio.  So the process  1 will work
like with priority of process 2.

I propose not to call cfq_init_prio_data() on io-priority change, but
only mark queue as queue with changed prority.  Every time when new
request comes cfq-scheduler checks for this flag and atomaticaly changes
priority of queue to new value.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-23 17:10:39 +02:00
Jens Axboe
b31dc66a54 [PATCH] Kill PF_SYNCWRITE flag
A process flag to indicate whether we are doing sync io is incredibly
ugly. It also causes performance problems when one does a lot of async
io and then proceeds to sync it. Part of the io will go out as async,
and the other part as sync. This causes a disconnect between the
previously submitted io and the synced io. For io schedulers such as CFQ,
this will cause us lost merges and suboptimal behaviour in scheduling.

Remove PF_SYNCWRITE completely from the fsync/msync paths, and let
the O_DIRECT path just directly indicate that the writes are sync
by using WRITE_SYNC instead.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-23 17:10:39 +02:00
Jens Axboe
271f18f102 [PATCH] cfq-iosched: Don't set the queue batching limits
We cannot update them if the user changes nr_requests, so don't
set it in the first place. The gains are pretty questionable as
well. The batching loss has been shown to decrease throughput.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-23 17:10:38 +02:00
Linus Torvalds
6b41fd1785 Fix up CFQ scheduler for recent rbtree node shrinkage
The color is now in the low bits of the parent pointer, and initializing
it to 0 happens as part of the whole memset above, so just remove the
unnecessary RB_CLEAR_COLOR.

Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-20 19:44:03 -07:00
Linus Torvalds
2edc322d42 Merge git://git.infradead.org/~dwmw2/rbtree-2.6
* git://git.infradead.org/~dwmw2/rbtree-2.6:
  [RBTREE] Switch rb_colour() et al to en_US spelling of 'color' for consistency
  Update UML kernel/physmem.c to use rb_parent() accessor macro
  [RBTREE] Update hrtimers to use rb_parent() accessor macro.
  [RBTREE] Add explicit alignment to sizeof(long) for struct rb_node.
  [RBTREE] Merge colour and parent fields of struct rb_node.
  [RBTREE] Remove dead code in rb_erase()
  [RBTREE] Update JFFS2 to use rb_parent() accessor macro.
  [RBTREE] Update eventpoll.c to use rb_parent() accessor macro.
  [RBTREE] Update key.c to use rb_parent() accessor macro.
  [RBTREE] Update ext3 to use rb_parent() accessor macro.
  [RBTREE] Change rbtree off-tree marking in I/O schedulers.
  [RBTREE] Add accessor macros for colour and parent fields of rb_node
2006-06-20 14:51:22 -07:00
Jens Axboe
553698f944 [PATCH] cfq-iosched: fix crash in do_div()
We don't clear the seek stat values in cfq_alloc_io_context(), and if
->seek_mean is unlucky enough to be set to -36 by chance, the first
invocation of cfq_update_io_seektime() will oops with a divide by zero
in do_div().

Just memset the entire cic instead of filling invididual values
independently.

Signed-off-by: Jens Axboe <axboe@suse.de>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-14 10:22:16 -07:00
Jens Axboe
bc1c116974 [PATCH] elevator switching race
There's a race between shutting down one io scheduler and firing up the
next, in which a new io could enter and cause the io scheduler to be
invoked with bad or NULL data.

To fix this, we need to maintain the queue lock for a bit longer.
Unfortunately we cannot do that, since the elevator init requires to be
run without the lock held.  This isn't easily fixable, without also
changing the mempool API.  So split the initialization into two parts,
and alloc-init operation and an attach operation.  Then we can
preallocate the io scheduler and related structures, and run the attach
inside the lock after we detach the old one.

This patch has survived 30 minutes of 1 second io scheduler switching
with a very busy io load.

Signed-off-by: Jens Axboe <axboe@suse.de>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-08 15:14:23 -07:00
Jens Axboe
b52a834892 [PATCH] cfq-iosched: busy_rr fairness fix
Now that we select busy_rr for possible service, insert entries at the
back of that list instead of at the front.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-01 18:53:43 +02:00
Jens Axboe
ae818a38d4 [PATCH] cfq-iosched: fix bug in timer handling for the idle class
There's a small window from when the timer is entered and we grab
the queue lock, where cfq_set_active_queue() could be rearming the
timer for us. Seen in the wild on a 12-way ppc box. Fix this by
just using mod_timer(), which will do the right thing for us.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-01 10:13:43 +02:00
Jens Axboe
25776e3594 [PATCH] cfq-iosched: Detect hardware queueing
If the hardware is doing real queueing, decide that it's worthless to
idle the hardware. It does reasonable simultaneous io in that case
anyways, and the idling hurts some work loads.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-01 10:12:26 +02:00
Jens Axboe
12e9fddd6e [PATCH] cfq-iosched: Detect idle process issuing async request
If we are anticipating a sync request from this process and we are
waiting for that and see an async request come in, expire that slice
and move on.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-01 10:09:56 +02:00
Jens Axboe
e0de0206a2 [PATCH] cfq-iosched: check busy queues before deciding we are idle
For just one busy queue (like async write out), we often overlooked
that we could queue more io and decided we were idle instead. This causes
us quite a bit of performance loss.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-06-01 10:07:26 +02:00
Jens Axboe
3793c65c13 [PATCH] cfq-iosched: fixup locking and ->queue_list list management
- Drop cic from the list when seen as dead.
- Fixup the locking, just use a simple spinlock.

Signed-off-by: Jens Axboe <axboe@suse.de>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-05-30 20:31:05 -07:00
David Woodhouse
3db3a44530 [RBTREE] Change rbtree off-tree marking in I/O schedulers.
They were abusing the rb_color field to mark nodes which weren't currently
on the tree. Fix that to use the same method as eventpoll did -- setting
the parent pointer to point back to itself. And use the appropriate
accessor macros for setting and reading the parent.

Signed-off-by: David Woodhouse <dwmw2@infradead.org>
2006-04-21 13:15:17 +01:00
OGAWA Hirofumi
be3b075354 [PATCH] cfq: Further rbtree traversal and cfq_exit_queue() race fix
In current code, we are re-reading cic->key after dead cic->key check.
So, in theory, it may really re-read *after* cfq_exit_queue() seted NULL.

To avoid race, we copy it to stack, then use it. With this change, I
guess gcc will assign cic->key to a register or stack, and it wouldn't
be re-readed.

Signed-off-by: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>
Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-18 19:18:31 +02:00
OGAWA Hirofumi
dbecf3ab40 [PATCH 2/2] cfq: fix cic's rbtree traversal
When queue dies, we set cic->key=NULL as dead mark. So, when we
traverse a rbtree, we must check whether it's still valid key. if it
was invalidated, drop it, then restart the traversal from top.

Signed-off-by: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>
Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-18 09:45:18 +02:00
OGAWA Hirofumi
fba822722e [PATCH 1/2] iosched: fix typo and barrier()
On rmmod path, cfq/as waits to make sure all io-contexts was
freed. However, it's using complete(), not wait_for_completion().

I think barrier() is not enough in here. To avoid the following case,
this patch replaces barrier() with smb_wmb().

	cpu0			visibility			cpu1
	                [ioc_gnone=NULL,ioc_count=1]

ioc_gnone = &all_gone		NULL,ioc_count=1
atomic_read(&ioc_count)		NULL,ioc_count=1
wait_for_completion()		NULL,ioc_count=0	atomic_sub_and_test()
				NULL,ioc_count=0	if ( && ioc_gone)
						    [ioc_gone==NULL,
						    so doesn't call complete()]
			   &all_gone,ioc_count=0

Signed-off-by: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>
Signed-off-by: Jens Axboe <axboe@suse.de>
2006-04-18 09:44:06 +02:00
Jens Axboe
206dc69b31 [BLOCK] cfq-iosched: seek and async performance fixes
Detect whether a given process is seeky and if so disable (mostly) the
idle window if it is. We still allow just a little idle time, just enough
to allow that process to submit a new request. That is needed to maintain
fairness across priority groups.

In some cases, we could setup several async queues. This is not optimal
from a performance POV, since we want all async io in one queue to perform
good sorting on it. It also impacted sync queues, as async io got too much
slice time.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-03-28 13:03:44 +02:00
Andreas Mohr
e8a99053ea [PATCH] cfq-iosched: small cfq_choose_req() optimization
this is a small optimization to cfq_choose_req() in the CFQ I/O scheduler
(this function is a semi-often invoked candidate in an oprofile log):
by using a bit mask variable, we can use a simple switch() to check
the various cases instead of having to query two variables for each check.
Benefit: 251 vs. 285 bytes footprint of cfq_choose_req().
Also, common case 0 (no request wrapping) is now checked first in code.

Signed-off-by: Andreas Mohr <andi@lisas.de>
Signed-off-by: Jens Axboe <axboe@suse.de>
2006-03-28 08:59:49 +02:00
Jens Axboe
e2d74ac066 [PATCH] [BLOCK] cfq-iosched: change cfq io context linking from list to tree
On setups with many disks, we spend a considerable amount of time
looking up the process-disk mapping on each queue of io. Testing with
a NULL based block driver, this costs 40-50% reduction in throughput
for 1000 disks.

Signed-off-by: Jens Axboe <axboe@suse.de>
2006-03-28 08:59:01 +02:00
Matthew Dobson
93d2341c75 [PATCH] mempool: use mempool_create_slab_pool()
Modify well over a dozen mempool users to call mempool_create_slab_pool()
rather than calling mempool_create() with extra arguments, saving about 30
lines of code and increasing readability.

Signed-off-by: Matthew Dobson <colpatch@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-26 08:57:00 -08:00
Al Viro
e572ec7e4e [PATCH] fix rmmod problems with elevator attributes, clean them up 2006-03-18 22:27:18 -05:00
Al Viro
3d1ab40f4c [PATCH] elevator_t lifetime rules and sysfs fixes 2006-03-18 18:35:43 -05:00
Al Viro
1cc9be68eb [PATCH] noise removal: cfq-iosched.c
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:35:08 -05:00
Al Viro
a90d742e4c [PATCH] don't bother with refcounting for cfq_data
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:35:05 -05:00
Al Viro
6f325a1344 [PATCH] fix cfq_get_queue()/ioprio_set(2) races
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:34:17 -05:00
Al Viro
334e94de9b [PATCH] deal with rmmod/put_io_context() races
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:34:15 -05:00
Al Viro
e17a9489b4 [PATCH] stop elv_unregister() from rogering other iosched's data, fix locking
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:34:12 -05:00
Al Viro
25975f863b [PATCH] stop cfq from pinning queue down
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:34:09 -05:00
Al Viro
d9ff418793 [PATCH] make cfq_exit_queue() prune the cfq_io_context for that queue
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:34:07 -05:00
Al Viro
a6a0763a60 [PATCH] fix the exclusion for ioprio_set()
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:34:04 -05:00
Al Viro
12a0573215 [PATCH] keep sync and async cfq_queue separate
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:34:02 -05:00
Al Viro
478a82b0ed [PATCH] switch to use of ->key to get cfq_data by cfq_io_context
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:33:59 -05:00
Al Viro
7670876d2d [PATCH] stop leaking cfq_data in cfq_set_request()
We don't need to pin ->key down; ->cfqq->cfqd will do that for us.
Incidentally, that stops the leak we had - that reference was never
dropped.

Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:33:56 -05:00
Al Viro
b0a6916bcc [PATCH] fix cfq hash lookups
If somebody does a hash lookup for cfq_queue while ioprio of an async queue
is elevated, they shouldn't end up stuck with lowered ioprio when we go back.
Fix is to use ->org_ioprio{,class} in hash lookups.

Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2006-03-18 18:33:54 -05:00
Jens Axboe
7b14e3b52f [PATCH] cfq-iosched: slice expiry fixups
During testing of SLES10, we encountered a hang in the CFQ io scheduler.
Turns out the deferred slice expiry logic is buggy, so remove that for
now.  We could be left with an idle queue that would never wake up.  So
kill that logic, always expire immediately.  Also fix a potential timer
race condition.

Patch looks bigger than it is, because it moves a function.

Signed-off-by: Jens Axboe <axboe@suse.de>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-02-28 00:38:02 -08:00
Arjan van de Ven
64100099ed [BLOCK] mark some block/ variables cons
the patch below marks various read-only variables in block/* as const,
so that gcc can optimize the use of them; eg gcc will replace the use by
the value directly now and will even remove the memory usage of these.

Signed-off-by: Arjan van de Ven <arjan@infradead.org>
Signed-off-by: Jens Axboe <axboe@suse.de>
2006-01-06 09:46:02 +01:00
Coywolf Qi Hunt
eb97b73d75 [BLOCK] new block/ directory comment tidy
Some leftover comments referring to drivers/block that are now block/.
They don't add any information we don't already have, so kill them.

Signed-off-by: Coywolf Qi Hunt <qiyong@fc-cn.com>
Signed-off-by: Jens Axboe <axboe@suse.de>
2005-11-18 21:59:31 +01:00
Tejun Heo
b740d98f56 [BLOCK] cfq-iosched: fix slice_left calculation
When cfq slice expires, remainder of slice is calculated and stored in
cfqq->slice_left.  Current code calculates the opposite of remainder -
how many jiffies the cfqq has used past slice end.  This patch fixes
the bug.

Signed-off-by: Tejun Heo <htejun@gmail.com>
Signed-off-by: Jens Axboe <axboe@suse.de>
2005-11-12 10:56:36 +01:00
Tejun Heo
1b5ed5e1f1 [BLOCK] cfq-iosched: cfq forced dispatching fix
cfq forced dispatching might not return all requests on the queue.
This bug can hang elevator switchinig and corrupt request ordering
during flush sequence.

Signed-off-by: Tejun Heo <htejun@gmail.com>
Signed-off-by: Jens Axboe <axboe@suse.de>
2005-11-12 10:55:51 +01:00
Jens Axboe
3a65dfe8c0 [BLOCK] Move all core block layer code to new block/ directory
drivers/block/ is right now a mix of core and driver parts. Lets move
the core parts to a new top level directory. Al will move the fs/
related block parts to block/ next.

Signed-off-by: Jens Axboe <axboe@suse.de>
2005-11-04 08:43:35 +01:00
Renamed from drivers/block/cfq-iosched.c (Browse further)