High Disk Queue Length

Hi all,
(This is a repost, as my post from an hour earlier
doesn't seem to have made it. Please excuse if
duplicate.)
We've got a problem at the moment with high disk queue
length, and it's puzzling us.
We've got a Compaq DL580G2 with 4 fast procs and 32 GB of
ram. It's running Windows 2003 Enterprise and SQL Server
2000 Enterprise.
It has Compaq's 5300 controller (four channels), and
we've got three drive enclosures attached to it (so using
3 of the four controller channels). Each of the
enclosures contains 7x36GB drives (15k), and 20 of the 21
are in a RAID 5 array, with the remainder being a hot
spare.
We process 800 - 1000 batches per second, and here are
some stats on the physical disk:
- Disk throughput is about 8 mb per second (per
perfmon), and is split almost exactly evenly between
reads and writes.
- Total disk queue is averaging 120 (way too high), with
2/3 of that coming from the write queue.
- Checkpoints are almost always occurring, only a second
or two between them.
We used to (until last night) have only one drive
enclosure and 14 drives, and were convinced that our high
disk queue was due to the bottleneck of using only one
controller channel. Now we have three enclosures, each
on their own channel, and 50% more drives, and disk queue
is exactly the same as before.
Plus, 8 mb per second seems very low to be a disk
bottleneck, and yet we have a high disk queue.
Any hints on where to look next? Is there any way to
pinpoint the exact bottleneck? (i.e. controller
channels, disk throughput, bus speed, anything else)
Thanks,
MyronIt did make it since I responded to it. You might want to check again.
--
Andrew J. Kelly
SQL Server MVP
"Myron Schroner" <myronschroner@.yahoo.com> wrote in message
news:036501c3470c$8d2cf7e0$a101280a@.phx.gbl...
> Hi all,
> (This is a repost, as my post from an hour earlier
> doesn't seem to have made it. Please excuse if
> duplicate.)
> We've got a problem at the moment with high disk queue
> length, and it's puzzling us.
> We've got a Compaq DL580G2 with 4 fast procs and 32 GB of
> ram. It's running Windows 2003 Enterprise and SQL Server
> 2000 Enterprise.
> It has Compaq's 5300 controller (four channels), and
> we've got three drive enclosures attached to it (so using
> 3 of the four controller channels). Each of the
> enclosures contains 7x36GB drives (15k), and 20 of the 21
> are in a RAID 5 array, with the remainder being a hot
> spare.
> We process 800 - 1000 batches per second, and here are
> some stats on the physical disk:
> - Disk throughput is about 8 mb per second (per
> perfmon), and is split almost exactly evenly between
> reads and writes.
> - Total disk queue is averaging 120 (way too high), with
> 2/3 of that coming from the write queue.
> - Checkpoints are almost always occurring, only a second
> or two between them.
> We used to (until last night) have only one drive
> enclosure and 14 drives, and were convinced that our high
> disk queue was due to the bottleneck of using only one
> controller channel. Now we have three enclosures, each
> on their own channel, and 50% more drives, and disk queue
> is exactly the same as before.
> Plus, 8 mb per second seems very low to be a disk
> bottleneck, and yet we have a high disk queue.
> Any hints on where to look next? Is there any way to
> pinpoint the exact bottleneck? (i.e. controller
> channels, disk throughput, bus speed, anything else)
> Thanks,
>
> Myron
>|||i dont' see the original post or Andrews reply
first, 2 drives should be in RAID1 for the log,
as for data, you are most likely doing random read/writes,
so raw bandwidth in MB/sec is irrelevent,
for 20 15k drives in RAID5 you can expect 150x20/4 = 600
write IO/sec. In RAID 1+0, you should get 1,200 write
IO/sec.
use perfmon to get the read/sec & writes/sec for the
drives with the data files, if the log are on the same
partition, move them somewhere else as that will muddy the
analysis
>--Original Message--
>Hi all,
>(This is a repost, as my post from an hour earlier
>doesn't seem to have made it. Please excuse if
>duplicate.)
>We've got a problem at the moment with high disk queue
>length, and it's puzzling us.
>We've got a Compaq DL580G2 with 4 fast procs and 32 GB of
>ram. It's running Windows 2003 Enterprise and SQL Server
>2000 Enterprise.
>It has Compaq's 5300 controller (four channels), and
>we've got three drive enclosures attached to it (so using
>3 of the four controller channels). Each of the
>enclosures contains 7x36GB drives (15k), and 20 of the 21
>are in a RAID 5 array, with the remainder being a hot
>spare.
>We process 800 - 1000 batches per second, and here are
>some stats on the physical disk:
> - Disk throughput is about 8 mb per second (per
>perfmon), and is split almost exactly evenly between
>reads and writes.
> - Total disk queue is averaging 120 (way too high), with
>2/3 of that coming from the write queue.
> - Checkpoints are almost always occurring, only a second
>or two between them.
>We used to (until last night) have only one drive
>enclosure and 14 drives, and were convinced that our high
>disk queue was due to the bottleneck of using only one
>controller channel. Now we have three enclosures, each
>on their own channel, and 50% more drives, and disk queue
>is exactly the same as before.
>Plus, 8 mb per second seems very low to be a disk
>bottleneck, and yet we have a high disk queue.
>Any hints on where to look next? Is there any way to
>pinpoint the exact bottleneck? (i.e. controller
>channels, disk throughput, bus speed, anything else)
>Thanks,
>
>Myron
>.
>|||Sorry, Andrew, I can now see my earlier post, but not your reply. Would
you mind re-posting?
Thanks!
Myron
In article <#WnlvSxRDHA.1556@.TK2MSFTNGP11.phx.gbl>,
sqlmvpnooospam@.shadhawk.com says...
> It did make it since I responded to it. You might want to check again.
>|||It was essentially what Joe just said in his post. The key is to make sure
the logs are on their own Raid 1 and check your recovery interval setting.
Sounds like someone may have changed it from the default. Make sure you
have Write back cached enabled on the controller and ensure it is battery
backed.
--
Andrew J. Kelly
SQL Server MVP
"Myron Schroner" <myronschroner@.yahoo.com> wrote in message
news:MPG.1977641bd4c1f416989680@.news.microsoft.com...
> Sorry, Andrew, I can now see my earlier post, but not your reply. Would
> you mind re-posting?
> Thanks!
> Myron
> In article <#WnlvSxRDHA.1556@.TK2MSFTNGP11.phx.gbl>,
> sqlmvpnooospam@.shadhawk.com says...
> > It did make it since I responded to it. You might want to check again.
> >
> >|||Thanks for that, Joe.
1. We do have the logs split off into their own raid 1 array, and there
are no issues with queueing on that array. It's only the data array
that has queueing.
2. That's a nifty formula for calculating expected write io's. Is that
an industry standard, or an experienced guess, or ... ? And is there
some formula for read IO?
3. Any chance that by setting up the 20 drives across 3 enclosures (each
with their own controller channel) that we're somehow still using only
one channel primarily? Doesn't seem possible, but figured I should ask.
4. The read/sec is ~725, and the write/sec is ~350. But total disk
queue is averaging 120, with 2/3 of that coming from the write queue.
Based on the fact that reads was higher than writes, we went with RAID
5, thinking that it's faster for reads. Maybe though it's just a matter
of not having enough drives?
Thanks again!
Myron
In article <01ee01c3471c$d3cc8c00$a501280a@.phx.gbl>, jchang6@.yahoo.com
says...
> i dont' see the original post or Andrews reply
> first, 2 drives should be in RAID1 for the log,
> as for data, you are most likely doing random read/writes,
> so raw bandwidth in MB/sec is irrelevent,
> for 20 15k drives in RAID5 you can expect 150x20/4 = 600
> write IO/sec. In RAID 1+0, you should get 1,200 write
> IO/sec.
> use perfmon to get the read/sec & writes/sec for the
> drives with the data files, if the log are on the same
> partition, move them somewhere else as that will muddy the
> analysis
> >--Original Message--
> >Hi all,
> >(This is a repost, as my post from an hour earlier
> >doesn't seem to have made it. Please excuse if
> >duplicate.)
> >
> >We've got a problem at the moment with high disk queue
> >length, and it's puzzling us.
> >
> >We've got a Compaq DL580G2 with 4 fast procs and 32 GB of
> >ram. It's running Windows 2003 Enterprise and SQL Server
> >2000 Enterprise.
> >
> >It has Compaq's 5300 controller (four channels), and
> >we've got three drive enclosures attached to it (so using
> >3 of the four controller channels). Each of the
> >enclosures contains 7x36GB drives (15k), and 20 of the 21
> >are in a RAID 5 array, with the remainder being a hot
> >spare.
> >
> >We process 800 - 1000 batches per second, and here are
> >some stats on the physical disk:
> >
> > - Disk throughput is about 8 mb per second (per
> >perfmon), and is split almost exactly evenly between
> >reads and writes.
> >
> > - Total disk queue is averaging 120 (way too high), with
> >2/3 of that coming from the write queue.
> >
> > - Checkpoints are almost always occurring, only a second
> >or two between them.
> >
> >We used to (until last night) have only one drive
> >enclosure and 14 drives, and were convinced that our high
> >disk queue was due to the bottleneck of using only one|||this is the common formula (used in TPC-C benchmarking
analysis) for small block random IO on disk drives. it is
based on the average rotational latency (2ms for a 15K
disk to rotate 180 deg) + the avg seek time (4ms) + data
xfer time (<0.2ms) for a total seek time of 6.2ms
this means that at an average queue depth of 1 (one
outstanding request to the disk)
the IO rate is 1000ms/sec / 6.2ms = 160/sec
at higher queue depth, the disk should be able to re-order
the seeks, and achieve higher IO rate, at the expense of
longer latency (avg sec/xfer)
as a rought approximation, the following applies:
7.2k 75 IO/sec
10k 100
15k 150
in any raid level, reads can occur from any disk, so the
total read rate for 15k disk is 150 x number of drives
for writes, RAID1 requires 2 writes, hence the net is 1/2
of the base rate
RAID5 requires 2 reads (for the data block and the parity
block) and 2 writes (for data and parity)
hence the overall random small block formula for 15k disks
is:
RAID0: reads/sec + writes/sec = 150 * number of drives
RAID1: reads/sec + 2x writes/sec = 150 * drives
RAID5: reads/sec + 4x writes/sec = 150 * drives
your situation is:
20 15k drives are capable of 20x150 = 3,000 IO/sec
your actual physical disk IO is
reads writes
725 + 4 x 350 = 2,125
if you were in RAID1, it would be
725 + 2 x 350 = 1,425
so you should be ok,
did you configure all 20 drives in one RAID5 array or
separate arrays. If 1, you should see a queue depth of <20
(1 per disk)
i don't know why your queue depth is so high
what is the avg sec/read & /write?
if these numbers are approx equal to or less than the avg
access time (6ms), don't worry about the queue depth,
Wait, are you dumping the batch of 800-1000 inserts all at
once?
if so, the queue depth may spike, but the avg should be <20
the compaq 5304 has battery backup (pretty much all high-
end controllers do) so you may as well enable write
caching on both the controller and in W2K on the disk
properties page
>--Original Message--
>Thanks for that, Joe.
>1. We do have the logs split off into their own raid 1
array, and there
>are no issues with queueing on that array. It's only the
data array
>that has queueing.
>2. That's a nifty formula for calculating expected write
io's. Is that
>an industry standard, or an experienced guess, or ... ?
And is there
>some formula for read IO?
>3. Any chance that by setting up the 20 drives across 3
enclosures (each
>with their own controller channel) that we're somehow
still using only
>one channel primarily? Doesn't seem possible, but
figured I should ask.
>4. The read/sec is ~725, and the write/sec is ~350. But
total disk
>queue is averaging 120, with 2/3 of that coming from the
write queue.
>Based on the fact that reads was higher than writes, we
went with RAID
>5, thinking that it's faster for reads. Maybe though
it's just a matter
>of not having enough drives?
>Thanks again!
>Myron
>
>In article <01ee01c3471c$d3cc8c00$a501280a@.phx.gbl>,
jchang6@.yahoo.com
>says...
>> i dont' see the original post or Andrews reply
>> first, 2 drives should be in RAID1 for the log,
>> as for data, you are most likely doing random
read/writes,
>> so raw bandwidth in MB/sec is irrelevent,
>> for 20 15k drives in RAID5 you can expect 150x20/4 =600
>> write IO/sec. In RAID 1+0, you should get 1,200 write
>> IO/sec.
>> use perfmon to get the read/sec & writes/sec for the
>> drives with the data files, if the log are on the same
>> partition, move them somewhere else as that will muddy
the
>> analysis
>> >--Original Message--
>> >Hi all,
>> >(This is a repost, as my post from an hour earlier
>> >doesn't seem to have made it. Please excuse if
>> >duplicate.)
>> >
>> >We've got a problem at the moment with high disk queue
>> >length, and it's puzzling us.
>> >
>> >We've got a Compaq DL580G2 with 4 fast procs and 32 GB
of
>> >ram. It's running Windows 2003 Enterprise and SQL
Server
>> >2000 Enterprise.
>> >
>> >It has Compaq's 5300 controller (four channels), and
>> >we've got three drive enclosures attached to it (so
using
>> >3 of the four controller channels). Each of the
>> >enclosures contains 7x36GB drives (15k), and 20 of the
21
>> >are in a RAID 5 array, with the remainder being a hot
>> >spare.
>> >
>> >We process 800 - 1000 batches per second, and here are
>> >some stats on the physical disk:
>> >
>> > - Disk throughput is about 8 mb per second (per
>> >perfmon), and is split almost exactly evenly between
>> >reads and writes.
>> >
>> > - Total disk queue is averaging 120 (way too high),
with
>> >2/3 of that coming from the write queue.
>> >
>> > - Checkpoints are almost always occurring, only a
second
>> >or two between them.
>> >
>> >We used to (until last night) have only one drive
>> >enclosure and 14 drives, and were convinced that our
high
>> >disk queue was due to the bottleneck of using only one
>.
>|||how did i forget,
change the 5304 cache setting from 50%/50% read/write to
100% write, ie, never try to cache reads,
you might send compaq an email on this
-joe
>--Original Message--
>Thanks for that, Joe.
>1. We do have the logs split off into their own raid 1
array, and there
>are no issues with queueing on that array. It's only the
data array
>that has queueing.
>2. That's a nifty formula for calculating expected write
io's. Is that
>an industry standard, or an experienced guess, or ... ?
And is there
>some formula for read IO?
>3. Any chance that by setting up the 20 drives across 3
enclosures (each
>with their own controller channel) that we're somehow
still using only
>one channel primarily? Doesn't seem possible, but
figured I should ask.
>4. The read/sec is ~725, and the write/sec is ~350. But
total disk
>queue is averaging 120, with 2/3 of that coming from the
write queue.
>Based on the fact that reads was higher than writes, we
went with RAID
>5, thinking that it's faster for reads. Maybe though
it's just a matter
>of not having enough drives?
>Thanks again!
>Myron
>
>In article <01ee01c3471c$d3cc8c00$a501280a@.phx.gbl>,
jchang6@.yahoo.com
>says...
>> i dont' see the original post or Andrews reply
>> first, 2 drives should be in RAID1 for the log,
>> as for data, you are most likely doing random
read/writes,
>> so raw bandwidth in MB/sec is irrelevent,
>> for 20 15k drives in RAID5 you can expect 150x20/4 =600
>> write IO/sec. In RAID 1+0, you should get 1,200 write
>> IO/sec.
>> use perfmon to get the read/sec & writes/sec for the
>> drives with the data files, if the log are on the same
>> partition, move them somewhere else as that will muddy
the
>> analysis
>> >--Original Message--
>> >Hi all,
>> >(This is a repost, as my post from an hour earlier
>> >doesn't seem to have made it. Please excuse if
>> >duplicate.)
>> >
>> >We've got a problem at the moment with high disk queue
>> >length, and it's puzzling us.
>> >
>> >We've got a Compaq DL580G2 with 4 fast procs and 32 GB
of
>> >ram. It's running Windows 2003 Enterprise and SQL
Server
>> >2000 Enterprise.
>> >
>> >It has Compaq's 5300 controller (four channels), and
>> >we've got three drive enclosures attached to it (so
using
>> >3 of the four controller channels). Each of the
>> >enclosures contains 7x36GB drives (15k), and 20 of the
21
>> >are in a RAID 5 array, with the remainder being a hot
>> >spare.
>> >
>> >We process 800 - 1000 batches per second, and here are
>> >some stats on the physical disk:
>> >
>> > - Disk throughput is about 8 mb per second (per
>> >perfmon), and is split almost exactly evenly between
>> >reads and writes.
>> >
>> > - Total disk queue is averaging 120 (way too high),
with
>> >2/3 of that coming from the write queue.
>> >
>> > - Checkpoints are almost always occurring, only a
second
>> >or two between them.
>> >
>> >We used to (until last night) have only one drive
>> >enclosure and 14 drives, and were convinced that our
high
>> >disk queue was due to the bottleneck of using only one
>.
>|||Thanks, Joe, we found this in a compaq white paper as well (after you
mentioned it), and have done so. No noticable difference yet, but every
little bit helps.
Myron
In article <072401c3475f$cb3e7880$a301280a@.phx.gbl>, jchang6@.yahoo.com
says...
> how did i forget,
> change the 5304 cache setting from 50%/50% read/write to
> 100% write, ie, never try to cache reads,
> you might send compaq an email on this
> -joe
>|||Joe, that's just fabulous information, thanks so much! Your email was
clearer on capacity planning than any number of books or white papers
I've read. Just pointing us to look at reads & writes instead of
throughput (since we're random io) makes a world of difference in our
focus.
We had the feeling that we should be ok, and your numbers seem to
confirm that. Sadly, we're not ok, but this gives us a place to start.
Answers to your questions:
- We did configure all 20 drives as one logical drive. (And were
hoping to see queue depth drop when we moved to those 20 drives on three
controller channels, but it's the same as when on 14 drives on one
channel).
- avg sec/read = .11, and avg sec/write = .29, both well above the 6 ms
mark.
- we are not inserting batches all at once; that is a sustained rate.
- we did enable (just today) write caching on the disk in Windows, and
changed to 100% write caching on the controller.
One other thing: we noticed in the Array Controller Information (in the
compaq system management tools) that the CPU Utilization is 0% (or
sometimes 1%). Does that seem unusual, given the load we're
experiencing?
And in the same screen, the "command count" is 888 (and changes when we
refresh). Do you know if that indicates the number of reads & writes
per second?
Thanks again for your help,
Myron
In article <042301c3473f$3938a070$a501280a@.phx.gbl>, jchang6@.yahoo.com
says...
> this is the common formula (used in TPC-C benchmarking
> analysis) for small block random IO on disk drives. it is
> based on the average rotational latency (2ms for a 15K
> disk to rotate 180 deg) + the avg seek time (4ms) + data
> xfer time (<0.2ms) for a total seek time of 6.2ms
> this means that at an average queue depth of 1 (one
> outstanding request to the disk)
> the IO rate is 1000ms/sec / 6.2ms = 160/sec
> at higher queue depth, the disk should be able to re-order
> the seeks, and achieve higher IO rate, at the expense of
> longer latency (avg sec/xfer)
> as a rought approximation, the following applies:
> 7.2k 75 IO/sec
> 10k 100
> 15k 150
> in any raid level, reads can occur from any disk, so the
> total read rate for 15k disk is 150 x number of drives
> for writes, RAID1 requires 2 writes, hence the net is 1/2
> of the base rate
> RAID5 requires 2 reads (for the data block and the parity
> block) and 2 writes (for data and parity)
> hence the overall random small block formula for 15k disks
> is:
> RAID0: reads/sec + writes/sec = 150 * number of drives
> RAID1: reads/sec + 2x writes/sec = 150 * drives
> RAID5: reads/sec + 4x writes/sec = 150 * drives
> your situation is:
> 20 15k drives are capable of 20x150 = 3,000 IO/sec
> your actual physical disk IO is
> reads writes
> 725 + 4 x 350 = 2,125
> if you were in RAID1, it would be
> 725 + 2 x 350 = 1,425
> so you should be ok,
> did you configure all 20 drives in one RAID5 array or
> separate arrays. If 1, you should see a queue depth of <20
> (1 per disk)
> i don't know why your queue depth is so high
> what is the avg sec/read & /write?
> if these numbers are approx equal to or less than the avg
> access time (6ms), don't worry about the queue depth,
> Wait, are you dumping the batch of 800-1000 inserts all at
> once?
> if so, the queue depth may spike, but the avg should be <20
> the compaq 5304 has battery backup (pretty much all high-
> end controllers do) so you may as well enable write
> caching on both the controller and in W2K on the disk
> properties page