High Pagelatch and threadpool

I see high pagelatch and threadpool wait types in SQL 2005. What does that
mean ? How can I resolve it ?
Also when i see a waitresource of 7:1:145620? I understand the first is the
dbid , whats the 3rd value ?
Any help would be appreciated. This is causing heavy blocking on our
systems.High pagelatch waits usually mean that you have some pages that are being
accessed frequently. You have some hot pages. In the case of 7:1:145620, 7 is
the database id, 1 is the primary data file, and 145620 is the page number in
the primary data file. To find what object owns this page, you can use DBCC
PAGE(dbid, file_id, page_num). See
http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
for more info on using DBCC PAGE to find the object name.
Once you find the hot object(s), the next step is to find the user process,
SQL statements, SPs, and so on that access the hot object, and determine if
you can modify these SQL statements/SPs and so on to reduce contention.
I'm pretty sure that threadpool waits are related to contention on the
worker threads, but I need to double check.
Linchi
"Hassan" wrote:
> I see high pagelatch and threadpool wait types in SQL 2005. What does that
> mean ? How can I resolve it ?
> Also when i see a waitresource of 7:1:145620? I understand the first is the
> dbid , whats the 3rd value ?
> Any help would be appreciated. This is causing heavy blocking on our
> systems.
>
>|||Yes we do have worker thread contention. Its all related to this massive
batch of many inserts against a single table and the pagelatch i believe
causes contention and hence the rest of the threads stack up and worker
threads become starved too.
As you mentioned I have some hot pages.. I'm sure I do.. Are these pages
that are being accessed in memory since its a PageLatch wait as opposed to
PageIOLatch ? Is that the only difference between the 2 wait types i.e. one
being accessed from memory vs other being accessed from disk ?
Also page 49 of Kalens Query Tuning and Optimization states the following:
PageIOLatch_ex : This wait occurs when a task is waiting on a latch for a
buffer that is NOT in an IO request.
What does that mean ? If its not IO related, then what is it ? And is a
reference to a buffer always means it in memory as in RAM or could it be
virtual memory and hence be a latch on a buffer on disk ?
Thanks
"Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
> High pagelatch waits usually mean that you have some pages that are being
> accessed frequently. You have some hot pages. In the case of 7:1:145620, 7
> is
> the database id, 1 is the primary data file, and 145620 is the page number
> in
> the primary data file. To find what object owns this page, you can use
> DBCC
> PAGE(dbid, file_id, page_num). See
> http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
> for more info on using DBCC PAGE to find the object name.
> Once you find the hot object(s), the next step is to find the user
> process,
> SQL statements, SPs, and so on that access the hot object, and determine
> if
> you can modify these SQL statements/SPs and so on to reduce contention.
> I'm pretty sure that threadpool waits are related to contention on the
> worker threads, but I need to double check.
> Linchi
> "Hassan" wrote:
>> I see high pagelatch and threadpool wait types in SQL 2005. What does
>> that
>> mean ? How can I resolve it ?
>> Also when i see a waitresource of 7:1:145620? I understand the first is
>> the
>> dbid , whats the 3rd value ?
>> Any help would be appreciated. This is causing heavy blocking on our
>> systems.
>>|||Perhaps you can consider using some form of bulk mechanism (there are
several) to load your data instead of a "massive number of inserts". Also,
consider using some form of queing mechanism to slow down the intensity of
the load process and spread out the crush on the server.
--
Kevin G. Boles
TheSQLGuru
Indicium Resources, Inc.
"Hassan" <hassan@.test.com> wrote in message
news:OEyk7JGMIHA.2064@.TK2MSFTNGP06.phx.gbl...
> Yes we do have worker thread contention. Its all related to this massive
> batch of many inserts against a single table and the pagelatch i believe
> causes contention and hence the rest of the threads stack up and worker
> threads become starved too.
> As you mentioned I have some hot pages.. I'm sure I do.. Are these pages
> that are being accessed in memory since its a PageLatch wait as opposed to
> PageIOLatch ? Is that the only difference between the 2 wait types i.e.
> one being accessed from memory vs other being accessed from disk ?
> Also page 49 of Kalens Query Tuning and Optimization states the following:
> PageIOLatch_ex : This wait occurs when a task is waiting on a latch for a
> buffer that is NOT in an IO request.
> What does that mean ? If its not IO related, then what is it ? And is a
> reference to a buffer always means it in memory as in RAM or could it be
> virtual memory and hence be a latch on a buffer on disk ?
> Thanks
>
> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>> High pagelatch waits usually mean that you have some pages that are being
>> accessed frequently. You have some hot pages. In the case of 7:1:145620,
>> 7 is
>> the database id, 1 is the primary data file, and 145620 is the page
>> number in
>> the primary data file. To find what object owns this page, you can use
>> DBCC
>> PAGE(dbid, file_id, page_num). See
>> http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>> for more info on using DBCC PAGE to find the object name.
>> Once you find the hot object(s), the next step is to find the user
>> process,
>> SQL statements, SPs, and so on that access the hot object, and determine
>> if
>> you can modify these SQL statements/SPs and so on to reduce contention.
>> I'm pretty sure that threadpool waits are related to contention on the
>> worker threads, but I need to double check.
>> Linchi
>> "Hassan" wrote:
>> I see high pagelatch and threadpool wait types in SQL 2005. What does
>> that
>> mean ? How can I resolve it ?
>> Also when i see a waitresource of 7:1:145620? I understand the first is
>> the
>> dbid , whats the 3rd value ?
>> Any help would be appreciated. This is causing heavy blocking on our
>> systems.
>>
>|||> What does that mean ? If its not IO related, then what is it ? And is a
> reference to a buffer always means it in memory as in RAM or could it be
> virtual memory and hence be a latch on a buffer on disk ?
I don't know the exact context in which this is stated. But PageIOLatch is
I/O related. You can easily confirm this yourself by running some quick tests.
Linchi
"Hassan" wrote:
> Yes we do have worker thread contention. Its all related to this massive
> batch of many inserts against a single table and the pagelatch i believe
> causes contention and hence the rest of the threads stack up and worker
> threads become starved too.
> As you mentioned I have some hot pages.. I'm sure I do.. Are these pages
> that are being accessed in memory since its a PageLatch wait as opposed to
> PageIOLatch ? Is that the only difference between the 2 wait types i.e. one
> being accessed from memory vs other being accessed from disk ?
> Also page 49 of Kalens Query Tuning and Optimization states the following:
> PageIOLatch_ex : This wait occurs when a task is waiting on a latch for a
> buffer that is NOT in an IO request.
> What does that mean ? If its not IO related, then what is it ? And is a
> reference to a buffer always means it in memory as in RAM or could it be
> virtual memory and hence be a latch on a buffer on disk ?
> Thanks
>
> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
> > High pagelatch waits usually mean that you have some pages that are being
> > accessed frequently. You have some hot pages. In the case of 7:1:145620, 7
> > is
> > the database id, 1 is the primary data file, and 145620 is the page number
> > in
> > the primary data file. To find what object owns this page, you can use
> > DBCC
> > PAGE(dbid, file_id, page_num). See
> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
> > for more info on using DBCC PAGE to find the object name.
> >
> > Once you find the hot object(s), the next step is to find the user
> > process,
> > SQL statements, SPs, and so on that access the hot object, and determine
> > if
> > you can modify these SQL statements/SPs and so on to reduce contention.
> >
> > I'm pretty sure that threadpool waits are related to contention on the
> > worker threads, but I need to double check.
> >
> > Linchi
> >
> > "Hassan" wrote:
> >
> >> I see high pagelatch and threadpool wait types in SQL 2005. What does
> >> that
> >> mean ? How can I resolve it ?
> >>
> >> Also when i see a waitresource of 7:1:145620? I understand the first is
> >> the
> >> dbid , whats the 3rd value ?
> >>
> >> Any help would be appreciated. This is causing heavy blocking on our
> >> systems.
> >>
> >>
> >>
>
>|||And is PageLatch from memory only ?
"Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>> What does that mean ? If its not IO related, then what is it ? And is a
>> reference to a buffer always means it in memory as in RAM or could it be
>> virtual memory and hence be a latch on a buffer on disk ?
> I don't know the exact context in which this is stated. But PageIOLatch is
> I/O related. You can easily confirm this yourself by running some quick
> tests.
> Linchi
> "Hassan" wrote:
>> Yes we do have worker thread contention. Its all related to this massive
>> batch of many inserts against a single table and the pagelatch i believe
>> causes contention and hence the rest of the threads stack up and worker
>> threads become starved too.
>> As you mentioned I have some hot pages.. I'm sure I do.. Are these pages
>> that are being accessed in memory since its a PageLatch wait as opposed
>> to
>> PageIOLatch ? Is that the only difference between the 2 wait types i.e.
>> one
>> being accessed from memory vs other being accessed from disk ?
>> Also page 49 of Kalens Query Tuning and Optimization states the
>> following:
>> PageIOLatch_ex : This wait occurs when a task is waiting on a latch for a
>> buffer that is NOT in an IO request.
>> What does that mean ? If its not IO related, then what is it ? And is a
>> reference to a buffer always means it in memory as in RAM or could it be
>> virtual memory and hence be a latch on a buffer on disk ?
>> Thanks
>>
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>> > High pagelatch waits usually mean that you have some pages that are
>> > being
>> > accessed frequently. You have some hot pages. In the case of
>> > 7:1:145620, 7
>> > is
>> > the database id, 1 is the primary data file, and 145620 is the page
>> > number
>> > in
>> > the primary data file. To find what object owns this page, you can use
>> > DBCC
>> > PAGE(dbid, file_id, page_num). See
>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>> > for more info on using DBCC PAGE to find the object name.
>> >
>> > Once you find the hot object(s), the next step is to find the user
>> > process,
>> > SQL statements, SPs, and so on that access the hot object, and
>> > determine
>> > if
>> > you can modify these SQL statements/SPs and so on to reduce contention.
>> >
>> > I'm pretty sure that threadpool waits are related to contention on the
>> > worker threads, but I need to double check.
>> >
>> > Linchi
>> >
>> > "Hassan" wrote:
>> >
>> >> I see high pagelatch and threadpool wait types in SQL 2005. What does
>> >> that
>> >> mean ? How can I resolve it ?
>> >>
>> >> Also when i see a waitresource of 7:1:145620? I understand the first
>> >> is
>> >> the
>> >> dbid , whats the 3rd value ?
>> >>
>> >> Any help would be appreciated. This is causing heavy blocking on our
>> >> systems.
>> >>
>> >>
>> >>
>>|||Yes, a PageIOLatch is when it is syncing with the physical disk and a plain
latch is totally memory related.
--
Andrew J. Kelly SQL MVP
Solid Quality Mentors
"Hassan" <hassan@.test.com> wrote in message
news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
> And is PageLatch from memory only ?
> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
> news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>> What does that mean ? If its not IO related, then what is it ? And is a
>> reference to a buffer always means it in memory as in RAM or could it be
>> virtual memory and hence be a latch on a buffer on disk ?
>> I don't know the exact context in which this is stated. But PageIOLatch
>> is
>> I/O related. You can easily confirm this yourself by running some quick
>> tests.
>> Linchi
>> "Hassan" wrote:
>> Yes we do have worker thread contention. Its all related to this massive
>> batch of many inserts against a single table and the pagelatch i believe
>> causes contention and hence the rest of the threads stack up and worker
>> threads become starved too.
>> As you mentioned I have some hot pages.. I'm sure I do.. Are these
>> pages
>> that are being accessed in memory since its a PageLatch wait as opposed
>> to
>> PageIOLatch ? Is that the only difference between the 2 wait types i.e.
>> one
>> being accessed from memory vs other being accessed from disk ?
>> Also page 49 of Kalens Query Tuning and Optimization states the
>> following:
>> PageIOLatch_ex : This wait occurs when a task is waiting on a latch for
>> a
>> buffer that is NOT in an IO request.
>> What does that mean ? If its not IO related, then what is it ? And is a
>> reference to a buffer always means it in memory as in RAM or could it be
>> virtual memory and hence be a latch on a buffer on disk ?
>> Thanks
>>
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>> > High pagelatch waits usually mean that you have some pages that are
>> > being
>> > accessed frequently. You have some hot pages. In the case of
>> > 7:1:145620, 7
>> > is
>> > the database id, 1 is the primary data file, and 145620 is the page
>> > number
>> > in
>> > the primary data file. To find what object owns this page, you can use
>> > DBCC
>> > PAGE(dbid, file_id, page_num). See
>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>> > for more info on using DBCC PAGE to find the object name.
>> >
>> > Once you find the hot object(s), the next step is to find the user
>> > process,
>> > SQL statements, SPs, and so on that access the hot object, and
>> > determine
>> > if
>> > you can modify these SQL statements/SPs and so on to reduce
>> > contention.
>> >
>> > I'm pretty sure that threadpool waits are related to contention on the
>> > worker threads, but I need to double check.
>> >
>> > Linchi
>> >
>> > "Hassan" wrote:
>> >
>> >> I see high pagelatch and threadpool wait types in SQL 2005. What does
>> >> that
>> >> mean ? How can I resolve it ?
>> >>
>> >> Also when i see a waitresource of 7:1:145620? I understand the first
>> >> is
>> >> the
>> >> dbid , whats the 3rd value ?
>> >>
>> >> Any help would be appreciated. This is causing heavy blocking on our
>> >> systems.
>> >>
>> >>
>> >>
>>
>|||So when I have high PageLatch, whats a fix ? Will more memory help ? Doesnt
seem like.. I guess I might just have to rearchitect the way we hit the page
in contention.
Since the table that undergoes these high inserts has an identity column
thats the clustered index and with all those inserts hitting that one pages
as its sequential, i guess we have our contention on that page.
"Andrew J. Kelly" <sqlmvpnooospam@.shadhawk.com> wrote in message
news:%23%23FCA4IMIHA.5208@.TK2MSFTNGP04.phx.gbl...
> Yes, a PageIOLatch is when it is syncing with the physical disk and a
> plain latch is totally memory related.
> --
> Andrew J. Kelly SQL MVP
> Solid Quality Mentors
>
> "Hassan" <hassan@.test.com> wrote in message
> news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
>> And is PageLatch from memory only ?
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>> What does that mean ? If its not IO related, then what is it ? And is a
>> reference to a buffer always means it in memory as in RAM or could it
>> be
>> virtual memory and hence be a latch on a buffer on disk ?
>> I don't know the exact context in which this is stated. But PageIOLatch
>> is
>> I/O related. You can easily confirm this yourself by running some quick
>> tests.
>> Linchi
>> "Hassan" wrote:
>> Yes we do have worker thread contention. Its all related to this
>> massive
>> batch of many inserts against a single table and the pagelatch i
>> believe
>> causes contention and hence the rest of the threads stack up and worker
>> threads become starved too.
>> As you mentioned I have some hot pages.. I'm sure I do.. Are these
>> pages
>> that are being accessed in memory since its a PageLatch wait as opposed
>> to
>> PageIOLatch ? Is that the only difference between the 2 wait types i.e.
>> one
>> being accessed from memory vs other being accessed from disk ?
>> Also page 49 of Kalens Query Tuning and Optimization states the
>> following:
>> PageIOLatch_ex : This wait occurs when a task is waiting on a latch for
>> a
>> buffer that is NOT in an IO request.
>> What does that mean ? If its not IO related, then what is it ? And is a
>> reference to a buffer always means it in memory as in RAM or could it
>> be
>> virtual memory and hence be a latch on a buffer on disk ?
>> Thanks
>>
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>> > High pagelatch waits usually mean that you have some pages that are
>> > being
>> > accessed frequently. You have some hot pages. In the case of
>> > 7:1:145620, 7
>> > is
>> > the database id, 1 is the primary data file, and 145620 is the page
>> > number
>> > in
>> > the primary data file. To find what object owns this page, you can
>> > use
>> > DBCC
>> > PAGE(dbid, file_id, page_num). See
>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>> > for more info on using DBCC PAGE to find the object name.
>> >
>> > Once you find the hot object(s), the next step is to find the user
>> > process,
>> > SQL statements, SPs, and so on that access the hot object, and
>> > determine
>> > if
>> > you can modify these SQL statements/SPs and so on to reduce
>> > contention.
>> >
>> > I'm pretty sure that threadpool waits are related to contention on
>> > the
>> > worker threads, but I need to double check.
>> >
>> > Linchi
>> >
>> > "Hassan" wrote:
>> >
>> >> I see high pagelatch and threadpool wait types in SQL 2005. What
>> >> does
>> >> that
>> >> mean ? How can I resolve it ?
>> >>
>> >> Also when i see a waitresource of 7:1:145620? I understand the first
>> >> is
>> >> the
>> >> dbid , whats the 3rd value ?
>> >>
>> >> Any help would be appreciated. This is causing heavy blocking on our
>> >> systems.
>> >>
>> >>
>> >>
>>
>>
>|||I am not convinced the insert process is the only culprit here. Have you
checked for blocking (sp_who2 active is simplest probably). dbcc
inputbuffer(spid) will get you the executing code. Are their other blocking
processes ongoing that are hitting the table at the same time? Are
transactions being held as short as possible? Are the locks being acquired
row, page or something larger? Lots of things to investigate here.
--
Kevin G. Boles
TheSQLGuru
Indicium Resources, Inc.
"Hassan" <hassan@.hotmail.com> wrote in message
news:utE8VeJMIHA.1204@.TK2MSFTNGP03.phx.gbl...
> So when I have high PageLatch, whats a fix ? Will more memory help ?
> Doesnt seem like.. I guess I might just have to rearchitect the way we hit
> the page in contention.
> Since the table that undergoes these high inserts has an identity column
> thats the clustered index and with all those inserts hitting that one
> pages as its sequential, i guess we have our contention on that page.
> "Andrew J. Kelly" <sqlmvpnooospam@.shadhawk.com> wrote in message
> news:%23%23FCA4IMIHA.5208@.TK2MSFTNGP04.phx.gbl...
>> Yes, a PageIOLatch is when it is syncing with the physical disk and a
>> plain latch is totally memory related.
>> --
>> Andrew J. Kelly SQL MVP
>> Solid Quality Mentors
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
>> And is PageLatch from memory only ?
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>> What does that mean ? If its not IO related, then what is it ? And is
>> a
>> reference to a buffer always means it in memory as in RAM or could it
>> be
>> virtual memory and hence be a latch on a buffer on disk ?
>> I don't know the exact context in which this is stated. But PageIOLatch
>> is
>> I/O related. You can easily confirm this yourself by running some quick
>> tests.
>> Linchi
>> "Hassan" wrote:
>> Yes we do have worker thread contention. Its all related to this
>> massive
>> batch of many inserts against a single table and the pagelatch i
>> believe
>> causes contention and hence the rest of the threads stack up and
>> worker
>> threads become starved too.
>> As you mentioned I have some hot pages.. I'm sure I do.. Are these
>> pages
>> that are being accessed in memory since its a PageLatch wait as
>> opposed to
>> PageIOLatch ? Is that the only difference between the 2 wait types
>> i.e. one
>> being accessed from memory vs other being accessed from disk ?
>> Also page 49 of Kalens Query Tuning and Optimization states the
>> following:
>> PageIOLatch_ex : This wait occurs when a task is waiting on a latch
>> for a
>> buffer that is NOT in an IO request.
>> What does that mean ? If its not IO related, then what is it ? And is
>> a
>> reference to a buffer always means it in memory as in RAM or could it
>> be
>> virtual memory and hence be a latch on a buffer on disk ?
>> Thanks
>>
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>> > High pagelatch waits usually mean that you have some pages that are
>> > being
>> > accessed frequently. You have some hot pages. In the case of
>> > 7:1:145620, 7
>> > is
>> > the database id, 1 is the primary data file, and 145620 is the page
>> > number
>> > in
>> > the primary data file. To find what object owns this page, you can
>> > use
>> > DBCC
>> > PAGE(dbid, file_id, page_num). See
>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>> > for more info on using DBCC PAGE to find the object name.
>> >
>> > Once you find the hot object(s), the next step is to find the user
>> > process,
>> > SQL statements, SPs, and so on that access the hot object, and
>> > determine
>> > if
>> > you can modify these SQL statements/SPs and so on to reduce
>> > contention.
>> >
>> > I'm pretty sure that threadpool waits are related to contention on
>> > the
>> > worker threads, but I need to double check.
>> >
>> > Linchi
>> >
>> > "Hassan" wrote:
>> >
>> >> I see high pagelatch and threadpool wait types in SQL 2005. What
>> >> does
>> >> that
>> >> mean ? How can I resolve it ?
>> >>
>> >> Also when i see a waitresource of 7:1:145620? I understand the
>> >> first is
>> >> the
>> >> dbid , whats the 3rd value ?
>> >>
>> >> Any help would be appreciated. This is causing heavy blocking on
>> >> our
>> >> systems.
>> >>
>> >>
>> >>
>>
>>
>>
>|||On Mon, 26 Nov 2007 11:27:51 -0800, "Hassan" <hassan@.test.com> wrote:
>Yes we do have worker thread contention. Its all related to this massive
>batch of many inserts against a single table and the pagelatch i believe
>causes contention and hence the rest of the threads stack up and worker
>threads become starved too.
So you have many spids inserting many rows to the same table at the
same time? Well, they will contend, won't they, and bottleneck at the
log if nowhere else, and block anybody trying to read the target table
while the inserts are going. Are there also readers, and have they
tried read-uncommitted (eg nolock)?
J.|||There is blocking and it appears that its so extensive that the inserts to
this single table caused by the same stored procedure causes others to be
blocked as its trying to insert and since all of them are contending for the
same page, it appears to be the bottleneck.
"TheSQLGuru" <kgboles@.earthlink.net> wrote in message
news:13kn1jqq6jgt969@.corp.supernews.com...
>I am not convinced the insert process is the only culprit here. Have you
>checked for blocking (sp_who2 active is simplest probably). dbcc
>inputbuffer(spid) will get you the executing code. Are their other
>blocking processes ongoing that are hitting the table at the same time?
>Are transactions being held as short as possible? Are the locks being
>acquired row, page or something larger? Lots of things to investigate
>here.
> --
> Kevin G. Boles
> TheSQLGuru
> Indicium Resources, Inc.
>
> "Hassan" <hassan@.hotmail.com> wrote in message
> news:utE8VeJMIHA.1204@.TK2MSFTNGP03.phx.gbl...
>> So when I have high PageLatch, whats a fix ? Will more memory help ?
>> Doesnt seem like.. I guess I might just have to rearchitect the way we
>> hit the page in contention.
>> Since the table that undergoes these high inserts has an identity column
>> thats the clustered index and with all those inserts hitting that one
>> pages as its sequential, i guess we have our contention on that page.
>> "Andrew J. Kelly" <sqlmvpnooospam@.shadhawk.com> wrote in message
>> news:%23%23FCA4IMIHA.5208@.TK2MSFTNGP04.phx.gbl...
>> Yes, a PageIOLatch is when it is syncing with the physical disk and a
>> plain latch is totally memory related.
>> --
>> Andrew J. Kelly SQL MVP
>> Solid Quality Mentors
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
>> And is PageLatch from memory only ?
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>> What does that mean ? If its not IO related, then what is it ? And is
>> a
>> reference to a buffer always means it in memory as in RAM or could it
>> be
>> virtual memory and hence be a latch on a buffer on disk ?
>> I don't know the exact context in which this is stated. But
>> PageIOLatch is
>> I/O related. You can easily confirm this yourself by running some
>> quick tests.
>> Linchi
>> "Hassan" wrote:
>> Yes we do have worker thread contention. Its all related to this
>> massive
>> batch of many inserts against a single table and the pagelatch i
>> believe
>> causes contention and hence the rest of the threads stack up and
>> worker
>> threads become starved too.
>> As you mentioned I have some hot pages.. I'm sure I do.. Are these
>> pages
>> that are being accessed in memory since its a PageLatch wait as
>> opposed to
>> PageIOLatch ? Is that the only difference between the 2 wait types
>> i.e. one
>> being accessed from memory vs other being accessed from disk ?
>> Also page 49 of Kalens Query Tuning and Optimization states the
>> following:
>> PageIOLatch_ex : This wait occurs when a task is waiting on a latch
>> for a
>> buffer that is NOT in an IO request.
>> What does that mean ? If its not IO related, then what is it ? And is
>> a
>> reference to a buffer always means it in memory as in RAM or could it
>> be
>> virtual memory and hence be a latch on a buffer on disk ?
>> Thanks
>>
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>> > High pagelatch waits usually mean that you have some pages that are
>> > being
>> > accessed frequently. You have some hot pages. In the case of
>> > 7:1:145620, 7
>> > is
>> > the database id, 1 is the primary data file, and 145620 is the page
>> > number
>> > in
>> > the primary data file. To find what object owns this page, you can
>> > use
>> > DBCC
>> > PAGE(dbid, file_id, page_num). See
>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>> > for more info on using DBCC PAGE to find the object name.
>> >
>> > Once you find the hot object(s), the next step is to find the user
>> > process,
>> > SQL statements, SPs, and so on that access the hot object, and
>> > determine
>> > if
>> > you can modify these SQL statements/SPs and so on to reduce
>> > contention.
>> >
>> > I'm pretty sure that threadpool waits are related to contention on
>> > the
>> > worker threads, but I need to double check.
>> >
>> > Linchi
>> >
>> > "Hassan" wrote:
>> >
>> >> I see high pagelatch and threadpool wait types in SQL 2005. What
>> >> does
>> >> that
>> >> mean ? How can I resolve it ?
>> >>
>> >> Also when i see a waitresource of 7:1:145620? I understand the
>> >> first is
>> >> the
>> >> dbid , whats the 3rd value ?
>> >>
>> >> Any help would be appreciated. This is causing heavy blocking on
>> >> our
>> >> systems.
>> >>
>> >>
>> >>
>>
>>
>>
>>
>|||Its just inserts for now.
"JXStern" <JXSternChangeX2R@.gte.net> wrote in message
news:c24nk354u8moo54lavqo9sdumfi91818a3@.4ax.com...
> On Mon, 26 Nov 2007 11:27:51 -0800, "Hassan" <hassan@.test.com> wrote:
>>Yes we do have worker thread contention. Its all related to this massive
>>batch of many inserts against a single table and the pagelatch i believe
>>causes contention and hence the rest of the threads stack up and worker
>>threads become starved too.
> So you have many spids inserting many rows to the same table at the
> same time? Well, they will contend, won't they, and bottleneck at the
> log if nowhere else, and block anybody trying to read the target table
> while the inserts are going. Are there also readers, and have they
> tried read-uncommitted (eg nolock)?
> J.
>|||try simply putting a waitfor delay... of a few hundredths of a second or
something in the insert sproc to allow other activities to be able to access
the page
--
Kevin G. Boles
TheSQLGuru
Indicium Resources, Inc.
"Hassan" <hassan@.test.com> wrote in message
news:O9LqSBLMIHA.5300@.TK2MSFTNGP04.phx.gbl...
> There is blocking and it appears that its so extensive that the inserts to
> this single table caused by the same stored procedure causes others to be
> blocked as its trying to insert and since all of them are contending for
> the same page, it appears to be the bottleneck.
> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
> news:13kn1jqq6jgt969@.corp.supernews.com...
>>I am not convinced the insert process is the only culprit here. Have you
>>checked for blocking (sp_who2 active is simplest probably). dbcc
>>inputbuffer(spid) will get you the executing code. Are their other
>>blocking processes ongoing that are hitting the table at the same time?
>>Are transactions being held as short as possible? Are the locks being
>>acquired row, page or something larger? Lots of things to investigate
>>here.
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.hotmail.com> wrote in message
>> news:utE8VeJMIHA.1204@.TK2MSFTNGP03.phx.gbl...
>> So when I have high PageLatch, whats a fix ? Will more memory help ?
>> Doesnt seem like.. I guess I might just have to rearchitect the way we
>> hit the page in contention.
>> Since the table that undergoes these high inserts has an identity column
>> thats the clustered index and with all those inserts hitting that one
>> pages as its sequential, i guess we have our contention on that page.
>> "Andrew J. Kelly" <sqlmvpnooospam@.shadhawk.com> wrote in message
>> news:%23%23FCA4IMIHA.5208@.TK2MSFTNGP04.phx.gbl...
>> Yes, a PageIOLatch is when it is syncing with the physical disk and a
>> plain latch is totally memory related.
>> --
>> Andrew J. Kelly SQL MVP
>> Solid Quality Mentors
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
>> And is PageLatch from memory only ?
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>> I don't know the exact context in which this is stated. But
>> PageIOLatch is
>> I/O related. You can easily confirm this yourself by running some
>> quick tests.
>> Linchi
>> "Hassan" wrote:
>>> Yes we do have worker thread contention. Its all related to this
>>> massive
>>> batch of many inserts against a single table and the pagelatch i
>>> believe
>>> causes contention and hence the rest of the threads stack up and
>>> worker
>>> threads become starved too.
>>>
>>> As you mentioned I have some hot pages.. I'm sure I do.. Are these
>>> pages
>>> that are being accessed in memory since its a PageLatch wait as
>>> opposed to
>>> PageIOLatch ? Is that the only difference between the 2 wait types
>>> i.e. one
>>> being accessed from memory vs other being accessed from disk ?
>>>
>>> Also page 49 of Kalens Query Tuning and Optimization states the
>>> following:
>>>
>>> PageIOLatch_ex : This wait occurs when a task is waiting on a latch
>>> for a
>>> buffer that is NOT in an IO request.
>>>
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> Thanks
>>>
>>>
>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>> message
>>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>>> > High pagelatch waits usually mean that you have some pages that
>>> > are being
>>> > accessed frequently. You have some hot pages. In the case of
>>> > 7:1:145620, 7
>>> > is
>>> > the database id, 1 is the primary data file, and 145620 is the
>>> > page number
>>> > in
>>> > the primary data file. To find what object owns this page, you can
>>> > use
>>> > DBCC
>>> > PAGE(dbid, file_id, page_num). See
>>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>>> > for more info on using DBCC PAGE to find the object name.
>>> >
>>> > Once you find the hot object(s), the next step is to find the user
>>> > process,
>>> > SQL statements, SPs, and so on that access the hot object, and
>>> > determine
>>> > if
>>> > you can modify these SQL statements/SPs and so on to reduce
>>> > contention.
>>> >
>>> > I'm pretty sure that threadpool waits are related to contention on
>>> > the
>>> > worker threads, but I need to double check.
>>> >
>>> > Linchi
>>> >
>>> > "Hassan" wrote:
>>> >
>>> >> I see high pagelatch and threadpool wait types in SQL 2005. What
>>> >> does
>>> >> that
>>> >> mean ? How can I resolve it ?
>>> >>
>>> >> Also when i see a waitresource of 7:1:145620? I understand the
>>> >> first is
>>> >> the
>>> >> dbid , whats the 3rd value ?
>>> >>
>>> >> Any help would be appreciated. This is causing heavy blocking on
>>> >> our
>>> >> systems.
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
>>
>>
>>
>|||Also, do you have any triggers by any chance?
--
Kevin G. Boles
TheSQLGuru
Indicium Resources, Inc.
"Hassan" <hassan@.test.com> wrote in message
news:O9LqSBLMIHA.5300@.TK2MSFTNGP04.phx.gbl...
> There is blocking and it appears that its so extensive that the inserts to
> this single table caused by the same stored procedure causes others to be
> blocked as its trying to insert and since all of them are contending for
> the same page, it appears to be the bottleneck.
> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
> news:13kn1jqq6jgt969@.corp.supernews.com...
>>I am not convinced the insert process is the only culprit here. Have you
>>checked for blocking (sp_who2 active is simplest probably). dbcc
>>inputbuffer(spid) will get you the executing code. Are their other
>>blocking processes ongoing that are hitting the table at the same time?
>>Are transactions being held as short as possible? Are the locks being
>>acquired row, page or something larger? Lots of things to investigate
>>here.
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.hotmail.com> wrote in message
>> news:utE8VeJMIHA.1204@.TK2MSFTNGP03.phx.gbl...
>> So when I have high PageLatch, whats a fix ? Will more memory help ?
>> Doesnt seem like.. I guess I might just have to rearchitect the way we
>> hit the page in contention.
>> Since the table that undergoes these high inserts has an identity column
>> thats the clustered index and with all those inserts hitting that one
>> pages as its sequential, i guess we have our contention on that page.
>> "Andrew J. Kelly" <sqlmvpnooospam@.shadhawk.com> wrote in message
>> news:%23%23FCA4IMIHA.5208@.TK2MSFTNGP04.phx.gbl...
>> Yes, a PageIOLatch is when it is syncing with the physical disk and a
>> plain latch is totally memory related.
>> --
>> Andrew J. Kelly SQL MVP
>> Solid Quality Mentors
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
>> And is PageLatch from memory only ?
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>> I don't know the exact context in which this is stated. But
>> PageIOLatch is
>> I/O related. You can easily confirm this yourself by running some
>> quick tests.
>> Linchi
>> "Hassan" wrote:
>>> Yes we do have worker thread contention. Its all related to this
>>> massive
>>> batch of many inserts against a single table and the pagelatch i
>>> believe
>>> causes contention and hence the rest of the threads stack up and
>>> worker
>>> threads become starved too.
>>>
>>> As you mentioned I have some hot pages.. I'm sure I do.. Are these
>>> pages
>>> that are being accessed in memory since its a PageLatch wait as
>>> opposed to
>>> PageIOLatch ? Is that the only difference between the 2 wait types
>>> i.e. one
>>> being accessed from memory vs other being accessed from disk ?
>>>
>>> Also page 49 of Kalens Query Tuning and Optimization states the
>>> following:
>>>
>>> PageIOLatch_ex : This wait occurs when a task is waiting on a latch
>>> for a
>>> buffer that is NOT in an IO request.
>>>
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> Thanks
>>>
>>>
>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>> message
>>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>>> > High pagelatch waits usually mean that you have some pages that
>>> > are being
>>> > accessed frequently. You have some hot pages. In the case of
>>> > 7:1:145620, 7
>>> > is
>>> > the database id, 1 is the primary data file, and 145620 is the
>>> > page number
>>> > in
>>> > the primary data file. To find what object owns this page, you can
>>> > use
>>> > DBCC
>>> > PAGE(dbid, file_id, page_num). See
>>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>>> > for more info on using DBCC PAGE to find the object name.
>>> >
>>> > Once you find the hot object(s), the next step is to find the user
>>> > process,
>>> > SQL statements, SPs, and so on that access the hot object, and
>>> > determine
>>> > if
>>> > you can modify these SQL statements/SPs and so on to reduce
>>> > contention.
>>> >
>>> > I'm pretty sure that threadpool waits are related to contention on
>>> > the
>>> > worker threads, but I need to double check.
>>> >
>>> > Linchi
>>> >
>>> > "Hassan" wrote:
>>> >
>>> >> I see high pagelatch and threadpool wait types in SQL 2005. What
>>> >> does
>>> >> that
>>> >> mean ? How can I resolve it ?
>>> >>
>>> >> Also when i see a waitresource of 7:1:145620? I understand the
>>> >> first is
>>> >> the
>>> >> dbid , whats the 3rd value ?
>>> >>
>>> >> Any help would be appreciated. This is causing heavy blocking on
>>> >> our
>>> >> systems.
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
>>
>>
>>
>|||No triggers..
This table does have a Foreign key constraint to another table though.Maybe
there is some checks done to validate every insert does not conflict on the
foreign key.
"TheSQLGuru" <kgboles@.earthlink.net> wrote in message
news:13kopa13iqppj8b@.corp.supernews.com...
> Also, do you have any triggers by any chance?
> --
> Kevin G. Boles
> TheSQLGuru
> Indicium Resources, Inc.
>
> "Hassan" <hassan@.test.com> wrote in message
> news:O9LqSBLMIHA.5300@.TK2MSFTNGP04.phx.gbl...
>> There is blocking and it appears that its so extensive that the inserts
>> to this single table caused by the same stored procedure causes others to
>> be blocked as its trying to insert and since all of them are contending
>> for the same page, it appears to be the bottleneck.
>> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
>> news:13kn1jqq6jgt969@.corp.supernews.com...
>>I am not convinced the insert process is the only culprit here. Have you
>>checked for blocking (sp_who2 active is simplest probably). dbcc
>>inputbuffer(spid) will get you the executing code. Are their other
>>blocking processes ongoing that are hitting the table at the same time?
>>Are transactions being held as short as possible? Are the locks being
>>acquired row, page or something larger? Lots of things to investigate
>>here.
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.hotmail.com> wrote in message
>> news:utE8VeJMIHA.1204@.TK2MSFTNGP03.phx.gbl...
>> So when I have high PageLatch, whats a fix ? Will more memory help ?
>> Doesnt seem like.. I guess I might just have to rearchitect the way we
>> hit the page in contention.
>> Since the table that undergoes these high inserts has an identity
>> column thats the clustered index and with all those inserts hitting
>> that one pages as its sequential, i guess we have our contention on
>> that page.
>> "Andrew J. Kelly" <sqlmvpnooospam@.shadhawk.com> wrote in message
>> news:%23%23FCA4IMIHA.5208@.TK2MSFTNGP04.phx.gbl...
>> Yes, a PageIOLatch is when it is syncing with the physical disk and a
>> plain latch is totally memory related.
>> --
>> Andrew J. Kelly SQL MVP
>> Solid Quality Mentors
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
>> And is PageLatch from memory only ?
>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in message
>> news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> I don't know the exact context in which this is stated. But
>>> PageIOLatch is
>>> I/O related. You can easily confirm this yourself by running some
>>> quick tests.
>>>
>>> Linchi
>>>
>>> "Hassan" wrote:
>>>
>>> Yes we do have worker thread contention. Its all related to this
>>> massive
>>> batch of many inserts against a single table and the pagelatch i
>>> believe
>>> causes contention and hence the rest of the threads stack up and
>>> worker
>>> threads become starved too.
>>>
>>> As you mentioned I have some hot pages.. I'm sure I do.. Are these
>>> pages
>>> that are being accessed in memory since its a PageLatch wait as
>>> opposed to
>>> PageIOLatch ? Is that the only difference between the 2 wait types
>>> i.e. one
>>> being accessed from memory vs other being accessed from disk ?
>>>
>>> Also page 49 of Kalens Query Tuning and Optimization states the
>>> following:
>>>
>>> PageIOLatch_ex : This wait occurs when a task is waiting on a latch
>>> for a
>>> buffer that is NOT in an IO request.
>>>
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> Thanks
>>>
>>>
>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>> message
>>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>>> > High pagelatch waits usually mean that you have some pages that
>>> > are being
>>> > accessed frequently. You have some hot pages. In the case of
>>> > 7:1:145620, 7
>>> > is
>>> > the database id, 1 is the primary data file, and 145620 is the
>>> > page number
>>> > in
>>> > the primary data file. To find what object owns this page, you
>>> > can use
>>> > DBCC
>>> > PAGE(dbid, file_id, page_num). See
>>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>>> > for more info on using DBCC PAGE to find the object name.
>>> >
>>> > Once you find the hot object(s), the next step is to find the
>>> > user
>>> > process,
>>> > SQL statements, SPs, and so on that access the hot object, and
>>> > determine
>>> > if
>>> > you can modify these SQL statements/SPs and so on to reduce
>>> > contention.
>>> >
>>> > I'm pretty sure that threadpool waits are related to contention
>>> > on the
>>> > worker threads, but I need to double check.
>>> >
>>> > Linchi
>>> >
>>> > "Hassan" wrote:
>>> >
>>> >> I see high pagelatch and threadpool wait types in SQL 2005. What
>>> >> does
>>> >> that
>>> >> mean ? How can I resolve it ?
>>> >>
>>> >> Also when i see a waitresource of 7:1:145620? I understand the
>>> >> first is
>>> >> the
>>> >> dbid , whats the 3rd value ?
>>> >>
>>> >> Any help would be appreciated. This is causing heavy blocking on
>>> >> our
>>> >> systems.
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
>>
>>
>>
>>
>|||No maybes about it. Every new row inserted must check to ensure it doesn't
violate the FK constraint. Does the other table have an index on that
column? You can do many thousands of rows per second inserted with a
clustered Identity column if the hardware is configured properly and there
are no other extenuating circumstances. But if you are blocking the inserts
all bets are off. Is it possible the selects are not using the proper index
so they are reading the page instead of the rows? But as for pagelatches
what version and service pack are you on? There was a known issue with one
of the sps in 2005 that caused high latch waits in this scenario. I think it
was SP1 but not sure.
--
Andrew J. Kelly SQL MVP
Solid Quality Mentors
"Hassan" <hassan@.hotmail.com> wrote in message
news:%23BVQVEgMIHA.5400@.TK2MSFTNGP04.phx.gbl...
> No triggers..
> This table does have a Foreign key constraint to another table
> though.Maybe there is some checks done to validate every insert does not
> conflict on the foreign key.
>
> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
> news:13kopa13iqppj8b@.corp.supernews.com...
>> Also, do you have any triggers by any chance?
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:O9LqSBLMIHA.5300@.TK2MSFTNGP04.phx.gbl...
>> There is blocking and it appears that its so extensive that the inserts
>> to this single table caused by the same stored procedure causes others
>> to be blocked as its trying to insert and since all of them are
>> contending for the same page, it appears to be the bottleneck.
>> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
>> news:13kn1jqq6jgt969@.corp.supernews.com...
>>I am not convinced the insert process is the only culprit here. Have
>>you checked for blocking (sp_who2 active is simplest probably). dbcc
>>inputbuffer(spid) will get you the executing code. Are their other
>>blocking processes ongoing that are hitting the table at the same time?
>>Are transactions being held as short as possible? Are the locks being
>>acquired row, page or something larger? Lots of things to investigate
>>here.
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.hotmail.com> wrote in message
>> news:utE8VeJMIHA.1204@.TK2MSFTNGP03.phx.gbl...
>> So when I have high PageLatch, whats a fix ? Will more memory help ?
>> Doesnt seem like.. I guess I might just have to rearchitect the way we
>> hit the page in contention.
>> Since the table that undergoes these high inserts has an identity
>> column thats the clustered index and with all those inserts hitting
>> that one pages as its sequential, i guess we have our contention on
>> that page.
>> "Andrew J. Kelly" <sqlmvpnooospam@.shadhawk.com> wrote in message
>> news:%23%23FCA4IMIHA.5208@.TK2MSFTNGP04.phx.gbl...
>> Yes, a PageIOLatch is when it is syncing with the physical disk and a
>> plain latch is totally memory related.
>> --
>> Andrew J. Kelly SQL MVP
>> Solid Quality Mentors
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
>>> And is PageLatch from memory only ?
>>>
>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>> message news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> I don't know the exact context in which this is stated. But
>>> PageIOLatch is
>>> I/O related. You can easily confirm this yourself by running some
>>> quick tests.
>>>
>>> Linchi
>>>
>>> "Hassan" wrote:
>>>
>>> Yes we do have worker thread contention. Its all related to this
>>> massive
>>> batch of many inserts against a single table and the pagelatch i
>>> believe
>>> causes contention and hence the rest of the threads stack up and
>>> worker
>>> threads become starved too.
>>>
>>> As you mentioned I have some hot pages.. I'm sure I do.. Are
>>> these pages
>>> that are being accessed in memory since its a PageLatch wait as
>>> opposed to
>>> PageIOLatch ? Is that the only difference between the 2 wait types
>>> i.e. one
>>> being accessed from memory vs other being accessed from disk ?
>>>
>>> Also page 49 of Kalens Query Tuning and Optimization states the
>>> following:
>>>
>>> PageIOLatch_ex : This wait occurs when a task is waiting on a
>>> latch for a
>>> buffer that is NOT in an IO request.
>>>
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> Thanks
>>>
>>>
>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>> message
>>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>>> > High pagelatch waits usually mean that you have some pages that
>>> > are being
>>> > accessed frequently. You have some hot pages. In the case of
>>> > 7:1:145620, 7
>>> > is
>>> > the database id, 1 is the primary data file, and 145620 is the
>>> > page number
>>> > in
>>> > the primary data file. To find what object owns this page, you
>>> > can use
>>> > DBCC
>>> > PAGE(dbid, file_id, page_num). See
>>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>>> > for more info on using DBCC PAGE to find the object name.
>>> >
>>> > Once you find the hot object(s), the next step is to find the
>>> > user
>>> > process,
>>> > SQL statements, SPs, and so on that access the hot object, and
>>> > determine
>>> > if
>>> > you can modify these SQL statements/SPs and so on to reduce
>>> > contention.
>>> >
>>> > I'm pretty sure that threadpool waits are related to contention
>>> > on the
>>> > worker threads, but I need to double check.
>>> >
>>> > Linchi
>>> >
>>> > "Hassan" wrote:
>>> >
>>> >> I see high pagelatch and threadpool wait types in SQL 2005.
>>> >> What does
>>> >> that
>>> >> mean ? How can I resolve it ?
>>> >>
>>> >> Also when i see a waitresource of 7:1:145620? I understand the
>>> >> first is
>>> >> the
>>> >> dbid , whats the 3rd value ?
>>> >>
>>> >> Any help would be appreciated. This is causing heavy blocking
>>> >> on our
>>> >> systems.
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>|||As Andrew says - the engine definitely has to check to verify every FK
insert. And if you don't have an appropriate index for that check - VOILA -
total I/O dog! :-)
Search the web for code that will report every FK that doesn't have an
index.
--
Kevin G. Boles
TheSQLGuru
Indicium Resources, Inc.
"Hassan" <hassan@.hotmail.com> wrote in message
news:%23BVQVEgMIHA.5400@.TK2MSFTNGP04.phx.gbl...
> No triggers..
> This table does have a Foreign key constraint to another table
> though.Maybe there is some checks done to validate every insert does not
> conflict on the foreign key.
>
> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
> news:13kopa13iqppj8b@.corp.supernews.com...
>> Also, do you have any triggers by any chance?
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:O9LqSBLMIHA.5300@.TK2MSFTNGP04.phx.gbl...
>> There is blocking and it appears that its so extensive that the inserts
>> to this single table caused by the same stored procedure causes others
>> to be blocked as its trying to insert and since all of them are
>> contending for the same page, it appears to be the bottleneck.
>> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
>> news:13kn1jqq6jgt969@.corp.supernews.com...
>>I am not convinced the insert process is the only culprit here. Have
>>you checked for blocking (sp_who2 active is simplest probably). dbcc
>>inputbuffer(spid) will get you the executing code. Are their other
>>blocking processes ongoing that are hitting the table at the same time?
>>Are transactions being held as short as possible? Are the locks being
>>acquired row, page or something larger? Lots of things to investigate
>>here.
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.hotmail.com> wrote in message
>> news:utE8VeJMIHA.1204@.TK2MSFTNGP03.phx.gbl...
>> So when I have high PageLatch, whats a fix ? Will more memory help ?
>> Doesnt seem like.. I guess I might just have to rearchitect the way we
>> hit the page in contention.
>> Since the table that undergoes these high inserts has an identity
>> column thats the clustered index and with all those inserts hitting
>> that one pages as its sequential, i guess we have our contention on
>> that page.
>> "Andrew J. Kelly" <sqlmvpnooospam@.shadhawk.com> wrote in message
>> news:%23%23FCA4IMIHA.5208@.TK2MSFTNGP04.phx.gbl...
>> Yes, a PageIOLatch is when it is syncing with the physical disk and a
>> plain latch is totally memory related.
>> --
>> Andrew J. Kelly SQL MVP
>> Solid Quality Mentors
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
>>> And is PageLatch from memory only ?
>>>
>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>> message news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> I don't know the exact context in which this is stated. But
>>> PageIOLatch is
>>> I/O related. You can easily confirm this yourself by running some
>>> quick tests.
>>>
>>> Linchi
>>>
>>> "Hassan" wrote:
>>>
>>> Yes we do have worker thread contention. Its all related to this
>>> massive
>>> batch of many inserts against a single table and the pagelatch i
>>> believe
>>> causes contention and hence the rest of the threads stack up and
>>> worker
>>> threads become starved too.
>>>
>>> As you mentioned I have some hot pages.. I'm sure I do.. Are
>>> these pages
>>> that are being accessed in memory since its a PageLatch wait as
>>> opposed to
>>> PageIOLatch ? Is that the only difference between the 2 wait types
>>> i.e. one
>>> being accessed from memory vs other being accessed from disk ?
>>>
>>> Also page 49 of Kalens Query Tuning and Optimization states the
>>> following:
>>>
>>> PageIOLatch_ex : This wait occurs when a task is waiting on a
>>> latch for a
>>> buffer that is NOT in an IO request.
>>>
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> Thanks
>>>
>>>
>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>> message
>>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>>> > High pagelatch waits usually mean that you have some pages that
>>> > are being
>>> > accessed frequently. You have some hot pages. In the case of
>>> > 7:1:145620, 7
>>> > is
>>> > the database id, 1 is the primary data file, and 145620 is the
>>> > page number
>>> > in
>>> > the primary data file. To find what object owns this page, you
>>> > can use
>>> > DBCC
>>> > PAGE(dbid, file_id, page_num). See
>>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>>> > for more info on using DBCC PAGE to find the object name.
>>> >
>>> > Once you find the hot object(s), the next step is to find the
>>> > user
>>> > process,
>>> > SQL statements, SPs, and so on that access the hot object, and
>>> > determine
>>> > if
>>> > you can modify these SQL statements/SPs and so on to reduce
>>> > contention.
>>> >
>>> > I'm pretty sure that threadpool waits are related to contention
>>> > on the
>>> > worker threads, but I need to double check.
>>> >
>>> > Linchi
>>> >
>>> > "Hassan" wrote:
>>> >
>>> >> I see high pagelatch and threadpool wait types in SQL 2005.
>>> >> What does
>>> >> that
>>> >> mean ? How can I resolve it ?
>>> >>
>>> >> Also when i see a waitresource of 7:1:145620? I understand the
>>> >> first is
>>> >> the
>>> >> dbid , whats the 3rd value ?
>>> >>
>>> >> Any help would be appreciated. This is causing heavy blocking
>>> >> on our
>>> >> systems.
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>|||Hassan
http://sqlblogcasts.com/blogs/tonyrogerson/archive/2007/03/30/blocking-from-foreign-key-constraint-look-ups-clustered-blocks-v-heap-doesn-t-block.aspx
"Hassan" <hassan@.hotmail.com> wrote in message
news:%23BVQVEgMIHA.5400@.TK2MSFTNGP04.phx.gbl...
> No triggers..
> This table does have a Foreign key constraint to another table
> though.Maybe there is some checks done to validate every insert does not
> conflict on the foreign key.
>
> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
> news:13kopa13iqppj8b@.corp.supernews.com...
>> Also, do you have any triggers by any chance?
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:O9LqSBLMIHA.5300@.TK2MSFTNGP04.phx.gbl...
>> There is blocking and it appears that its so extensive that the inserts
>> to this single table caused by the same stored procedure causes others
>> to be blocked as its trying to insert and since all of them are
>> contending for the same page, it appears to be the bottleneck.
>> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
>> news:13kn1jqq6jgt969@.corp.supernews.com...
>>I am not convinced the insert process is the only culprit here. Have
>>you checked for blocking (sp_who2 active is simplest probably). dbcc
>>inputbuffer(spid) will get you the executing code. Are their other
>>blocking processes ongoing that are hitting the table at the same time?
>>Are transactions being held as short as possible? Are the locks being
>>acquired row, page or something larger? Lots of things to investigate
>>here.
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.hotmail.com> wrote in message
>> news:utE8VeJMIHA.1204@.TK2MSFTNGP03.phx.gbl...
>> So when I have high PageLatch, whats a fix ? Will more memory help ?
>> Doesnt seem like.. I guess I might just have to rearchitect the way we
>> hit the page in contention.
>> Since the table that undergoes these high inserts has an identity
>> column thats the clustered index and with all those inserts hitting
>> that one pages as its sequential, i guess we have our contention on
>> that page.
>> "Andrew J. Kelly" <sqlmvpnooospam@.shadhawk.com> wrote in message
>> news:%23%23FCA4IMIHA.5208@.TK2MSFTNGP04.phx.gbl...
>> Yes, a PageIOLatch is when it is syncing with the physical disk and a
>> plain latch is totally memory related.
>> --
>> Andrew J. Kelly SQL MVP
>> Solid Quality Mentors
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
>>> And is PageLatch from memory only ?
>>>
>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>> message news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> I don't know the exact context in which this is stated. But
>>> PageIOLatch is
>>> I/O related. You can easily confirm this yourself by running some
>>> quick tests.
>>>
>>> Linchi
>>>
>>> "Hassan" wrote:
>>>
>>> Yes we do have worker thread contention. Its all related to this
>>> massive
>>> batch of many inserts against a single table and the pagelatch i
>>> believe
>>> causes contention and hence the rest of the threads stack up and
>>> worker
>>> threads become starved too.
>>>
>>> As you mentioned I have some hot pages.. I'm sure I do.. Are
>>> these pages
>>> that are being accessed in memory since its a PageLatch wait as
>>> opposed to
>>> PageIOLatch ? Is that the only difference between the 2 wait types
>>> i.e. one
>>> being accessed from memory vs other being accessed from disk ?
>>>
>>> Also page 49 of Kalens Query Tuning and Optimization states the
>>> following:
>>>
>>> PageIOLatch_ex : This wait occurs when a task is waiting on a
>>> latch for a
>>> buffer that is NOT in an IO request.
>>>
>>> What does that mean ? If its not IO related, then what is it ? And
>>> is a
>>> reference to a buffer always means it in memory as in RAM or could
>>> it be
>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> Thanks
>>>
>>>
>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>> message
>>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>>> > High pagelatch waits usually mean that you have some pages that
>>> > are being
>>> > accessed frequently. You have some hot pages. In the case of
>>> > 7:1:145620, 7
>>> > is
>>> > the database id, 1 is the primary data file, and 145620 is the
>>> > page number
>>> > in
>>> > the primary data file. To find what object owns this page, you
>>> > can use
>>> > DBCC
>>> > PAGE(dbid, file_id, page_num). See
>>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>>> > for more info on using DBCC PAGE to find the object name.
>>> >
>>> > Once you find the hot object(s), the next step is to find the
>>> > user
>>> > process,
>>> > SQL statements, SPs, and so on that access the hot object, and
>>> > determine
>>> > if
>>> > you can modify these SQL statements/SPs and so on to reduce
>>> > contention.
>>> >
>>> > I'm pretty sure that threadpool waits are related to contention
>>> > on the
>>> > worker threads, but I need to double check.
>>> >
>>> > Linchi
>>> >
>>> > "Hassan" wrote:
>>> >
>>> >> I see high pagelatch and threadpool wait types in SQL 2005.
>>> >> What does
>>> >> that
>>> >> mean ? How can I resolve it ?
>>> >>
>>> >> Also when i see a waitresource of 7:1:145620? I understand the
>>> >> first is
>>> >> the
>>> >> dbid , whats the 3rd value ?
>>> >>
>>> >> Any help would be appreciated. This is causing heavy blocking
>>> >> on our
>>> >> systems.
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>|||Oh, MY! That was an interesting read! I need to put that one in the memory
banks.
--
Kevin G. Boles
TheSQLGuru
Indicium Resources, Inc.
"Uri Dimant" <urid@.iscar.co.il> wrote in message
news:uBD663lMIHA.5040@.TK2MSFTNGP04.phx.gbl...
> Hassan
> http://sqlblogcasts.com/blogs/tonyrogerson/archive/2007/03/30/blocking-from-foreign-key-constraint-look-ups-clustered-blocks-v-heap-doesn-t-block.aspx
>
>
> "Hassan" <hassan@.hotmail.com> wrote in message
> news:%23BVQVEgMIHA.5400@.TK2MSFTNGP04.phx.gbl...
>> No triggers..
>> This table does have a Foreign key constraint to another table
>> though.Maybe there is some checks done to validate every insert does not
>> conflict on the foreign key.
>>
>> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
>> news:13kopa13iqppj8b@.corp.supernews.com...
>> Also, do you have any triggers by any chance?
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.test.com> wrote in message
>> news:O9LqSBLMIHA.5300@.TK2MSFTNGP04.phx.gbl...
>> There is blocking and it appears that its so extensive that the inserts
>> to this single table caused by the same stored procedure causes others
>> to be blocked as its trying to insert and since all of them are
>> contending for the same page, it appears to be the bottleneck.
>> "TheSQLGuru" <kgboles@.earthlink.net> wrote in message
>> news:13kn1jqq6jgt969@.corp.supernews.com...
>>I am not convinced the insert process is the only culprit here. Have
>>you checked for blocking (sp_who2 active is simplest probably). dbcc
>>inputbuffer(spid) will get you the executing code. Are their other
>>blocking processes ongoing that are hitting the table at the same time?
>>Are transactions being held as short as possible? Are the locks being
>>acquired row, page or something larger? Lots of things to investigate
>>here.
>> --
>> Kevin G. Boles
>> TheSQLGuru
>> Indicium Resources, Inc.
>>
>> "Hassan" <hassan@.hotmail.com> wrote in message
>> news:utE8VeJMIHA.1204@.TK2MSFTNGP03.phx.gbl...
>> So when I have high PageLatch, whats a fix ? Will more memory help ?
>> Doesnt seem like.. I guess I might just have to rearchitect the way
>> we hit the page in contention.
>> Since the table that undergoes these high inserts has an identity
>> column thats the clustered index and with all those inserts hitting
>> that one pages as its sequential, i guess we have our contention on
>> that page.
>> "Andrew J. Kelly" <sqlmvpnooospam@.shadhawk.com> wrote in message
>> news:%23%23FCA4IMIHA.5208@.TK2MSFTNGP04.phx.gbl...
>>> Yes, a PageIOLatch is when it is syncing with the physical disk and
>>> a plain latch is totally memory related.
>>>
>>> --
>>> Andrew J. Kelly SQL MVP
>>> Solid Quality Mentors
>>>
>>>
>>> "Hassan" <hassan@.test.com> wrote in message
>>> news:eOP7PjIMIHA.6060@.TK2MSFTNGP05.phx.gbl...
>>> And is PageLatch from memory only ?
>>>
>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>> message news:6677072C-AA1D-4E87-993F-C606D2C0AC20@.microsoft.com...
>>>> What does that mean ? If its not IO related, then what is it ?
>>>> And is a
>>>> reference to a buffer always means it in memory as in RAM or
>>>> could it be
>>>> virtual memory and hence be a latch on a buffer on disk ?
>>>
>>> I don't know the exact context in which this is stated. But
>>> PageIOLatch is
>>> I/O related. You can easily confirm this yourself by running some
>>> quick tests.
>>>
>>> Linchi
>>>
>>> "Hassan" wrote:
>>>
>>>> Yes we do have worker thread contention. Its all related to this
>>>> massive
>>>> batch of many inserts against a single table and the pagelatch i
>>>> believe
>>>> causes contention and hence the rest of the threads stack up and
>>>> worker
>>>> threads become starved too.
>>>>
>>>> As you mentioned I have some hot pages.. I'm sure I do.. Are
>>>> these pages
>>>> that are being accessed in memory since its a PageLatch wait as
>>>> opposed to
>>>> PageIOLatch ? Is that the only difference between the 2 wait
>>>> types i.e. one
>>>> being accessed from memory vs other being accessed from disk ?
>>>>
>>>> Also page 49 of Kalens Query Tuning and Optimization states the
>>>> following:
>>>>
>>>> PageIOLatch_ex : This wait occurs when a task is waiting on a
>>>> latch for a
>>>> buffer that is NOT in an IO request.
>>>>
>>>> What does that mean ? If its not IO related, then what is it ?
>>>> And is a
>>>> reference to a buffer always means it in memory as in RAM or
>>>> could it be
>>>> virtual memory and hence be a latch on a buffer on disk ?
>>>>
>>>> Thanks
>>>>
>>>>
>>>> "Linchi Shea" <LinchiShea@.discussions.microsoft.com> wrote in
>>>> message
>>>> news:20BE1711-42E7-485F-84E8-3769A78AA1E5@.microsoft.com...
>>>> > High pagelatch waits usually mean that you have some pages that
>>>> > are being
>>>> > accessed frequently. You have some hot pages. In the case of
>>>> > 7:1:145620, 7
>>>> > is
>>>> > the database id, 1 is the primary data file, and 145620 is the
>>>> > page number
>>>> > in
>>>> > the primary data file. To find what object owns this page, you
>>>> > can use
>>>> > DBCC
>>>> > PAGE(dbid, file_id, page_num). See
>>>> > http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/10/625659.aspx
>>>> > for more info on using DBCC PAGE to find the object name.
>>>> >
>>>> > Once you find the hot object(s), the next step is to find the
>>>> > user
>>>> > process,
>>>> > SQL statements, SPs, and so on that access the hot object, and
>>>> > determine
>>>> > if
>>>> > you can modify these SQL statements/SPs and so on to reduce
>>>> > contention.
>>>> >
>>>> > I'm pretty sure that threadpool waits are related to contention
>>>> > on the
>>>> > worker threads, but I need to double check.
>>>> >
>>>> > Linchi
>>>> >
>>>> > "Hassan" wrote:
>>>> >
>>>> >> I see high pagelatch and threadpool wait types in SQL 2005.
>>>> >> What does
>>>> >> that
>>>> >> mean ? How can I resolve it ?
>>>> >>
>>>> >> Also when i see a waitresource of 7:1:145620? I understand the
>>>> >> first is
>>>> >> the
>>>> >> dbid , whats the 3rd value ?
>>>> >>
>>>> >> Any help would be appreciated. This is causing heavy blocking
>>>> >> on our
>>>> >> systems.
>>>> >>
>>>> >>
>>>> >>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>