This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Idle Blocker - the real reason it is difficult to analyze in DPA - missing feature object locks

In DPA an idle blocker is a connection that is not currently running a statement but is holding up connections.

What the connection IS doing, is holding onto locks that are preventing other connections from getting their work done.  It is a bit of a mystery because my idle blockers show CPU/MEMORY and do not reveal that they are holding locks, and more importantly do not reveal what objects/rows they are holding locks for.  This makes it really hard to track down.  But some of the other connections blocked show shared lock waits, and some do not.  I don't see intent locks and am not sure why.

Also the connection is idle in the current time period that is shown, but the former commands run may be available in earlier time periods.  The problem is, the last command run is not at all necessarily the one that is holding the locks that are blocking other sessions.  So for example, "sp_tables" is my last command run, and I don't know for sure if there are system table locks involved, but since it completed I think earlier application SQL is to blame.

What is needed is for DPA to keep track of what objects are locked and for what period of time they are locked, in the event that the locking time is excessive.  Also, what is needed is the intent locking, so that connections that are waiting for locks can be analyzed. Connections that are in a wait state can have the locks held/desired analyzed.  The "Details.." , "statements executed by session NN while holding the lock" feature is helpful when it appears, but it is tricky to get the link to appear by manipulating the time period.  Ideally, we would get a summary of lock counts and types by object , such as table lock, row lock rowct=nn, and we would further know which statement caused the lock.

Also ideally, we would be able to know about transaction start and end.  Due to connection pooling and filtering of commands by whether are not they are logged at all, I cannot tell if subsequent commands are in the same transaction or not.  It would be nice to know what the grouping of commands is, that a bunch of SQL statements frequently appear in the same transactions together.  Because ultimately, transactions block each other, not connections. You need a connection to create a transaction, and some transactions are single statement, but most of our blocking issues will be due to large transactions and we have trouble identifying what those are, and where they start and end in the application code.  It should be fairly straightforward to figure out where they start and end on the database using transaction nest level and transaction ids.  Just knowing the nest level when not 0 would be a great easy start.

Another tool that I have used, Precise I3, was able to show historical object contention analysis indicating which table and index objects were frequently locked, which statements locked them, and which statements had lock contention over an object.  I replaced Precise I3 with Confio Ignite because of the difficulty licensing and installing and upgrading Precise I3 agents on monitored machines.  But this feature was awesome, and another related feature that kept track of table index usage using the same data to identify important indexes and indexes that were not actually used at all.

Anyway, if there is a way to identify what objects are locked, please let me know.  Otherwise, it is time for a new product feature, because I am spending an inordinate amount of time in forensic analysis of locking contention in order to make proper tuning recommendations.

  • I agree completely. I already have a couple of related feature requests submitted that you can vote on that are small steps toward a more comprehensive way to address the idle blockers mysteries:

  • FormerMember
    0 FormerMember

    Hi Chip,

    To Marks point below, we have some feature requests for this specifically.  As a Product Management team we HIGHLY value feature requests that are requested by our existing customer base and the voting for a feature request is one way to show us at SolarWinds how important a specific feature is to the customer base.

    That being said, I will add your request to our internal feature request system as well.

    Thank you very much for your feedback.  Hopefully when we get to this work, we can work with you to have you beta test the feature.

    Kathy Gibbs

  • Kathy,

    When I go to our developers and say something like "you had an idle blocker

    that held up a half dozen sessions that were trying to access table foo for

    20 minutes" and I explain yet again what an idle blocker is, they always

    come back with "Can you provide any other information that would help me

    track this down?" My answer is always no, and that's the end of the

    conversation. They won't spend time hunting through a large code base when

    they don't know anything more specific about what they should be looking

    for.

    Anything DPA can provide that would improve that conversation would be very

    welcome.

    - Mark

    Mark Freeman

    Database Administrator | Rogue Fitness

    1080 Steelwood Road, Columbus, OH 43212

    MFreeman@RogueFitness.com

    On Mon, May 11, 2015 at 12:53 PM, kathleen.gibbs <

  • Thank you Kathleen, I have confidence in your team and I enjoyed working on the prior beta.

    I tried finding an answer to the Idle Blocker issue in the Q and A.  I don't think any of the product feature requests nailed the issue.

    So my suggestion is new data capture vs. tweaks to the UI.  Capture data on Objects that are locked, capture data on transactions that have member statements and object locks and contend with other transactions.  This allows analysis of "what transactions lock the XYZ table", "what statements lock/query the XYZ" table, etc. 

    Please reply with related feature links so that folks can vote on them.  I would rather vote on a feature that you can complete soon than a blue sky dream that is too hard for reasons that we are not aware.

  • One thing I always asked my developers when I'd see idle blockers was is there a dependency within the transaction that's external to the database query?

    If yes, is there another way to achieve the same transaction unit level (like do that external dependency first, then open transaction in the DB) without adding that external dependency?

    We actually fixed some major contention at the DB layer by working it from this angle regarding idle sessions.

  • Absolutely, thanks. I see that a lot in SQL Server with shared intent locks on objects selected after an update. Why not select before the update, as you are saying.

    My issue is with improving the tool to help show the statements that are holding the locks.

  • mandevil,

    Exactly right. But we have lots of old code that uses web services and

    other external calls, possibly inside transactions, and rather than

    creating a project to find and fix all of them, it would be helpful to give

    the developers the information they need to find the one that is causing

    the specific problem I'm reporting to them.

    That's not to say they shouldn't find and fix all of them just because it's

    the right thing to do, but that may not rise to the top of their priority

    list for a long time. Like most places, there's plenty of other technical

    debt to get paid down.

    Mark Freeman

    Database Administrator | Rogue Fitness

    1080 Steelwood Road, Columbus, OH 43212

    MFreeman@RogueFitness.com

    On Mon, May 11, 2015 at 1:39 PM, mandevil <