[infinispan-dev] JGroupsDistSync and ISPN-83

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] JGroupsDistSync and ISPN-83

Vladimir Blagojevic
Hi,

Would you please confirm my understanding and reading of javadoc for
ReentrantReadWriteLock under section for "lock downgrading". It says:
"Reentrancy also allows downgrading from the write lock to a read lock,
by acquiring the write lock, then the read lock and then releasing the
write lock. However, upgrading from a read lock to the write lock is not
possible." ReentrantReadWriteLock javadoc code example with class
CacheData also shows how a thread owning a read lock first has to
release it in order to obtain a write lock! So a thread owning a read
lock first has to release a read lock in order to obtain a write lock?

This is very symptomatic in logs of ISPN-83 such as this one
https://issues.jboss.org/secure/attachment/12341409/server1.log

Recall how FLUSH stops all invocations on cluster and therefore all read
lock acquisitions in JGroupsDistSync ultimately enabling smooth write
lock acquisitions for state transfer and what not. In conclusion this
leads me wondering if the culprit of all this FLUSH mess is rooted in
read/write lock semantics from above?

Regards,
Vladimir




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Vladimir Blagojevic
The more I research ReentrantReadWriteLock the more shocked I am. Not
only that a thread wanting to acquire write lock first has to release
read lock, but we can block forever even if we release the read lock if
we have acquired that read lock reentrantly. Each call to unlock just
reduces the hold count, and the lock is only actually released when the
hold count hits zero. Surreal!

People have already debated this issue:
http://stackoverflow.com/questions/464784/java-reentrantreadwritelocks-how-to-safely-acquire-write-lock

In conlusion we have to seriously revisit all our uses of
ReentrantReadWriteLock!

Vladimir


On 11-05-10 11:31 AM, Vladimir Blagojevic wrote:

> Hi,
>
> Would you please confirm my understanding and reading of javadoc for
> ReentrantReadWriteLock under section for "lock downgrading". It says:
> "Reentrancy also allows downgrading from the write lock to a read lock,
> by acquiring the write lock, then the read lock and then releasing the
> write lock. However, upgrading from a read lock to the write lock is not
> possible." ReentrantReadWriteLock javadoc code example with class
> CacheData also shows how a thread owning a read lock first has to
> release it in order to obtain a write lock! So a thread owning a read
> lock first has to release a read lock in order to obtain a write lock?
>
> This is very symptomatic in logs of ISPN-83 such as this one
> https://issues.jboss.org/secure/attachment/12341409/server1.log
>
> Recall how FLUSH stops all invocations on cluster and therefore all read
> lock acquisitions in JGroupsDistSync ultimately enabling smooth write
> lock acquisitions for state transfer and what not. In conclusion this
> leads me wondering if the culprit of all this FLUSH mess is rooted in
> read/write lock semantics from above?
>
> Regards,
> Vladimir
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Dan Berindei
On Wed, May 11, 2011 at 11:25 AM, Vladimir Blagojevic
<[hidden email]> wrote:
> The more I research ReentrantReadWriteLock the more shocked I am. Not
> only that a thread wanting to acquire write lock first has to release
> read lock, but we can block forever even if we release the read lock if
> we have acquired that read lock reentrantly. Each call to unlock just
> reduces the hold count, and the lock is only actually released when the
> hold count hits zero. Surreal!
>

If ReentrantReadWriteLock would allow upgrades then you would get a
deadlock when two threads both hold the read lock and try to upgrade
to a write lock at the same time.
There's always a trade-off...

I'm not familiar with the code, but are you sure the read lock is
being held by the same thread that is trying to acquire the write
lock?

Dan


> People have already debated this issue:
> http://stackoverflow.com/questions/464784/java-reentrantreadwritelocks-how-to-safely-acquire-write-lock
>
> In conlusion we have to seriously revisit all our uses of
> ReentrantReadWriteLock!
>
> Vladimir
>
>
> On 11-05-10 11:31 AM, Vladimir Blagojevic wrote:
>> Hi,
>>
>> Would you please confirm my understanding and reading of javadoc for
>> ReentrantReadWriteLock under section for "lock downgrading". It says:
>> "Reentrancy also allows downgrading from the write lock to a read lock,
>> by acquiring the write lock, then the read lock and then releasing the
>> write lock. However, upgrading from a read lock to the write lock is not
>> possible." ReentrantReadWriteLock javadoc code example with class
>> CacheData also shows how a thread owning a read lock first has to
>> release it in order to obtain a write lock! So a thread owning a read
>> lock first has to release a read lock in order to obtain a write lock?
>>
>> This is very symptomatic in logs of ISPN-83 such as this one
>> https://issues.jboss.org/secure/attachment/12341409/server1.log
>>
>> Recall how FLUSH stops all invocations on cluster and therefore all read
>> lock acquisitions in JGroupsDistSync ultimately enabling smooth write
>> lock acquisitions for state transfer and what not. In conclusion this
>> leads me wondering if the culprit of all this FLUSH mess is rooted in
>> read/write lock semantics from above?
>>
>> Regards,
>> Vladimir
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Vladimir Blagojevic
On 11-05-11 11:23 AM, Dan Berindei wrote:
> If ReentrantReadWriteLock would allow upgrades then you would get a
> deadlock when two threads both hold the read lock and try to upgrade
> to a write lock at the same time.
> There's always a trade-off...
>
> I'm not familiar with the code, but are you sure the read lock is
> being held by the same thread that is trying to acquire the write
> lock?
>

Not sure and it sounds counter intuitive that a thread holding a read
lock from cluster invocation is doing state generation for state
transfer as well. But this lock business is fishy and I plan to get to
the bottom of it...

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Manik Surtani-2
Yes, please have a look. If we are relying on lock upgrades then that's really bad. I am aware of the inability to (safely) upgrade a RWL and I'm pretty sure we don't try, but the dist sync codebase has evolved a lot and could do with some careful analysis.

Sent from my mobile phone

On 12 May 2011, at 09:24, Vladimir Blagojevic <[hidden email]> wrote:

> On 11-05-11 11:23 AM, Dan Berindei wrote:
>> If ReentrantReadWriteLock would allow upgrades then you would get a
>> deadlock when two threads both hold the read lock and try to upgrade
>> to a write lock at the same time.
>> There's always a trade-off...
>>
>> I'm not familiar with the code, but are you sure the read lock is
>> being held by the same thread that is trying to acquire the write
>> lock?
>>
>
> Not sure and it sounds counter intuitive that a thread holding a read
> lock from cluster invocation is doing state generation for state
> transfer as well. But this lock business is fishy and I plan to get to
> the bottom of it...
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Vladimir Blagojevic
I agree Manik. Codebase has evolved and the solution I have for this
problem involves detecting such a lockup happening again but this time
we will capture which threads locked RWL and why thus leading to a final
resolution.

I will include the fix in the next CR and follow up in later releases
(if necessary).

Vladimir


On 11-05-13 4:53 PM, Manik Surtani wrote:
> Yes, please have a look. If we are relying on lock upgrades then that's really bad. I am aware of the inability to (safely) upgrade a RWL and I'm pretty sure we don't try, but the dist sync codebase has evolved a lot and could do with some careful analysis.
>
> Sent from my mobile phone

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Vladimir Blagojevic
In reply to this post by Manik Surtani-2
Manik (and others),

Can you run this code on your laptops and let me know what happened!

Vladimir

    public static void main (String [] arg) throws Exception {
       final JGroupsDistSync ds = new JGroupsDistSync();

       ds.acquireProcessingLock(false, 3, TimeUnit.SECONDS);
       System.out.println("Got read lock");

       Thread t = new Thread(){
         public void run (){
            try {
             ds.acquireProcessingLock(true, 3, TimeUnit.SECONDS);
             System.out.println("Got write lock");
          } catch (TimeoutException e) {
             System.out.println(e);
          }
         }
       };
       t.start();
    }



On 11-05-13 4:53 PM, Manik Surtani wrote:

> Yes, please have a look. If we are relying on lock upgrades then that's really bad. I am aware of the inability to (safely) upgrade a RWL and I'm pretty sure we don't try, but the dist sync codebase has evolved a lot and could do with some careful analysis.
>
> Sent from my mobile phone
>
> On 12 May 2011, at 09:24, Vladimir Blagojevic<[hidden email]>  wrote:
>
>> On 11-05-11 11:23 AM, Dan Berindei wrote:
>>> If ReentrantReadWriteLock would allow upgrades then you would get a
>>> deadlock when two threads both hold the read lock and try to upgrade
>>> to a write lock at the same time.
>>> There's always a trade-off...
>>>
>>> I'm not familiar with the code, but are you sure the read lock is
>>> being held by the same thread that is trying to acquire the write
>>> lock?
>>>
>> Not sure and it sounds counter intuitive that a thread holding a read
>> lock from cluster invocation is doing state generation for state
>> transfer as well. But this lock business is fishy and I plan to get to
>> the bottom of it...
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Erik Salter
EDIT:  Originally posted in the wrong thread.  I blame Outlook.

I guess I qualify as "others", since I'm looking at similar issues.

Got read lock
java.util.concurrent.TimeoutException: Thread-1 could not obtain exclusive processing lock after 3 seconds.  Locks in question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@16aa37a6[Read locks = 1] and java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@12b7eea[Unlocked]

This is on the latest master.

Erik

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Vladimir Blagojevic
Sent: Monday, May 16, 2011 4:25 PM
To: infinispan -Dev List
Cc: Manik Surtani
Subject: Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Manik (and others),

Can you run this code on your laptops and let me know what happened!

Vladimir

    public static void main (String [] arg) throws Exception {
       final JGroupsDistSync ds = new JGroupsDistSync();

       ds.acquireProcessingLock(false, 3, TimeUnit.SECONDS);
       System.out.println("Got read lock");

       Thread t = new Thread(){
         public void run (){
            try {
             ds.acquireProcessingLock(true, 3, TimeUnit.SECONDS);
             System.out.println("Got write lock");
          } catch (TimeoutException e) {
             System.out.println(e);
          }
         }
       };
       t.start();
    }



On 11-05-13 4:53 PM, Manik Surtani wrote:

> Yes, please have a look. If we are relying on lock upgrades then that's really bad. I am aware of the inability to (safely) upgrade a RWL and I'm pretty sure we don't try, but the dist sync codebase has evolved a lot and could do with some careful analysis.
>
> Sent from my mobile phone
>
> On 12 May 2011, at 09:24, Vladimir Blagojevic<[hidden email]>  wrote:
>
>> On 11-05-11 11:23 AM, Dan Berindei wrote:
>>> If ReentrantReadWriteLock would allow upgrades then you would get a
>>> deadlock when two threads both hold the read lock and try to upgrade
>>> to a write lock at the same time.
>>> There's always a trade-off...
>>>
>>> I'm not familiar with the code, but are you sure the read lock is
>>> being held by the same thread that is trying to acquire the write
>>> lock?
>>>
>> Not sure and it sounds counter intuitive that a thread holding a read
>> lock from cluster invocation is doing state generation for state
>> transfer as well. But this lock business is fishy and I plan to get
>> to the bottom of it...
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

The information contained in this message is legally privileged and confidential, and is intended for the individual or entity to whom it is addressed (or their designee). If this message is read by anyone other than the intended recipient, please be advised that distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete or destroy all copies of this message.

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Sanne Grinovero
Same result here - where you expecting something different?

Cheers,
Sanne

2011/5/16 Erik Salter <[hidden email]>:

> EDIT:  Originally posted in the wrong thread.  I blame Outlook.
>
> I guess I qualify as "others", since I'm looking at similar issues.
>
> Got read lock
> java.util.concurrent.TimeoutException: Thread-1 could not obtain exclusive processing lock after 3 seconds.  Locks in question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@16aa37a6[Read locks = 1] and java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@12b7eea[Unlocked]
>
> This is on the latest master.
>
> Erik
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Vladimir Blagojevic
> Sent: Monday, May 16, 2011 4:25 PM
> To: infinispan -Dev List
> Cc: Manik Surtani
> Subject: Re: [infinispan-dev] JGroupsDistSync and ISPN-83
>
> Manik (and others),
>
> Can you run this code on your laptops and let me know what happened!
>
> Vladimir
>
>    public static void main (String [] arg) throws Exception {
>       final JGroupsDistSync ds = new JGroupsDistSync();
>
>       ds.acquireProcessingLock(false, 3, TimeUnit.SECONDS);
>       System.out.println("Got read lock");
>
>       Thread t = new Thread(){
>         public void run (){
>            try {
>             ds.acquireProcessingLock(true, 3, TimeUnit.SECONDS);
>             System.out.println("Got write lock");
>          } catch (TimeoutException e) {
>             System.out.println(e);
>          }
>         }
>       };
>       t.start();
>    }
>
>
>
> On 11-05-13 4:53 PM, Manik Surtani wrote:
>> Yes, please have a look. If we are relying on lock upgrades then that's really bad. I am aware of the inability to (safely) upgrade a RWL and I'm pretty sure we don't try, but the dist sync codebase has evolved a lot and could do with some careful analysis.
>>
>> Sent from my mobile phone
>>
>> On 12 May 2011, at 09:24, Vladimir Blagojevic<[hidden email]>  wrote:
>>
>>> On 11-05-11 11:23 AM, Dan Berindei wrote:
>>>> If ReentrantReadWriteLock would allow upgrades then you would get a
>>>> deadlock when two threads both hold the read lock and try to upgrade
>>>> to a write lock at the same time.
>>>> There's always a trade-off...
>>>>
>>>> I'm not familiar with the code, but are you sure the read lock is
>>>> being held by the same thread that is trying to acquire the write
>>>> lock?
>>>>
>>> Not sure and it sounds counter intuitive that a thread holding a read
>>> lock from cluster invocation is doing state generation for state
>>> transfer as well. But this lock business is fishy and I plan to get
>>> to the bottom of it...
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> The information contained in this message is legally privileged and confidential, and is intended for the individual or entity to whom it is addressed (or their designee). If this message is read by anyone other than the intended recipient, please be advised that distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete or destroy all copies of this message.
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Vladimir Blagojevic
Apparently I did not understand semantics of RWL. I thought that writer
can obtain a write lock even though read lock has not been released. But
no, after all read locks have been released only then a writer can
obtain a lock.



On 11-05-16 11:18 PM, Sanne Grinovero wrote:

> Same result here - where you expecting something different?
>
> Cheers,
> Sanne
>
> 2011/5/16 Erik Salter<[hidden email]>:
>> EDIT:  Originally posted in the wrong thread.  I blame Outlook.
>>
>> I guess I qualify as "others", since I'm looking at similar issues.
>>
>> Got read lock
>> java.util.concurrent.TimeoutException: Thread-1 could not obtain exclusive processing lock after 3 seconds.  Locks in question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@16aa37a6[Read locks = 1] and java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@12b7eea[Unlocked]
>>
>> This is on the latest master.
>>
>> Erik
>>
>> -----Original Message-----
>> From: [hidden email] [mailto:[hidden email]] On Behalf Of Vladimir Blagojevic
>> Sent: Monday, May 16, 2011 4:25 PM
>> To: infinispan -Dev List
>> Cc: Manik Surtani
>> Subject: Re: [infinispan-dev] JGroupsDistSync and ISPN-83
>>
>> Manik (and others),
>>
>> Can you run this code on your laptops and let me know what happened!
>>
>> Vladimir
>>
>>     public static void main (String [] arg) throws Exception {
>>        final JGroupsDistSync ds = new JGroupsDistSync();
>>
>>        ds.acquireProcessingLock(false, 3, TimeUnit.SECONDS);
>>        System.out.println("Got read lock");
>>
>>        Thread t = new Thread(){
>>          public void run (){
>>             try {
>>              ds.acquireProcessingLock(true, 3, TimeUnit.SECONDS);
>>              System.out.println("Got write lock");
>>           } catch (TimeoutException e) {
>>              System.out.println(e);
>>           }
>>          }
>>        };
>>        t.start();
>>     }
>>
>>
>>
>> On 11-05-13 4:53 PM, Manik Surtani wrote:
>>> Yes, please have a look. If we are relying on lock upgrades then that's really bad. I am aware of the inability to (safely) upgrade a RWL and I'm pretty sure we don't try, but the dist sync codebase has evolved a lot and could do with some careful analysis.
>>>
>>> Sent from my mobile phone
>>>
>>> On 12 May 2011, at 09:24, Vladimir Blagojevic<[hidden email]>    wrote:
>>>
>>>> On 11-05-11 11:23 AM, Dan Berindei wrote:
>>>>> If ReentrantReadWriteLock would allow upgrades then you would get a
>>>>> deadlock when two threads both hold the read lock and try to upgrade
>>>>> to a write lock at the same time.
>>>>> There's always a trade-off...
>>>>>
>>>>> I'm not familiar with the code, but are you sure the read lock is
>>>>> being held by the same thread that is trying to acquire the write
>>>>> lock?
>>>>>
>>>> Not sure and it sounds counter intuitive that a thread holding a read
>>>> lock from cluster invocation is doing state generation for state
>>>> transfer as well. But this lock business is fishy and I plan to get
>>>> to the bottom of it...
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> The information contained in this message is legally privileged and confidential, and is intended for the individual or entity to whom it is addressed (or their designee). If this message is read by anyone other than the intended recipient, please be advised that distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete or destroy all copies of this message.
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JGroupsDistSync and ISPN-83

Vladimir Blagojevic
In conclusion this is a what we know so far regarding ISPN-83:

One of the reader threads (gets R lock on JGroupsDistSync) fails to
complete (release R lock) in 10 sec stateTransferTimeout and therefore
prevents state receiver from properly installing the state thus throwing
ISPN00095 error code. You can observe this phenomena by running
StateTransferCacheLoaderFunctionalTest

ISPN00095 is what the users where reporting after they have removed
FLUSH from the stack

Unfortunately I would say that since ISPN00095 is a symptom of a problem
but not the cause I have to continue to dig to see why it is happening
to begin with an I will continue to do so as soon as the master
testsuite is stabilized.

Let me know if you have some comments and suggestions!

Regards,
Vladimir


On 11-05-17 8:27 AM, Vladimir Blagojevic wrote:

> Apparently I did not understand semantics of RWL. I thought that writer
> can obtain a write lock even though read lock has not been released. But
> no, after all read locks have been released only then a writer can
> obtain a lock.
>
>
>
> On 11-05-16 11:18 PM, Sanne Grinovero wrote:
>> Same result here - where you expecting something different?
>>
>> Cheers,
>> Sanne
>>
>> 2011/5/16 Erik Salter<[hidden email]>:
>>> EDIT:  Originally posted in the wrong thread.  I blame Outlook.
>>>
>>> I guess I qualify as "others", since I'm looking at similar issues.
>>>
>>> Got read lock
>>> java.util.concurrent.TimeoutException: Thread-1 could not obtain exclusive processing lock after 3 seconds.  Locks in question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@16aa37a6[Read locks = 1] and java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@12b7eea[Unlocked]
>>>
>>> This is on the latest master.
>>>
>>> Erik
>>>
>>> -----Original Message-----
>>> From: [hidden email] [mailto:[hidden email]] On Behalf Of Vladimir Blagojevic
>>> Sent: Monday, May 16, 2011 4:25 PM
>>> To: infinispan -Dev List
>>> Cc: Manik Surtani
>>> Subject: Re: [infinispan-dev] JGroupsDistSync and ISPN-83
>>>
>>> Manik (and others),
>>>
>>> Can you run this code on your laptops and let me know what happened!
>>>
>>> Vladimir
>>>
>>>      public static void main (String [] arg) throws Exception {
>>>         final JGroupsDistSync ds = new JGroupsDistSync();
>>>
>>>         ds.acquireProcessingLock(false, 3, TimeUnit.SECONDS);
>>>         System.out.println("Got read lock");
>>>
>>>         Thread t = new Thread(){
>>>           public void run (){
>>>              try {
>>>               ds.acquireProcessingLock(true, 3, TimeUnit.SECONDS);
>>>               System.out.println("Got write lock");
>>>            } catch (TimeoutException e) {
>>>               System.out.println(e);
>>>            }
>>>           }
>>>         };
>>>         t.start();
>>>      }
>>>
>>>
>>>
>>> On 11-05-13 4:53 PM, Manik Surtani wrote:
>>>> Yes, please have a look. If we are relying on lock upgrades then that's really bad. I am aware of the inability to (safely) upgrade a RWL and I'm pretty sure we don't try, but the dist sync codebase has evolved a lot and could do with some careful analysis.
>>>>
>>>> Sent from my mobile phone
>>>>
>>>> On 12 May 2011, at 09:24, Vladimir Blagojevic<[hidden email]>     wrote:
>>>>
>>>>> On 11-05-11 11:23 AM, Dan Berindei wrote:
>>>>>> If ReentrantReadWriteLock would allow upgrades then you would get a
>>>>>> deadlock when two threads both hold the read lock and try to upgrade
>>>>>> to a write lock at the same time.
>>>>>> There's always a trade-off...
>>>>>>
>>>>>> I'm not familiar with the code, but are you sure the read lock is
>>>>>> being held by the same thread that is trying to acquire the write
>>>>>> lock?
>>>>>>
>>>>> Not sure and it sounds counter intuitive that a thread holding a read
>>>>> lock from cluster invocation is doing state generation for state
>>>>> transfer as well. But this lock business is fishy and I plan to get
>>>>> to the bottom of it...
>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> The information contained in this message is legally privileged and confidential, and is intended for the individual or entity to whom it is addressed (or their designee). If this message is read by anyone other than the intended recipient, please be advised that distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please notify the sender immediately and delete or destroy all copies of this message.
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev