[infinispan-dev] Rehashing and ongoing transactions

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] Rehashing and ongoing transactions

Manik Surtani
Guys,

I see this in the code base:


which has come in as a part of this commit:


This is no good, since throwing an IllegalStateException *after* a transaction has prepared and is attempting to commit will leave the transaction in an inconsistent state.  This is why I see heuristic exceptions when trying to stress a lot of transactions while having nodes join/leave.

A few points:

1) Mircea, are you aware of this, and is this a part of the transaction refactoring you are working on?
2) If not, it probably should be.  ;)
3) Vladimir, why is this check necessary in the first place?  If prepares have happened on nodes A and B, and the commit should now go to nodes B and C due to a rehash, can't we send the commit to a union of A, B and C, and let the nodes that don't care about the tx anymore just ignore the commit?

Cheers
Manik

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Rehashing and ongoing transactions

Mircea Markus

On 31 Jan 2011, at 21:24, Manik Surtani wrote:

Guys,

I see this in the code base:


which has come in as a part of this commit:


This is no good, since throwing an IllegalStateException *after* a transaction has prepared and is attempting to commit will leave the transaction in an inconsistent state.  This is why I see heuristic exceptions when trying to stress a lot of transactions while having nodes join/leave.

A few points:

1) Mircea, are you aware of this, and is this a part of the transaction refactoring you are working on?
2) If not, it probably should be.  ;)
TransactionXAResource would raise an heuristic exception in this case, so it will be the admin that would need to take care of the commit. 
This code will be revisited as part of the tx refactoring
3) Vladimir, why is this check necessary in the first place?  If prepares have happened on nodes A and B, and the commit should now go to nodes B and C due to a rehash, can't we send the commit to a union of A, B and C, and let the nodes that don't care about the tx anymore just ignore the commit?

Cheers
Manik
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Rehashing and ongoing transactions

Manik Surtani
This is actually worse than I thought, since as prepare has already been issued and remote locks already acquired.  With this code in place, you're guaranteed to have stale locks in the system, and the likelihood is very high if you have any ongoing transactions when a node joins or leaves.  Stale locks then cascades to causing other subsequent transactions to fail with timeouts.

We need to handle this (i.e., not throw this exception!)

On 31 Jan 2011, at 21:40, Mircea Markus wrote:


On 31 Jan 2011, at 21:24, Manik Surtani wrote:

Guys,

I see this in the code base:


which has come in as a part of this commit:


This is no good, since throwing an IllegalStateException *after* a transaction has prepared and is attempting to commit will leave the transaction in an inconsistent state.  This is why I see heuristic exceptions when trying to stress a lot of transactions while having nodes join/leave.

A few points:

1) Mircea, are you aware of this, and is this a part of the transaction refactoring you are working on?
2) If not, it probably should be.  ;)
TransactionXAResource would raise an heuristic exception in this case, so it will be the admin that would need to take care of the commit. 
This code will be revisited as part of the tx refactoring
3) Vladimir, why is this check necessary in the first place?  If prepares have happened on nodes A and B, and the commit should now go to nodes B and C due to a rehash, can't we send the commit to a union of A, B and C, and let the nodes that don't care about the tx anymore just ignore the commit?

Cheers
Manik
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Rehashing and ongoing transactions

Manik Surtani
https://issues.jboss.org/browse/ISPN-902 is the related JIRA.  I'm adding a test that demonstrates this.

On 31 Jan 2011, at 21:43, Manik Surtani wrote:

This is actually worse than I thought, since as prepare has already been issued and remote locks already acquired.  With this code in place, you're guaranteed to have stale locks in the system, and the likelihood is very high if you have any ongoing transactions when a node joins or leaves.  Stale locks then cascades to causing other subsequent transactions to fail with timeouts.

We need to handle this (i.e., not throw this exception!)

On 31 Jan 2011, at 21:40, Mircea Markus wrote:


On 31 Jan 2011, at 21:24, Manik Surtani wrote:

Guys,

I see this in the code base:


which has come in as a part of this commit:


This is no good, since throwing an IllegalStateException *after* a transaction has prepared and is attempting to commit will leave the transaction in an inconsistent state.  This is why I see heuristic exceptions when trying to stress a lot of transactions while having nodes join/leave.

A few points:

1) Mircea, are you aware of this, and is this a part of the transaction refactoring you are working on?
2) If not, it probably should be.  ;)
TransactionXAResource would raise an heuristic exception in this case, so it will be the admin that would need to take care of the commit. 
This code will be revisited as part of the tx refactoring
3) Vladimir, why is this check necessary in the first place?  If prepares have happened on nodes A and B, and the commit should now go to nodes B and C due to a rehash, can't we send the commit to a union of A, B and C, and let the nodes that don't care about the tx anymore just ignore the commit?

Cheers
Manik
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev