[infinispan-dev] weekly questions

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] weekly questions

Mircea Markus
Hi Jonathan,

We're thinking about doing some significant changes/improvements around transactions[1]. I'd like to pick your brain especially on improvement #1, as this changes the way we acquire locks quite a bit.

With this new locking approach, we don't acquire WL for a transaction until prepare phase. On each write  we keep a copy of the written key on the node where transaction is executed so that each subsequent access, in the same transaction, would read/update this copy. Same goes for reads.
When transaction commits we acquire WL during prepare time. This looks to me very similar to the optimistic CC, just that there's no conflict verification before commit.
Also, it seems to preserve repeatable read transaction isolation correctly.
This is a very significant change in our transaction model - do you see any problems/suggestions around it?

Cheers,
Mircea 
  


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] weekly questions

Jonathan Halliday

some general comments:

- preserving some subset of the existing transactional guarantees is all
well and good BUT if the user is relying on additional 'guarantees' that
are not preserved in all cases then they'll be in trouble. Therefore,
it's essential to document not just what the minimum expected guarantees
for a given config are, but also that no additional properties that may
coincidently be observed are actually guaranteed. Some vendors go
further and explicitly document the bad things that may happen with
given settings, since in some cases these are not easy to reproduce in
testing.

- aquiring locks in defined (e.g. HC) order is kind of obvious and I'm
surprised it's not default behaviour. It probably should be as soon as
the implementation is stable.  Additionally, rather than making
consecutive RPCs to a node for each of the keys it owns, why not locally
walk the full set of objects in the tx first and batch them by owning
node, then make one call to each node with the whole batch of lock requests?

- I'm rather suspicions of any discussion that includes phrases like 'it
will have to rollback its running transactions'.  A node in an XA
transaction that has already prepared CAN'T rollback, the protocol does
not allow it. So, optimisations using non-replicated locks for
replicated data should be disabled if the tx is under external control
e.g. XA rather than local to infinispan. For that matter I think you'll
have trouble with internal transactions too in some failure cases, but
at least they won't affect me :-)

Jonathan.

On 05/25/2011 12:55 PM, Mircea Markus wrote:

> Hi Jonathan,
>
> We're thinking about doing some significant changes/improvements around
> transactions[1]. I'd like to pick your brain especially on improvement
> #1, as this changes the way we acquire locks quite a bit.
>
> With this new locking approach, we don't acquire WL for a transaction
> until prepare phase. On each write we keep a copy of the written key on
> the node where transaction is executed so that each subsequent access,
> in the same transaction, would read/update this copy. Same goes for reads.
> When transaction commits we acquire WL during prepare time. This looks
> to me very similar to the optimistic CC, just that there's no conflict
> verification before commit.
> Also, it seems to preserve repeatable read transaction isolation correctly.
> This is a very significant change in our transaction model - do you see
> any problems/suggestions around it?
>
> Cheers,
> Mircea
>
> [1] http://community.jboss.org/wiki/PossibleLockingImprovements

--
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903
Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons
(USA) and Brendan Lane (Ireland).
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] weekly questions

Sanne Grinovero
2011/6/1 Jonathan Halliday <[hidden email]>:

>
> some general comments:
>
> - preserving some subset of the existing transactional guarantees is all
> well and good BUT if the user is relying on additional 'guarantees' that
> are not preserved in all cases then they'll be in trouble. Therefore,
> it's essential to document not just what the minimum expected guarantees
> for a given config are, but also that no additional properties that may
> coincidently be observed are actually guaranteed. Some vendors go
> further and explicitly document the bad things that may happen with
> given settings, since in some cases these are not easy to reproduce in
> testing.

Agreed on documenting it properly, as I'm still confused myself. But
just to clarify this thread
is not about preserving a subset of existing transactional guarantees,
it's more about having
the same guarantees on each node: we observed that some locking
patterns have a different semantic
when run on the node owning the locked keys, compared to run them on
nodes which need to acquire
the same locks by using a remote call.

>
> - aquiring locks in defined (e.g. HC) order is kind of obvious and I'm
> surprised it's not default behaviour. It probably should be as soon as
> the implementation is stable.  Additionally, rather than making
> consecutive RPCs to a node for each of the keys it owns, why not locally
> walk the full set of objects in the tx first and batch them by owning
> node, then make one call to each node with the whole batch of lock requests?

It was surprising to me as well to find they where not acquired in
order, but the main use case of a key value store is to operate on
single keys; also if we don't postpone the lock acquisition to the
prepare phase (as with the current proposal), it all depends in the
order of execution of the user operations so there was no point in
having such a logic.

It would be nice to be able to batch the lock commands in single
invocations per owner, but it's going to be tricky to implement that
if you also want to acquire them all in a specific order. We'll need
to make sure that the defined order in which locks are acquired
matches the order of keys on the wheel: we first define the sequence
of locks, and then split this sequence in chunks of commands only when
contiguous keys are owned by the same node.
Another good reason to keep a single lock owner per key, that would
make this much simpler.

>
> - I'm rather suspicions of any discussion that includes phrases like 'it
> will have to rollback its running transactions'.  A node in an XA
> transaction that has already prepared CAN'T rollback, the protocol does
> not allow it. So, optimisations using non-replicated locks for
> replicated data should be disabled if the tx is under external control
> e.g. XA rather than local to infinispan. For that matter I think you'll
> have trouble with internal transactions too in some failure cases, but
> at least they won't affect me :-)

What do you mean by "me" ?

Thanks for the very valuable feedback.
Sanne

>
> Jonathan.
>
> On 05/25/2011 12:55 PM, Mircea Markus wrote:
>> Hi Jonathan,
>>
>> We're thinking about doing some significant changes/improvements around
>> transactions[1]. I'd like to pick your brain especially on improvement
>> #1, as this changes the way we acquire locks quite a bit.
>>
>> With this new locking approach, we don't acquire WL for a transaction
>> until prepare phase. On each write we keep a copy of the written key on
>> the node where transaction is executed so that each subsequent access,
>> in the same transaction, would read/update this copy. Same goes for reads.
>> When transaction commits we acquire WL during prepare time. This looks
>> to me very similar to the optimistic CC, just that there's no conflict
>> verification before commit.
>> Also, it seems to preserve repeatable read transaction isolation correctly.
>> This is a very significant change in our transaction model - do you see
>> any problems/suggestions around it?
>>
>> Cheers,
>> Mircea
>>
>> [1] http://community.jboss.org/wiki/PossibleLockingImprovements
>
> --
> Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
> Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
> Registered in UK and Wales under Company Registration No. 3798903
> Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons
> (USA) and Brendan Lane (Ireland).
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] weekly questions

Mircea Markus
In reply to this post by Jonathan Halliday
On 1 Jun 2011, at 08:26, Jonathan Halliday wrote:

some general comments:

- preserving some subset of the existing transactional guarantees is all well and good BUT if the user is relying on additional 'guarantees' that are not preserved in all cases then they'll be in trouble. Therefore, it's essential to document not just what the minimum expected guarantees for a given config are, but also that no additional properties that may coincidently be observed are actually guaranteed. Some vendors go further and explicitly document the bad things that may happen with given settings, since in some cases these are not easy to reproduce in testing.
Agreed: https://issues.jboss.org/browse/ISPN-1146

- aquiring locks in defined (e.g. HC) order is kind of obvious and I'm surprised it's not default behaviour.
It is the default behaviour indeed. We are considering an option to reorder lock acquisition in order to avoid deadlocks[1]. Enablable through a configuration.
It probably should be as soon as the implementation is stable.  Additionally, rather than making consecutive RPCs to a node for each of the keys it owns, why not locally walk the full set of objects in the tx first and batch them by owning node, then make one call to each node with the whole batch of lock requests?
The futures[2] API does something similar, but not quite that though. Interesting optimisation for the Future API.

- I'm rather suspicions of any discussion that includes phrases like 'it will have to rollback its running transactions'.   
A node in an XA transaction that has already prepared CAN'T rollback, the protocol does not allow it.
yes, totally agree. And since 5.0 we don't do that anymore. 
So, optimisations using non-replicated locks for replicated data should be disabled if the tx is under external control e.g. XA rather than local to infinispan. For that matter I think you'll have trouble with internal transactions too in some failure cases, but at least they won't affect me :-)

[1] http://community.jboss.org/wiki/PossibleLockingImprovements, see section #3.
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] weekly questions

Mircea Markus
In reply to this post by Sanne Grinovero

On 1 Jun 2011, at 09:37, Sanne Grinovero wrote:

> 2011/6/1 Jonathan Halliday <[hidden email]>:
>>
>> some general comments:
>>
>> - preserving some subset of the existing transactional guarantees is all
>> well and good BUT if the user is relying on additional 'guarantees' that
>> are not preserved in all cases then they'll be in trouble. Therefore,
>> it's essential to document not just what the minimum expected guarantees
>> for a given config are, but also that no additional properties that may
>> coincidently be observed are actually guaranteed. Some vendors go
>> further and explicitly document the bad things that may happen with
>> given settings, since in some cases these are not easy to reproduce in
>> testing.
>
> Agreed on documenting it properly, as I'm still confused myself. But
> just to clarify this thread
> is not about preserving a subset of existing transactional guarantees,
> it's more about having
> the same guarantees on each node: we observed that some locking
> patterns have a different semantic
> when run on the node owning the locked keys, compared to run them on
> nodes which need to acquire
> the same locks by using a remote call.
>
>>
>> - aquiring locks in defined (e.g. HC) order is kind of obvious and I'm
>> surprised it's not default behaviour. It probably should be as soon as
>> the implementation is stable.  Additionally, rather than making
>> consecutive RPCs to a node for each of the keys it owns, why not locally
>> walk the full set of objects in the tx first and batch them by owning
>> node, then make one call to each node with the whole batch of lock requests?
>
> It was surprising to me as well to find they where not acquired in
> order,
But they are acquired in order :-)

> but the main use case of a key value store is to operate on
> single keys; also if we don't postpone the lock acquisition to the
> prepare phase (as with the current proposal), it all depends in the
> order of execution of the user operations so there was no point in
> having such a logic.
>
> It would be nice to be able to batch the lock commands in single
> invocations per owner, but it's going to be tricky to implement that
> if you also want to acquire them all in a specific order. We'll need
> to make sure that the defined order in which locks are acquired
> matches the order of keys on the wheel: we first define the sequence
> of locks, and then split this sequence in chunks of commands only when
> contiguous keys are owned by the same node.
> Another good reason to keep a single lock owner per key, that would
> make this much simpler.
>
>>
>> - I'm rather suspicions of any discussion that includes phrases like 'it
>> will have to rollback its running transactions'.  A node in an XA
>> transaction that has already prepared CAN'T rollback, the protocol does
>> not allow it. So, optimisations using non-replicated locks for
>> replicated data should be disabled if the tx is under external control
>> e.g. XA rather than local to infinispan. For that matter I think you'll
>> have trouble with internal transactions too in some failure cases, but
>> at least they won't affect me :-)
>
> What do you mean by "me" ?
>
> Thanks for the very valuable feedback.
> Sanne
>
>>
>> Jonathan.
>>
>> On 05/25/2011 12:55 PM, Mircea Markus wrote:
>>> Hi Jonathan,
>>>
>>> We're thinking about doing some significant changes/improvements around
>>> transactions[1]. I'd like to pick your brain especially on improvement
>>> #1, as this changes the way we acquire locks quite a bit.
>>>
>>> With this new locking approach, we don't acquire WL for a transaction
>>> until prepare phase. On each write we keep a copy of the written key on
>>> the node where transaction is executed so that each subsequent access,
>>> in the same transaction, would read/update this copy. Same goes for reads.
>>> When transaction commits we acquire WL during prepare time. This looks
>>> to me very similar to the optimistic CC, just that there's no conflict
>>> verification before commit.
>>> Also, it seems to preserve repeatable read transaction isolation correctly.
>>> This is a very significant change in our transaction model - do you see
>>> any problems/suggestions around it?
>>>
>>> Cheers,
>>> Mircea
>>>
>>> [1] http://community.jboss.org/wiki/PossibleLockingImprovements
>>
>> --
>> Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
>> Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
>> Registered in UK and Wales under Company Registration No. 3798903
>> Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons
>> (USA) and Brendan Lane (Ireland).
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev