[infinispan-dev] Introducing JGroups scopes

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] Introducing JGroups scopes

Vladimir Blagojevic
Hey,

JGroups scopes [1] have been implemented for a while now and we should
think about using them in Infinispan. I looked around a bit to see how
we can use scope and satisfy requirements Manik outlined [2]. I think a
good idea would be to introduce scope method in InvocationContext? We
can start by implementing scope to return hash of cache name where
invocation originated and subsequently refine this scope by adding
GlobalTransaction and so on. Users, if they really need to scope their
calls could do so by attaching additional markers to InvocationContext
or smth similar.

WDYT?


[1]https://issues.jboss.org/browse/JGRP-822
[2]https://issues.jboss.org/browse/ISPN-3
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Introducing JGroups scopes

Vladimir Blagojevic
Forgot to mention that scopes require addition of SCOPE protocol to the
stack. So we can probably do this on 5.0 branch only.
On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:

> Hey,
>
> JGroups scopes [1] have been implemented for a while now and we should
> think about using them in Infinispan. I looked around a bit to see how
> we can use scope and satisfy requirements Manik outlined [2]. I think a
> good idea would be to introduce scope method in InvocationContext? We
> can start by implementing scope to return hash of cache name where
> invocation originated and subsequently refine this scope by adding
> GlobalTransaction and so on. Users, if they really need to scope their
> calls could do so by attaching additional markers to InvocationContext
> or smth similar.
>
> WDYT?
>
>
> [1]https://issues.jboss.org/browse/JGRP-822
> [2]https://issues.jboss.org/browse/ISPN-3
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Introducing JGroups scopes

Galder Zamarreno
Hmmmm, not sure about adding a method to IC, cos that would lead to code like this which is very verbose:

cache.getAdvancedCache().getInvocationContextContainer().getInvocationContext().setScope("X")

As a user, I'd prefer something like:

cache.getAdvancedCache().withScope("X")

However, this would only be used to attach a particular scope. I envision a global, cache manager level configuration such as that marks all caches in that cache manager to be scoped by the cache name. That way, HTTP session repl code would need no changing at all and with a single flag, they could make each individual cache to act in its own scope. The code above would allow two caches to share the same scope for those situations where we want sequential calls for the two caches, i.e.:

cache1.getAdvancedCache().withScope("X")
cache1.put()
cache2.getAdvancedCache().withScope("X")
cache2.put()

By the way, I don't understand your comment of "refine this scope by adding GlobalTransaction and so on".  If we take the cache1/cache2 example and imagine that these two would be called within a transaction, I'd imagine that the transaction itself would act as a scope and so would not need for a particular scope definition? i.e.

tx.begin()
cache1.put()
cache2.put()
tx.commit()

At first glance, 5.0 would be the perfect target for this, but we'd need to take in account for any requirements coming from Paul since HTTP session repl could greatly benefit from it.

As far as 2LC is concerned, within a deployment multiple txs will be operating on same caches so scoping could not be used. However different deployments could benefit from deployment level scoping so that messages for different caches can be sent in parallel. In AS environment different deployments could share the same 2LC cache manager and so the configuration option above would not work, but I'd need a deployment level scope that I'd use with cache.withScope().

Cheers,

On Jan 28, 2011, at 8:54 PM, Vladimir Blagojevic wrote:

> Forgot to mention that scopes require addition of SCOPE protocol to the
> stack. So we can probably do this on 5.0 branch only.
> On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:
>> Hey,
>>
>> JGroups scopes [1] have been implemented for a while now and we should
>> think about using them in Infinispan. I looked around a bit to see how
>> we can use scope and satisfy requirements Manik outlined [2]. I think a
>> good idea would be to introduce scope method in InvocationContext? We
>> can start by implementing scope to return hash of cache name where
>> invocation originated and subsequently refine this scope by adding
>> GlobalTransaction and so on. Users, if they really need to scope their
>> calls could do so by attaching additional markers to InvocationContext
>> or smth similar.
>>
>> WDYT?
>>
>>
>> [1]https://issues.jboss.org/browse/JGRP-822
>> [2]https://issues.jboss.org/browse/ISPN-3
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Introducing JGroups scopes

Manik Surtani
Just to clarify, there are 2 phases to this, if you will.

Phase 1: Infinispan makes use of scopes internally, for transactional and non-transactional calls to gain greater concurrency in the way JGroups broadcasts messages.

Phase 2: Allow user defined scopes.

Lets start with 1 and make sure we have a workable structure first.  So to revisit the problem (Vladimir/Bela - pls correct me if I am wrong), JGroups serializes the delivery of messages off the wire per-sender.  This means that Node A may send many messages, but a given recipient, say Node B, would queue up these messages and process them in order.  This means that even though we may have many threads on A doing non-related work, on disjoint data sets, and although this happens in parallel on A, by the time they get to B they get processed in order.  

To work around this, we use the OOB flag for synchronous messages (since we hold locks for these until the RPC returns and we know that data from such transactions cannot get reordered leading to inconsistent data).

The issue is that if we use _async_ mode, we cannot use OOB since things can get reordered and hence inconsistent, and we _rely_ on JGroups processing messages in order to maintain consistency.

Scopes helps us parallelize this.  On a very basic level, if the scope is a cache name, then at least transactions on different caches in the same cache manager can be handled in parallel.  But it can get better: I propose using some mechanism to represent the lock or locks touched by the transaction as your scope.  So any other transaction touching one or more of the same locks will be serialized, everything else could happen in parallel.  Perhaps a Bloom filter?

Cheers
Manik


On 31 Jan 2011, at 13:19, Galder Zamarreño wrote:

> Hmmmm, not sure about adding a method to IC, cos that would lead to code like this which is very verbose:
>
> cache.getAdvancedCache().getInvocationContextContainer().getInvocationContext().setScope("X")
>
> As a user, I'd prefer something like:
>
> cache.getAdvancedCache().withScope("X")
>
> However, this would only be used to attach a particular scope. I envision a global, cache manager level configuration such as that marks all caches in that cache manager to be scoped by the cache name. That way, HTTP session repl code would need no changing at all and with a single flag, they could make each individual cache to act in its own scope. The code above would allow two caches to share the same scope for those situations where we want sequential calls for the two caches, i.e.:
>
> cache1.getAdvancedCache().withScope("X")
> cache1.put()
> cache2.getAdvancedCache().withScope("X")
> cache2.put()
>
> By the way, I don't understand your comment of "refine this scope by adding GlobalTransaction and so on".  If we take the cache1/cache2 example and imagine that these two would be called within a transaction, I'd imagine that the transaction itself would act as a scope and so would not need for a particular scope definition? i.e.
>
> tx.begin()
> cache1.put()
> cache2.put()
> tx.commit()
>
> At first glance, 5.0 would be the perfect target for this, but we'd need to take in account for any requirements coming from Paul since HTTP session repl could greatly benefit from it.
>
> As far as 2LC is concerned, within a deployment multiple txs will be operating on same caches so scoping could not be used. However different deployments could benefit from deployment level scoping so that messages for different caches can be sent in parallel. In AS environment different deployments could share the same 2LC cache manager and so the configuration option above would not work, but I'd need a deployment level scope that I'd use with cache.withScope().
>
> Cheers,
>
> On Jan 28, 2011, at 8:54 PM, Vladimir Blagojevic wrote:
>
>> Forgot to mention that scopes require addition of SCOPE protocol to the
>> stack. So we can probably do this on 5.0 branch only.
>> On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:
>>> Hey,
>>>
>>> JGroups scopes [1] have been implemented for a while now and we should
>>> think about using them in Infinispan. I looked around a bit to see how
>>> we can use scope and satisfy requirements Manik outlined [2]. I think a
>>> good idea would be to introduce scope method in InvocationContext? We
>>> can start by implementing scope to return hash of cache name where
>>> invocation originated and subsequently refine this scope by adding
>>> GlobalTransaction and so on. Users, if they really need to scope their
>>> calls could do so by attaching additional markers to InvocationContext
>>> or smth similar.
>>>
>>> WDYT?
>>>
>>>
>>> [1]https://issues.jboss.org/browse/JGRP-822
>>> [2]https://issues.jboss.org/browse/ISPN-3
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Introducing JGroups scopes

Bela Ban
In reply to this post by Galder Zamarreno

Let's take a step back and ask ourselves what it is that we want to
scope ? Off of the top of my head the following things come to mind:

- HTTP sessions. I guess they're independent, so we can have a scope per
session. Do you guys currently use 1 Cache per session ? Then Galder's
suggestion below makes sense

- SFSBs: do we have 1 scope per bean ? What if bean X has a ref to bean
Y ? Would they have to have the same scope ?

- Hibernate 2LC: scope per entity ? I don't know how entities are broken
into tuples in the 2LC.



On 1/31/11 2:19 PM, Galder Zamarreño wrote:

> Hmmmm, not sure about adding a method to IC, cos that would lead to code like this which is very verbose:
>
> cache.getAdvancedCache().getInvocationContextContainer().getInvocationContext().setScope("X")
>
> As a user, I'd prefer something like:
>
> cache.getAdvancedCache().withScope("X")
>
> However, this would only be used to attach a particular scope. I envision a global, cache manager level configuration such as that marks all caches in that cache manager to be scoped by the cache name. That way, HTTP session repl code would need no changing at all and with a single flag, they could make each individual cache to act in its own scope. The code above would allow two caches to share the same scope for those situations where we want sequential calls for the two caches, i.e.:
>
> cache1.getAdvancedCache().withScope("X")
> cache1.put()
> cache2.getAdvancedCache().withScope("X")
> cache2.put()
>
> By the way, I don't understand your comment of "refine this scope by adding GlobalTransaction and so on".  If we take the cache1/cache2 example and imagine that these two would be called within a transaction, I'd imagine that the transaction itself would act as a scope and so would not need for a particular scope definition? i.e.
>
> tx.begin()
> cache1.put()
> cache2.put()
> tx.commit()
>
> At first glance, 5.0 would be the perfect target for this, but we'd need to take in account for any requirements coming from Paul since HTTP session repl could greatly benefit from it.
>
> As far as 2LC is concerned, within a deployment multiple txs will be operating on same caches so scoping could not be used. However different deployments could benefit from deployment level scoping so that messages for different caches can be sent in parallel. In AS environment different deployments could share the same 2LC cache manager and so the configuration option above would not work, but I'd need a deployment level scope that I'd use with cache.withScope().
>
> Cheers,
>
> On Jan 28, 2011, at 8:54 PM, Vladimir Blagojevic wrote:
>
>> Forgot to mention that scopes require addition of SCOPE protocol to the
>> stack. So we can probably do this on 5.0 branch only.
>> On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:
>>> Hey,
>>>
>>> JGroups scopes [1] have been implemented for a while now and we should
>>> think about using them in Infinispan. I looked around a bit to see how
>>> we can use scope and satisfy requirements Manik outlined [2]. I think a
>>> good idea would be to introduce scope method in InvocationContext? We
>>> can start by implementing scope to return hash of cache name where
>>> invocation originated and subsequently refine this scope by adding
>>> GlobalTransaction and so on. Users, if they really need to scope their
>>> calls could do so by attaching additional markers to InvocationContext

--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Introducing JGroups scopes

Manik Surtani
Even more fine grained than that, any 2 txs should be allowed to complete in parallel (if they competed for locks, one would block on the other in the LockManager and wouldn't progress in parallel anyway).  This is the logic that allows us to use OOB for sync calls.  The hard part is, how do we encode this information into a scope ID.  AIUI, scope is a String?  And you use simple string matching?  In which case Bloom filters won't work ...

On 31 Jan 2011, at 13:39, Bela Ban wrote:

>
> Let's take a step back and ask ourselves what it is that we want to
> scope ? Off of the top of my head the following things come to mind:
>
> - HTTP sessions. I guess they're independent, so we can have a scope per
> session. Do you guys currently use 1 Cache per session ? Then Galder's
> suggestion below makes sense
>
> - SFSBs: do we have 1 scope per bean ? What if bean X has a ref to bean
> Y ? Would they have to have the same scope ?
>
> - Hibernate 2LC: scope per entity ? I don't know how entities are broken
> into tuples in the 2LC.
>
>
>
> On 1/31/11 2:19 PM, Galder Zamarreño wrote:
>> Hmmmm, not sure about adding a method to IC, cos that would lead to code like this which is very verbose:
>>
>> cache.getAdvancedCache().getInvocationContextContainer().getInvocationContext().setScope("X")
>>
>> As a user, I'd prefer something like:
>>
>> cache.getAdvancedCache().withScope("X")
>>
>> However, this would only be used to attach a particular scope. I envision a global, cache manager level configuration such as that marks all caches in that cache manager to be scoped by the cache name. That way, HTTP session repl code would need no changing at all and with a single flag, they could make each individual cache to act in its own scope. The code above would allow two caches to share the same scope for those situations where we want sequential calls for the two caches, i.e.:
>>
>> cache1.getAdvancedCache().withScope("X")
>> cache1.put()
>> cache2.getAdvancedCache().withScope("X")
>> cache2.put()
>>
>> By the way, I don't understand your comment of "refine this scope by adding GlobalTransaction and so on".  If we take the cache1/cache2 example and imagine that these two would be called within a transaction, I'd imagine that the transaction itself would act as a scope and so would not need for a particular scope definition? i.e.
>>
>> tx.begin()
>> cache1.put()
>> cache2.put()
>> tx.commit()
>>
>> At first glance, 5.0 would be the perfect target for this, but we'd need to take in account for any requirements coming from Paul since HTTP session repl could greatly benefit from it.
>>
>> As far as 2LC is concerned, within a deployment multiple txs will be operating on same caches so scoping could not be used. However different deployments could benefit from deployment level scoping so that messages for different caches can be sent in parallel. In AS environment different deployments could share the same 2LC cache manager and so the configuration option above would not work, but I'd need a deployment level scope that I'd use with cache.withScope().
>>
>> Cheers,
>>
>> On Jan 28, 2011, at 8:54 PM, Vladimir Blagojevic wrote:
>>
>>> Forgot to mention that scopes require addition of SCOPE protocol to the
>>> stack. So we can probably do this on 5.0 branch only.
>>> On 11-01-28 4:52 PM, Vladimir Blagojevic wrote:
>>>> Hey,
>>>>
>>>> JGroups scopes [1] have been implemented for a while now and we should
>>>> think about using them in Infinispan. I looked around a bit to see how
>>>> we can use scope and satisfy requirements Manik outlined [2]. I think a
>>>> good idea would be to introduce scope method in InvocationContext? We
>>>> can start by implementing scope to return hash of cache name where
>>>> invocation originated and subsequently refine this scope by adding
>>>> GlobalTransaction and so on. Users, if they really need to scope their
>>>> calls could do so by attaching additional markers to InvocationContext
>
> --
> Bela Ban
> Lead JGroups / Clustering Team
> JBoss
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Introducing JGroups scopes

Bela Ban
In reply to this post by Manik Surtani


On 1/31/11 2:33 PM, Manik Surtani wrote:
> Just to clarify, there are 2 phases to this, if you will.
>
> Phase 1: Infinispan makes use of scopes internally, for transactional and non-transactional calls to gain greater concurrency in the way JGroups broadcasts messages.
>
> Phase 2: Allow user defined scopes.
>
> Lets start with 1 and make sure we have a workable structure first.  So to revisit the problem (Vladimir/Bela - pls correct me if I am wrong), JGroups serializes the delivery of messages off the wire per-sender.


Correct


>  This means that Node A may send many messages, but a given recipient, say Node B, would queue up these messages and process them in order.  This means that even though we may have many threads on A doing non-related work, on disjoint data sets, and although this happens in parallel on A, by the time they get to B they get processed in order


Yes


> To work around this, we use the OOB flag for synchronous messages (since we hold locks for these until the RPC returns and we know that data from such transactions cannot get reordered leading to inconsistent data).
>
> The issue is that if we use _async_ mode, we cannot use OOB since things can get reordered and hence inconsistent, and we _rely_ on JGroups processing messages in order to maintain consistency.


Yes

> Scopes helps us parallelize this.  On a very basic level, if the scope is a cache name, then at least transactions on different caches in the same cache manager can be handled in parallel.

Correct


>  But it can get better: I propose using some mechanism to represent the lock or locks touched by the transaction as your scope.  So any other transaction touching one or more of the same locks will be serialized, everything else could happen in parallel.  Perhaps a Bloom filter?

Do you always use locks, even if there's no TX, or the locking level is
NONE ?

If you have a TX, you could use the TX-ID as scope.

--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Introducing JGroups scopes

Bela Ban
In reply to this post by Manik Surtani
A scope is a short.

When discussing scopes, this was seen as sufficient, but if you want
bloom filters this won't suffice...

The question is can you do with roughly 32'000 scopes per member ? If
this is the case, you could maintain a hashmap of scope-ids and bloom
filters (for example)...

If not, we need to revisit this. If you need a different type, let's
discuss this, but it should be a type that can be used as a key into a
hashmap...


On 1/31/11 2:42 PM, Manik Surtani wrote:
> Even more fine grained than that, any 2 txs should be allowed to complete in parallel (if they competed for locks, one would block on the other in the LockManager and wouldn't progress in parallel anyway).  This is the logic that allows us to use OOB for sync calls.  The hard part is, how do we encode this information into a scope ID.  AIUI, scope is a String?  And you use simple string matching?  In which case Bloom filters won't work ...


--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Introducing JGroups scopes

Manik Surtani
In reply to this post by Bela Ban

On 31 Jan 2011, at 13:55, Bela Ban wrote:

>
>
>> But it can get better: I propose using some mechanism to represent the lock or locks touched by the transaction as your scope.  So any other transaction touching one or more of the same locks will be serialized, everything else could happen in parallel.  Perhaps a Bloom filter?
>
> Do you always use locks, even if there's no TX, or the locking level is
> NONE ?

If you skip locking then things could get reordered/inconsistent even on one node.  So it is irrelevant in a distributed sense.

> If you have a TX, you could use the TX-ID as scope.

I know this is what we originally thought, but I think this might be insufficient.  Different scopes are delivered in parallel.  So tx1 and tx2 *could* be delivered in parallel, even though they touch the same keys.  E.g., they happen in sequence on node A (due to locking), but since we're talking async RPC, the 1PC prepare/commit messages could get reordered and applied on B in a different order.  And this is wrong.

Some form of understanding which keys are touched, and grouping transactions that share locks into the same scope, is what is needed.

If there are external hints, e.g., http session IDs as you suggested, this makes it much easier.

Cheers
Manik

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Introducing JGroups scopes

Mircea Markus
>
> Some form of understanding which keys are touched, and grouping transactions that share locks into the same scope, is what is needed.
What if  we run outside  tx. Each key would be in its own scope?


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Introducing JGroups scopes

Manik Surtani
Yes.  Essentially the same logic as to the locks acquired.

On 3 Feb 2011, at 12:27, Mircea Markus wrote:

>>
>> Some form of understanding which keys are touched, and grouping transactions that share locks into the same scope, is what is needed.
> What if  we run outside  tx. Each key would be in its own scope?
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev