[infinispan-dev] distributed execution - invoking commands on self

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] distributed execution - invoking commands on self

Vladimir Blagojevic
Hi,

I discovered a problem with distributed framework in cases where
Callables submitted for distributed execution contain mutable instance
fields. Just before Callable is dispersed across cluster it gets invoked
locally where instance fields of a submitted Callable can be possibly
mutated; Callable is in turn sent to remote nodes with mutated values
instead of "original" field values as submitted by user. For example,
consider pi approximation example from wiki [1]. If CircleTest Callable
declared insideCircleCount as an instance field instead of local field
of call method we would ultimately get a wrong final result.

Therefore we need to ensure that each Infinispan node gets an unmodified
instance of Callable for execution. I was looking at the possibility to
send command to self thus causing creation of a marshaled copy of a
Callable, pretty much like any other remote node receives it. However, I
was unable to invoke a command on self. Any way to do this? How about
invoking marshaling locally to create a Callable copy without going
through rpc invocation layer? I though about skipping master/self node
altogether but that's an easy way out;-)

Got suggestions, ideas?

Regards,
Vladimir


[1] http://community.jboss.org/wiki/InfinispanDistributedExecutionFramework
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Sanne Grinovero
2011/4/11 Vladimir Blagojevic <[hidden email]>:

> Hi,
>
> I discovered a problem with distributed framework in cases where
> Callables submitted for distributed execution contain mutable instance
> fields. Just before Callable is dispersed across cluster it gets invoked
> locally where instance fields of a submitted Callable can be possibly
> mutated; Callable is in turn sent to remote nodes with mutated values
> instead of "original" field values as submitted by user. For example,
> consider pi approximation example from wiki [1]. If CircleTest Callable
> declared insideCircleCount as an instance field instead of local field
> of call method we would ultimately get a wrong final result.
>
> Therefore we need to ensure that each Infinispan node gets an unmodified
> instance of Callable for execution. I was looking at the possibility to
> send command to self thus causing creation of a marshaled copy of a
> Callable, pretty much like any other remote node receives it. However, I
> was unable to invoke a command on self. Any way to do this? How about
> invoking marshaling locally to create a Callable copy without going
> through rpc invocation layer? I though about skipping master/self node
> altogether but that's an easy way out;-)
>
> Got suggestions, ideas?

Hi Vladimir,
I've no clue about invoking a marshalled command on self, but could
you not just make sure the local callable is being processed *after*
you have created a serialized copy (sent it to others) ?

Regards,
Sanne

>
> Regards,
> Vladimir
>
>
> [1] http://community.jboss.org/wiki/InfinispanDistributedExecutionFramework
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Vladimir Blagojevic
Sanne,

I thought about that as well but I do not see a way to ensure that all
other commands have been sent to other nodes other than waiting for
corresponding futures to return from get and then invoke locally in a
serial fashion.

I do not see a hook anywhere that will tell me when a command has been
dropped on the wire!

Vladimir


On 11-04-11 2:21 PM, Sanne Grinovero wrote:

> 2011/4/11 Vladimir Blagojevic<[hidden email]>:
>> Hi,
>>
>> I discovered a problem with distributed framework in cases where
>> Callables submitted for distributed execution contain mutable instance
>> fields. Just before Callable is dispersed across cluster it gets invoked
>> locally where instance fields of a submitted Callable can be possibly
>> mutated; Callable is in turn sent to remote nodes with mutated values
>> instead of "original" field values as submitted by user. For example,
>> consider pi approximation example from wiki [1]. If CircleTest Callable
>> declared insideCircleCount as an instance field instead of local field
>> of call method we would ultimately get a wrong final result.
>>
>> Therefore we need to ensure that each Infinispan node gets an unmodified
>> instance of Callable for execution. I was looking at the possibility to
>> send command to self thus causing creation of a marshaled copy of a
>> Callable, pretty much like any other remote node receives it. However, I
>> was unable to invoke a command on self. Any way to do this? How about
>> invoking marshaling locally to create a Callable copy without going
>> through rpc invocation layer? I though about skipping master/self node
>> altogether but that's an easy way out;-)
>>
>> Got suggestions, ideas?
> Hi Vladimir,
> I've no clue about invoking a marshalled command on self, but could
> you not just make sure the local callable is being processed *after*
> you have created a serialized copy (sent it to others) ?
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Mircea Markus
can't you run the Callable locally on a clone of it? If Callable is Clonable then clone it in an orthodox manner, otherwise clone it by serializing/deseralizing it.

On 11 Apr 2011, at 19:58, Vladimir Blagojevic wrote:

> Sanne,
>
> I thought about that as well but I do not see a way to ensure that all
> other commands have been sent to other nodes other than waiting for
> corresponding futures to return from get and then invoke locally in a
> serial fashion.
>
> I do not see a hook anywhere that will tell me when a command has been
> dropped on the wire!
>
> Vladimir
>
>
> On 11-04-11 2:21 PM, Sanne Grinovero wrote:
>> 2011/4/11 Vladimir Blagojevic<[hidden email]>:
>>> Hi,
>>>
>>> I discovered a problem with distributed framework in cases where
>>> Callables submitted for distributed execution contain mutable instance
>>> fields. Just before Callable is dispersed across cluster it gets invoked
>>> locally where instance fields of a submitted Callable can be possibly
>>> mutated; Callable is in turn sent to remote nodes with mutated values
>>> instead of "original" field values as submitted by user. For example,
>>> consider pi approximation example from wiki [1]. If CircleTest Callable
>>> declared insideCircleCount as an instance field instead of local field
>>> of call method we would ultimately get a wrong final result.
>>>
>>> Therefore we need to ensure that each Infinispan node gets an unmodified
>>> instance of Callable for execution. I was looking at the possibility to
>>> send command to self thus causing creation of a marshaled copy of a
>>> Callable, pretty much like any other remote node receives it. However, I
>>> was unable to invoke a command on self. Any way to do this? How about
>>> invoking marshaling locally to create a Callable copy without going
>>> through rpc invocation layer? I though about skipping master/self node
>>> altogether but that's an easy way out;-)
>>>
>>> Got suggestions, ideas?
>> Hi Vladimir,
>> I've no clue about invoking a marshalled command on self, but could
>> you not just make sure the local callable is being processed *after*
>> you have created a serialized copy (sent it to others) ?
>>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Vladimir Blagojevic
Yeah, it would be too strict to require Callable to be Cloneable so I
would opt out to serialization/deserialization as a clone method.

Galder, do out have a recommended mechanism for this in your marshalers?

Thanks Mircea

On 11-04-12 12:59 PM, Mircea Markus wrote:

> can't you run the Callable locally on a clone of it? If Callable is Clonable then clone it in an orthodox manner, otherwise clone it by serializing/deseralizing it.
>
> On 11 Apr 2011, at 19:58, Vladimir Blagojevic wrote:
>> Sanne,
>>
>> I thought about that as well but I do not see a way to ensure that all
>> other commands have been sent to other nodes other than waiting for
>> corresponding futures to return from get and then invoke locally in a
>> serial fashion.
>>
>> I do not see a hook anywhere that will tell me when a command has been
>> dropped on the wire!
>>
>> Vladimir
>>
>>
>> On 11-04-11 2:21 PM, Sanne Grinovero wrote:
>>> 2011/4/11 Vladimir Blagojevic<[hidden email]>:
>>>> Hi,
>>>>
>>>> I discovered a problem with distributed framework in cases where
>>>> Callables submitted for distributed execution contain mutable instance
>>>> fields. Just before Callable is dispersed across cluster it gets invoked
>>>> locally where instance fields of a submitted Callable can be possibly
>>>> mutated; Callable is in turn sent to remote nodes with mutated values
>>>> instead of "original" field values as submitted by user. For example,
>>>> consider pi approximation example from wiki [1]. If CircleTest Callable
>>>> declared insideCircleCount as an instance field instead of local field
>>>> of call method we would ultimately get a wrong final result.
>>>>
>>>> Therefore we need to ensure that each Infinispan node gets an unmodified
>>>> instance of Callable for execution. I was looking at the possibility to
>>>> send command to self thus causing creation of a marshaled copy of a
>>>> Callable, pretty much like any other remote node receives it. However, I
>>>> was unable to invoke a command on self. Any way to do this? How about
>>>> invoking marshaling locally to create a Callable copy without going
>>>> through rpc invocation layer? I though about skipping master/self node
>>>> altogether but that's an easy way out;-)
>>>>
>>>> Got suggestions, ideas?
>>> Hi Vladimir,
>>> I've no clue about invoking a marshalled command on self, but could
>>> you not just make sure the local callable is being processed *after*
>>> you have created a serialized copy (sent it to others) ?
>>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Mircea Markus

On 12 Apr 2011, at 18:45, Vladimir Blagojevic wrote:

> Yeah, it would be too strict to require Callable to be Cloneable so I would opt out to serialization/deserialization as a clone method.
yes, but you can still check to see if the Callable is Clonable first. Only if it is not use serialization (less effcicient)

>
> Galder, do out have a recommended mechanism for this in your marshalers?
>
> Thanks Mircea
>
> On 11-04-12 12:59 PM, Mircea Markus wrote:
>> can't you run the Callable locally on a clone of it? If Callable is Clonable then clone it in an orthodox manner, otherwise clone it by serializing/deseralizing it.
>>
>> On 11 Apr 2011, at 19:58, Vladimir Blagojevic wrote:
>>> Sanne,
>>>
>>> I thought about that as well but I do not see a way to ensure that all
>>> other commands have been sent to other nodes other than waiting for
>>> corresponding futures to return from get and then invoke locally in a
>>> serial fashion.
>>>
>>> I do not see a hook anywhere that will tell me when a command has been
>>> dropped on the wire!
>>>
>>> Vladimir
>>>
>>>
>>> On 11-04-11 2:21 PM, Sanne Grinovero wrote:
>>>> 2011/4/11 Vladimir Blagojevic<[hidden email]>:
>>>>> Hi,
>>>>>
>>>>> I discovered a problem with distributed framework in cases where
>>>>> Callables submitted for distributed execution contain mutable instance
>>>>> fields. Just before Callable is dispersed across cluster it gets invoked
>>>>> locally where instance fields of a submitted Callable can be possibly
>>>>> mutated; Callable is in turn sent to remote nodes with mutated values
>>>>> instead of "original" field values as submitted by user. For example,
>>>>> consider pi approximation example from wiki [1]. If CircleTest Callable
>>>>> declared insideCircleCount as an instance field instead of local field
>>>>> of call method we would ultimately get a wrong final result.
>>>>>
>>>>> Therefore we need to ensure that each Infinispan node gets an unmodified
>>>>> instance of Callable for execution. I was looking at the possibility to
>>>>> send command to self thus causing creation of a marshaled copy of a
>>>>> Callable, pretty much like any other remote node receives it. However, I
>>>>> was unable to invoke a command on self. Any way to do this? How about
>>>>> invoking marshaling locally to create a Callable copy without going
>>>>> through rpc invocation layer? I though about skipping master/self node
>>>>> altogether but that's an easy way out;-)
>>>>>
>>>>> Got suggestions, ideas?
>>>> Hi Vladimir,
>>>> I've no clue about invoking a marshalled command on self, but could
>>>> you not just make sure the local callable is being processed *after*
>>>> you have created a serialized copy (sent it to others) ?
>>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Galder Zamarreno
In reply to this post by Vladimir Blagojevic

On Apr 12, 2011, at 7:45 PM, Vladimir Blagojevic wrote:

> Yeah, it would be too strict to require Callable to be Cloneable so I
> would opt out to serialization/deserialization as a clone method.
>
> Galder, do out have a recommended mechanism for this in your marshalers?

Not really. I think you should marshall or clone it before executing operations against it.

For marshalling data, you just have to see what is done in the lazy deserialization code, just get hold of the VersionAwareMarshaller and marshall it.

>
> Thanks Mircea
>
> On 11-04-12 12:59 PM, Mircea Markus wrote:
>> can't you run the Callable locally on a clone of it? If Callable is Clonable then clone it in an orthodox manner, otherwise clone it by serializing/deseralizing it.
>>
>> On 11 Apr 2011, at 19:58, Vladimir Blagojevic wrote:
>>> Sanne,
>>>
>>> I thought about that as well but I do not see a way to ensure that all
>>> other commands have been sent to other nodes other than waiting for
>>> corresponding futures to return from get and then invoke locally in a
>>> serial fashion.
>>>
>>> I do not see a hook anywhere that will tell me when a command has been
>>> dropped on the wire!
>>>
>>> Vladimir
>>>
>>>
>>> On 11-04-11 2:21 PM, Sanne Grinovero wrote:
>>>> 2011/4/11 Vladimir Blagojevic<[hidden email]>:
>>>>> Hi,
>>>>>
>>>>> I discovered a problem with distributed framework in cases where
>>>>> Callables submitted for distributed execution contain mutable instance
>>>>> fields. Just before Callable is dispersed across cluster it gets invoked
>>>>> locally where instance fields of a submitted Callable can be possibly
>>>>> mutated; Callable is in turn sent to remote nodes with mutated values
>>>>> instead of "original" field values as submitted by user. For example,
>>>>> consider pi approximation example from wiki [1]. If CircleTest Callable
>>>>> declared insideCircleCount as an instance field instead of local field
>>>>> of call method we would ultimately get a wrong final result.
>>>>>
>>>>> Therefore we need to ensure that each Infinispan node gets an unmodified
>>>>> instance of Callable for execution. I was looking at the possibility to
>>>>> send command to self thus causing creation of a marshaled copy of a
>>>>> Callable, pretty much like any other remote node receives it. However, I
>>>>> was unable to invoke a command on self. Any way to do this? How about
>>>>> invoking marshaling locally to create a Callable copy without going
>>>>> through rpc invocation layer? I though about skipping master/self node
>>>>> altogether but that's an easy way out;-)
>>>>>
>>>>> Got suggestions, ideas?
>>>> Hi Vladimir,
>>>> I've no clue about invoking a marshalled command on self, but could
>>>> you not just make sure the local callable is being processed *after*
>>>> you have created a serialized copy (sent it to others) ?
>>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Vladimir Blagojevic
On 11-04-15 4:34 AM, Galder Zamarreño wrote:
> On Apr 12, 2011, at 7:45 PM, Vladimir Blagojevic wrote:
>
>> Yeah, it would be too strict to require Callable to be Cloneable so I
>> would opt out to serialization/deserialization as a clone method.
>>
>> Galder, do out have a recommended mechanism for this in your marshalers?
> Not really. I think you should marshall or clone it before executing operations against it.
>
> For marshalling data, you just have to see what is done in the lazy deserialization code, just get hold of the VersionAwareMarshaller and marshall it.
How do I get a hold of it? I found Marshaller registered in
GlobalComponentRegistry but I could not find API to access it. I used a
regular serialization for cloning and it works ok but I'd definitely
prefer to use Marshaller.


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Galder Zamarreno

On Apr 15, 2011, at 3:23 PM, Vladimir Blagojevic wrote:

> On 11-04-15 4:34 AM, Galder Zamarreño wrote:
>> On Apr 12, 2011, at 7:45 PM, Vladimir Blagojevic wrote:
>>
>>> Yeah, it would be too strict to require Callable to be Cloneable so I
>>> would opt out to serialization/deserialization as a clone method.
>>>
>>> Galder, do out have a recommended mechanism for this in your marshalers?
>> Not really. I think you should marshall or clone it before executing operations against it.
>>
>> For marshalling data, you just have to see what is done in the lazy deserialization code, just get hold of the VersionAwareMarshaller and marshall it.
> How do I get a hold of it? I found Marshaller registered in
> GlobalComponentRegistry but I could not find API to access it. I used a
> regular serialization for cloning and it works ok but I'd definitely
> prefer to use Marshaller.

Seems like you figured out eventually :) -  globalRegistry.getComponent(StreamingMarshaller.class)

>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Manik Surtani
I still think Mircea's suggestion is very valid.  Try cloning first.  If cloning is not supported then serialize/deserialize.  And in any case you only need 1 copy of the callable to send across to all other nodes, and the original copy for local execution, right?


On 18 Apr 2011, at 11:24, Galder Zamarreño wrote:

>
> On Apr 15, 2011, at 3:23 PM, Vladimir Blagojevic wrote:
>
>> On 11-04-15 4:34 AM, Galder Zamarreño wrote:
>>> On Apr 12, 2011, at 7:45 PM, Vladimir Blagojevic wrote:
>>>
>>>> Yeah, it would be too strict to require Callable to be Cloneable so I
>>>> would opt out to serialization/deserialization as a clone method.
>>>>
>>>> Galder, do out have a recommended mechanism for this in your marshalers?
>>> Not really. I think you should marshall or clone it before executing operations against it.
>>>
>>> For marshalling data, you just have to see what is done in the lazy deserialization code, just get hold of the VersionAwareMarshaller and marshall it.
>> How do I get a hold of it? I found Marshaller registered in
>> GlobalComponentRegistry but I could not find API to access it. I used a
>> regular serialization for cloning and it works ok but I'd definitely
>> prefer to use Marshaller.
>
> Seems like you figured out eventually :) -  globalRegistry.getComponent(StreamingMarshaller.class)
>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Vladimir Blagojevic
On 11-04-18 7:58 AM, Manik Surtani wrote:
> I still think Mircea's suggestion is very valid.  Try cloning first.  If cloning is not supported then serialize/deserialize.  And in any case you only need 1 copy of the callable to send across to all other nodes, and the original copy for local execution, right?
>
Even if a type implements Cloneable we can not cast it to Cloneable and
invoke clone as Cloneable is just a marker interface. One can invoke
clone only on types known to implement clone method (ArrayList, Set etc
etc). Yes, if type implements Cloneable we could reasonably assume that
it implemented clone method but still - we have to resort to reflection
to invoke clone method but that route is even worse IMHO. I'd avoid this
altogether.

It is just one copy for local execution. Time it takes to do this is
several magnitudes smaller than the task of migrating Callables around
the cluster executing them and collecting results from all those non
local Callables.

Vladimir
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] distributed execution - invoking commands on self

Manik Surtani
Fair enough.  Keep it simple.

On 18 Apr 2011, at 14:12, Vladimir Blagojevic wrote:

> On 11-04-18 7:58 AM, Manik Surtani wrote:
>> I still think Mircea's suggestion is very valid.  Try cloning first.  If cloning is not supported then serialize/deserialize.  And in any case you only need 1 copy of the callable to send across to all other nodes, and the original copy for local execution, right?
>>
> Even if a type implements Cloneable we can not cast it to Cloneable and invoke clone as Cloneable is just a marker interface. One can invoke clone only on types known to implement clone method (ArrayList, Set etc etc). Yes, if type implements Cloneable we could reasonably assume that it implemented clone method but still - we have to resort to reflection to invoke clone method but that route is even worse IMHO. I'd avoid this altogether.
>
> It is just one copy for local execution. Time it takes to do this is several magnitudes smaller than the task of migrating Callables around the cluster executing them and collecting results from all those non local Callables.
>
> Vladimir

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev