[infinispan-dev] Invoking distributed exec and mapreduce over hotrod

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Vladimir Blagojevic
Galder,

I believe the ability to invoke distributed executors and mapreduce over
hotrod would be very interesting. However, I quickly realized that
internals of both DistributedExecutorService and MapReduceTask rely
heavily on some Cache internals (RpcManager, CommandsFactory,
InterceptoChain) that are only available in non-remote caches. There is
no way to fake this by simply passing RemoteCache instead of Cache.
Either we rethink the internals of DistributedExecutorService and
MapReduceTask or we somehow bridge to these abstractions from a thin
client.

Any thoughts how we could potentially achieve hotrodization of dist. exec?

Regards,
Vladimir



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Manik Surtani
Well, I feel exposing M/R over Hot Rod - from a protocol standpoint - would require a platform-independent mechanism of defining a closure (keeping in mind we need to allow this from non-Java clients too).

So I reckon Javascript is the way to go, at least from a protocol standpoint.  Now how we expose this in remote client APIs (Java, Python, etc) needs some thought, but at first glance it would seem as though we won't have a direct mapping to what we do on the embedded side of things.  E.g., http://www.mongodb.org/display/DOCS/MapReduce

- Manik

On 4 May 2011, at 04:50, Vladimir Blagojevic wrote:

Galder,

I believe the ability to invoke distributed executors and mapreduce over
hotrod would be very interesting. However, I quickly realized that
internals of both DistributedExecutorService and MapReduceTask rely
heavily on some Cache internals (RpcManager, CommandsFactory,
InterceptoChain) that are only available in non-remote caches. There is
no way to fake this by simply passing RemoteCache instead of Cache.
Either we rethink the internals of DistributedExecutorService and
MapReduceTask or we somehow bridge to these abstractions from a thin
client.

Any thoughts how we could potentially achieve hotrodization of dist. exec?

Regards,
Vladimir



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Tristan Tarrant
On Wed, May 4, 2011 at 23:51, Manik Surtani <[hidden email]> wrote:

So I reckon Javascript is the way to go, at least from a protocol standpoint.  Now how we expose this in remote client APIs (Java, Python, etc) needs some thought, but at first glance it would seem as though we won't have a direct mapping to what we do on the embedded side of things.  E.g., http://www.mongodb.org/display/DOCS/MapReduce

Actually, using javax.script, we could default to Javascript (which is included in every Java6 implementation out there) but also be able to specify alternatives via a mimetype. In this way one could use any of the JSR 223 languages (Jython, JRuby, Groovy, etc).

Tristan

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Galder Zamarreno
In reply to this post by Manik Surtani
Vladimir, from a HotRod protocol perspective we'd need a new operation but after that the body, as Manik suggested, you could have javascript which the HotRod server ignores and passes it to the distexec code which deals with it accordingly.

I was wondering too whether we'd need streaming here or not but at first glance it does not appear to be as important as in remote querying.

On May 4, 2011, at 11:51 PM, Manik Surtani wrote:

> Well, I feel exposing M/R over Hot Rod - from a protocol standpoint - would require a platform-independent mechanism of defining a closure (keeping in mind we need to allow this from non-Java clients too).
>
> So I reckon Javascript is the way to go, at least from a protocol standpoint.  Now how we expose this in remote client APIs (Java, Python, etc) needs some thought, but at first glance it would seem as though we won't have a direct mapping to what we do on the embedded side of things.  E.g., http://www.mongodb.org/display/DOCS/MapReduce
>
> - Manik
>
> On 4 May 2011, at 04:50, Vladimir Blagojevic wrote:
>
>> Galder,
>>
>> I believe the ability to invoke distributed executors and mapreduce over
>> hotrod would be very interesting. However, I quickly realized that
>> internals of both DistributedExecutorService and MapReduceTask rely
>> heavily on some Cache internals (RpcManager, CommandsFactory,
>> InterceptoChain) that are only available in non-remote caches. There is
>> no way to fake this by simply passing RemoteCache instead of Cache.
>> Either we rethink the internals of DistributedExecutorService and
>> MapReduceTask or we somehow bridge to these abstractions from a thin
>> client.
>>
>> Any thoughts how we could potentially achieve hotrodization of dist. exec?
>>
>> Regards,
>> Vladimir
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Manik Surtani
> [hidden email]
> twitter.com/maniksurtani
>
> Lead, Infinispan
> http://www.infinispan.org
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Mircea Markus

On 5 May 2011, at 03:18, Galder Zamarreño wrote:

> Vladimir, from a HotRod protocol perspective we'd need a new operation but after that the body, as Manik suggested, you could have javascript which the HotRod server ignores and passes it to the distexec code which deals with it accordingly.
Can't we just handle this through a interceptor rather than adding a operation?

>
> I was wondering too whether we'd need streaming here or not but at first glance it does not appear to be as important as in remote querying.
>
> On May 4, 2011, at 11:51 PM, Manik Surtani wrote:
>
>> Well, I feel exposing M/R over Hot Rod - from a protocol standpoint - would require a platform-independent mechanism of defining a closure (keeping in mind we need to allow this from non-Java clients too).
>>
>> So I reckon Javascript is the way to go, at least from a protocol standpoint.  Now how we expose this in remote client APIs (Java, Python, etc) needs some thought, but at first glance it would seem as though we won't have a direct mapping to what we do on the embedded side of things.  E.g., http://www.mongodb.org/display/DOCS/MapReduce
>>
>> - Manik
>>
>> On 4 May 2011, at 04:50, Vladimir Blagojevic wrote:
>>
>>> Galder,
>>>
>>> I believe the ability to invoke distributed executors and mapreduce over
>>> hotrod would be very interesting. However, I quickly realized that
>>> internals of both DistributedExecutorService and MapReduceTask rely
>>> heavily on some Cache internals (RpcManager, CommandsFactory,
>>> InterceptoChain) that are only available in non-remote caches. There is
>>> no way to fake this by simply passing RemoteCache instead of Cache.
>>> Either we rethink the internals of DistributedExecutorService and
>>> MapReduceTask or we somehow bridge to these abstractions from a thin
>>> client.
>>>
>>> Any thoughts how we could potentially achieve hotrodization of dist. exec?
>>>
>>> Regards,
>>> Vladimir
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> --
>> Manik Surtani
>> [hidden email]
>> twitter.com/maniksurtani
>>
>> Lead, Infinispan
>> http://www.infinispan.org
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Galder Zamarreno

On May 6, 2011, at 12:14 AM, Mircea Markus wrote:

>
> On 5 May 2011, at 03:18, Galder Zamarreño wrote:
>
>> Vladimir, from a HotRod protocol perspective we'd need a new operation but after that the body, as Manik suggested, you could have javascript which the HotRod server ignores and passes it to the distexec code which deals with it accordingly.
> Can't we just handle this through a interceptor rather than adding a operation?

Hmmm, that doesn't sound very intuitive from a user perspective in the sense of hijacking one of the existing operations to send a dist exec or m/r call


>>
>> I was wondering too whether we'd need streaming here or not but at first glance it does not appear to be as important as in remote querying.
>>
>> On May 4, 2011, at 11:51 PM, Manik Surtani wrote:
>>
>>> Well, I feel exposing M/R over Hot Rod - from a protocol standpoint - would require a platform-independent mechanism of defining a closure (keeping in mind we need to allow this from non-Java clients too).
>>>
>>> So I reckon Javascript is the way to go, at least from a protocol standpoint.  Now how we expose this in remote client APIs (Java, Python, etc) needs some thought, but at first glance it would seem as though we won't have a direct mapping to what we do on the embedded side of things.  E.g., http://www.mongodb.org/display/DOCS/MapReduce
>>>
>>> - Manik
>>>
>>> On 4 May 2011, at 04:50, Vladimir Blagojevic wrote:
>>>
>>>> Galder,
>>>>
>>>> I believe the ability to invoke distributed executors and mapreduce over
>>>> hotrod would be very interesting. However, I quickly realized that
>>>> internals of both DistributedExecutorService and MapReduceTask rely
>>>> heavily on some Cache internals (RpcManager, CommandsFactory,
>>>> InterceptoChain) that are only available in non-remote caches. There is
>>>> no way to fake this by simply passing RemoteCache instead of Cache.
>>>> Either we rethink the internals of DistributedExecutorService and
>>>> MapReduceTask or we somehow bridge to these abstractions from a thin
>>>> client.
>>>>
>>>> Any thoughts how we could potentially achieve hotrodization of dist. exec?
>>>>
>>>> Regards,
>>>> Vladimir
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> --
>>> Manik Surtani
>>> [hidden email]
>>> twitter.com/maniksurtani
>>>
>>> Lead, Infinispan
>>> http://www.infinispan.org
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> --
>> Galder Zamarreño
>> Sr. Software Engineer
>> Infinispan, JBoss Cache
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Pete Muir
In reply to this post by Tristan Tarrant

On 4 May 2011, at 23:47, Tristan Tarrant wrote:

> On Wed, May 4, 2011 at 23:51, Manik Surtani <[hidden email]> wrote:
>
> So I reckon Javascript is the way to go, at least from a protocol standpoint.  Now how we expose this in remote client APIs (Java, Python, etc) needs some thought, but at first glance it would seem as though we won't have a direct mapping to what we do on the embedded side of things.  E.g., http://www.mongodb.org/display/DOCS/MapReduce
>
> Actually, using javax.script, we could default to Javascript (which is included in every Java6 implementation out there) but also be able to specify alternatives via a mimetype. In this way one could use any of the JSR 223 languages (Jython, JRuby, Groovy, etc).

This is a pretty good idea I think.
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Galder Zamarreno
I've created https://issues.jboss.org/browse/ISPN-1094

On May 6, 2011, at 4:30 PM, Pete Muir wrote:

>
> On 4 May 2011, at 23:47, Tristan Tarrant wrote:
>
>> On Wed, May 4, 2011 at 23:51, Manik Surtani <[hidden email]> wrote:
>>
>> So I reckon Javascript is the way to go, at least from a protocol standpoint.  Now how we expose this in remote client APIs (Java, Python, etc) needs some thought, but at first glance it would seem as though we won't have a direct mapping to what we do on the embedded side of things.  E.g., http://www.mongodb.org/display/DOCS/MapReduce
>>
>> Actually, using javax.script, we could default to Javascript (which is included in every Java6 implementation out there) but also be able to specify alternatives via a mimetype. In this way one could use any of the JSR 223 languages (Jython, JRuby, Groovy, etc).
>
> This is a pretty good idea I think.
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Manik Surtani
In reply to this post by Tristan Tarrant

On 5 May 2011, at 04:47, Tristan Tarrant wrote:

On Wed, May 4, 2011 at 23:51, Manik Surtani <[hidden email]> wrote:

So I reckon Javascript is the way to go, at least from a protocol standpoint.  Now how we expose this in remote client APIs (Java, Python, etc) needs some thought, but at first glance it would seem as though we won't have a direct mapping to what we do on the embedded side of things.  E.g., http://www.mongodb.org/display/DOCS/MapReduce

Actually, using javax.script, we could default to Javascript (which is included in every Java6 implementation out there) but also be able to specify alternatives via a mimetype. In this way one could use any of the JSR 223 languages (Jython, JRuby, Groovy, etc).

Definitely an interesting thought.  However this would require the various script parsers being installed and available.


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Manik Surtani
In reply to this post by Galder Zamarreno

On 5 May 2011, at 08:18, Galder Zamarreño wrote:

> Vladimir, from a HotRod protocol perspective we'd need a new operation but after that the body, as Manik suggested, you could have javascript which the HotRod server ignores and passes it to the distexec code which deals with it accordingly.
>
> I was wondering too whether we'd need streaming here or not but at first glance it does not appear to be as important as in remote querying.

Well, streaming would be important for both.  Both operations (querying and M/R) have a commonality in that they both could return an arbitrarily large amount of data.  

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Manik Surtani
In reply to this post by Galder Zamarreno
On 9 May 2011, at 16:53, Galder Zamarreño wrote:

> I've created https://issues.jboss.org/browse/ISPN-1094

Great.  So I know this is a way off (maybe Infinispan 5.2 or 6?) but does someone want to "own" it for now, maintain a design wiki page to capture ideas and thoughts, link to the JIRA?  Tristan's JSR-233 suggestion could be very valid, and I spoke to Sanne about this offline at JBW as well and he had some interesting thoughts to contribute.

So - any takers?  :-)

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Mircea Markus-2
I'll take it.

On 18 May 2011, at 11:20, Manik Surtani <[hidden email]> wrote:

> On 9 May 2011, at 16:53, Galder Zamarreño wrote:
>
>> I've created https://issues.jboss.org/browse/ISPN-1094
>
> Great.  So I know this is a way off (maybe Infinispan 5.2 or 6?) but does someone want to "own" it for now, maintain a design wiki page to capture ideas and thoughts, link to the JIRA?  Tristan's JSR-233 suggestion could be very valid, and I spoke to Sanne about this offline at JBW as well and he had some interesting thoughts to contribute.
>
> So - any takers?  :-)
>
> --
> Manik Surtani
> [hidden email]
> twitter.com/maniksurtani
>
> Lead, Infinispan
> http://www.infinispan.org
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Vladimir Blagojevic
I would like to co-lead this effort with Mircea. Having two of us on it
would be perfect I believe.

I am personally very interested in this as I envision dist.exec and
map/reduce to be a service API similar to what hibernate, hornetq etc
are today. Imagine firing up Infinispan cluster with 100+ nodes on EC2
and invoking distributed execution remotely from your locally managed
JBoss AS instance/cluster. You could offload any long running
computational tasks to a remote cluster and when you are done turn off
EC2 cluster until it is needed again....

Vladimir

On 11-05-18 12:23 PM, Mircea Markus wrote:

> I'll take it.
>
> On 18 May 2011, at 11:20, Manik Surtani<[hidden email]>  wrote:
>
>> On 9 May 2011, at 16:53, Galder Zamarreño wrote:
>>
>>> I've created https://issues.jboss.org/browse/ISPN-1094
>> Great.  So I know this is a way off (maybe Infinispan 5.2 or 6?) but does someone want to "own" it for now, maintain a design wiki page to capture ideas and thoughts, link to the JIRA?  Tristan's JSR-233 suggestion could be very valid, and I spoke to Sanne about this offline at JBW as well and he had some interesting thoughts to contribute.
>>
>> So - any takers?  :-)
>>
>> --
>> Manik Surtani
>> [hidden email]
>> twitter.com/maniksurtani
>>
>> Lead, Infinispan
>> http://www.infinispan.org
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Invoking distributed exec and mapreduce over hotrod

Manik Surtani

On 18 May 2011, at 12:15, Vladimir Blagojevic wrote:

> I would like to co-lead this effort with Mircea. Having two of us on it
> would be perfect I believe.

Sure.

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev