[infinispan-dev] JAR distribution once the grid has been deployed

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] JAR distribution once the grid has been deployed

Emmanuel Bernard
Hi all,

I know this has been discussed in the past (by Tristan I think), but I don’t know how concrete the plans have come since then.

One major issue with all the distributed execution code interfaces we have is that it requires to have in the classpath of each node both the implementation of these interfaces and the class files corresponding to the key and value being processed. My understanding is that this is true of the distexec, Map / Reduce and (clustered) listener.

Evangelos from the LEADS project sort of worked around this problem by creating specialized versions of his distexec that loads the necessary JARs from the grid itself (in a set of keys) and creates a classloader that references these JARs. In a sequence, it conceptually looks like that:

- have the generic classloader distexec version in the each of grid nodes classpath at start time
- when a new remote execution is required, load each necessary JAR in a specific key in a specific cache
- the generic distexec basically receives the necessary keys, load each jar and create a classloader out of them
- the generic distexec load and launch the specific code that needs to be executed (based on the fqcn of the code to execute) from the created classloader

There are a few problems with that including:
- it requires a lot of manual work from the user
- big JARs make the key / value per JAR logic explode a bit. The algorithms LEADS use have 300 MB sized JARs
- god know what security leak this can lead to

So I wondered if we have a better alternative and plans and if there was a wiki page discussing the needs and potential approaches.
As an intermediary step we could make this approach a tutorial or side classes that people can borrow from for each of the use cases.

Emmanuel
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JAR distribution once the grid has been deployed

Vladimir Blagojevic
Emmanuel,

I don't think we have any plans in place. I agree with you - we should
at least provide hooks for these classloaders and possibly implement a
certain simple approach as a proof-of-concept/tutorial for others to
hook in their own mechanism of class loading. We can reference
Evangelos' approach as one example of how this could be done.

Vladimir



On 2014-10-16, 5:23 AM, Emmanuel Bernard wrote:

> Hi all,
>
> I know this has been discussed in the past (by Tristan I think), but I don’t know how concrete the plans have come since then.
>
> One major issue with all the distributed execution code interfaces we have is that it requires to have in the classpath of each node both the implementation of these interfaces and the class files corresponding to the key and value being processed. My understanding is that this is true of the distexec, Map / Reduce and (clustered) listener.
>
> Evangelos from the LEADS project sort of worked around this problem by creating specialized versions of his distexec that loads the necessary JARs from the grid itself (in a set of keys) and creates a classloader that references these JARs. In a sequence, it conceptually looks like that:
>
> - have the generic classloader distexec version in the each of grid nodes classpath at start time
> - when a new remote execution is required, load each necessary JAR in a specific key in a specific cache
> - the generic distexec basically receives the necessary keys, load each jar and create a classloader out of them
> - the generic distexec load and launch the specific code that needs to be executed (based on the fqcn of the code to execute) from the created classloader
>
> There are a few problems with that including:
> - it requires a lot of manual work from the user
> - big JARs make the key / value per JAR logic explode a bit. The algorithms LEADS use have 300 MB sized JARs
> - god know what security leak this can lead to
>
> So I wondered if we have a better alternative and plans and if there was a wiki page discussing the needs and potential approaches.
> As an intermediary step we could make this approach a tutorial or side classes that people can borrow from for each of the use cases.
>
> Emmanuel
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] JAR distribution once the grid has been deployed

Tristan Tarrant-2
In reply to this post by Emmanuel Bernard
Hi Emmanuel,

the plan is to leverage both the domain management and deployment
features that are part of server. Galder has already introduced
deployment of custom filters/converters and this code can be extended to
support all of the other extension points we support (distexec,
mapreduce, entities, custom cache loaders/stores, etc).
The other alternative I am going to explore is server-side scripting
based on JSR-223 which follows the general philosophy of Infinispan
Server to be language and protocol independent.
I have some initial POC code for this @ [1].

Tristan

[1]
https://github.com/tristantarrant/infinispan/commit/3e5ec12a071ff489447a611c9da0657d9641d306

On 16/10/14 11:23, Emmanuel Bernard wrote:

> Hi all,
>
> I know this has been discussed in the past (by Tristan I think), but I don’t know how concrete the plans have come since then.
>
> One major issue with all the distributed execution code interfaces we have is that it requires to have in the classpath of each node both the implementation of these interfaces and the class files corresponding to the key and value being processed. My understanding is that this is true of the distexec, Map / Reduce and (clustered) listener.
>
> Evangelos from the LEADS project sort of worked around this problem by creating specialized versions of his distexec that loads the necessary JARs from the grid itself (in a set of keys) and creates a classloader that references these JARs. In a sequence, it conceptually looks like that:
>
> - have the generic classloader distexec version in the each of grid nodes classpath at start time
> - when a new remote execution is required, load each necessary JAR in a specific key in a specific cache
> - the generic distexec basically receives the necessary keys, load each jar and create a classloader out of them
> - the generic distexec load and launch the specific code that needs to be executed (based on the fqcn of the code to execute) from the created classloader
>
> There are a few problems with that including:
> - it requires a lot of manual work from the user
> - big JARs make the key / value per JAR logic explode a bit. The algorithms LEADS use have 300 MB sized JARs
> - god know what security leak this can lead to
>
> So I wondered if we have a better alternative and plans and if there was a wiki page discussing the needs and potential approaches.
> As an intermediary step we could make this approach a tutorial or side classes that people can borrow from for each of the use cases.
>
> Emmanuel
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev