[infinispan-dev] Executing server tasks that contain POJOs

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] Executing server tasks that contain POJOs

Galder Zamarreno
Hi all,

For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs.

To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.

We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:

1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.

2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].

3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.

Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.

Cheers,

[1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
[2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
[3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
[4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
--
Galder Zamarreño
Infinispan, Red Hat


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Executing server tasks that contain POJOs

Ramesh Reddy
Glader,

FWIW, I am rewriting Teiid translator to Infinispan, where I needed a portable marshaller that is simply based on .proto file as you describe issues with your step #2. The use of predefined custom java marshaller is not viable in my usecase, as I want to dynamically convert Relational Tables defined in the Teiid into POJO's in Infinispan, and ability to query them. I have written a such marshaller for my usecase you can see testcase at [1]. Basically I capture the metadata from .proto file and use that information in encoding/decoding the protostream, *most* of the needed code is already there in the ProtoStream libraries.

BTW, with your Task example you gave me another idea how I can even further enhance the this integration layer in terms of updating multiple POJOs in single call :)

Ramesh..

[1] https://github.com/rareddy/infinispan/blob/master/translator-infinispan-hotrod/src/test/java/org/teiid/translator/infinispan/hotrod/TestTeiidTableMarsheller.java


----- Original Message -----

> Hi all,
>
> For a demo I'm giving next week, I'd like to show how to use distributed
> streams via a remote server task. All server tasks that we have in testsuite
> rely on primitives but in my case I wanted to use POJOs.
>
> To do that, I needed to get compatibility mode working in such way that those
> POJOs could be unmarshalled for the server task. Since in another demo I'm
> showing Protostream based POJOs, I thought I'd try to use that as mechanism
> to unmarshall POJOs server side.
>
> We have a test for such scenario [1], but the reality (running on a proper
> server) is anything that simple. Here's a list of things I've found out
> while creating a WordCount example that relies on a POJO:
>
> 1. Out of the box, it's impossible to set compatibility marshaller to
> org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because
> "org.infinispan.main" classloader can't access that class. I worked around
> that by tweaking the module.xml to have an optional dependency to
> "org.infinispan.remote-query.server" module.
>
> 2. After doing that, I had to register the protofile and associated classes
> remotely in the server. Again, there's no out of the box mechanism for that,
> so I created a remote server task that would do that [3].
>
> 3. Finally, with all that in place, I was able to complete the WordCount test
> [4] with a final caveat: the return of the word count, and words protofile
> registration, tasks return objects that are not marshalled by the
> compatibility marshaller, so I had to make sure that the remote cache
> manager used for those tasks uses the default marshaller.
>
> Clearly we need to improve on this, and we have plans to address these issues
> (with new upcoming transcoding capabilities), but I thought it'd be worth
> mentioning the problems found in case anyone else encounters them before
> transcoding is in place.
>
> Cheers,
>
> [1]
> https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
> [2]
> https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
> [3]
> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> [4]
> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Executing server tasks that contain POJOs

Galder Zamarreno
Hey Ramesh,

I don't know your use case very well, so allow me to ask you some qs:

1. When does your marshaller come into play? At the compatibility layer? Or is it used as a client marshaller?

1a. If it's at the compatibility layer, why can't you use CompatibilityProtoStreamMarshaller?

Your description below makes it sound like your marshaller does similar work to CompatibilityProtoStreamMarshaller, hence the questions and see whether it could fit your use case.

Cheers,
--
Galder Zamarreño
Infinispan, Red Hat

> On 30 Mar 2017, at 16:30, Ramesh Reddy <[hidden email]> wrote:
>
> Glader,
>
> FWIW, I am rewriting Teiid translator to Infinispan, where I needed a portable marshaller that is simply based on .proto file as you describe issues with your step #2. The use of predefined custom java marshaller is not viable in my usecase, as I want to dynamically convert Relational Tables defined in the Teiid into POJO's in Infinispan, and ability to query them. I have written a such marshaller for my usecase you can see testcase at [1]. Basically I capture the metadata from .proto file and use that information in encoding/decoding the protostream, *most* of the needed code is already there in the ProtoStream libraries.
>
> BTW, with your Task example you gave me another idea how I can even further enhance the this integration layer in terms of updating multiple POJOs in single call :)
>
> Ramesh..
>
> [1] https://github.com/rareddy/infinispan/blob/master/translator-infinispan-hotrod/src/test/java/org/teiid/translator/infinispan/hotrod/TestTeiidTableMarsheller.java
>
>
> ----- Original Message -----
>> Hi all,
>>
>> For a demo I'm giving next week, I'd like to show how to use distributed
>> streams via a remote server task. All server tasks that we have in testsuite
>> rely on primitives but in my case I wanted to use POJOs.
>>
>> To do that, I needed to get compatibility mode working in such way that those
>> POJOs could be unmarshalled for the server task. Since in another demo I'm
>> showing Protostream based POJOs, I thought I'd try to use that as mechanism
>> to unmarshall POJOs server side.
>>
>> We have a test for such scenario [1], but the reality (running on a proper
>> server) is anything that simple. Here's a list of things I've found out
>> while creating a WordCount example that relies on a POJO:
>>
>> 1. Out of the box, it's impossible to set compatibility marshaller to
>> org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because
>> "org.infinispan.main" classloader can't access that class. I worked around
>> that by tweaking the module.xml to have an optional dependency to
>> "org.infinispan.remote-query.server" module.
>>
>> 2. After doing that, I had to register the protofile and associated classes
>> remotely in the server. Again, there's no out of the box mechanism for that,
>> so I created a remote server task that would do that [3].
>>
>> 3. Finally, with all that in place, I was able to complete the WordCount test
>> [4] with a final caveat: the return of the word count, and words protofile
>> registration, tasks return objects that are not marshalled by the
>> compatibility marshaller, so I had to make sure that the remote cache
>> manager used for those tasks uses the default marshaller.
>>
>> Clearly we need to improve on this, and we have plans to address these issues
>> (with new upcoming transcoding capabilities), but I thought it'd be worth
>> mentioning the problems found in case anyone else encounters them before
>> transcoding is in place.
>>
>> Cheers,
>>
>> [1]
>> https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
>> [2]
>> https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
>> [3]
>> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
>> [4]
>> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
>> --
>> Galder Zamarreño
>> Infinispan, Red Hat
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Executing server tasks that contain POJOs

Gustavo Fernandes-2
In reply to this post by Galder Zamarreno


On Thu, Mar 30, 2017 at 1:51 PM, Galder Zamarreño <[hidden email]> wrote:
Hi all,

For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs.

To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.

We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:

1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.

2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].


AFAICT, you should be able to do that doing a PUT in the Protobuf_Metadata cache, which entails having auth enabled. This cache should be REPL_SYNC, so no need to run a server task.

 

3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.

Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.

Cheers,

[1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
[2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
[3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
[4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
--
Galder Zamarreño
Infinispan, Red Hat


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Executing server tasks that contain POJOs

Ramesh Reddy
In reply to this post by Galder Zamarreno
Yes, it is at the client layer. When I write the server tasks to I will look into CompatibilityProtoStreamMarshaller.


----- Original Message -----

> Hey Ramesh,
>
> I don't know your use case very well, so allow me to ask you some qs:
>
> 1. When does your marshaller come into play? At the compatibility layer? Or
> is it used as a client marshaller?
>
> 1a. If it's at the compatibility layer, why can't you use
> CompatibilityProtoStreamMarshaller?
>
> Your description below makes it sound like your marshaller does similar work
> to CompatibilityProtoStreamMarshaller, hence the questions and see whether
> it could fit your use case.
>
> Cheers,
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
> > On 30 Mar 2017, at 16:30, Ramesh Reddy <[hidden email]> wrote:
> >
> > Glader,
> >
> > FWIW, I am rewriting Teiid translator to Infinispan, where I needed a
> > portable marshaller that is simply based on .proto file as you describe
> > issues with your step #2. The use of predefined custom java marshaller is
> > not viable in my usecase, as I want to dynamically convert Relational
> > Tables defined in the Teiid into POJO's in Infinispan, and ability to
> > query them. I have written a such marshaller for my usecase you can see
> > testcase at [1]. Basically I capture the metadata from .proto file and use
> > that information in encoding/decoding the protostream, *most* of the
> > needed code is already there in the ProtoStream libraries.
> >
> > BTW, with your Task example you gave me another idea how I can even further
> > enhance the this integration layer in terms of updating multiple POJOs in
> > single call :)
> >
> > Ramesh..
> >
> > [1]
> > https://github.com/rareddy/infinispan/blob/master/translator-infinispan-hotrod/src/test/java/org/teiid/translator/infinispan/hotrod/TestTeiidTableMarsheller.java
> >
> >
> > ----- Original Message -----
> >> Hi all,
> >>
> >> For a demo I'm giving next week, I'd like to show how to use distributed
> >> streams via a remote server task. All server tasks that we have in
> >> testsuite
> >> rely on primitives but in my case I wanted to use POJOs.
> >>
> >> To do that, I needed to get compatibility mode working in such way that
> >> those
> >> POJOs could be unmarshalled for the server task. Since in another demo I'm
> >> showing Protostream based POJOs, I thought I'd try to use that as
> >> mechanism
> >> to unmarshall POJOs server side.
> >>
> >> We have a test for such scenario [1], but the reality (running on a proper
> >> server) is anything that simple. Here's a list of things I've found out
> >> while creating a WordCount example that relies on a POJO:
> >>
> >> 1. Out of the box, it's impossible to set compatibility marshaller to
> >> org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because
> >> "org.infinispan.main" classloader can't access that class. I worked around
> >> that by tweaking the module.xml to have an optional dependency to
> >> "org.infinispan.remote-query.server" module.
> >>
> >> 2. After doing that, I had to register the protofile and associated
> >> classes
> >> remotely in the server. Again, there's no out of the box mechanism for
> >> that,
> >> so I created a remote server task that would do that [3].
> >>
> >> 3. Finally, with all that in place, I was able to complete the WordCount
> >> test
> >> [4] with a final caveat: the return of the word count, and words protofile
> >> registration, tasks return objects that are not marshalled by the
> >> compatibility marshaller, so I had to make sure that the remote cache
> >> manager used for those tasks uses the default marshaller.
> >>
> >> Clearly we need to improve on this, and we have plans to address these
> >> issues
> >> (with new upcoming transcoding capabilities), but I thought it'd be worth
> >> mentioning the problems found in case anyone else encounters them before
> >> transcoding is in place.
> >>
> >> Cheers,
> >>
> >> [1]
> >> https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
> >> [2]
> >> https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
> >> [3]
> >> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> >> [4]
> >> https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> >> --
> >> Galder Zamarreño
> >> Infinispan, Red Hat
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> [hidden email]
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > [hidden email]
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Executing server tasks that contain POJOs

Galder Zamarreno
In reply to this post by Gustavo Fernandes-2

--
Galder Zamarreño
Infinispan, Red Hat

> On 30 Mar 2017, at 17:15, Gustavo Fernandes <[hidden email]> wrote:
>
>
>
> On Thu, Mar 30, 2017 at 1:51 PM, Galder Zamarreño <[hidden email]> wrote:
> Hi all,
>
> For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs.
>
> To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.
>
> We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:
>
> 1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.
>
> 2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].
>
>
> AFAICT, you should be able to do that doing a PUT in the Protobuf_Metadata cache, which entails having auth enabled. This cache should be REPL_SYNC, so no need to run a server task.

Good point but not so sure it completely removes the need for the task. The task does two things:

1. Call ProtobufMetadataManager.registerProtofile, which as you say could be swapped with a cache.put on the metadata cache.

2. Call ProtobufMetadataManager.registerMarshaller. This goes deep into updating SerializationContextImpl, which seems independent of any replicated cache.

In fact, I had originally set up the task to execute in only in one node, but when I did that I found that marshallers were not registered in all nodes, so I had to execute the task in all nodes.

I guess the task could be just limited to only executing 2.) in all nodes (and store the protofile contents by accessing the cache remotely), but I can't see how I can avoid the task altogether right now.

>
>  
>
> 3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.
>
> Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.
>
> Cheers,
>
> [1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
> [2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
> [3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> [4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Executing server tasks that contain POJOs

Gustavo Fernandes-2
On Thu, Mar 30, 2017 at 4:49 PM, Galder Zamarreño <[hidden email]> wrote:

--
Galder Zamarreño
Infinispan, Red Hat

> On 30 Mar 2017, at 17:15, Gustavo Fernandes <[hidden email]> wrote:
>
>
>
> On Thu, Mar 30, 2017 at 1:51 PM, Galder Zamarreño <[hidden email]> wrote:
> Hi all,
>
> For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs.
>
> To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.
>
> We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:
>
> 1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.
>
> 2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].
>
>
> AFAICT, you should be able to do that doing a PUT in the Protobuf_Metadata cache, which entails having auth enabled. This cache should be REPL_SYNC, so no need to run a server task.

Good point but not so sure it completely removes the need for the task. The task does two things:

1. Call ProtobufMetadataManager.registerProtofile, which as you say could be swapped with a cache.put on the metadata cache.

2. Call ProtobufMetadataManager.registerMarshaller. This goes deep into updating SerializationContextImpl, which seems independent of any replicated cache.


AFAICT (again), there is an internal listener or interceptor that upon metadata cache change, will update the SerCtx on all nodes.

Gustavo
 

In fact, I had originally set up the task to execute in only in one node, but when I did that I found that marshallers were not registered in all nodes, so I had to execute the task in all nodes.

I guess the task could be just limited to only executing 2.) in all nodes (and store the protofile contents by accessing the cache remotely), but I can't see how I can avoid the task altogether right now.

>
>
>
> 3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.
>
> Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.
>
> Cheers,
>
> [1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
> [2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
> [3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> [4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Executing server tasks that contain POJOs

Gustavo Fernandes-2


On Thu, Mar 30, 2017 at 5:01 PM, Gustavo Fernandes <[hidden email]> wrote:
On Thu, Mar 30, 2017 at 4:49 PM, Galder Zamarreño <[hidden email]> wrote:

--
Galder Zamarreño
Infinispan, Red Hat

> On 30 Mar 2017, at 17:15, Gustavo Fernandes <[hidden email]> wrote:
>
>
>
> On Thu, Mar 30, 2017 at 1:51 PM, Galder Zamarreño <[hidden email]> wrote:
> Hi all,
>
> For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs.
>
> To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.
>
> We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:
>
> 1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.
>
> 2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].
>
>
> AFAICT, you should be able to do that doing a PUT in the Protobuf_Metadata cache, which entails having auth enabled. This cache should be REPL_SYNC, so no need to run a server task.

Good point but not so sure it completely removes the need for the task. The task does two things:

1. Call ProtobufMetadataManager.registerProtofile, which as you say could be swapped with a cache.put on the metadata cache.

2. Call ProtobufMetadataManager.registerMarshaller. This goes deep into updating SerializationContextImpl, which seems independent of any replicated cache.


AFAICT (again), there is an internal listener or interceptor that upon metadata cache change, will update the SerCtx on all nodes.

Gustavo


Ops, misread that, you are registering marshallers *in the server*, so yes, you probably need a task for that :)

 
 

In fact, I had originally set up the task to execute in only in one node, but when I did that I found that marshallers were not registered in all nodes, so I had to execute the task in all nodes.

I guess the task could be just limited to only executing 2.) in all nodes (and store the protofile contents by accessing the cache remotely), but I can't see how I can avoid the task altogether right now.

>
>
>
> 3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.
>
> Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.
>
> Cheers,
>
> [1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
> [2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
> [3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> [4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Executing server tasks that contain POJOs

Dan Berindei
In reply to this post by Galder Zamarreno
On Thu, Mar 30, 2017 at 3:51 PM, Galder Zamarreño <[hidden email]> wrote:

> Hi all,
>
> For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs.
>
> To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.
>
> We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:
>
> 1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.
>

I know Sanne also wanted to add one of the query modules as an
optional dependency to the core module for a similar reason, but it
seems really hacky.

Could the server create the GlobalConfigurationBuilder with a
classloader that has access to the query module instead?

Alternatively, I know ModularClassResolver prefixes class names with
the slot and module name, and can load a class from any module. Maybe
we could also allow a slot:module:class format everywhere the
configuration currently accepts a class name?

> 2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].
>
> 3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.
>
> Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.
>
> Cheers,
>
> [1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
> [2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
> [3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
> [4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Executing server tasks that contain POJOs

Galder Zamarreno

--
Galder Zamarreño
Infinispan, Red Hat

> On 30 Mar 2017, at 18:33, Dan Berindei <[hidden email]> wrote:
>
> On Thu, Mar 30, 2017 at 3:51 PM, Galder Zamarreño <[hidden email]> wrote:
>> Hi all,
>>
>> For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs.
>>
>> To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.
>>
>> We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:
>>
>> 1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.
>>
>
> I know Sanne also wanted to add one of the query modules as an
> optional dependency to the core module for a similar reason, but it
> seems really hacky.

Yeah, might be a bit hacky but it's just a configuration change as opposed to a code change.

> Could the server create the GlobalConfigurationBuilder with a
> classloader that has access to the query module instead?

Hmmm, not sure. I mean, in theory you could plug any marshaller, even com.acme.WhateverMarshaller, so it's not really an issue about having access to the query module, but having access a module that contains that marshaller.

> Alternatively, I know ModularClassResolver prefixes class names with
> the slot and module name, and can load a class from any module. Maybe
> we could also allow a slot:module:class format everywhere the
> configuration currently accepts a class name?

That could be handy. I'm about to create a JIRA on this, so I'll add it as an idea to it.

>
>> 2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].
>>
>> 3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.
>>
>> Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.
>>
>> Cheers,
>>
>> [1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
>> [2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
>> [3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
>> [4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
>> --
>> Galder Zamarreño
>> Infinispan, Red Hat
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Executing server tasks that contain POJOs

Galder Zamarreno
FYI, I've created these JIRAs to improve some of the issues highlighted here:
https://issues.jboss.org/browse/ISPN-7710
https://issues.jboss.org/browse/ISPN-7711

Cheers,
--
Galder Zamarreño
Infinispan, Red Hat

> On 7 Apr 2017, at 14:20, Galder Zamarreño <[hidden email]> wrote:
>
>
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>> On 30 Mar 2017, at 18:33, Dan Berindei <[hidden email]> wrote:
>>
>> On Thu, Mar 30, 2017 at 3:51 PM, Galder Zamarreño <[hidden email]> wrote:
>>> Hi all,
>>>
>>> For a demo I'm giving next week, I'd like to show how to use distributed streams via a remote server task. All server tasks that we have in testsuite rely on primitives but in my case I wanted to use POJOs.
>>>
>>> To do that, I needed to get compatibility mode working in such way that those POJOs could be unmarshalled for the server task. Since in another demo I'm showing Protostream based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server side.
>>>
>>> We have a test for such scenario [1], but the reality (running on a proper server) is anything that simple. Here's a list of things I've found out while creating a WordCount example that relies on a POJO:
>>>
>>> 1. Out of the box, it's impossible to set compatibility marshaller to org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because "org.infinispan.main" classloader can't access that class. I worked around that by tweaking the module.xml to have an optional dependency to "org.infinispan.remote-query.server" module.
>>>
>>
>> I know Sanne also wanted to add one of the query modules as an
>> optional dependency to the core module for a similar reason, but it
>> seems really hacky.
>
> Yeah, might be a bit hacky but it's just a configuration change as opposed to a code change.
>
>> Could the server create the GlobalConfigurationBuilder with a
>> classloader that has access to the query module instead?
>
> Hmmm, not sure. I mean, in theory you could plug any marshaller, even com.acme.WhateverMarshaller, so it's not really an issue about having access to the query module, but having access a module that contains that marshaller.
>
>> Alternatively, I know ModularClassResolver prefixes class names with
>> the slot and module name, and can load a class from any module. Maybe
>> we could also allow a slot:module:class format everywhere the
>> configuration currently accepts a class name?
>
> That could be handy. I'm about to create a JIRA on this, so I'll add it as an idea to it.
>
>>
>>> 2. After doing that, I had to register the protofile and associated classes remotely in the server. Again, there's no out of the box mechanism for that, so I created a remote server task that would do that [3].
>>>
>>> 3. Finally, with all that in place, I was able to complete the WordCount test [4] with a final caveat: the return of the word count, and words protofile registration, tasks return objects that are not marshalled by the compatibility marshaller, so I had to make sure that the remote cache manager used for those tasks uses the default marshaller.
>>>
>>> Clearly we need to improve on this, and we have plans to address these issues (with new upcoming transcoding capabilities), but I thought it'd be worth mentioning the problems found in case anyone else encounters them before transcoding is in place.
>>>
>>> Cheers,
>>>
>>> [1] https://github.com/galderz/datagrid-patterns/blob/master/server-config/domain/domain.xml#L139
>>> [2] https://github.com/galderz/datagrid-patterns/blob/master/server-config/org.infinispan.main_module.xml#L18
>>> [3] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-server/src/main/java/test/WordsProtoTask.java
>>> [4] https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream/tasks-client/src/test/java/test/WordCountTest.java
>>> --
>>> Galder Zamarreño
>>> Infinispan, Red Hat
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev