[infinispan-dev] transaction/query/mapreduce over hotrod

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] transaction/query/mapreduce over hotrod

Mircea Markus
Hi,

As there is a high community demand for having these operations in place, and most of these are targeted for post 5.1 releases, I thought about a workaround for having this functionality in place. 
I hijacked Hotrod's put operation and added a custom interceptor, so that if a certain object is being "put" into remote cache, the server side interceptor jumps in and runs transactions.
This doesn't look too bad for the user, e.g. for supporting transactions:

RemoteCache rc = getRemoteCache();//from somewhere...

//this is what we'll use for running remote transactions over hotrod
BatchEnabledRemoteCache berc = new BatchEnabledRemoteCache(rc);

berc.startBatch(); //everything from here to endBatch call is a single transaction
berc.put("k", "v1");
berc.put("k2", "v2");
berc.put("k3", "v3");
berc.endBatch(true); // all or nothing!


Of course this won't work with other clients than the java client, but I think most of our users are using that one ATM. 
Currently there's only support for transactions but this approach (and the code) can be easily extended to mapreduce and querying. 
I added s short description on how this can be used [1], also the source code is available here[2].

What do you think about it? Is it worth suggesting to the users this approach(and possibly the code as well)?

Cheers,
Mircea


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] transaction/query/mapreduce over hotrod

Sanne Grinovero-3

Hi,
in the case of Lucene Queries my main concern is about how to a) properly serialize the query b) how to stream the resusts back, possibly having some pagination / flow control.
I didn't look into hotrod sources, but I was not assuming the hard part would to make it possible to send some new commands?

So if that's all what is needed for transactions to work properly, can't you just add the commands before 5.1?

Cheers,
Sanne

On 15 Jul 2011 12:53, "Mircea Markus" <[hidden email]> wrote:
> Hi,
>
> As there is a high community demand for having these operations in place, and most of these are targeted for post 5.1 releases, I thought about a workaround for having this functionality in place.
> I hijacked Hotrod's put operation and added a custom interceptor, so that if a certain object is being "put" into remote cache, the server side interceptor jumps in and runs transactions.
> This doesn't look too bad for the user, e.g. for supporting transactions:
>
> RemoteCache rc = getRemoteCache();//from somewhere...
>
> //this is what we'll use for running remote transactions over hotrod
> BatchEnabledRemoteCache berc = new BatchEnabledRemoteCache(rc);
>
> berc.startBatch(); //everything from here to endBatch call is a single transaction
> berc.put("k", "v1");
> berc.put("k2", "v2");
> berc.put("k3", "v3");
> berc.endBatch(true); // all or nothing!
>
>
> Of course this won't work with other clients than the java client, but I think most of our users are using that one ATM.
> Currently there's only support for transactions but this approach (and the code) can be easily extended to mapreduce and querying.
> I added s short description on how this can be used [1], also the source code is available here[2].
>
> What do you think about it? Is it worth suggesting to the users this approach(and possibly the code as well)?
>
> Cheers,
> Mircea
>
> [1] https://github.com/mmarkus/ops_over_hotrod/wiki/Usage-guide
> [2]https://github.com/mmarkus/ops_over_hotrod

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] transaction/query/mapreduce over hotrod

Navin Surtani

> Hi,
> in the case of Lucene Queries my main concern is about how to a)
> properly serialize the query b) how to stream the resusts back,
> possibly having some pagination / flow control.
> I didn't look into hotrod sources, but I was not assuming the hard
> part would to make it possible to send some new commands?
>
Would there be any effect in terms of not allowing usage of the lazy
loader? In other words, all of the query results get sent over the
protocol as opposed to as and when the user would require them?

Just a thought...


> So if that's all what is needed for transactions to work properly,
> can't you just add the commands before 5.1?
>
> Cheers,
> Sanne
>
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] transaction/query/mapreduce over hotrod

Mircea Markus
In reply to this post by Sanne Grinovero-3

On 15 Jul 2011, at 13:10, Sanne Grinovero wrote:

Hi,
in the case of Lucene Queries my main concern is about how to a) properly serialize the query b) how to stream the resusts back, possibly having some pagination / flow control.

If you want we can have a look together to see wether this can work for queries.

I didn't look into hotrod sources, but I was not assuming the hard part would to make it possible to send some new commands So if that's all what is needed for transactions to work properly, can't you just add the commands before 5.1?

This approach has some limitations: only works with the java client, also doesn't support eager locking transactions, nor recovery/distributed tx. 
Even so, I think it covers most of the transactional scenarios I've seen in the community. 


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] transaction/query/mapreduce over hotrod

Manik Surtani
In reply to this post by Mircea Markus
I think this is fine as a wiki page, and let users extend/hack in this functionality for now, but IMO we need to properly engineer distributed XA over Hot Rod (with a tx broker, etc. as we designed some months back).

It's good that we have detailed this prototype, but I would say put it up on the wiki (not official docs) and point people to it each time they ask.  Maybe someone will implement it and contribute it back.  Otherwise, I would say wait for 5.2 for a full distributed XA impl.  :)

On 15 Jul 2011, at 12:53, Mircea Markus wrote:

Hi,

As there is a high community demand for having these operations in place, and most of these are targeted for post 5.1 releases, I thought about a workaround for having this functionality in place. 
I hijacked Hotrod's put operation and added a custom interceptor, so that if a certain object is being "put" into remote cache, the server side interceptor jumps in and runs transactions.
This doesn't look too bad for the user, e.g. for supporting transactions:

RemoteCache rc = getRemoteCache();//from somewhere...

//this is what we'll use for running remote transactions over hotrod
BatchEnabledRemoteCache berc = new BatchEnabledRemoteCache(rc);

berc.startBatch(); //everything from here to endBatch call is a single transaction
berc.put("k", "v1");
berc.put("k2", "v2");
berc.put("k3", "v3");
berc.endBatch(true); // all or nothing!


Of course this won't work with other clients than the java client, but I think most of our users are using that one ATM. 
Currently there's only support for transactions but this approach (and the code) can be easily extended to mapreduce and querying. 
I added s short description on how this can be used [1], also the source code is available here[2].

What do you think about it? Is it worth suggesting to the users this approach(and possibly the code as well)?

Cheers,
Mircea

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] transaction/query/mapreduce over hotrod

Vladimir Blagojevic
In reply to this post by Mircea Markus
Mircea,

I applaud the effort and your experiment! Having said that I think we have to engineer a much more plumbing for more complicated cases, e.g map/reduce (marshalling results, classloading etc etc). I am more and more convinced nowadays that all of this plumbing is better provided by AS itself rather than us reinventing the wheel here...

WDYT?

Vladimir
On 11-07-15 7:53 AM, Mircea Markus wrote:
Hi,

As there is a high community demand for having these operations in place, and most of these are targeted for post 5.1 releases, I thought about a workaround for having this functionality in place. 
I hijacked Hotrod's put operation and added a custom interceptor, so that if a certain object is being "put" into remote cache, the server side interceptor jumps in and runs transactions.
This doesn't look too bad for the user, e.g. for supporting transactions:

RemoteCache rc = getRemoteCache();//from somewhere...

//this is what we'll use for running remote transactions over hotrod
BatchEnabledRemoteCache berc = new BatchEnabledRemoteCache(rc);

berc.startBatch(); //everything from here to endBatch call is a single transaction
berc.put("k", "v1");
berc.put("k2", "v2");
berc.put("k3", "v3");
berc.endBatch(true); // all or nothing!


Of course this won't work with other clients than the java client, but I think most of our users are using that one ATM. 
Currently there's only support for transactions but this approach (and the code) can be easily extended to mapreduce and querying. 
I added s short description on how this can be used [1], also the source code is available here[2].

What do you think about it? Is it worth suggesting to the users this approach(and possibly the code as well)?

Cheers,
Mircea



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] transaction/query/mapreduce over hotrod

Mircea Markus
In reply to this post by Manik Surtani

On 15 Jul 2011, at 18:06, Manik Surtani wrote:

I think this is fine as a wiki page, and let users extend/hack in this functionality for now, but IMO we need to properly engineer distributed XA over Hot Rod (with a tx broker, etc. as we designed some months back).
+1. I've added an wiki[1] and started by making it clear to the users that this is a *workaround* and proper functionality should be used when available.

It's good that we have detailed this prototype, but I would say put it up on the wiki (not official docs) and point people to it each time they ask.  Maybe someone will implement it and contribute it back.  Otherwise, I would say wait for 5.2 for a full distributed XA impl.  :)
my thought exactly :-)

On 15 Jul 2011, at 12:53, Mircea Markus wrote:

Hi,

As there is a high community demand for having these operations in place, and most of these are targeted for post 5.1 releases, I thought about a workaround for having this functionality in place. 
I hijacked Hotrod's put operation and added a custom interceptor, so that if a certain object is being "put" into remote cache, the server side interceptor jumps in and runs transactions.
This doesn't look too bad for the user, e.g. for supporting transactions:

RemoteCache rc = getRemoteCache();//from somewhere...

//this is what we'll use for running remote transactions over hotrod
BatchEnabledRemoteCache berc = new BatchEnabledRemoteCache(rc);

berc.startBatch(); //everything from here to endBatch call is a single transaction
berc.put("k", "v1");
berc.put("k2", "v2");
berc.put("k3", "v3");
berc.endBatch(true); // all or nothing!


Of course this won't work with other clients than the java client, but I think most of our users are using that one ATM. 
Currently there's only support for transactions but this approach (and the code) can be easily extended to mapreduce and querying. 
I added s short description on how this can be used [1], also the source code is available here[2].

What do you think about it? Is it worth suggesting to the users this approach(and possibly the code as well)?

Cheers,
Mircea

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] transaction/query/mapreduce over hotrod

Mircea Markus
In reply to this post by Vladimir Blagojevic

On 15 Jul 2011, at 18:17, Vladimir Blagojevic wrote:

Mircea,

I applaud the effort and your experiment! Having said that I think we have to engineer a much more plumbing for more complicated cases, e.g map/reduce (marshalling results, classloading etc etc). I am more and more convinced nowadays that all of this plumbing is better provided by AS itself rather than us reinventing the wheel here...

WDYT?
+1. This is workaround only works with java clients, which way too restrictive for Hotrod, so we cannot consider this as a solution even if we'd want to :-)  

Vladimir
On 11-07-15 7:53 AM, Mircea Markus wrote:
Hi,

As there is a high community demand for having these operations in place, and most of these are targeted for post 5.1 releases, I thought about a workaround for having this functionality in place. 
I hijacked Hotrod's put operation and added a custom interceptor, so that if a certain object is being "put" into remote cache, the server side interceptor jumps in and runs transactions.
This doesn't look too bad for the user, e.g. for supporting transactions:

RemoteCache rc = getRemoteCache();//from somewhere...

//this is what we'll use for running remote transactions over hotrod
BatchEnabledRemoteCache berc = new BatchEnabledRemoteCache(rc);

berc.startBatch(); //everything from here to endBatch call is a single transaction
berc.put("k", "v1");
berc.put("k2", "v2");
berc.put("k3", "v3");
berc.endBatch(true); // all or nothing!


Of course this won't work with other clients than the java client, but I think most of our users are using that one ATM. 
Currently there's only support for transactions but this approach (and the code) can be easily extended to mapreduce and querying. 
I added s short description on how this can be used [1], also the source code is available here[2].

What do you think about it? Is it worth suggesting to the users this approach(and possibly the code as well)?

Cheers,
Mircea



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev