Re: [infinispan-dev] ISPN 200

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] ISPN 200

Manik Surtani
Hi Israel

Any updates on ISPN-200?  How are you getting on?

Cheers
Manik

On 15 Sep 2010, at 12:56, Israel Lacerra wrote:

Thanks Manik!! I just did not understand the last item..

> - on an implementation level, the GetHItsCommand (or something like that) could return with a single hit, or N hits, with a flag of whether more hits are available or not.


Israel

On Wed, Sep 15, 2010 at 7:16 AM, Manik Surtani <[hidden email]> wrote:
How about something like:

- Broadcast the query
- Every node creates the QueryHits inst, runs the query and collects results.  Starts streaming the results back immediately.
- The lazy iterator returns to the user immediately, and maintains an internal cache of results coming in from N remote QueryHits instances
- iterator.next() blocks until this cache has available entries to return.
- on an implementation level, the GetHItsCommand (or something like that) could return with a single hit, or N hits, with a flag of whether more hits are available or not.

Does that help?

Manik


On 14 Sep 2010, at 20:46, Israel Lacerra wrote:

Hi guys,

I'm still thinking about it, but I don't have a really good idea about the lazy iterator yet. The only way (that I see) I could make it more lazily is:

- Broadcast the query.
- Every node creates a QueryHits instance with the query and keep it in a simple little cache (array, hash, etc)
- A "state" of the query is created and every lazyIterator.next() must send a command to a node and get the next hit (the next key).
- After a certain time, the instances of queryHits "dies".

It seems to me that this is not too efficient. But I don't have any other ideas.

Do you have any suggestions about it?

thanks!
Israel

AnyoneOn Tue, Aug 31, 2010 at 9:14 AM, Israel Lacerra <[hidden email]> wrote:
Hi Navin,

I'm trying to do ISPN-200 (https://jira.jboss.org/browse/ISPN-200) and I don't know how to implement a good lazy iterator in a "distributed way".

(Sorry... my english is not soo good. If you don't understand again, please just ask again!  :)


Israel


On Tue, Aug 31, 2010 at 6:27 AM, Navin Surtani <[hidden email]> wrote:
Hi Israel,

Just a quick question to your issue. What do you mean by you do not have
the LazyIterator? I suppose I'm not really understanding what your issue
is. Is it just that you don't want the lazy version or you don't know
how to use it? :S


On 25/08/10 14:39, Israel Lacerra wrote:
> Ok, Manik!
>
> I've already have a code working, but not too "lazy" (and without the
> "sort" part). I get the keys on all nodes, and then I use them. So I
> have a EagerIterator, but not the LazyIterator. :/
>
> I'll think about it...
>
> thanks
>
> On Wed, Aug 25, 2010 at 10:23 AM, Manik Surtani <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>     On 24 Aug 2010, at 17:20, Israel Lacerra wrote:
>
>      > Manik,
>      >
>      > What you mean by:
>      > "   * The calling node returns a CacheQuery impl that lazily fetches
>      > and collates results from the cluster." (JIRA)
>      >
>      > Is enough if each node returns a list of keys and then, we lazily
>     get the values using the keys? Or the process has to be more lazy yet?
>
>     I think it can be "more lazy" as you said.  :)
>     --
>     Manik Surtani
>     [hidden email] <mailto:[hidden email]>
>     Lead, Infinispan
>     Lead, JBoss Cache
>     http://www.infinispan.org
>     http://www.jbosscache.org
>
>
>
>
>
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Navin Surtani
Intern Infinispan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
Lead, Infinispan
Lead, JBoss Cache





_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] ISPN 200

Israel Lacerra
Hi Manik,

I've finished the LazyIterator. Now I need to:

- Code some good tests (now I'm testing using InfinispanDemo with some changes)
- Implement all methods of iterators (the most simple are not ready yet)
- Migrate to git

I will send the code to you in the end of the next week (I think git will help with this) and probably you will show a lot of things to improve!

Cheers
Israel

On Wed, Dec 15, 2010 at 5:23 PM, Manik Surtani <[hidden email]> wrote:
Hi Israel

Any updates on ISPN-200?  How are you getting on?

Cheers
Manik

On 15 Sep 2010, at 12:56, Israel Lacerra wrote:

Thanks Manik!! I just did not understand the last item..

> - on an implementation level, the GetHItsCommand (or something like that) could return with a single hit, or N hits, with a flag of whether more hits are available or not.


Israel

On Wed, Sep 15, 2010 at 7:16 AM, Manik Surtani <[hidden email]> wrote:
How about something like:

- Broadcast the query
- Every node creates the QueryHits inst, runs the query and collects results.  Starts streaming the results back immediately.
- The lazy iterator returns to the user immediately, and maintains an internal cache of results coming in from N remote QueryHits instances
- iterator.next() blocks until this cache has available entries to return.
- on an implementation level, the GetHItsCommand (or something like that) could return with a single hit, or N hits, with a flag of whether more hits are available or not.

Does that help?

Manik


On 14 Sep 2010, at 20:46, Israel Lacerra wrote:

Hi guys,

I'm still thinking about it, but I don't have a really good idea about the lazy iterator yet. The only way (that I see) I could make it more lazily is:

- Broadcast the query.
- Every node creates a QueryHits instance with the query and keep it in a simple little cache (array, hash, etc)
- A "state" of the query is created and every lazyIterator.next() must send a command to a node and get the next hit (the next key).
- After a certain time, the instances of queryHits "dies".

It seems to me that this is not too efficient. But I don't have any other ideas.

Do you have any suggestions about it?

thanks!
Israel

AnyoneOn Tue, Aug 31, 2010 at 9:14 AM, Israel Lacerra <[hidden email]> wrote:
Hi Navin,

I'm trying to do ISPN-200 (https://jira.jboss.org/browse/ISPN-200) and I don't know how to implement a good lazy iterator in a "distributed way".

(Sorry... my english is not soo good. If you don't understand again, please just ask again!  :)


Israel


On Tue, Aug 31, 2010 at 6:27 AM, Navin Surtani <[hidden email]> wrote:
Hi Israel,

Just a quick question to your issue. What do you mean by you do not have
the LazyIterator? I suppose I'm not really understanding what your issue
is. Is it just that you don't want the lazy version or you don't know
how to use it? :S


On 25/08/10 14:39, Israel Lacerra wrote:
> Ok, Manik!
>
> I've already have a code working, but not too "lazy" (and without the
> "sort" part). I get the keys on all nodes, and then I use them. So I
> have a EagerIterator, but not the LazyIterator. :/
>
> I'll think about it...
>
> thanks
>
> On Wed, Aug 25, 2010 at 10:23 AM, Manik Surtani <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>     On 24 Aug 2010, at 17:20, Israel Lacerra wrote:
>
>      > Manik,
>      >
>      > What you mean by:
>      > "   * The calling node returns a CacheQuery impl that lazily fetches
>      > and collates results from the cluster." (JIRA)
>      >
>      > Is enough if each node returns a list of keys and then, we lazily
>     get the values using the keys? Or the process has to be more lazy yet?
>
>     I think it can be "more lazy" as you said.  :)
>     --
>     Manik Surtani
>     [hidden email] <mailto:[hidden email]>
>     Lead, Infinispan
>     Lead, JBoss Cache
>     http://www.infinispan.org
>     http://www.jbosscache.org
>
>
>
>
>
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Navin Surtani
Intern Infinispan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
Lead, Infinispan
Lead, JBoss Cache





_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] ISPN 200

Manik Surtani

On 16 Dec 2010, at 11:15, Israel Lacerra wrote:

Hi Manik,

I've finished the LazyIterator. Now I need to:

- Code some good tests (now I'm testing using InfinispanDemo with some changes)
- Implement all methods of iterators (the most simple are not ready yet)
- Migrate to git

I will send the code to you in the end of the next week (I think git will help with this) and probably you will show a lot of things to improve!

No worries.  I'll only be able to look at it in the first week of Jan, but I'm sure others on this list will be able to comment in the meanwhile.  :-)


Cheers
Israel

On Wed, Dec 15, 2010 at 5:23 PM, Manik Surtani <[hidden email]> wrote:
Hi Israel

Any updates on ISPN-200?  How are you getting on?

Cheers
Manik

On 15 Sep 2010, at 12:56, Israel Lacerra wrote:

Thanks Manik!! I just did not understand the last item..

> - on an implementation level, the GetHItsCommand (or something like that) could return with a single hit, or N hits, with a flag of whether more hits are available or not.


Israel

On Wed, Sep 15, 2010 at 7:16 AM, Manik Surtani <[hidden email]> wrote:
How about something like:

- Broadcast the query
- Every node creates the QueryHits inst, runs the query and collects results.  Starts streaming the results back immediately.
- The lazy iterator returns to the user immediately, and maintains an internal cache of results coming in from N remote QueryHits instances
- iterator.next() blocks until this cache has available entries to return.
- on an implementation level, the GetHItsCommand (or something like that) could return with a single hit, or N hits, with a flag of whether more hits are available or not.

Does that help?

Manik


On 14 Sep 2010, at 20:46, Israel Lacerra wrote:

Hi guys,

I'm still thinking about it, but I don't have a really good idea about the lazy iterator yet. The only way (that I see) I could make it more lazily is:

- Broadcast the query.
- Every node creates a QueryHits instance with the query and keep it in a simple little cache (array, hash, etc)
- A "state" of the query is created and every lazyIterator.next() must send a command to a node and get the next hit (the next key).
- After a certain time, the instances of queryHits "dies".

It seems to me that this is not too efficient. But I don't have any other ideas.

Do you have any suggestions about it?

thanks!
Israel

AnyoneOn Tue, Aug 31, 2010 at 9:14 AM, Israel Lacerra <[hidden email]> wrote:
Hi Navin,

I'm trying to do ISPN-200 (https://jira.jboss.org/browse/ISPN-200) and I don't know how to implement a good lazy iterator in a "distributed way".

(Sorry... my english is not soo good. If you don't understand again, please just ask again!  :)


Israel


On Tue, Aug 31, 2010 at 6:27 AM, Navin Surtani <[hidden email]> wrote:
Hi Israel,

Just a quick question to your issue. What do you mean by you do not have
the LazyIterator? I suppose I'm not really understanding what your issue
is. Is it just that you don't want the lazy version or you don't know
how to use it? :S


On 25/08/10 14:39, Israel Lacerra wrote:
> Ok, Manik!
>
> I've already have a code working, but not too "lazy" (and without the
> "sort" part). I get the keys on all nodes, and then I use them. So I
> have a EagerIterator, but not the LazyIterator. :/
>
> I'll think about it...
>
> thanks
>
> On Wed, Aug 25, 2010 at 10:23 AM, Manik Surtani <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>     On 24 Aug 2010, at 17:20, Israel Lacerra wrote:
>
>      > Manik,
>      >
>      > What you mean by:
>      > "   * The calling node returns a CacheQuery impl that lazily fetches
>      > and collates results from the cluster." (JIRA)
>      >
>      > Is enough if each node returns a list of keys and then, we lazily
>     get the values using the keys? Or the process has to be more lazy yet?
>
>     I think it can be "more lazy" as you said.  :)
>     --
>     Manik Surtani
>     [hidden email] <mailto:[hidden email]>
>     Lead, Infinispan
>     Lead, JBoss Cache
>     http://www.infinispan.org
>     http://www.jbosscache.org
>
>
>
>
>
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Navin Surtani
Intern Infinispan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
Lead, Infinispan
Lead, JBoss Cache





_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev






_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] ISPN 200

Israel Lacerra
Manik,

I'm gonna need a few more days. I'll probably send you the code on the middle of the next week (about jan/6).

Guys,

About the way I'll send the code, I think the best is to make a fork on github and then I push my changes on this fork and send you the url of the fork. Right?


Israel

On Thu, Dec 16, 2010 at 9:23 AM, Manik Surtani <[hidden email]> wrote:

On 16 Dec 2010, at 11:15, Israel Lacerra wrote:

Hi Manik,

I've finished the LazyIterator. Now I need to:

- Code some good tests (now I'm testing using InfinispanDemo with some changes)
- Implement all methods of iterators (the most simple are not ready yet)
- Migrate to git

I will send the code to you in the end of the next week (I think git will help with this) and probably you will show a lot of things to improve!

No worries.  I'll only be able to look at it in the first week of Jan, but I'm sure others on this list will be able to comment in the meanwhile.  :-)


Cheers
Israel

On Wed, Dec 15, 2010 at 5:23 PM, Manik Surtani <[hidden email]> wrote:
Hi Israel

Any updates on ISPN-200?  How are you getting on?

Cheers
Manik

On 15 Sep 2010, at 12:56, Israel Lacerra wrote:

Thanks Manik!! I just did not understand the last item..

> - on an implementation level, the GetHItsCommand (or something like that) could return with a single hit, or N hits, with a flag of whether more hits are available or not.


Israel

On Wed, Sep 15, 2010 at 7:16 AM, Manik Surtani <[hidden email]> wrote:
How about something like:

- Broadcast the query
- Every node creates the QueryHits inst, runs the query and collects results.  Starts streaming the results back immediately.
- The lazy iterator returns to the user immediately, and maintains an internal cache of results coming in from N remote QueryHits instances
- iterator.next() blocks until this cache has available entries to return.
- on an implementation level, the GetHItsCommand (or something like that) could return with a single hit, or N hits, with a flag of whether more hits are available or not.

Does that help?

Manik


On 14 Sep 2010, at 20:46, Israel Lacerra wrote:

Hi guys,

I'm still thinking about it, but I don't have a really good idea about the lazy iterator yet. The only way (that I see) I could make it more lazily is:

- Broadcast the query.
- Every node creates a QueryHits instance with the query and keep it in a simple little cache (array, hash, etc)
- A "state" of the query is created and every lazyIterator.next() must send a command to a node and get the next hit (the next key).
- After a certain time, the instances of queryHits "dies".

It seems to me that this is not too efficient. But I don't have any other ideas.

Do you have any suggestions about it?

thanks!
Israel

AnyoneOn Tue, Aug 31, 2010 at 9:14 AM, Israel Lacerra <[hidden email]> wrote:
Hi Navin,

I'm trying to do ISPN-200 (https://jira.jboss.org/browse/ISPN-200) and I don't know how to implement a good lazy iterator in a "distributed way".

(Sorry... my english is not soo good. If you don't understand again, please just ask again!  :)


Israel


On Tue, Aug 31, 2010 at 6:27 AM, Navin Surtani <[hidden email]> wrote:
Hi Israel,

Just a quick question to your issue. What do you mean by you do not have
the LazyIterator? I suppose I'm not really understanding what your issue
is. Is it just that you don't want the lazy version or you don't know
how to use it? :S


On 25/08/10 14:39, Israel Lacerra wrote:
> Ok, Manik!
>
> I've already have a code working, but not too "lazy" (and without the
> "sort" part). I get the keys on all nodes, and then I use them. So I
> have a EagerIterator, but not the LazyIterator. :/
>
> I'll think about it...
>
> thanks
>
> On Wed, Aug 25, 2010 at 10:23 AM, Manik Surtani <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>     On 24 Aug 2010, at 17:20, Israel Lacerra wrote:
>
>      > Manik,
>      >
>      > What you mean by:
>      > "   * The calling node returns a CacheQuery impl that lazily fetches
>      > and collates results from the cluster." (JIRA)
>      >
>      > Is enough if each node returns a list of keys and then, we lazily
>     get the values using the keys? Or the process has to be more lazy yet?
>
>     I think it can be "more lazy" as you said.  :)
>     --
>     Manik Surtani
>     [hidden email] <mailto:[hidden email]>
>     Lead, Infinispan
>     Lead, JBoss Cache
>     http://www.infinispan.org
>     http://www.jbosscache.org
>
>
>
>
>
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Navin Surtani
Intern Infinispan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
Lead, Infinispan
Lead, JBoss Cache





_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev





_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] ISPN 200

Manik Surtani

On 30 Dec 2010, at 18:34, Israel Lacerra wrote:

> Manik,
>
> I'm gonna need a few more days. I'll probably send you the code on the middle of the next week (about jan/6).
>
> Guys,
>
> About the way I'll send the code, I think the best is to make a fork on github and then I push my changes on this fork and send you the url of the fork. Right?

Yes.  Pls post the link to this mail list so others can take a look as well.

Cheers
Manik

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] ISPN 200

Israel Lacerra
I've pushed some code in https://github.com/israeldl/infinispan/

This is not the final version. Is just to you take a look and tell me what you think. This commit have some problems (I'm fighting with Git yet). Ignore the changes in demos/*.

To use the distributed query, we have to call QueryFactory.getDistributedQuery.

There must be a lot things that I have to improve in the code, but I hope it is not in a totally wrong way!

Some thoughts:
- In the QueryBox, I made a simple eviction policy, and probably not the best. I maintain up to 300 lazyIterators. What you think is a good policy??
- To make the sort, I had to force the fields used in that sort to implement Comparable. I could not use comparator methods behind the Sort class of Lucene. One option would be to create another way to user create the sort method and then from there we would create the lucene Sort. What you think?
- I created a ClusteredQueryFacade on org.infinispan.commands, because I initialize this command in CommandsFactoryImpl.initializeReplicableCommand. There is a better way??
- The sort is made in a naive way. If we have too many nodes, probably we will have problems. Should I improve this?
In a real case, the number of nodes will be too big?


thanks!

Israel

On Tue, Jan 4, 2011 at 2:49 PM, Manik Surtani <[hidden email]> wrote:

On 30 Dec 2010, at 18:34, Israel Lacerra wrote:

> Manik,
>
> I'm gonna need a few more days. I'll probably send you the code on the middle of the next week (about jan/6).
>
> Guys,
>
> About the way I'll send the code, I think the best is to make a fork on github and then I push my changes on this fork and send you the url of the fork. Right?

Yes.  Pls post the link to this mail list so others can take a look as well.

Cheers
Manik


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] ISPN 200

Sanne Grinovero
Hi Israel,
thank you very much, I've been very eager to read some code on this.

I've started to add some very minor comments to github; it would
greatly help if you could add some
heading javadoc to each class you created, for example what's the
purpose and role of QueryBox?

About sorting, it's good to have a starter, but I don't think we can
use reflection so massively, nor
rely on Comparator.

Sort fields aren't mandated to be stored, so it's not an option to
pre-extract them from the index either;
bottom line, it's not enough to retrieve only the matching objects
from the remote node, you have to fetch
also some index lowlevel metadata to be able to properly merge the
results using the appropriate sorting.

I just found this code which might be very useful as inspiration, it's
designed to merge results
from searches performed in different threads, but if you re-think the
executor as remote node executors,
you should be set:
org.apache.lucene.search.ParallelMultiSearcher.search(Weight, Filter, int, Sort)

So you have each node perform a Query on his local index, but then the
final sort is performed on the requesting
node again processing the same TopFieldDocs which each remote node created.
When returning the matching value objects, return the TopFieldDocs
too. Actually I'd say it should be fine to retrieve
the values at a later (second phase) step, as it's quite possible
people want to retrieve only the "top 100" or similar queries.

Finally, it would be great to see some test. We don't need extreme
code coverage at this point, but it would greatly help to
have at least a two nodes scenario with a simple sort being performed,
it's also nice for people
wanting to look into it, as an example of how to use your code and
what are the entry points.

thank you, great job.
Feel free to ask anything, by email or on IRC.

Sanne


2011/1/7 Israel Lacerra <[hidden email]>:

> I've pushed some code in https://github.com/israeldl/infinispan/
>
> This is not the final version. Is just to you take a look and tell me what
> you think. This commit have some problems (I'm fighting with Git yet).
> Ignore the changes in demos/*.
>
> To use the distributed query, we have to call
> QueryFactory.getDistributedQuery.
>
> There must be a lot things that I have to improve in the code, but I hope it
> is not in a totally wrong way!
>
> Some thoughts:
> - In the QueryBox, I made a simple eviction policy, and probably not the
> best. I maintain up to 300 lazyIterators. What you think is a good policy??
> - To make the sort, I had to force the fields used in that sort to implement
> Comparable. I could not use comparator methods behind the Sort class of
> Lucene. One option would be to create another way to user create the sort
> method and then from there we would create the lucene Sort. What you think?
> - I created a ClusteredQueryFacade on org.infinispan.commands, because I
> initialize this command in CommandsFactoryImpl.initializeReplicableCommand.
> There is a better way??
> - The sort is made in a naive way. If we have too many nodes, probably we
> will have problems. Should I improve this? In a real case, the number of
> nodes will be too big?
>
>
> thanks!
>
> Israel
>
> On Tue, Jan 4, 2011 at 2:49 PM, Manik Surtani <[hidden email]> wrote:
>>
>> On 30 Dec 2010, at 18:34, Israel Lacerra wrote:
>>
>> > Manik,
>> >
>> > I'm gonna need a few more days. I'll probably send you the code on the
>> > middle of the next week (about jan/6).
>> >
>> > Guys,
>> >
>> > About the way I'll send the code, I think the best is to make a fork on
>> > github and then I push my changes on this fork and send you the url of the
>> > fork. Right?
>>
>> Yes.  Pls post the link to this mail list so others can take a look as
>> well.
>>
>> Cheers
>> Manik
>>
>> --
>> Manik Surtani
>> [hidden email]
>> twitter.com/maniksurtani
>>
>> Lead, Infinispan
>> http://www.infinispan.org
>>
>>
>>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] ISPN 200

Israel Lacerra
Hi Sanne,

Thank you for the feedback! I will see this ParallelMultiSearcher today and all you other comments.

I have a little doubt, about the development environment that you use. I'm using Eclipse, but since the migration to Git, I'm having some troubles. The Git plugin is not so good...

Are you using Eclipse or another IDE? If you using Eclipse, are you using another GUI (outside Eclipse) to synchronize and see all changes?

thanks!

Israel

On Mon, Jan 10, 2011 at 10:53 AM, Sanne Grinovero <[hidden email]> wrote:
Hi Israel,
thank you very much, I've been very eager to read some code on this.

I've started to add some very minor comments to github; it would
greatly help if you could add some
heading javadoc to each class you created, for example what's the
purpose and role of QueryBox?

About sorting, it's good to have a starter, but I don't think we can
use reflection so massively, nor
rely on Comparator.

Sort fields aren't mandated to be stored, so it's not an option to
pre-extract them from the index either;
bottom line, it's not enough to retrieve only the matching objects
from the remote node, you have to fetch
also some index lowlevel metadata to be able to properly merge the
results using the appropriate sorting.

I just found this code which might be very useful as inspiration, it's
designed to merge results
from searches performed in different threads, but if you re-think the
executor as remote node executors,
you should be set:
org.apache.lucene.search.ParallelMultiSearcher.search(Weight, Filter, int, Sort)

So you have each node perform a Query on his local index, but then the
final sort is performed on the requesting
node again processing the same TopFieldDocs which each remote node created.
When returning the matching value objects, return the TopFieldDocs
too. Actually I'd say it should be fine to retrieve
the values at a later (second phase) step, as it's quite possible
people want to retrieve only the "top 100" or similar queries.

Finally, it would be great to see some test. We don't need extreme
code coverage at this point, but it would greatly help to
have at least a two nodes scenario with a simple sort being performed,
it's also nice for people
wanting to look into it, as an example of how to use your code and
what are the entry points.

thank you, great job.
Feel free to ask anything, by email or on IRC.

Sanne


2011/1/7 Israel Lacerra <[hidden email]>:
> I've pushed some code in https://github.com/israeldl/infinispan/
>
> This is not the final version. Is just to you take a look and tell me what
> you think. This commit have some problems (I'm fighting with Git yet).
> Ignore the changes in demos/*.
>
> To use the distributed query, we have to call
> QueryFactory.getDistributedQuery.
>
> There must be a lot things that I have to improve in the code, but I hope it
> is not in a totally wrong way!
>
> Some thoughts:
> - In the QueryBox, I made a simple eviction policy, and probably not the
> best. I maintain up to 300 lazyIterators. What you think is a good policy??
> - To make the sort, I had to force the fields used in that sort to implement
> Comparable. I could not use comparator methods behind the Sort class of
> Lucene. One option would be to create another way to user create the sort
> method and then from there we would create the lucene Sort. What you think?
> - I created a ClusteredQueryFacade on org.infinispan.commands, because I
> initialize this command in CommandsFactoryImpl.initializeReplicableCommand.
> There is a better way??
> - The sort is made in a naive way. If we have too many nodes, probably we
> will have problems. Should I improve this? In a real case, the number of
> nodes will be too big?
>
>
> thanks!
>
> Israel
>
> On Tue, Jan 4, 2011 at 2:49 PM, Manik Surtani <[hidden email]> wrote:
>>
>> On 30 Dec 2010, at 18:34, Israel Lacerra wrote:
>>
>> > Manik,
>> >
>> > I'm gonna need a few more days. I'll probably send you the code on the
>> > middle of the next week (about jan/6).
>> >
>> > Guys,
>> >
>> > About the way I'll send the code, I think the best is to make a fork on
>> > github and then I push my changes on this fork and send you the url of the
>> > fork. Right?
>>
>> Yes.  Pls post the link to this mail list so others can take a look as
>> well.
>>
>> Cheers
>> Manik
>>
>> --
>> Manik Surtani
>> [hidden email]
>> twitter.com/maniksurtani
>>
>> Lead, Infinispan
>> http://www.infinispan.org
>>
>>
>>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] ISPN 200

Manik Surtani

On 13 Jan 2011, at 17:05, Israel Lacerra wrote:

Are you using Eclipse or another IDE? If you using Eclipse, are you using another GUI (outside Eclipse) to synchronize and see all changes?

In general you are probably best off using the cmd line to work with git.

See


for tips, etc.



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] ISPN 200

Israel Lacerra
Sanne,

I'm making the changes you suggest. My strategy is: when a distributed query is "initialized", each node creates and keeps a Searcher for this query and return the TopDocs to the requester node. So, the requester node merge the results, and when he needs to retrieves the next value, he sends the ScoreDoc.doc to the correct node. To merge the values I'm using a FieldDocSortedHitQueue class copy, with some changes. Is this a good strategy?

I have a "dumb" question. How a node can get a own org.infinispan.remoting.transport.Address instance? Or how we can get all Address of the cluster?

thanks

Israel

On Thu, Jan 13, 2011 at 3:57 PM, Manik Surtani <[hidden email]> wrote:

On 13 Jan 2011, at 17:05, Israel Lacerra wrote:

Are you using Eclipse or another IDE? If you using Eclipse, are you using another GUI (outside Eclipse) to synchronize and see all changes?

In general you are probably best off using the cmd line to work with git.

See


for tips, etc.

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev