[infinispan-dev] Amanuensis: the infinispan clustered indexwriter

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[infinispan-dev] Amanuensis: the infinispan clustered indexwriter

Tristan Tarrant
Dear all,

just wanted to let you know that I have published Amanuensis on GitHub: https://github.com/tristantarrant/amanuensis

As the subject says, Amanuensis is a clustered IndexWriter for Infinispan's Lucene Directory which overcomes the usual limitation of having only one writer open at any one time on a Directory by using JGroups muxed channels to stream changes from the slaves to the coordinator.

I have to thank Sanne for help in learning about JGroups muxed channels and also for pointing me at Hibernate Search's backend. My approach is nearly identical, but I wanted something that was a bit closer to a Lucene IndexWriter.

Tell me what you think.

Tristan

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Amanuensis: the infinispan clustered indexwriter

Tristan Tarrant
I have posted a few more details on my blog: http://www.dataforte.net/blog/2011/01/12/amanuensis/

Tristan

On Wed, Jan 12, 2011 at 10:05, Tristan Tarrant <[hidden email]> wrote:
Dear all,

just wanted to let you know that I have published Amanuensis on GitHub: https://github.com/tristantarrant/amanuensis

As the subject says, Amanuensis is a clustered IndexWriter for Infinispan's Lucene Directory which overcomes the usual limitation of having only one writer open at any one time on a Directory by using JGroups muxed channels to stream changes from the slaves to the coordinator.

I have to thank Sanne for help in learning about JGroups muxed channels and also for pointing me at Hibernate Search's backend. My approach is nearly identical, but I wanted something that was a bit closer to a Lucene IndexWriter.

Tell me what you think.

Tristan


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Amanuensis: the infinispan clustered indexwriter

Sanne Grinovero
In reply to this post by Tristan Tarrant
Hi Tristan,
thank you very much, this is looking great; indeed I see some familiar
patterns, but also some interesting new approaches.

Some doubts:
1) I see you don't depend on the mux implementation
org.jboss.ha.core.channelfactory.MuxChannel but in the one provided in
jgroups. I was recommended to use
org.jboss.ha.core.channelfactory.MuxChannel, but don't remember the
reason, if any it might be outdated. Did you find out there's no need
for this additional dependency?

2) Could you clarify the licensing, I'd like to reuse some of this
code if possible.
I hope it will be Hibernate Search compatible, also because I see you
copied some code.. could you then restore the original @authors where
due? (I might be wrong and didn't check in depth, just the methods in
ExecutorContext look very similar to Search's Workspace. Actually it
seems you was inspired by an older copy, while some bugs where now
fixed *1, so you might as well want to depend on that code instead, to
get more fixes in future).

3) I can't compile it:
Non-resolvable parent POM: Could not find artifact
net.dataforte:dataforte-parent:pom:6

4) I'd recommend you to make the IndexOperation implementations
immutable, it seems you're suffering a bit from index visibility.

5) No tests ? :P

*1 exceptions in backend threads wheren't guaranteed to be handled and
could kill your helpers - seems you're affected too.

Cheers,
Sanne

2011/1/12 Tristan Tarrant <[hidden email]>:

> Dear all,
>
> just wanted to let you know that I have published Amanuensis on GitHub:
> https://github.com/tristantarrant/amanuensis
>
> As the subject says, Amanuensis is a clustered IndexWriter for Infinispan's
> Lucene Directory which overcomes the usual limitation of having only one
> writer open at any one time on a Directory by using JGroups muxed channels
> to stream changes from the slaves to the coordinator.
>
> I have to thank Sanne for help in learning about JGroups muxed channels and
> also for pointing me at Hibernate Search's backend. My approach is nearly
> identical, but I wanted something that was a bit closer to a Lucene
> IndexWriter.
>
> Tell me what you think.
>
> Tristan
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Amanuensis: the infinispan clustered indexwriter

Tristan Tarrant

Some doubts:
1) I see you don't depend on the mux implementation
org.jboss.ha.core.channelfactory.MuxChannel but in the one provided in
jgroups. I was recommended to use
org.jboss.ha.core.channelfactory.MuxChannel, but don't remember the
reason, if any it might be outdated. Did you find out there's no need
for this additional dependency?

I can't remember exactly the difference between the two, although I don't think there were any real benefits in the org.jboss.ha stuff, and having one class instead of another dependency is better :)
 

2) Could you clarify the licensing, I'd like to reuse some of this
code if possible.
I hope it will be Hibernate Search compatible, also because I see you
copied some code.. could you then restore the original @authors where
due? (I might be wrong and didn't check in depth, just the methods in
ExecutorContext look very similar to Search's Workspace. Actually it
seems you was inspired by an older copy, while some bugs where now
fixed *1, so you might as well want to depend on that code instead, to
get more fixes in future).

Hmm, I usually use the Apache License, and I didn't really think about licensing conflicts here. I can't remember if I copied the code, but I need to look at that in more detail. Don't want to ruffle any feathers :) If any of my code ends up in Hibernate Search's backend, I'd rather use that if possible. Obviously it needs to be decoupled from the Hibernate part :)
 

3) I can't compile it:
Non-resolvable parent POM: Could not find artifact
net.dataforte:dataforte-parent:pom:6

It's on my repo, look at my blog post.
 

4) I'd recommend you to make the IndexOperation implementations
immutable, it seems you're suffering a bit from index visibility.

:)
 

5) No tests ? :P


<guilty_look>I'm sure they were there a minute ago :)</guilty_look>

Thanks for your comments.

Tristan

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Amanuensis: the infinispan clustered indexwriter

Bela Ban
In reply to this post by Sanne Grinovero
Hey Tristan, Sanne,

make sure you didn't use any classes in the org.jgroups.mux package; the
new multiplexer (written by Paul) is in org.jgroups.blocks.mux. There is
a test MuxRpcDispatcherTest, which shows you how to use it. Otherwise, I
hang out in #jgroups @ irc.freenode.net...



On 1/12/11 12:31 PM, Sanne Grinovero wrote:

> Hi Tristan,
> thank you very much, this is looking great; indeed I see some familiar
> patterns, but also some interesting new approaches.
>
> Some doubts:
> 1) I see you don't depend on the mux implementation
> org.jboss.ha.core.channelfactory.MuxChannel but in the one provided in
> jgroups. I was recommended to use
> org.jboss.ha.core.channelfactory.MuxChannel, but don't remember the
> reason, if any it might be outdated. Did you find out there's no need
> for this additional dependency?
>
> 2) Could you clarify the licensing, I'd like to reuse some of this
> code if possible.
> I hope it will be Hibernate Search compatible, also because I see you
> copied some code.. could you then restore the original @authors where
> due? (I might be wrong and didn't check in depth, just the methods in
> ExecutorContext look very similar to Search's Workspace. Actually it
> seems you was inspired by an older copy, while some bugs where now
> fixed *1, so you might as well want to depend on that code instead, to
> get more fixes in future).
>
> 3) I can't compile it:
> Non-resolvable parent POM: Could not find artifact
> net.dataforte:dataforte-parent:pom:6
>
> 4) I'd recommend you to make the IndexOperation implementations
> immutable, it seems you're suffering a bit from index visibility.
>
> 5) No tests ? :P
>
> *1 exceptions in backend threads wheren't guaranteed to be handled and
> could kill your helpers - seems you're affected too.
>
> Cheers,
> Sanne
>
> 2011/1/12 Tristan Tarrant<[hidden email]>:
>> Dear all,
>>
>> just wanted to let you know that I have published Amanuensis on GitHub:
>> https://github.com/tristantarrant/amanuensis
>>
>> As the subject says, Amanuensis is a clustered IndexWriter for Infinispan's
>> Lucene Directory which overcomes the usual limitation of having only one
>> writer open at any one time on a Directory by using JGroups muxed channels
>> to stream changes from the slaves to the coordinator.
>>
>> I have to thank Sanne for help in learning about JGroups muxed channels and
>> also for pointing me at Hibernate Search's backend. My approach is nearly
>> identical, but I wanted something that was a bit closer to a Lucene
>> IndexWriter.
>>
>> Tell me what you think.
>>
>> Tristan
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Amanuensis: the infinispan clustered indexwriter

Bela Ban
In reply to this post by Tristan Tarrant


On 1/12/11 12:46 PM, Tristan Tarrant wrote:

>> Some doubts:
>> 1) I see you don't depend on the mux implementation
>> org.jboss.ha.core.channelfactory.MuxChannel but in the one provided in
>> jgroups. I was recommended to use
>> org.jboss.ha.core.channelfactory.MuxChannel, but don't remember the
>> reason, if any it might be outdated. Did you find out there's no need
>> for this additional dependency?
>>
>
> I can't remember exactly the difference between the two, although I don't
> think there were any real benefits in the org.jboss.ha stuff, and having one
> class instead of another dependency is better :)


I copied Paul, who's in charge of the multiplexer. IIRC, the multiplexer
in org.jboss.ha is deprecated and the one in JGroups is used, but I
might be wrong... Paul ?



Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Amanuensis: the infinispan clustered indexwriter

Tristan Tarrant
In reply to this post by Bela Ban
Hi Bela,,

I did indeed use org.jgroups.blocks.mux.MuxUpHandler and org.jgroups.blocks.mux.MuxMessageDispatcher so I guess I'm on the right track.

Thanks for the pointer

Tristan

On Wed, Jan 12, 2011 at 12:52, Bela Ban <[hidden email]> wrote:
Hey Tristan, Sanne,

make sure you didn't use any classes in the org.jgroups.mux package; the
new multiplexer (written by Paul) is in org.jgroups.blocks.mux. There is
a test MuxRpcDispatcherTest, which shows you how to use it. Otherwise, I
hang out in #jgroups @ irc.freenode.net...



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Amanuensis: the infinispan clustered indexwriter

Bela Ban
Excellent !

On 1/12/11 3:09 PM, Tristan Tarrant wrote:
> Hi Bela,,
>
> I did indeed use org.jgroups.blocks.mux.MuxUpHandler and
> org.jgroups.blocks.mux.MuxMessageDispatcher so I guess I'm on the right
> track.
>
> Thanks for the pointer

--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...