[infinispan-dev] new Infinispan Query API - ISPN-194

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] new Infinispan Query API - ISPN-194

Sanne Grinovero-2
Infinispan 4.0 contained an integration with Hibernate Search to index
stored POJOs using the well-known Hibernate Search annotations; but
the configuration was tricky and the API quite limited.
With the latest changes in H.Search, we could now make a better integration.

Quick recap on configuration/usage on Infinispan 4:
1) indexing must be enabled in Infinispan configuration
2) a queryHelper class must be started and pointed to known types to
be indexed, passing it HS configuration properties
 - this would start a Hibernate Search engine
 - modifications done on the grid before the queryHelper was started
are ignored by the indexing engine
 - list of "known entities" could not be changed
3) use a queryFactory to create queries, which needs to be hooked up
the previously started queryHelper
 - the query options where a bit limited

What's coming in Infinispan 5:
1) indexing is still be enabled in Infinispan configuration
  - Hibernate Search configuration properties are embedded in the
Infinispan configuration

<infinispan>
   <default>
      <indexing enabled="true" indexLocalOnly="true">
         <properties>
            <property
name="hibernate.search.default.directory_provider" value="ram" />
         </properties>
      </indexing>
   </default>
</infinispan>


2) no queryHelper is needed
 - lifecycle of the indexing engine is managed by Infinispan
 - new entity types are autodiscovered
   (each time a previously unseen class is put/removed it's annotation
scanned and HS is potentially reconfigured)

3) you still use a QueryFactory, like in the examples linked below
 - exposes all features of Hibernate Search query builder DSL
 - Faceting
 - projections, etc..

API, most significant classes:
https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/main/java/org/infinispan/query/CacheQuery.java
https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/main/java/org/infinispan/query/QueryFactory.java

Two usage examples:
https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/test/java/org/infinispan/query/queries/faceting/SimpleFacetingTest.java
https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/test/java/org/infinispan/query/indexedembedded/CollectionsIndexingTest.java

there's one catch:
when searching for a class type, it will only include results from
known subtypes. The targeted type is automatically added to the known
classes, but eventually existing subtypes are not discovered.

Bringing this issue to an extreme, if the query is not targeting any
type, and no indexed types where added to the grid (even if some exist
already as they might have been inserted by other JVMs or previous
runs), all queries will return no results.
How to solve this?
 - class scanning?
 - explicitly list indexed entities in Infinispan configuration?
 - a metadata cache maintaining a distributed&stored copy of known types
 - search for types in the index :)  would be nice, but the current
design in Search mandates to know the entities first to know how the
indexes are named. without name I can't open the index, but maybe we
could have the user specify the index names instead of all entity
classtypes.

Feedback on the API is more urgent than to solve this, so it could
make it for Infinispan 5 Beta1.

Cheers,
Sanne
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Emmanuel Bernard
Nice job :)

- getBasicQuery methods are deprecated. That's from the previous design right? Was it a tech preview or was it named as full fledged API?
In the former case I'd get rid of them.

- buildQueryBuilderForClass is that really needed?

qf.buildQueryBuilderForClass( Car.class ).get();
The alternative is
qf.getSearchFactory().buildQueryBuilder(Car.class).get();

- I'm not a big fan of the constructor approach to get QueryFactory for various reasons (including the fact that it forces a concrete type and no interfaces) but it seems to be an ISPN-wide design decision
That's definitely something that could be improved in a Seam Infinispan module.

- I'd rename QueryFactory to something else as conceptually it's more an entry point to anything related to Hibernate Search, potentially indexing, stats etc: maybe SearchManager, SearchProvider, GridSearcher, TheGridReaper?

- is there an easy metaconfig to ask ISPN to store the HSearch indexes in ISPN :)

- you can't do cross Cache queries today. Is that expected? for me CacheManager.getGridSearcher almost makes sense.

- for the discovery issue, option 2 and 3 are the only two viable to me

On 4 avr. 2011, at 18:23, Sanne Grinovero wrote:

> Infinispan 4.0 contained an integration with Hibernate Search to index
> stored POJOs using the well-known Hibernate Search annotations; but
> the configuration was tricky and the API quite limited.
> With the latest changes in H.Search, we could now make a better integration.
>
> Quick recap on configuration/usage on Infinispan 4:
> 1) indexing must be enabled in Infinispan configuration
> 2) a queryHelper class must be started and pointed to known types to
> be indexed, passing it HS configuration properties
> - this would start a Hibernate Search engine
> - modifications done on the grid before the queryHelper was started
> are ignored by the indexing engine
> - list of "known entities" could not be changed
> 3) use a queryFactory to create queries, which needs to be hooked up
> the previously started queryHelper
> - the query options where a bit limited
>
> What's coming in Infinispan 5:
> 1) indexing is still be enabled in Infinispan configuration
>  - Hibernate Search configuration properties are embedded in the
> Infinispan configuration
>
> <infinispan>
>   <default>
>      <indexing enabled="true" indexLocalOnly="true">
>         <properties>
>            <property
> name="hibernate.search.default.directory_provider" value="ram" />
>         </properties>
>      </indexing>
>   </default>
> </infinispan>
>
>
> 2) no queryHelper is needed
> - lifecycle of the indexing engine is managed by Infinispan
> - new entity types are autodiscovered
>   (each time a previously unseen class is put/removed it's annotation
> scanned and HS is potentially reconfigured)
>
> 3) you still use a QueryFactory, like in the examples linked below
> - exposes all features of Hibernate Search query builder DSL
> - Faceting
> - projections, etc..
>
> API, most significant classes:
> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/main/java/org/infinispan/query/CacheQuery.java
> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/main/java/org/infinispan/query/QueryFactory.java
>
> Two usage examples:
> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/test/java/org/infinispan/query/queries/faceting/SimpleFacetingTest.java
> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/test/java/org/infinispan/query/indexedembedded/CollectionsIndexingTest.java
>
> there's one catch:
> when searching for a class type, it will only include results from
> known subtypes. The targeted type is automatically added to the known
> classes, but eventually existing subtypes are not discovered.
>
> Bringing this issue to an extreme, if the query is not targeting any
> type, and no indexed types where added to the grid (even if some exist
> already as they might have been inserted by other JVMs or previous
> runs), all queries will return no results.
> How to solve this?
> - class scanning?
> - explicitly list indexed entities in Infinispan configuration?
> - a metadata cache maintaining a distributed&stored copy of known types
> - search for types in the index :)  would be nice, but the current
> design in Search mandates to know the entities first to know how the
> indexes are named. without name I can't open the index, but maybe we
> could have the user specify the index names instead of all entity
> classtypes.
>
> Feedback on the API is more urgent than to solve this, so it could
> make it for Infinispan 5 Beta1.
>
> Cheers,
> Sanne


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Sanne Grinovero-2
thanks for the quick and comprehensive feedback.
diving inline:

2011/4/4 Emmanuel Bernard <[hidden email]>:
> Nice job :)
>
> - getBasicQuery methods are deprecated. That's from the previous design right? Was it a tech preview or was it named as full fledged API?
> In the former case I'd get rid of them.

Yes that was a tech preview, I'll remove them, but maybe better keep
them for a single release?
There's a grails plugin which exposes these methods.

> - buildQueryBuilderForClass is that really needed?
>
> qf.buildQueryBuilderForClass( Car.class ).get();
> The alternative is
> qf.getSearchFactory().buildQueryBuilder(Car.class).get();

Ah, you found a "security hole".
If you do
gf.getSearchFactory().buildQueryBuilder()
I won't be able to intercept the type and reconfigure H.S. so it might
fail if you're asking for a builder for a type which is yet unknown to
the SearchFactory.
I guess I should remove my method, and have the type discovery fixed later.

> - I'm not a big fan of the constructor approach to get QueryFactory for various reasons (including the fact that it forces a concrete type and no interfaces) but it seems to be an ISPN-wide design decision
> That's definitely something that could be improved in a Seam Infinispan module.

I knew you where going to say that :) but yes this seems to be more
Infinispan-style.
I've considered something like a Search.getQueryFactory(Cache c) .. WDYT?
This wouldn't return a "FullTextCache" but the manager, I see no
reason to delegate a cache like we do with a o.h.Session.

> - I'd rename QueryFactory to something else as conceptually it's more an entry point to anything related to Hibernate Search, potentially indexing, stats etc: maybe SearchManager, SearchProvider, GridSearcher, TheGridReaper?
right!
which one? I like the first three.

>
> - is there an easy metaconfig to ask ISPN to store the HSearch indexes in ISPN :)
No, but agree that would be awesome. would require changes though in
all involved projects:
1) H.S. would bee to be able to start on a DirectoryProvider instance we pass in
2) Infinispan Directory module (the submodule of H.S.) would need to
be able to reconfigure an existing cachemanager configuration during
the startup
3) Infinispan Core needs to be adapted for the additional configuration fields
currently it's not that hard, just a single property:

<indexing enabled="true" indexLocalOnly="true">
   <properties>
      <property
 name="hibernate.search.default.directory_provider" value="infinispan" />
      </properties>
</indexing>

 downside is that it will start a second cachemanager with a second
JGroups channel, instead of reusing the same.

>
> - you can't do cross Cache queries today. Is that expected? for me CacheManager.getGridSearcher almost makes sense.
Very interesting concept. I'll see how that fits in the code; I can't
add methods to the CacheManager itself, but maybe I can make sure that
interceptors registered on different caches share the same
MutableSearchFactory.

>
> - for the discovery issue, option 2 and 3 are the only two viable to me
I'll work towards 2 first, looks simpler. BTW 3 requires the user to
configure a permanent cachestore, or at the bare minimum to be
dist/repl.

Cheers,
Sanne

> On 4 avr. 2011, at 18:23, Sanne Grinovero wrote:
>
>> Infinispan 4.0 contained an integration with Hibernate Search to index
>> stored POJOs using the well-known Hibernate Search annotations; but
>> the configuration was tricky and the API quite limited.
>> With the latest changes in H.Search, we could now make a better integration.
>>
>> Quick recap on configuration/usage on Infinispan 4:
>> 1) indexing must be enabled in Infinispan configuration
>> 2) a queryHelper class must be started and pointed to known types to
>> be indexed, passing it HS configuration properties
>> - this would start a Hibernate Search engine
>> - modifications done on the grid before the queryHelper was started
>> are ignored by the indexing engine
>> - list of "known entities" could not be changed
>> 3) use a queryFactory to create queries, which needs to be hooked up
>> the previously started queryHelper
>> - the query options where a bit limited
>>
>> What's coming in Infinispan 5:
>> 1) indexing is still be enabled in Infinispan configuration
>>  - Hibernate Search configuration properties are embedded in the
>> Infinispan configuration
>>
>> <infinispan>
>>   <default>
>>      <indexing enabled="true" indexLocalOnly="true">
>>         <properties>
>>            <property
>> name="hibernate.search.default.directory_provider" value="ram" />
>>         </properties>
>>      </indexing>
>>   </default>
>> </infinispan>
>>
>>
>> 2) no queryHelper is needed
>> - lifecycle of the indexing engine is managed by Infinispan
>> - new entity types are autodiscovered
>>   (each time a previously unseen class is put/removed it's annotation
>> scanned and HS is potentially reconfigured)
>>
>> 3) you still use a QueryFactory, like in the examples linked below
>> - exposes all features of Hibernate Search query builder DSL
>> - Faceting
>> - projections, etc..
>>
>> API, most significant classes:
>> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/main/java/org/infinispan/query/CacheQuery.java
>> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/main/java/org/infinispan/query/QueryFactory.java
>>
>> Two usage examples:
>> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/test/java/org/infinispan/query/queries/faceting/SimpleFacetingTest.java
>> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/test/java/org/infinispan/query/indexedembedded/CollectionsIndexingTest.java
>>
>> there's one catch:
>> when searching for a class type, it will only include results from
>> known subtypes. The targeted type is automatically added to the known
>> classes, but eventually existing subtypes are not discovered.
>>
>> Bringing this issue to an extreme, if the query is not targeting any
>> type, and no indexed types where added to the grid (even if some exist
>> already as they might have been inserted by other JVMs or previous
>> runs), all queries will return no results.
>> How to solve this?
>> - class scanning?
>> - explicitly list indexed entities in Infinispan configuration?
>> - a metadata cache maintaining a distributed&stored copy of known types
>> - search for types in the index :)  would be nice, but the current
>> design in Search mandates to know the entities first to know how the
>> indexes are named. without name I can't open the index, but maybe we
>> could have the user specify the index names instead of all entity
>> classtypes.
>>
>> Feedback on the API is more urgent than to solve this, so it could
>> make it for Infinispan 5 Beta1.
>>
>> Cheers,
>> Sanne
>
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Emmanuel Bernard

On 4 avr. 2011, at 19:24, Sanne Grinovero wrote:

> thanks for the quick and comprehensive feedback.
> diving inline:
>
> 2011/4/4 Emmanuel Bernard <[hidden email]>:
>> Nice job :)
>>
>> - getBasicQuery methods are deprecated. That's from the previous design right? Was it a tech preview or was it named as full fledged API?
>> In the former case I'd get rid of them.
>
> Yes that was a tech preview, I'll remove them, but maybe better keep
> them for a single release?
> There's a grails plugin which exposes these methods.

Discuss with the maintainer. There is probably a solution.

>
>> - buildQueryBuilderForClass is that really needed?
>>
>> qf.buildQueryBuilderForClass( Car.class ).get();
>> The alternative is
>> qf.getSearchFactory().buildQueryBuilder(Car.class).get();
>
> Ah, you found a "security hole".
> If you do
> gf.getSearchFactory().buildQueryBuilder()
> I won't be able to intercept the type and reconfigure H.S. so it might
> fail if you're asking for a builder for a type which is yet unknown to
> the SearchFactory.
> I guess I should remove my method, and have the type discovery fixed later.

That's not going to be for Hibernate Search 3.4 but do you think it's a feature that should be part of Hibernate Search per se?

>
>> - I'm not a big fan of the constructor approach to get QueryFactory for various reasons (including the fact that it forces a concrete type and no interfaces) but it seems to be an ISPN-wide design decision
>> That's definitely something that could be improved in a Seam Infinispan module.
>
> I knew you where going to say that :) but yes this seems to be more
> Infinispan-style.
> I've considered something like a Search.getQueryFactory(Cache c) .. WDYT?
> This wouldn't return a "FullTextCache" but the manager, I see no
> reason to delegate a cache like we do with a o.h.Session.

For sure the reasons are weaker, but the idea of a FullTextSession is also to get all useful interaction methods from a single object.
I've never been extremely happy with the static helper method but it has the advantage of letting you return an interface. I guess the most important is to be consistent with the rest of ISPN's API.
See also the unwrap() approach I've just described on Hibernate-Dev.

>
>> - I'd rename QueryFactory to something else as conceptually it's more an entry point to anything related to Hibernate Search, potentially indexing, stats etc: maybe SearchManager, SearchProvider, GridSearcher, TheGridReaper?
> right!
> which one? I like the first three.

I guess GridSearcher is fine but collect other opinions

>
>>
>> - is there an easy metaconfig to ask ISPN to store the HSearch indexes in ISPN :)
> No, but agree that would be awesome. would require changes though in
> all involved projects:
> 1) H.S. would be to be able to start on a DirectoryProvider instance we pass in
> 2) Infinispan Directory module (the submodule of H.S.) would need to
> be able to reconfigure an existing cachemanager configuration during
> the startup
> 3) Infinispan Core needs to be adapted for the additional configuration fields
> currently it's not that hard, just a single property:
>
> <indexing enabled="true" indexLocalOnly="true">
>   <properties>
>      <property
> name="hibernate.search.default.directory_provider" value="infinispan" />
>      </properties>
> </indexing>
>
> downside is that it will start a second cachemanager with a second
> JGroups channel, instead of reusing the same.

OK not for today, let's put that in the TODO truck


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Galder Zamarreno
In reply to this post by Sanne Grinovero-2
See comments below:

On Apr 4, 2011, at 7:24 PM, Sanne Grinovero wrote:

> thanks for the quick and comprehensive feedback.
> diving inline:
>
> 2011/4/4 Emmanuel Bernard <[hidden email]>:
>> </snip>
>>
>> - I'm not a big fan of the constructor approach to get QueryFactory for various reasons (including the fact that it forces a concrete type and no interfaces) but it seems to be an ISPN-wide design decision
>> That's definitely something that could be improved in a Seam Infinispan module.
>
> I knew you where going to say that :) but yes this seems to be more
> Infinispan-style.
> I've considered something like a Search.getQueryFactory(Cache c) .. WDYT?
> This wouldn't return a "FullTextCache" but the manager, I see no
> reason to delegate a cache like we do with a o.h.Session.

Is this Hibernate Search API you're talking about?


>
>> - I'd rename QueryFactory to something else as conceptually it's more an entry point to anything related to Hibernate Search, potentially indexing, stats etc: maybe SearchManager, SearchProvider, GridSearcher, TheGridReaper?
> right!
> which one? I like the first three.

I'd go with SearchManager to follow other naming patterns such as CacheManager.

>
>>
>> - is there an easy metaconfig to ask ISPN to store the HSearch indexes in ISPN :)
> No, but agree that would be awesome. would require changes though in
> all involved projects:
> 1) H.S. would bee to be able to start on a DirectoryProvider instance we pass in
> 2) Infinispan Directory module (the submodule of H.S.) would need to
> be able to reconfigure an existing cachemanager configuration during
> the startup
> 3) Infinispan Core needs to be adapted for the additional configuration fields
> currently it's not that hard, just a single property:
>
> <indexing enabled="true" indexLocalOnly="true">
>   <properties>
>      <property
> name="hibernate.search.default.directory_provider" value="infinispan" />
>      </properties>
> </indexing>
>
> downside is that it will start a second cachemanager with a second
> JGroups channel, instead of reusing the same.

JGroupsChannelLookup is there to enable multiple cache managers to share the same CacheManager. We don't have an impl for standalone cases but that's what AS uses to feed in the JGroups channel to the various cache managers.

>
>>
>> - you can't do cross Cache queries today. Is that expected? for me CacheManager.getGridSearcher almost makes sense.
> Very interesting concept. I'll see how that fits in the code; I can't
> add methods to the CacheManager itself, but maybe I can make sure that
> interceptors registered on different caches share the same
> MutableSearchFactory.
>
>>
>> - for the discovery issue, option 2 and 3 are the only two viable to me
> I'll work towards 2 first, looks simpler. BTW 3 requires the user to
> configure a permanent cachestore, or at the bare minimum to be
> dist/repl.
>
> Cheers,
> Sanne
>
>> On 4 avr. 2011, at 18:23, Sanne Grinovero wrote:
>>
>>> Infinispan 4.0 contained an integration with Hibernate Search to index
>>> stored POJOs using the well-known Hibernate Search annotations; but
>>> the configuration was tricky and the API quite limited.
>>> With the latest changes in H.Search, we could now make a better integration.
>>>
>>> Quick recap on configuration/usage on Infinispan 4:
>>> 1) indexing must be enabled in Infinispan configuration
>>> 2) a queryHelper class must be started and pointed to known types to
>>> be indexed, passing it HS configuration properties
>>> - this would start a Hibernate Search engine
>>> - modifications done on the grid before the queryHelper was started
>>> are ignored by the indexing engine
>>> - list of "known entities" could not be changed
>>> 3) use a queryFactory to create queries, which needs to be hooked up
>>> the previously started queryHelper
>>> - the query options where a bit limited
>>>
>>> What's coming in Infinispan 5:
>>> 1) indexing is still be enabled in Infinispan configuration
>>>  - Hibernate Search configuration properties are embedded in the
>>> Infinispan configuration
>>>
>>> <infinispan>
>>>   <default>
>>>      <indexing enabled="true" indexLocalOnly="true">
>>>         <properties>
>>>            <property
>>> name="hibernate.search.default.directory_provider" value="ram" />
>>>         </properties>
>>>      </indexing>
>>>   </default>
>>> </infinispan>
>>>
>>>
>>> 2) no queryHelper is needed
>>> - lifecycle of the indexing engine is managed by Infinispan
>>> - new entity types are autodiscovered
>>>   (each time a previously unseen class is put/removed it's annotation
>>> scanned and HS is potentially reconfigured)
>>>
>>> 3) you still use a QueryFactory, like in the examples linked below
>>> - exposes all features of Hibernate Search query builder DSL
>>> - Faceting
>>> - projections, etc..
>>>
>>> API, most significant classes:
>>> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/main/java/org/infinispan/query/CacheQuery.java
>>> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/main/java/org/infinispan/query/QueryFactory.java
>>>
>>> Two usage examples:
>>> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/test/java/org/infinispan/query/queries/faceting/SimpleFacetingTest.java
>>> https://github.com/Sanne/infinispan/blob/eafe084a2ffa7e539768c838334ad42da3e8e565/query/src/test/java/org/infinispan/query/indexedembedded/CollectionsIndexingTest.java
>>>
>>> there's one catch:
>>> when searching for a class type, it will only include results from
>>> known subtypes. The targeted type is automatically added to the known
>>> classes, but eventually existing subtypes are not discovered.
>>>
>>> Bringing this issue to an extreme, if the query is not targeting any
>>> type, and no indexed types where added to the grid (even if some exist
>>> already as they might have been inserted by other JVMs or previous
>>> runs), all queries will return no results.
>>> How to solve this?
>>> - class scanning?
>>> - explicitly list indexed entities in Infinispan configuration?
>>> - a metadata cache maintaining a distributed&stored copy of known types
>>> - search for types in the index :)  would be nice, but the current
>>> design in Search mandates to know the entities first to know how the
>>> indexes are named. without name I can't open the index, but maybe we
>>> could have the user specify the index names instead of all entity
>>> classtypes.
>>>
>>> Feedback on the API is more urgent than to solve this, so it could
>>> make it for Infinispan 5 Beta1.
>>>
>>> Cheers,
>>> Sanne
>>
>>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Galder Zamarreno
In reply to this post by Sanne Grinovero-2

On Apr 4, 2011, at 6:23 PM, Sanne Grinovero wrote:

> </snip>
>
> there's one catch:
> when searching for a class type, it will only include results from
> known subtypes. The targeted type is automatically added to the known
> classes, but eventually existing subtypes are not discovered.
>
> Bringing this issue to an extreme, if the query is not targeting any
> type, and no indexed types where added to the grid (even if some exist
> already as they might have been inserted by other JVMs or previous
> runs), all queries will return no results.
> How to solve this?
> - class scanning?

Nope, too expensive.

> - explicitly list indexed entities in Infinispan configuration?

No

> - a metadata cache maintaining a distributed&stored copy of known types

That sounds more appealing. It could be a good middle ground until Search can search for types.

> - search for types in the index :)  would be nice, but the current
> design in Search mandates to know the entities first to know how the
> indexes are named. without name I can't open the index, but maybe we
> could have the user specify the index names instead of all entity
> classtypes.
>
> Feedback on the API is more urgent than to solve this, so it could
> make it for Infinispan 5 Beta1.
>
> Cheers,
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Sanne Grinovero
I need to pick the final name for the Query entry point ASAP,
proposals so far:

- SearchManager (1 vote)
- SearchProvider (1 vote)
- GridSearcher (1 vote)
- TheGridReaper (no votes)

Galder pointed out that "SearchManager" is being consistent with other
Infinispan names.

More ideas?

Sanne

2011/4/5 Galder Zamarreño <[hidden email]>:

>
> On Apr 4, 2011, at 6:23 PM, Sanne Grinovero wrote:
>
>> </snip>
>>
>> there's one catch:
>> when searching for a class type, it will only include results from
>> known subtypes. The targeted type is automatically added to the known
>> classes, but eventually existing subtypes are not discovered.
>>
>> Bringing this issue to an extreme, if the query is not targeting any
>> type, and no indexed types where added to the grid (even if some exist
>> already as they might have been inserted by other JVMs or previous
>> runs), all queries will return no results.
>> How to solve this?
>> - class scanning?
>
> Nope, too expensive.
>
>> - explicitly list indexed entities in Infinispan configuration?
>
> No
>
>> - a metadata cache maintaining a distributed&stored copy of known types
>
> That sounds more appealing. It could be a good middle ground until Search can search for types.
>
>> - search for types in the index :)  would be nice, but the current
>> design in Search mandates to know the entities first to know how the
>> indexes are named. without name I can't open the index, but maybe we
>> could have the user specify the index names instead of all entity
>> classtypes.
>>
>> Feedback on the API is more urgent than to solve this, so it could
>> make it for Infinispan 5 Beta1.
>>
>> Cheers,
>> Sanne
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Emmanuel Bernard
In reply to this post by Galder Zamarreno

On 5 avr. 2011, at 12:03, Galder Zamarreño wrote:

>>>
>>> - I'm not a big fan of the constructor approach to get QueryFactory for various reasons (including the fact that it forces a concrete type and no interfaces) but it seems to be an ISPN-wide design decision
>>> That's definitely something that could be improved in a Seam Infinispan module.
>>
>> I knew you where going to say that :) but yes this seems to be more
>> Infinispan-style.
>> I've considered something like a Search.getQueryFactory(Cache c) .. WDYT?
>> This wouldn't return a "FullTextCache" but the manager, I see no
>> reason to delegate a cache like we do with a o.h.Session.
>
> Is this Hibernate Search API you're talking about?

Nope, Infinispan.
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Sanne Grinovero
2011/4/5 Emmanuel Bernard <[hidden email]>:

>
> On 5 avr. 2011, at 12:03, Galder Zamarreño wrote:
>
>>>>
>>>> - I'm not a big fan of the constructor approach to get QueryFactory for various reasons (including the fact that it forces a concrete type and no interfaces) but it seems to be an ISPN-wide design decision
>>>> That's definitely something that could be improved in a Seam Infinispan module.
>>>
>>> I knew you where going to say that :) but yes this seems to be more
>>> Infinispan-style.
>>> I've considered something like a Search.getQueryFactory(Cache c) .. WDYT?
>>> This wouldn't return a "FullTextCache" but the manager, I see no
>>> reason to delegate a cache like we do with a o.h.Session.
>>
>> Is this Hibernate Search API you're talking about?
>
> Nope, Infinispan.

I think it's looking better indeed:

(search for Search.getSearchManager at url: )
https://github.com/Sanne/infinispan/commit/f8e0a46bb7b4b66af893f74bad358e4faacb033e



> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Emmanuel Bernard
In reply to this post by Galder Zamarreno

On 5 avr. 2011, at 12:20, Galder Zamarreño wrote:

>
> On Apr 4, 2011, at 6:23 PM, Sanne Grinovero wrote:
>
>> </snip>
>>
>> there's one catch:
>> when searching for a class type, it will only include results from
>> known subtypes. The targeted type is automatically added to the known
>> classes, but eventually existing subtypes are not discovered.
>>
>> Bringing this issue to an extreme, if the query is not targeting any
>> type, and no indexed types where added to the grid (even if some exist
>> already as they might have been inserted by other JVMs or previous
>> runs), all queries will return no results.
>> How to solve this?
>> - class scanning?
>
> Nope, too expensive.
>
>> - explicitly list indexed entities in Infinispan configuration?
>
> No
>
>> - a metadata cache maintaining a distributed&stored copy of known types
>
> That sounds more appealing. It could be a good middle ground until Search can search for types.

Do you have any specific idea in mind?

To magically find types:
 - we scan every file system, databases, caches available to the app and look for Lucene metadata => unrealistic
 - there is some kind of convention on where the indexes are and we do index scanning at startup => scanning are very likely to be slower that class scanning (potential remote access, bigger dataset etc)
 - we enforce one or a fixed number of Lucene indexes for all data in Infinispan => not sure that's a good idea but this can be explored
 - we somehow ask the framework using HSearch to fill up classes

other approaches?
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Sanne Grinovero
2011/4/5 Emmanuel Bernard <[hidden email]>:

>
> On 5 avr. 2011, at 12:20, Galder Zamarreño wrote:
>
>>
>> On Apr 4, 2011, at 6:23 PM, Sanne Grinovero wrote:
>>
>>> </snip>
>>>
>>> there's one catch:
>>> when searching for a class type, it will only include results from
>>> known subtypes. The targeted type is automatically added to the known
>>> classes, but eventually existing subtypes are not discovered.
>>>
>>> Bringing this issue to an extreme, if the query is not targeting any
>>> type, and no indexed types where added to the grid (even if some exist
>>> already as they might have been inserted by other JVMs or previous
>>> runs), all queries will return no results.
>>> How to solve this?
>>> - class scanning?
>>
>> Nope, too expensive.
>>
>>> - explicitly list indexed entities in Infinispan configuration?
>>
>> No
>>
>>> - a metadata cache maintaining a distributed&stored copy of known types
>>
>> That sounds more appealing. It could be a good middle ground until Search can search for types.
>
> Do you have any specific idea in mind?
>
> To magically find types:
>  - we scan every file system, databases, caches available to the app and look for Lucene metadata => unrealistic
>  - there is some kind of convention on where the indexes are and we do index scanning at startup => scanning are very likely to be slower that class scanning (potential remote access, bigger dataset etc)
>  - we enforce one or a fixed number of Lucene indexes for all data in Infinispan => not sure that's a good idea but this can be explored
>  - we somehow ask the framework using HSearch to fill up classes
>
> other approaches?

why was class scanning discarded in the first answer? as H. Search can
auto-discover classes by working on top of JPA entity autodiscovery, I
guess that each application node could look into it's own known
classpath.
After all if some type is not visible to him as it was added from
another node from a different app, he won't be able to return
instances of it either.
We could face the opposite problem of building metadata of classes
people doesn't mean to index in this cache.

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Emmanuel Bernard

On 5 avr. 2011, at 13:38, Sanne Grinovero wrote:

> 2011/4/5 Emmanuel Bernard <[hidden email]>:
>>
>> On 5 avr. 2011, at 12:20, Galder Zamarreño wrote:
>>
>>>
>>> On Apr 4, 2011, at 6:23 PM, Sanne Grinovero wrote:
>>>
>>>> </snip>
>>>>
>>>> there's one catch:
>>>> when searching for a class type, it will only include results from
>>>> known subtypes. The targeted type is automatically added to the known
>>>> classes, but eventually existing subtypes are not discovered.
>>>>
>>>> Bringing this issue to an extreme, if the query is not targeting any
>>>> type, and no indexed types where added to the grid (even if some exist
>>>> already as they might have been inserted by other JVMs or previous
>>>> runs), all queries will return no results.
>>>> How to solve this?
>>>> - class scanning?
>>>
>>> Nope, too expensive.
>>>
>>>> - explicitly list indexed entities in Infinispan configuration?
>>>
>>> No
>>>
>>>> - a metadata cache maintaining a distributed&stored copy of known types
>>>
>>> That sounds more appealing. It could be a good middle ground until Search can search for types.
>>
>> Do you have any specific idea in mind?
>>
>> To magically find types:
>>  - we scan every file system, databases, caches available to the app and look for Lucene metadata => unrealistic
>>  - there is some kind of convention on where the indexes are and we do index scanning at startup => scanning are very likely to be slower that class scanning (potential remote access, bigger dataset etc)
>>  - we enforce one or a fixed number of Lucene indexes for all data in Infinispan => not sure that's a good idea but this can be explored
>>  - we somehow ask the framework using HSearch to fill up classes
>>
>> other approaches?
>
> why was class scanning discarded in the first answer? as H. Search can
> auto-discover classes by working on top of JPA entity autodiscovery, I
> guess that each application node could look into it's own known
> classpath.
> After all if some type is not visible to him as it was added from
> another node from a different app, he won't be able to return
> instances of it either.
> We could face the opposite problem of building metadata of classes
> people doesn't mean to index in this cache.

Right. scanning (class or index) will be a bit aggressive and could build unneeded metadata (or even worse, return unexpected classes).
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Israel Lacerra
Guys,

Do you think we can have a getQueryHits() on HSQueryImpl? To do ISPN-200 I was using the TopDocs and now, with Infinispan using HSQueryImpl, TopDocs is a little more hided....

Anyway, take this just as a "suggestion". I can get the TopDocs in another less fashion way :).

cheers

Israel

On Tue, Apr 5, 2011 at 9:23 AM, Emmanuel Bernard <[hidden email]> wrote:

On 5 avr. 2011, at 13:38, Sanne Grinovero wrote:

> 2011/4/5 Emmanuel Bernard <[hidden email]>:
>>
>> On 5 avr. 2011, at 12:20, Galder Zamarreño wrote:
>>
>>>
>>> On Apr 4, 2011, at 6:23 PM, Sanne Grinovero wrote:
>>>
>>>> </snip>
>>>>
>>>> there's one catch:
>>>> when searching for a class type, it will only include results from
>>>> known subtypes. The targeted type is automatically added to the known
>>>> classes, but eventually existing subtypes are not discovered.
>>>>
>>>> Bringing this issue to an extreme, if the query is not targeting any
>>>> type, and no indexed types where added to the grid (even if some exist
>>>> already as they might have been inserted by other JVMs or previous
>>>> runs), all queries will return no results.
>>>> How to solve this?
>>>> - class scanning?
>>>
>>> Nope, too expensive.
>>>
>>>> - explicitly list indexed entities in Infinispan configuration?
>>>
>>> No
>>>
>>>> - a metadata cache maintaining a distributed&stored copy of known types
>>>
>>> That sounds more appealing. It could be a good middle ground until Search can search for types.
>>
>> Do you have any specific idea in mind?
>>
>> To magically find types:
>>  - we scan every file system, databases, caches available to the app and look for Lucene metadata => unrealistic
>>  - there is some kind of convention on where the indexes are and we do index scanning at startup => scanning are very likely to be slower that class scanning (potential remote access, bigger dataset etc)
>>  - we enforce one or a fixed number of Lucene indexes for all data in Infinispan => not sure that's a good idea but this can be explored
>>  - we somehow ask the framework using HSearch to fill up classes
>>
>> other approaches?
>
> why was class scanning discarded in the first answer? as H. Search can
> auto-discover classes by working on top of JPA entity autodiscovery, I
> guess that each application node could look into it's own known
> classpath.
> After all if some type is not visible to him as it was added from
> another node from a different app, he won't be able to return
> instances of it either.
> We could face the opposite problem of building metadata of classes
> people doesn't mean to index in this cache.

Right. scanning (class or index) will be a bit aggressive and could build unneeded metadata (or even worse, return unexpected classes).
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Emmanuel Bernard
Why do you need TopDocs? What's the use case?

On 10 avr. 2011, at 03:42, Israel Lacerra wrote:

Guys,

Do you think we can have a getQueryHits() on HSQueryImpl? To do ISPN-200 I was using the TopDocs and now, with Infinispan using HSQueryImpl, TopDocs is a little more hided....

Anyway, take this just as a "suggestion". I can get the TopDocs in another less fashion way :).

cheers

Israel

On Tue, Apr 5, 2011 at 9:23 AM, Emmanuel Bernard <[hidden email]> wrote:

On 5 avr. 2011, at 13:38, Sanne Grinovero wrote:

> 2011/4/5 Emmanuel Bernard <[hidden email]>:
>>
>> On 5 avr. 2011, at 12:20, Galder Zamarreño wrote:
>>
>>>
>>> On Apr 4, 2011, at 6:23 PM, Sanne Grinovero wrote:
>>>
>>>> </snip>
>>>>
>>>> there's one catch:
>>>> when searching for a class type, it will only include results from
>>>> known subtypes. The targeted type is automatically added to the known
>>>> classes, but eventually existing subtypes are not discovered.
>>>>
>>>> Bringing this issue to an extreme, if the query is not targeting any
>>>> type, and no indexed types where added to the grid (even if some exist
>>>> already as they might have been inserted by other JVMs or previous
>>>> runs), all queries will return no results.
>>>> How to solve this?
>>>> - class scanning?
>>>
>>> Nope, too expensive.
>>>
>>>> - explicitly list indexed entities in Infinispan configuration?
>>>
>>> No
>>>
>>>> - a metadata cache maintaining a distributed&stored copy of known types
>>>
>>> That sounds more appealing. It could be a good middle ground until Search can search for types.
>>
>> Do you have any specific idea in mind?
>>
>> To magically find types:
>>  - we scan every file system, databases, caches available to the app and look for Lucene metadata => unrealistic
>>  - there is some kind of convention on where the indexes are and we do index scanning at startup => scanning are very likely to be slower that class scanning (potential remote access, bigger dataset etc)
>>  - we enforce one or a fixed number of Lucene indexes for all data in Infinispan => not sure that's a good idea but this can be explored
>>  - we somehow ask the framework using HSearch to fill up classes
>>
>> other approaches?
>
> why was class scanning discarded in the first answer? as H. Search can
> auto-discover classes by working on top of JPA entity autodiscovery, I
> guess that each application node could look into it's own known
> classpath.
> After all if some type is not visible to him as it was added from
> another node from a different app, he won't be able to return
> instances of it either.
> We could face the opposite problem of building metadata of classes
> people doesn't mean to index in this cache.

Right. scanning (class or index) will be a bit aggressive and could build unneeded metadata (or even worse, return unexpected classes).
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Israel Lacerra
To make a LazyIterator for ISPN-200 (https://issues.jboss.org/browse/ISPN-200) I was starting a query in each node and then merging the results (TopDocs) in the requester node...

On Sun, Apr 10, 2011 at 6:10 AM, Emmanuel Bernard <[hidden email]> wrote:
Why do you need TopDocs? What's the use case?

On 10 avr. 2011, at 03:42, Israel Lacerra wrote:

Guys,

Do you think we can have a getQueryHits() on HSQueryImpl? To do ISPN-200 I was using the TopDocs and now, with Infinispan using HSQueryImpl, TopDocs is a little more hided....

Anyway, take this just as a "suggestion". I can get the TopDocs in another less fashion way :).

cheers

Israel

On Tue, Apr 5, 2011 at 9:23 AM, Emmanuel Bernard <[hidden email]> wrote:

On 5 avr. 2011, at 13:38, Sanne Grinovero wrote:

> 2011/4/5 Emmanuel Bernard <[hidden email]>:
>>
>> On 5 avr. 2011, at 12:20, Galder Zamarreño wrote:
>>
>>>
>>> On Apr 4, 2011, at 6:23 PM, Sanne Grinovero wrote:
>>>
>>>> </snip>
>>>>
>>>> there's one catch:
>>>> when searching for a class type, it will only include results from
>>>> known subtypes. The targeted type is automatically added to the known
>>>> classes, but eventually existing subtypes are not discovered.
>>>>
>>>> Bringing this issue to an extreme, if the query is not targeting any
>>>> type, and no indexed types where added to the grid (even if some exist
>>>> already as they might have been inserted by other JVMs or previous
>>>> runs), all queries will return no results.
>>>> How to solve this?
>>>> - class scanning?
>>>
>>> Nope, too expensive.
>>>
>>>> - explicitly list indexed entities in Infinispan configuration?
>>>
>>> No
>>>
>>>> - a metadata cache maintaining a distributed&stored copy of known types
>>>
>>> That sounds more appealing. It could be a good middle ground until Search can search for types.
>>
>> Do you have any specific idea in mind?
>>
>> To magically find types:
>>  - we scan every file system, databases, caches available to the app and look for Lucene metadata => unrealistic
>>  - there is some kind of convention on where the indexes are and we do index scanning at startup => scanning are very likely to be slower that class scanning (potential remote access, bigger dataset etc)
>>  - we enforce one or a fixed number of Lucene indexes for all data in Infinispan => not sure that's a good idea but this can be explored
>>  - we somehow ask the framework using HSearch to fill up classes
>>
>> other approaches?
>
> why was class scanning discarded in the first answer? as H. Search can
> auto-discover classes by working on top of JPA entity autodiscovery, I
> guess that each application node could look into it's own known
> classpath.
> After all if some type is not visible to him as it was added from
> another node from a different app, he won't be able to return
> instances of it either.
> We could face the opposite problem of building metadata of classes
> people doesn't mean to index in this cache.

Right. scanning (class or index) will be a bit aggressive and could build unneeded metadata (or even worse, return unexpected classes).
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Sanne Grinovero
Hi Israel,
I'm looking for ways on how to make this easier for you, but as I'm
not completely sure on what you need, let's discuss on how to get it
more flexible:

We could make the classes
org.hibernate.search.query.engine.impl.HSQueryImpl
org.hibernate.search.query.engine.impl.QueryHits
suitable for being extended (changing most private methods to
protected, and also refactor the HSQueryImpl to possibly use a
different type for QueryHits)

and then making sure that it's possible to have
org.hibernate.search.spi.SearchFactoryIntegrator.createHSQuery()
return a custom type.

(I'm assuming you have the source code of hibernate Search; if not,
you can get it from https://github.com/hibernate/hibernate-search ).

How does that look to you, is that going to suite the needs of ISPN-200 ?

Cheers,
Sanne

2011/4/10 Israel Lacerra <[hidden email]>:

> To make a LazyIterator for ISPN-200
> (https://issues.jboss.org/browse/ISPN-200) I was starting a query in each
> node and then merging the results (TopDocs) in the requester node...
>
> On Sun, Apr 10, 2011 at 6:10 AM, Emmanuel Bernard <[hidden email]>
> wrote:
>>
>> Why do you need TopDocs? What's the use case?
>> On 10 avr. 2011, at 03:42, Israel Lacerra wrote:
>>
>> Guys,
>>
>> Do you think we can have a getQueryHits() on HSQueryImpl? To do ISPN-200 I
>> was using the TopDocs and now, with Infinispan using HSQueryImpl, TopDocs is
>> a little more hided....
>>
>> Anyway, take this just as a "suggestion". I can get the TopDocs in another
>> less fashion way :).
>>
>> cheers
>>
>> Israel
>>
>> On Tue, Apr 5, 2011 at 9:23 AM, Emmanuel Bernard <[hidden email]>
>> wrote:
>>>
>>> On 5 avr. 2011, at 13:38, Sanne Grinovero wrote:
>>>
>>> > 2011/4/5 Emmanuel Bernard <[hidden email]>:
>>> >>
>>> >> On 5 avr. 2011, at 12:20, Galder Zamarreño wrote:
>>> >>
>>> >>>
>>> >>> On Apr 4, 2011, at 6:23 PM, Sanne Grinovero wrote:
>>> >>>
>>> >>>> </snip>
>>> >>>>
>>> >>>> there's one catch:
>>> >>>> when searching for a class type, it will only include results from
>>> >>>> known subtypes. The targeted type is automatically added to the
>>> >>>> known
>>> >>>> classes, but eventually existing subtypes are not discovered.
>>> >>>>
>>> >>>> Bringing this issue to an extreme, if the query is not targeting any
>>> >>>> type, and no indexed types where added to the grid (even if some
>>> >>>> exist
>>> >>>> already as they might have been inserted by other JVMs or previous
>>> >>>> runs), all queries will return no results.
>>> >>>> How to solve this?
>>> >>>> - class scanning?
>>> >>>
>>> >>> Nope, too expensive.
>>> >>>
>>> >>>> - explicitly list indexed entities in Infinispan configuration?
>>> >>>
>>> >>> No
>>> >>>
>>> >>>> - a metadata cache maintaining a distributed&stored copy of known
>>> >>>> types
>>> >>>
>>> >>> That sounds more appealing. It could be a good middle ground until
>>> >>> Search can search for types.
>>> >>
>>> >> Do you have any specific idea in mind?
>>> >>
>>> >> To magically find types:
>>> >>  - we scan every file system, databases, caches available to the app
>>> >> and look for Lucene metadata => unrealistic
>>> >>  - there is some kind of convention on where the indexes are and we do
>>> >> index scanning at startup => scanning are very likely to be slower that
>>> >> class scanning (potential remote access, bigger dataset etc)
>>> >>  - we enforce one or a fixed number of Lucene indexes for all data in
>>> >> Infinispan => not sure that's a good idea but this can be explored
>>> >>  - we somehow ask the framework using HSearch to fill up classes
>>> >>
>>> >> other approaches?
>>> >
>>> > why was class scanning discarded in the first answer? as H. Search can
>>> > auto-discover classes by working on top of JPA entity autodiscovery, I
>>> > guess that each application node could look into it's own known
>>> > classpath.
>>> > After all if some type is not visible to him as it was added from
>>> > another node from a different app, he won't be able to return
>>> > instances of it either.
>>> > We could face the opposite problem of building metadata of classes
>>> > people doesn't mean to index in this cache.
>>>
>>> Right. scanning (class or index) will be a bit aggressive and could build
>>> unneeded metadata (or even worse, return unexpected classes).
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Israel Lacerra
Hi Sanne,

When I create a LazyIterator, every node makes a local normal query and thern I return the TopDocs to the requester node. With TopDocs I can merge the results and maintain them ordered. I'm using the idea behind
org.apache.lucene.search.ParallelMultiSearcher.search(Weight, Filter, int, Sort).


We could make the classes
org.hibernate.search.query.engine.impl.HSQueryImpl
org.hibernate.search.query.engine.impl.QueryHits
suitable for being extended (changing most private methods to
protected, and also refactor the HSQueryImpl to possibly use a
different type for QueryHits)

As I see now, I don't need changes in QueryHits. The current implementation has a getTopDocs() that solves my problem. On HSQueryImpl, I need the QueryHits that could be created by HSQueryImpl.getQueryHits( buildSearcher(), calculateTopDocsRetrievalSize()). So, changing these three methods to protected or creating a getQueryHits() that hides them, would solve my problem.

and then making sure that it's possible to have
org.hibernate.search.spi.SearchFactoryIntegrator.createHSQuery()
return a custom type.

This could be a little hard, no? I don't wanna create problems to you!

If you can't make these changes, I can find another way. I just want to create a query and get the TopDocs. This could be done in another way, probably writing more code....

And yes, I have the source code!

Let me know what you think.

Thanks!

Israel

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Sanne Grinovero
Hi,
answering inline:

2011/4/11 Israel Lacerra <[hidden email]>:
> Hi Sanne,
>
> When I create a LazyIterator, every node makes a local normal query and
> thern I return the TopDocs to the requester node. With TopDocs I can merge
> the results and maintain them ordered. I'm using the idea behind
> org.apache.lucene.search.ParallelMultiSearcher.search(Weight, Filter, int,
> Sort).

sure, I remember the idea.

>
>
>> We could make the classes
>> org.hibernate.search.query.engine.impl.HSQueryImpl
>> org.hibernate.search.query.engine.impl.QueryHits
>> suitable for being extended (changing most private methods to
>> protected, and also refactor the HSQueryImpl to possibly use a
>> different type for QueryHits)
>
> As I see now, I don't need changes in QueryHits. The current implementation
> has a getTopDocs() that solves my problem. On HSQueryImpl, I need the
> QueryHits that could be created by HSQueryImpl.getQueryHits(
> buildSearcher(), calculateTopDocsRetrievalSize()). So, changing these three
> methods to protected or creating a getQueryHits() that hides them, would
> solve my problem.

Yes it's not hard, I was looking into making those methods protected,
we can also easily destructure some methods in smaller ones so that
you can override specific factories, like the QueryHits in case you
need.
Then instead of making a HSQueryImpl you use your own extension. It
would still be nice to hide your magic behind the HSQuery interface.
>
>> and then making sure that it's possible to have
>> org.hibernate.search.spi.SearchFactoryIntegrator.createHSQuery()
>> return a custom type.
>
> This could be a little hard, no? I don't wanna create problems to you!
It's not a problem, actually as mentioned above you can avoid invoking
the method. so it's very simple :)

>
> If you can't make these changes, I can find another way. I just want to
> create a query and get the TopDocs. This could be done in another way,
> probably writing more code....
I hope we find a way to write less code actually ;)
exposing TopDocs on the HSQuery doesn't seem right, let me know if the
extension points would work. you can change the code directly and play
with it to see how it works; when it does, send us a pull request so
we can discuss this to be merged on Search very soon.

>
> And yes, I have the source code!
>
> Let me know what you think.
>
> Thanks!
>
> Israel
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Manik Surtani
In reply to this post by Sanne Grinovero-2

On 4 Apr 2011, at 18:24, Sanne Grinovero wrote:


- getBasicQuery methods are deprecated. That's from the previous design right? Was it a tech preview or was it named as full fledged API?
In the former case I'd get rid of them.

Yes that was a tech preview, I'll remove them, but maybe better keep
them for a single release?
There's a grails plugin which exposes these methods.

Mark them as deprecated and ping the maintainer of the grails plugin?


- I'm not a big fan of the constructor approach to get QueryFactory for various reasons (including the fact that it forces a concrete type and no interfaces) but it seems to be an ISPN-wide design decision
That's definitely something that could be improved in a Seam Infinispan module.

I knew you where going to say that :) but yes this seems to be more
Infinispan-style.
I've considered something like a Search.getQueryFactory(Cache c) .. WDYT?

+1.

This wouldn't return a "FullTextCache" but the manager, I see no
reason to delegate a cache like we do with a o.h.Session.

- I'd rename QueryFactory to something else as conceptually it's more an entry point to anything related to Hibernate Search, potentially indexing, stats etc: maybe SearchManager, SearchProvider, GridSearcher, TheGridReaper?
right!
which one? I like the first three.

SearchManager would be inline with other Infinispan components.  QueryManager perhaps?

- is there an easy metaconfig to ask ISPN to store the HSearch indexes in ISPN :)
No, but agree that would be awesome. would require changes though in
all involved projects:
1) H.S. would bee to be able to start on a DirectoryProvider instance we pass in
2) Infinispan Directory module (the submodule of H.S.) would need to
be able to reconfigure an existing cachemanager configuration during
the startup
3) Infinispan Core needs to be adapted for the additional configuration fields
currently it's not that hard, just a single property:

<indexing enabled="true" indexLocalOnly="true">
  <properties>
     <property
name="hibernate.search.default.directory_provider" value="infinispan" />

Could this be the default, if no other directory provider is specified?

     </properties>
</indexing>

downside is that it will start a second cachemanager with a second
JGroups channel, instead of reusing the same.

Could it not use the same cache manager?



- you can't do cross Cache queries today. Is that expected? for me CacheManager.getGridSearcher almost makes sense.
Very interesting concept. I'll see how that fits in the code; I can't
add methods to the CacheManager itself, but maybe I can make sure that
interceptors registered on different caches share the same
MutableSearchFactory.

Perhaps something for SearchManager/QueryManager... 

Cheers
Manik
--
Manik Surtani

Lead, Infinispan




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] new Infinispan Query API - ISPN-194

Thomas P. Fuller
Re: Grails plugin:

"Mark them as deprecated and ping the maintainer of the grails plugin?"

I've finished the refactoring for Pagoa today and my unit tests are passing at 100%.

I've kept the query method which takes a field and search param and this just delegates to the other query method which takes a Lucene Query param.

One issue I discovered when upgrading was that the version of Hibernate that Infinispan Pagoa uses is hibernate-core-3.6.2.Final.jar whereas Grails 1.3.7 uses hibernate-core-3.3.1.GA.jar. I plan to contact the Grails developers tomorrow and ask a few questions about this.

T
 
 
Thomas P. Fuller, MSc
Managing Director

Coherent Logic Limited
High Performance Software Engineering

[hidden email]
IM: thospfuller (Yahoo)

Registered in England, #05560634



145-157 St. John Street
London, EC1V 4PY United Kingdom

work:  44.[0]207.788.7654
mobile:  44.[0]781.828.7465



From: Manik Surtani <[hidden email]>
To: infinispan -Dev List <[hidden email]>
Cc: Hardy Ferentschik <[hidden email]>
Sent: Wed, 20 April, 2011 18:51:39
Subject: Re: [infinispan-dev] new Infinispan Query API - ISPN-194


On 4 Apr 2011, at 18:24, Sanne Grinovero wrote:


- getBasicQuery methods are deprecated. That's from the previous design right? Was it a tech preview or was it named as full fledged API?
In the former case I'd get rid of them.

Yes that was a tech preview, I'll remove them, but maybe better keep
them for a single release?
There's a grails plugin which exposes these methods.

Mark them as deprecated and ping the maintainer of the grails plugin?

I've finished the refactoring for Pagoa today and my unit tests are passing.

I've kept the query method



- I'm not a big fan of the constructor approach to get QueryFactory for various reasons (including the fact that it forces a concrete type and no interfaces) but it seems to be an ISPN-wide design decision
That's definitely something that could be improved in a Seam Infinispan module.

I knew you where going to say that :) but yes this seems to be more
Infinispan-style.
I've considered something like a Search.getQueryFactory(Cache c) .. WDYT?

+1.

This wouldn't return a "FullTextCache" but the manager, I see no
reason to delegate a cache like we do with a o.h.Session.

- I'd rename QueryFactory to something else as conceptually it's more an entry point to anything related to Hibernate Search, potentially indexing, stats etc: maybe SearchManager, SearchProvider, GridSearcher, TheGridReaper?
right!
which one? I like the first three.

SearchManager would be inline with other Infinispan components.  QueryManager perhaps?

- is there an easy metaconfig to ask ISPN to store the HSearch indexes in ISPN :)
No, but agree that would be awesome. would require changes though in
all involved projects:
1) H.S. would bee to be able to start on a DirectoryProvider instance we pass in
2) Infinispan Directory module (the submodule of H.S.) would need to
be able to reconfigure an existing cachemanager configuration during
the startup
3) Infinispan Core needs to be adapted for the additional configuration fields
currently it's not that hard, just a single property:

<indexing enabled="true" indexLocalOnly="true">
  <properties>
     <property
name="hibernate.search.default.directory_provider" value="infinispan" />

Could this be the default, if no other directory provider is specified?

     </properties>
</indexing>

downside is that it will start a second cachemanager with a second
JGroups channel, instead of reusing the same.

Could it not use the same cache manager?



- you can't do cross Cache queries today. Is that expected? for me CacheManager.getGridSearcher almost makes sense.
Very interesting concept. I'll see how that fits in the code; I can't
add methods to the CacheManager itself, but maybe I can make sure that
interceptors registered on different caches share the same
MutableSearchFactory.

Perhaps something for SearchManager/QueryManager... 

Cheers
Manik
--
Manik Surtani

Lead, Infinispan




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev