[infinispan-dev] Data Container configuration changes

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[infinispan-dev] Data Container configuration changes

William Burns-3
I have been working on adding in off heap support for a given cache.  I wanted to check in and let you all know what I was thinking for the configuration and changes that would come about with it.

TLDR;
New config under data container to enable off heap, StoreAsBinary removed, Equivalence removed

First I was planning on adding new sub elements of data container.  These would be instance, binary and off-heap.  Only of the three could be picked as they are mutually exclusive.  Instance is as we operate now where we store the instance of the object passed to us.  Binary is essentially what we have now that is called storeAsBinary with both keys and values converted.  Lastly off-heap would store the entry as a byte[] store completely in native memory.

Example:

 <data-container>
   <off-heap/>
  </data-container>

The reason it is a subelement instead of a property is because off-heap will most likely require some additional configuration to tell how many entries to store in the a bucket (think non resizing HashMap).

With these changes storeAsBinary becomes redundant, so I was planning on removing this configuration completely.  I would rather remove since this is 9.0 and not deprecate.  As far as I know nobody really used it before.

Also another side effect is I was removing all of the Equivalence classes.  I am not sure if I can plainly remove them since they have lived in commons for quite a while, but it would be best if I could, although I am fine deprecating.  In its place the instance setting for data-container will always wrap byte[] to satisfy equals and hashCode methods.

Any feedback would be appreciated.

Thanks,
 - Will

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Data Container configuration changes

Pedro Ruivo-2


On 26-10-2016 23:06, William Burns wrote:

> I have been working on adding in off heap support for a given cache.  I
> wanted to check in and let you all know what I was thinking for the
> configuration and changes that would come about with it.
>
> TLDR;
> New config under data container to enable off heap, StoreAsBinary
> removed, Equivalence removed
>
> First I was planning on adding new sub elements of data container.
> These would be instance, binary and off-heap.  Only of the three could
> be picked as they are mutually exclusive.  Instance is as we operate now
> where we store the instance of the object passed to us.  Binary is
> essentially what we have now that is called storeAsBinary with both keys
> and values converted.  Lastly off-heap would store the entry as a byte[]
> store completely in native memory.

I prefer 'object' instead of 'instance'.

Are you also planning to remove the expiration and/or eviction
configuration elements and set them in the data-container sub elements?


>
> Example:
>
>  <data-container>
>    <off-heap/>
>   </data-container>
>
> The reason it is a subelement instead of a property is because off-heap
> will most likely require some additional configuration to tell how many
> entries to store in the a bucket (think non resizing HashMap).
>
> With these changes storeAsBinary becomes redundant, so I was planning on
> removing this configuration completely.  I would rather remove since
> this is 9.0 and not deprecate.  As far as I know nobody really used it
> before.
>
> Also another side effect is I was removing all of the Equivalence
> classes.  I am not sure if I can plainly remove them since they have
> lived in commons for quite a while, but it would be best if I could,
> although I am fine deprecating.  In its place the instance setting for
> data-container will always wrap byte[] to satisfy equals and hashCode
> methods.
>
> Any feedback would be appreciated.
>
> Thanks,
>  - Will
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Data Container configuration changes

William Burns-3


On Thu, Oct 27, 2016 at 4:56 AM Pedro Ruivo <[hidden email]> wrote:


On 26-10-2016 23:06, William Burns wrote:
> I have been working on adding in off heap support for a given cache.  I
> wanted to check in and let you all know what I was thinking for the
> configuration and changes that would come about with it.
>
> TLDR;
> New config under data container to enable off heap, StoreAsBinary
> removed, Equivalence removed
>
> First I was planning on adding new sub elements of data container.
> These would be instance, binary and off-heap.  Only of the three could
> be picked as they are mutually exclusive.  Instance is as we operate now
> where we store the instance of the object passed to us.  Binary is
> essentially what we have now that is called storeAsBinary with both keys
> and values converted.  Lastly off-heap would store the entry as a byte[]
> store completely in native memory.

I prefer 'object' instead of 'instance'.

Sounds fine with me.
 

Are you also planning to remove the expiration and/or eviction
configuration elements and set them in the data-container sub elements?

I wasn't planning on touching those.  But now that you mention it, eviction only makes sense in the data container, where as expiration is container and cache store.  And taking this further storeAsBinary is both as well, only off-heap is container only.  I wonder if instead we should have a separate storage element at the same level as data-container.  I can see it either way.
 


>
> Example:
>
>  <data-container>
>    <off-heap/>
>   </data-container>
>
> The reason it is a subelement instead of a property is because off-heap
> will most likely require some additional configuration to tell how many
> entries to store in the a bucket (think non resizing HashMap).
>
> With these changes storeAsBinary becomes redundant, so I was planning on
> removing this configuration completely.  I would rather remove since
> this is 9.0 and not deprecate.  As far as I know nobody really used it
> before.
>
> Also another side effect is I was removing all of the Equivalence
> classes.  I am not sure if I can plainly remove them since they have
> lived in commons for quite a while, but it would be best if I could,
> although I am fine deprecating.  In its place the instance setting for
> data-container will always wrap byte[] to satisfy equals and hashCode
> methods.
>
> Any feedback would be appreciated.
>
> Thanks,
>  - Will
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Data Container configuration changes

Tristan Tarrant-2
In reply to this post by William Burns-3
On 27/10/16 00:06, William Burns wrote:
> The reason it is a subelement instead of a property is because off-heap
> will most likely require some additional configuration to tell how many
> entries to store in the a bucket (think non resizing HashMap).

Aside from that, it is best to use elements when possible over
attributes, as XSD allows more control over constraints and exclusivity.

> With these changes storeAsBinary becomes redundant, so I was planning on
> removing this configuration completely.  I would rather remove since
> this is 9.0 and not deprecate.  As far as I know nobody really used it
> before.

While this can be easily removed from the parser, since it supports
versioning, I'm not sure I'd want to see it go away from the API. We
haven't been very good at doing this in the past and I'd rather go down
the conservative route.

> Also another side effect is I was removing all of the Equivalence
> classes.  I am not sure if I can plainly remove them since they have
> lived in commons for quite a while, but it would be best if I could,
> although I am fine deprecating.  In its place the instance setting for
> data-container will always wrap byte[] to satisfy equals and hashCode
> methods.

Equivalence is only really used in the context of compatibility mode and
that has uses (hybrid servers). How would that continue working (until
we get transcoding) ?

Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Data Container configuration changes

Pedro Ruivo-2
In reply to this post by William Burns-3


On 27-10-2016 14:20, William Burns wrote:

>
>
> On Thu, Oct 27, 2016 at 4:56 AM Pedro Ruivo <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>
>     On 26-10-2016 23:06, William Burns wrote:
>     > I have been working on adding in off heap support for a given
>     cache.  I
>     > wanted to check in and let you all know what I was thinking for the
>     > configuration and changes that would come about with it.
>     >
>     > TLDR;
>     > New config under data container to enable off heap, StoreAsBinary
>     > removed, Equivalence removed
>     >
>     > First I was planning on adding new sub elements of data container.
>     > These would be instance, binary and off-heap.  Only of the three could
>     > be picked as they are mutually exclusive.  Instance is as we
>     operate now
>     > where we store the instance of the object passed to us.  Binary is
>     > essentially what we have now that is called storeAsBinary with
>     both keys
>     > and values converted.  Lastly off-heap would store the entry as a
>     byte[]
>     > store completely in native memory.
>
>     I prefer 'object' instead of 'instance'.
>
>
> Sounds fine with me.
>
>
>
>     Are you also planning to remove the expiration and/or eviction
>     configuration elements and set them in the data-container sub elements?
>
>
> I wasn't planning on touching those.  But now that you mention it,
> eviction only makes sense in the data container, where as expiration is
> container and cache store.  And taking this further storeAsBinary is
> both as well, only off-heap is container only.  I wonder if instead we
> should have a separate storage element at the same level as
> data-container.  I can see it either way.

Let me know if this makes sense:

<expiration> //no changes here
<memory evictionStrategy=... evictionPolicy=...>

//one of the following
     <on-heap max-entries=.../>
     <on-heap-binary max-size=.../>
     <off-heap ...max-size? and off-heap config.../>

</memory>
<persistence> //no changes here

wdyt?

>
>
>
>
>     >
>     > Example:
>     >
>     >  <data-container>
>     >    <off-heap/>
>     >   </data-container>
>     >
>     > The reason it is a subelement instead of a property is because
>     off-heap
>     > will most likely require some additional configuration to tell how
>     many
>     > entries to store in the a bucket (think non resizing HashMap).
>     >
>     > With these changes storeAsBinary becomes redundant, so I was
>     planning on
>     > removing this configuration completely.  I would rather remove since
>     > this is 9.0 and not deprecate.  As far as I know nobody really used it
>     > before.
>     >
>     > Also another side effect is I was removing all of the Equivalence
>     > classes.  I am not sure if I can plainly remove them since they have
>     > lived in commons for quite a while, but it would be best if I could,
>     > although I am fine deprecating.  In its place the instance setting for
>     > data-container will always wrap byte[] to satisfy equals and hashCode
>     > methods.
>     >
>     > Any feedback would be appreciated.
>     >
>     > Thanks,
>     >  - Will
>     >
>     >
>     > _______________________________________________
>     > infinispan-dev mailing list
>     > [hidden email] <mailto:[hidden email]>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     >
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Data Container configuration changes

William Burns-3
In reply to this post by Tristan Tarrant-2


On Thu, Oct 27, 2016 at 9:40 AM Tristan Tarrant <[hidden email]> wrote:
On 27/10/16 00:06, William Burns wrote:
> The reason it is a subelement instead of a property is because off-heap
> will most likely require some additional configuration to tell how many
> entries to store in the a bucket (think non resizing HashMap).

Aside from that, it is best to use elements when possible over
attributes, as XSD allows more control over constraints and exclusivity.

> With these changes storeAsBinary becomes redundant, so I was planning on
> removing this configuration completely.  I would rather remove since
> this is 9.0 and not deprecate.  As far as I know nobody really used it
> before.

While this can be easily removed from the parser, since it supports
versioning, I'm not sure I'd want to see it go away from the API. We
haven't been very good at doing this in the past and I'd rather go down
the conservative route.

Not sure I understand, so you are saying to remove it from the parser/xsd but keep it in the programmatic configuration?  And just to clarify the new binary element would replace storeAsBinary.
 

> Also another side effect is I was removing all of the Equivalence
> classes.  I am not sure if I can plainly remove them since they have
> lived in commons for quite a while, but it would be best if I could,
> although I am fine deprecating.  In its place the instance setting for
> data-container will always wrap byte[] to satisfy equals and hashCode
> methods.

Equivalence is only really used in the context of compatibility mode and
that has uses (hybrid servers). How would that continue working (until
we get transcoding) ?

Equivalence is only used when you are storing byte[] objects, which Infinispan would support directly with no configuration with the new changes (we would wrap the byte[] to ensure equals/hashCode).  Compatibility would work the same way it does right now, I just had to add in some handling for the wrapper in some parts of code.
 

Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Data Container configuration changes

Tristan Tarrant-2
On 27/10/16 16:01, William Burns wrote:
> Not sure I understand, so you are saying to remove it from the
> parser/xsd but keep it in the programmatic configuration?  And just to
> clarify the new binary element would replace storeAsBinary.

Yes. The parser, when confronted with a pre 9.0 schema, would print out
a warning, but it would use the new binary element internally.

> Equivalence is only used when you are storing byte[] objects, which
> Infinispan would support directly with no configuration with the new
> changes (we would wrap the byte[] to ensure equals/hashCode).
> Compatibility would work the same way it does right now, I just had to
> add in some handling for the wrapper in some parts of code.

Great. Go ahead.

Tristan


--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Data Container configuration changes

William Burns-3
In reply to this post by Pedro Ruivo-2


On Thu, Oct 27, 2016 at 9:56 AM Pedro Ruivo <[hidden email]> wrote:


On 27-10-2016 14:20, William Burns wrote:
>
>
> On Thu, Oct 27, 2016 at 4:56 AM Pedro Ruivo <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>
>     On 26-10-2016 23:06, William Burns wrote:
>     > I have been working on adding in off heap support for a given
>     cache.  I
>     > wanted to check in and let you all know what I was thinking for the
>     > configuration and changes that would come about with it.
>     >
>     > TLDR;
>     > New config under data container to enable off heap, StoreAsBinary
>     > removed, Equivalence removed
>     >
>     > First I was planning on adding new sub elements of data container.
>     > These would be instance, binary and off-heap.  Only of the three could
>     > be picked as they are mutually exclusive.  Instance is as we
>     operate now
>     > where we store the instance of the object passed to us.  Binary is
>     > essentially what we have now that is called storeAsBinary with
>     both keys
>     > and values converted.  Lastly off-heap would store the entry as a
>     byte[]
>     > store completely in native memory.
>
>     I prefer 'object' instead of 'instance'.
>
>
> Sounds fine with me.
>
>
>
>     Are you also planning to remove the expiration and/or eviction
>     configuration elements and set them in the data-container sub elements?
>
>
> I wasn't planning on touching those.  But now that you mention it,
> eviction only makes sense in the data container, where as expiration is
> container and cache store.  And taking this further storeAsBinary is
> both as well, only off-heap is container only.  I wonder if instead we
> should have a separate storage element at the same level as
> data-container.  I can see it either way.

Let me know if this makes sense:

<expiration> //no changes here
<memory evictionStrategy=... evictionPolicy=...>

//one of the following
     <on-heap max-entries=.../>
     <on-heap-binary max-size=.../>
     <off-heap ...max-size? and off-heap config.../>

</memory>
<persistence> //no changes here

wdyt?

Seems fine to me.  And to be honest I forgot to mention this but EvictionStrategy and EvictionPolicy are completely redundant now.  Policy has been for a while as we always used the same thread and Strategy is only Caffeine and for off heap I was thinking of a simple LRU.

This means that the data container element would be removed in favor of "memory"?  The reason being is that equivalence will be gone and afaik we never really supported a custom data container (eviction wouldn't work with it and neither would off heap).  In that case why not just keep using data container element?  
 
>
>
>
>
>     >
>     > Example:
>     >
>     >  <data-container>
>     >    <off-heap/>
>     >   </data-container>
>     >
>     > The reason it is a subelement instead of a property is because
>     off-heap
>     > will most likely require some additional configuration to tell how
>     many
>     > entries to store in the a bucket (think non resizing HashMap).
>     >
>     > With these changes storeAsBinary becomes redundant, so I was
>     planning on
>     > removing this configuration completely.  I would rather remove since
>     > this is 9.0 and not deprecate.  As far as I know nobody really used it
>     > before.
>     >
>     > Also another side effect is I was removing all of the Equivalence
>     > classes.  I am not sure if I can plainly remove them since they have
>     > lived in commons for quite a while, but it would be best if I could,
>     > although I am fine deprecating.  In its place the instance setting for
>     > data-container will always wrap byte[] to satisfy equals and hashCode
>     > methods.
>     >
>     > Any feedback would be appreciated.
>     >
>     > Thanks,
>     >  - Will
>     >
>     >
>     > _______________________________________________
>     > infinispan-dev mailing list
>     > [hidden email] <mailto:[hidden email]>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     >
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Data Container configuration changes

Radim Vansa
On 10/27/2016 05:26 PM, William Burns wrote:

>
>
> On Thu, Oct 27, 2016 at 9:56 AM Pedro Ruivo <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>
>     On 27-10-2016 14:20, William Burns wrote:
>     >
>     >
>     > On Thu, Oct 27, 2016 at 4:56 AM Pedro Ruivo
>     <[hidden email] <mailto:[hidden email]>
>     > <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >
>     >
>     >
>     >     On 26-10-2016 23:06, William Burns wrote:
>     >     > I have been working on adding in off heap support for a given
>     >     cache.  I
>     >     > wanted to check in and let you all know what I was
>     thinking for the
>     >     > configuration and changes that would come about with it.
>     >     >
>     >     > TLDR;
>     >     > New config under data container to enable off heap,
>     StoreAsBinary
>     >     > removed, Equivalence removed
>     >     >
>     >     > First I was planning on adding new sub elements of data
>     container.
>     >     > These would be instance, binary and off-heap. Only of the
>     three could
>     >     > be picked as they are mutually exclusive. Instance is as we
>     >     operate now
>     >     > where we store the instance of the object passed to us.
>     Binary is
>     >     > essentially what we have now that is called storeAsBinary with
>     >     both keys
>     >     > and values converted.  Lastly off-heap would store the
>     entry as a
>     >     byte[]
>     >     > store completely in native memory.
>     >
>     >     I prefer 'object' instead of 'instance'.
>     >
>     >
>     > Sounds fine with me.
>     >
>     >
>     >
>     >     Are you also planning to remove the expiration and/or eviction
>     >     configuration elements and set them in the data-container
>     sub elements?
>     >
>     >
>     > I wasn't planning on touching those.  But now that you mention it,
>     > eviction only makes sense in the data container, where as
>     expiration is
>     > container and cache store.  And taking this further storeAsBinary is
>     > both as well, only off-heap is container only.  I wonder if
>     instead we
>     > should have a separate storage element at the same level as
>     > data-container.  I can see it either way.
>
>     Let me know if this makes sense:
>
>     <expiration> //no changes here
>     <memory evictionStrategy=... evictionPolicy=...>
>
>     //one of the following
>          <on-heap max-entries=.../>
>          <on-heap-binary max-size=.../>
>          <off-heap ...max-size? and off-heap config.../>
>
>     </memory>
>     <persistence> //no changes here
>
>     wdyt?
>

While I prefer "on-heap" instead of "object" or "instance", I don't
think that "binary" should be its own element. Are there any attributes
specific to that (do you plan to have keys="false" values="true"? I
guess not). "on-heap" and "off-heap" is a binary ( :-) ) option,
"on-heap-binary" is just a flavour of "on-heap".

For comparison, HC uses
<in-memory-format>OBJECT|BINARY|NATIVE</in-memory-format> where "NATIVE"
means off-heap. I like "format= OBJECT|BINARY" as a child of on-heap,
either as attribute or element. I haven't found similar settings in
Coherence - seems that they store data in a serialized form only when
they persist to disk/flash or offload to non-managed memory.

Regarding Equivalence: can't we wrap objects in a similar way we do with
byte[] if Equivalance (different from AnyInstance) is defined? I can
definitely see use case when the hashCode() is not very well defined and
user can't change the class - he has to wrap the objects themselves.

R.

>
> Seems fine to me.  And to be honest I forgot to mention this but
> EvictionStrategy and EvictionPolicy are completely redundant now.  
> Policy has been for a while as we always used the same thread and
> Strategy is only Caffeine and for off heap I was thinking of a simple LRU.
>
> This means that the data container element would be removed in favor
> of "memory"?  The reason being is that equivalence will be gone and
> afaik we never really supported a custom data container (eviction
> wouldn't work with it and neither would off heap).  In that case why
> not just keep using data container element?
>
>     >
>     >
>     >
>     >
>     >     >
>     >     > Example:
>     >     >
>     >     >  <data-container>
>     >     >    <off-heap/>
>     >     >   </data-container>
>     >     >
>     >     > The reason it is a subelement instead of a property is because
>     >     off-heap
>     >     > will most likely require some additional configuration to
>     tell how
>     >     many
>     >     > entries to store in the a bucket (think non resizing HashMap).
>     >     >
>     >     > With these changes storeAsBinary becomes redundant, so I was
>     >     planning on
>     >     > removing this configuration completely.  I would rather
>     remove since
>     >     > this is 9.0 and not deprecate.  As far as I know nobody
>     really used it
>     >     > before.
>     >     >
>     >     > Also another side effect is I was removing all of the
>     Equivalence
>     >     > classes.  I am not sure if I can plainly remove them since
>     they have
>     >     > lived in commons for quite a while, but it would be best
>     if I could,
>     >     > although I am fine deprecating.  In its place the instance
>     setting for
>     >     > data-container will always wrap byte[] to satisfy equals
>     and hashCode
>     >     > methods.
>     >     >
>     >     > Any feedback would be appreciated.
>     >     >
>     >     > Thanks,
>     >     >  - Will
>     >     >
>     >     >
>     >     > _______________________________________________
>     >     > infinispan-dev mailing list
>     >     > [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     >     >
>     >     _______________________________________________
>     >     infinispan-dev mailing list
>     > [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     >
>     >
>     >
>     > _______________________________________________
>     > infinispan-dev mailing list
>     > [hidden email]
>     <mailto:[hidden email]>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     >
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Data Container configuration changes

William Burns-3
Since I have had some time to think it over I like Pedro's layout, unless someone has a strong objection.

Responses inline.

On Mon, Oct 31, 2016 at 4:51 AM Radim Vansa <[hidden email]> wrote:
On 10/27/2016 05:26 PM, William Burns wrote:
>
>
> On Thu, Oct 27, 2016 at 9:56 AM Pedro Ruivo <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>
>     On 27-10-2016 14:20, William Burns wrote:
>     >
>     >
>     > On Thu, Oct 27, 2016 at 4:56 AM Pedro Ruivo
>     <[hidden email] <mailto:[hidden email]>
>     > <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >
>     >
>     >
>     >     On 26-10-2016 23:06, William Burns wrote:
>     >     > I have been working on adding in off heap support for a given
>     >     cache.  I
>     >     > wanted to check in and let you all know what I was
>     thinking for the
>     >     > configuration and changes that would come about with it.
>     >     >
>     >     > TLDR;
>     >     > New config under data container to enable off heap,
>     StoreAsBinary
>     >     > removed, Equivalence removed
>     >     >
>     >     > First I was planning on adding new sub elements of data
>     container.
>     >     > These would be instance, binary and off-heap. Only of the
>     three could
>     >     > be picked as they are mutually exclusive. Instance is as we
>     >     operate now
>     >     > where we store the instance of the object passed to us.
>     Binary is
>     >     > essentially what we have now that is called storeAsBinary with
>     >     both keys
>     >     > and values converted.  Lastly off-heap would store the
>     entry as a
>     >     byte[]
>     >     > store completely in native memory.
>     >
>     >     I prefer 'object' instead of 'instance'.
>     >
>     >
>     > Sounds fine with me.
>     >
>     >
>     >
>     >     Are you also planning to remove the expiration and/or eviction
>     >     configuration elements and set them in the data-container
>     sub elements?
>     >
>     >
>     > I wasn't planning on touching those.  But now that you mention it,
>     > eviction only makes sense in the data container, where as
>     expiration is
>     > container and cache store.  And taking this further storeAsBinary is
>     > both as well, only off-heap is container only.  I wonder if
>     instead we
>     > should have a separate storage element at the same level as
>     > data-container.  I can see it either way.
>
>     Let me know if this makes sense:
>
>     <expiration> //no changes here
>     <memory evictionStrategy=... evictionPolicy=...>

There wouldn't be any elements here.  The strategy would always be TinyLFU as this is all Caffeine supports.  The thread policy though is interesting, I would say for now don't let it be configured and it will run in the same thead.  However Caffeine does have async eviction which we could utilize in the future.  But I think we can look into that later and not part of this.
 
>
>     //one of the following
>          <on-heap max-entries=.../>
>          <on-heap-binary max-size=.../>
>          <off-heap ...max-size? and off-heap config.../>

I think I like the naming of the elements as OBJECT, BINARY and OFF-HEAP/NATIVE that Radim mentioned.

I am thinking for now OBJECT would only support max-entries and the other 2 could support both max-entries or max-size.
 
>
>     </memory>
>     <persistence> //no changes here
>
>     wdyt?
>

While I prefer "on-heap" instead of "object" or "instance", I don't
think that "binary" should be its own element. Are there any attributes
specific to that (do you plan to have keys="false" values="true"? I
guess not). "on-heap" and "off-heap" is a binary ( :-) ) option,
"on-heap-binary" is just a flavour of "on-heap".

Right now the plan is to remove keys and values and always do both.  But we may have to change that later, I don't know.  Having separate elements gives us more flexibility which I like.
 

For comparison, HC uses
<in-memory-format>OBJECT|BINARY|NATIVE</in-memory-format> where "NATIVE"
means off-heap. I like "format= OBJECT|BINARY" as a child of on-heap,
either as attribute or element. I haven't found similar settings in
Coherence - seems that they store data in a serialized form only when
they persist to disk/flash or offload to non-managed memory.

Regarding Equivalence: can't we wrap objects in a similar way we do with
byte[] if Equivalance (different from AnyInstance) is defined? I can
definitely see use case when the hashCode() is not very well defined and
user can't change the class - he has to wrap the objects themselves.

Hrmm we could add that back in if it is really needed.  As far as I know though no one has really used this.  I would rather make it more minimalistic to begin with and add back on as we find we need features.
 

R.

>
> Seems fine to me.  And to be honest I forgot to mention this but
> EvictionStrategy and EvictionPolicy are completely redundant now.
> Policy has been for a while as we always used the same thread and
> Strategy is only Caffeine and for off heap I was thinking of a simple LRU.
>
> This means that the data container element would be removed in favor
> of "memory"?  The reason being is that equivalence will be gone and
> afaik we never really supported a custom data container (eviction
> wouldn't work with it and neither would off heap).  In that case why
> not just keep using data container element?
>
>     >
>     >
>     >
>     >
>     >     >
>     >     > Example:
>     >     >
>     >     >  <data-container>
>     >     >    <off-heap/>
>     >     >   </data-container>
>     >     >
>     >     > The reason it is a subelement instead of a property is because
>     >     off-heap
>     >     > will most likely require some additional configuration to
>     tell how
>     >     many
>     >     > entries to store in the a bucket (think non resizing HashMap).
>     >     >
>     >     > With these changes storeAsBinary becomes redundant, so I was
>     >     planning on
>     >     > removing this configuration completely.  I would rather
>     remove since
>     >     > this is 9.0 and not deprecate.  As far as I know nobody
>     really used it
>     >     > before.
>     >     >
>     >     > Also another side effect is I was removing all of the
>     Equivalence
>     >     > classes.  I am not sure if I can plainly remove them since
>     they have
>     >     > lived in commons for quite a while, but it would be best
>     if I could,
>     >     > although I am fine deprecating.  In its place the instance
>     setting for
>     >     > data-container will always wrap byte[] to satisfy equals
>     and hashCode
>     >     > methods.
>     >     >
>     >     > Any feedback would be appreciated.
>     >     >
>     >     > Thanks,
>     >     >  - Will
>     >     >
>     >     >
>     >     > _______________________________________________
>     >     > infinispan-dev mailing list
>     >     > [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     >     >
>     >     _______________________________________________
>     >     infinispan-dev mailing list
>     > [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     >
>     >
>     >
>     > _______________________________________________
>     > infinispan-dev mailing list
>     > [hidden email]
>     <mailto:[hidden email]>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     >
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...