[infinispan-dev] Some memory usage data points of the Hibernate Search / Lucene query engine.

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] Some memory usage data points of the Hibernate Search / Lucene query engine.

Sanne Grinovero-3
Hi all,
at our last meeting we had some chats about heap usage at runtime.

I wasn't actually investigating this, but since I was running some
tests now I thought it was interesting to share the following figures
about the Hibernate Search / Lucene query engine.

# Index size
My test is running with an off-heap index of about 5 million entries,
which translates to about 170MB of disk space.

# Query type
We're running a large query which matches every single one of the 5
million entries, and sorting the results on a Float property.

# TLAB usage
Running such a query takes 2.78 MB of TLAB; this data is never
promoted so it's "cheap" to collect as there's no much interaction
with other threads.

# Beyond TLAB
Allocating zero bytes ;)

In terms of memory usage this looks quite good; kudos to the Lucene
team of course as they do the heavy lifting, but I'm proud of the
Hibernate Search team as well as we don't add significant overhead.
We still have some work to do, as while memory looks good the sorting
on Floats is still very heavy.. but we'll figure it out.

Looking forward to see similar metrics from get/put tests on Infinispan ;-)

N.B. this is an "off heap" index, but even if I use an "on heap"
index, above figures are pretty much the same, and the performance
doesn't get higher, but slower.

Thanks,
Sanne
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Some memory usage data points of the Hibernate Search / Lucene query engine.

Emmanuel Bernard
To put some context, that's a very unfavorable query for Lucene (sort by
data other than score, retrieve all the data set).

On Mon 16-10-24 17:42, Sanne Grinovero wrote:

>Hi all,
>at our last meeting we had some chats about heap usage at runtime.
>
>I wasn't actually investigating this, but since I was running some
>tests now I thought it was interesting to share the following figures
>about the Hibernate Search / Lucene query engine.
>
># Index size
>My test is running with an off-heap index of about 5 million entries,
>which translates to about 170MB of disk space.
>
># Query type
>We're running a large query which matches every single one of the 5
>million entries, and sorting the results on a Float property.
>
># TLAB usage
>Running such a query takes 2.78 MB of TLAB; this data is never
>promoted so it's "cheap" to collect as there's no much interaction
>with other threads.
>
># Beyond TLAB
>Allocating zero bytes ;)
>
>In terms of memory usage this looks quite good; kudos to the Lucene
>team of course as they do the heavy lifting, but I'm proud of the
>Hibernate Search team as well as we don't add significant overhead.
>We still have some work to do, as while memory looks good the sorting
>on Floats is still very heavy.. but we'll figure it out.
>
>Looking forward to see similar metrics from get/put tests on Infinispan ;-)
>
>N.B. this is an "off heap" index, but even if I use an "on heap"
>index, above figures are pretty much the same, and the performance
>doesn't get higher, but slower.
>
>Thanks,
>Sanne
>_______________________________________________
>infinispan-dev mailing list
>[hidden email]
>https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev