[infinispan-dev] State transfer is transferring the same data twice

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] State transfer is transferring the same data twice

Dan Berindei
Hi guys

While fixing https://issues.jboss.org/browse/ISPN-1243 I found that in
certain cases state transfer will copy the in-memory data twice: first
when copying the in-memory data and then again while copying the
persistent data.

I was getting duplicate key exceptions in
StateTransferManagerImpl.applyPersistentState(), so I changed
StateTransferManagerImpl.applyInMemoryState() to skip writing to the
cache store if persistent state transfer is enabled and passivation is
disabled. That seems to work, but I'm wondering if there are other
cases that I missed.

For 5.1, since we're going to use the rehashing code, we can check if
a key is in memory and not include it again in the persistent state,
but that seems too intrusive to include in 5.0. WDYT?

Cheers
Dan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] State transfer is transferring the same data twice

Manik Surtani

On 15 Jul 2011, at 20:00, Dan Berindei wrote:

> Hi guys
>
> While fixing https://issues.jboss.org/browse/ISPN-1243 I found that in
> certain cases state transfer will copy the in-memory data twice: first
> when copying the in-memory data and then again while copying the
> persistent data.
>
> I was getting duplicate key exceptions in
> StateTransferManagerImpl.applyPersistentState(), so I changed
> StateTransferManagerImpl.applyInMemoryState() to skip writing to the
> cache store if persistent state transfer is enabled and passivation is
> disabled. That seems to work, but I'm wondering if there are other
> cases that I missed.
>
> For 5.1, since we're going to use the rehashing code, we can check if
> a key is in memory and not include it again in the persistent state,
> but that seems too intrusive to include in 5.0. WDYT?

Agreed - too intrusive for 5.0 now.

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] State transfer is transferring the same data twice

Mircea Markus
In reply to this post by Dan Berindei

On 15 Jul 2011, at 20:00, Dan Berindei wrote:

> Hi guys
>
> While fixing https://issues.jboss.org/browse/ISPN-1243 I found that in
> certain cases state transfer will copy the in-memory data twice: first
> when copying the in-memory data and then again while copying the
> persistent data.
>
> I was getting duplicate key exceptions in
> StateTransferManagerImpl.applyPersistentState(), so I changed
> StateTransferManagerImpl.applyInMemoryState() to skip writing to the
> cache store if persistent state transfer is enabled and passivation is
> disabled.
can't you make StateTransferManagerImpl.applyInMemoryState() to always write to the cache store and modify StateTransferManagerImpl.generatePersistentState to skip sending a key if it is already in the memory/DataContainer?
That would require passing a filter to CacheStore.toStrem...

> That seems to work, but I'm wondering if there are other
> cases that I missed.
>
> For 5.1, since we're going to use the rehashing code, we can check if
> a key is in memory and not include it again in the persistent state,
> but that seems too intrusive to include in 5.0. WDYT?
>
> Cheers
> Dan
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] State transfer is transferring the same data twice

Manik Surtani

On 19 Jul 2011, at 17:17, Mircea Markus wrote:

can't you make StateTransferManagerImpl.applyInMemoryState() to always write to the cache store and modify StateTransferManagerImpl.generatePersistentState to skip sending a key if it is already in the memory/DataContainer?
That would require passing a filter to CacheStore.toStrem…

…. and potentially changing a lot of the cache store impls… 



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev