[infinispan-dev] Late end invalidation delivery stops put from load - ISPN-8114

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[infinispan-dev] Late end invalidation delivery stops put from load - ISPN-8114

Galder Zamarreño
Hey Radim,

Re: https://issues.jboss.org/browse/ISPN-8114

I've been looking at the trace logs of this failure. I've extracted the most interesting parts of this failure into [1].

What happens is that after loading the entries into the cache, the end invalidation message to allow put from loads to succeed does not get executed in time before the put from load is attempted. As a result of this, the put from load does not happen and hence the entry is not loaded into the cache.

The end invalidation message eventually gets through. There's a gap between receiving the JGroups message and the actual execution, but that's due to the delivery mode of the message.

I'm not sure how we should fix this. Options:

1) A thread sleep before loading entries "might work" but for a CI test this could always backfire with the right timing sets.

2) Find a way to hook into the PFLValidator class and only load after we know end invalidation has been received by all nodes.

3) Make end invalidation message sync? This would expensive. Even with async, changing delivery mode might have worked here... but under the right circumstances you could still get the same issue with async.

I'm keen on trying to find a potential solution using 2), but wondered if you have other ideas.

Cheers,

[1] https://gist.github.com/galderz/0bce6dce16de018375e43e25c0cf3913
--
Galder Zamarreño
Infinispan, Red Hat


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Late end invalidation delivery stops put from load - ISPN-8114

Radim Vansa
On 08/02/2017 01:02 PM, Galder Zamarreño wrote:

> Hey Radim,
>
> Re: https://issues.jboss.org/browse/ISPN-8114
>
> I've been looking at the trace logs of this failure. I've extracted the most interesting parts of this failure into [1].
>
> What happens is that after loading the entries into the cache, the end invalidation message to allow put from loads to succeed does not get executed in time before the put from load is attempted. As a result of this, the put from load does not happen and hence the entry is not loaded into the cache.
>
> The end invalidation message eventually gets through. There's a gap between receiving the JGroups message and the actual execution, but that's due to the delivery mode of the message.
>
> I'm not sure how we should fix this. Options:
>
> 1) A thread sleep before loading entries "might work" but for a CI test this could always backfire with the right timing sets.
>
> 2) Find a way to hook into the PFLValidator class and only load after we know end invalidation has been received by all nodes.
>
> 3) Make end invalidation message sync? This would expensive. Even with async, changing delivery mode might have worked here... but under the right circumstances you could still get the same issue with async.

1) is unreliable and wrong from a testsuite perspective, and 3) is
completely wrong (making sth sync just because it's easier to test it
that way).

Spying on PFVL is an option, but I would rather wrap
PerCacheInboundInvocationHandler (I hope that's the correct way). Note
that there's even a TestingUtil.wrapInboundInvocationHandler() helper
method.

If I am missing the complexity, please elaborate.

Radim

> I'm keen on trying to find a potential solution using 2), but wondered if you have other ideas.
>
> Cheers,
>
> [1] https://gist.github.com/galderz/0bce6dce16de018375e43e25c0cf3913
> --
> Galder Zamarreño
> Infinispan, Red Hat
>


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Late end invalidation delivery stops put from load - ISPN-8114

Galder Zamarreño
Thx Radim, I'll look into wrapping PerCacheInboundInvocationHandler.

Cheers,
--
Galder Zamarreño
Infinispan, Red Hat

> On 2 Aug 2017, at 18:41, Radim Vansa <[hidden email]> wrote:
>
> On 08/02/2017 01:02 PM, Galder Zamarreño wrote:
>> Hey Radim,
>>
>> Re: https://issues.jboss.org/browse/ISPN-8114
>>
>> I've been looking at the trace logs of this failure. I've extracted the most interesting parts of this failure into [1].
>>
>> What happens is that after loading the entries into the cache, the end invalidation message to allow put from loads to succeed does not get executed in time before the put from load is attempted. As a result of this, the put from load does not happen and hence the entry is not loaded into the cache.
>>
>> The end invalidation message eventually gets through. There's a gap between receiving the JGroups message and the actual execution, but that's due to the delivery mode of the message.
>>
>> I'm not sure how we should fix this. Options:
>>
>> 1) A thread sleep before loading entries "might work" but for a CI test this could always backfire with the right timing sets.
>>
>> 2) Find a way to hook into the PFLValidator class and only load after we know end invalidation has been received by all nodes.
>>
>> 3) Make end invalidation message sync? This would expensive. Even with async, changing delivery mode might have worked here... but under the right circumstances you could still get the same issue with async.
>
> 1) is unreliable and wrong from a testsuite perspective, and 3) is completely wrong (making sth sync just because it's easier to test it that way).
>
> Spying on PFVL is an option, but I would rather wrap PerCacheInboundInvocationHandler (I hope that's the correct way). Note that there's even a TestingUtil.wrapInboundInvocationHandler() helper method.
>
> If I am missing the complexity, please elaborate.
>
> Radim
>
>> I'm keen on trying to find a potential solution using 2), but wondered if you have other ideas.
>>
>> Cheers,
>>
>> [1] https://gist.github.com/galderz/0bce6dce16de018375e43e25c0cf3913
>> --
>> Galder Zamarreño
>> Infinispan, Red Hat
>>
>
>
> --
> Radim Vansa <[hidden email]>
> JBoss Performance Team


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Late end invalidation delivery stops put from load - ISPN-8114

Galder Zamarreño
FYI: https://github.com/infinispan/infinispan/pull/5350

--
Galder Zamarreño
Infinispan, Red Hat

> On 3 Aug 2017, at 11:34, Galder Zamarreño <[hidden email]> wrote:
>
> Thx Radim, I'll look into wrapping PerCacheInboundInvocationHandler.
>
> Cheers,
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>> On 2 Aug 2017, at 18:41, Radim Vansa <[hidden email]> wrote:
>>
>> On 08/02/2017 01:02 PM, Galder Zamarreño wrote:
>>> Hey Radim,
>>>
>>> Re: https://issues.jboss.org/browse/ISPN-8114
>>>
>>> I've been looking at the trace logs of this failure. I've extracted the most interesting parts of this failure into [1].
>>>
>>> What happens is that after loading the entries into the cache, the end invalidation message to allow put from loads to succeed does not get executed in time before the put from load is attempted. As a result of this, the put from load does not happen and hence the entry is not loaded into the cache.
>>>
>>> The end invalidation message eventually gets through. There's a gap between receiving the JGroups message and the actual execution, but that's due to the delivery mode of the message.
>>>
>>> I'm not sure how we should fix this. Options:
>>>
>>> 1) A thread sleep before loading entries "might work" but for a CI test this could always backfire with the right timing sets.
>>>
>>> 2) Find a way to hook into the PFLValidator class and only load after we know end invalidation has been received by all nodes.
>>>
>>> 3) Make end invalidation message sync? This would expensive. Even with async, changing delivery mode might have worked here... but under the right circumstances you could still get the same issue with async.
>>
>> 1) is unreliable and wrong from a testsuite perspective, and 3) is completely wrong (making sth sync just because it's easier to test it that way).
>>
>> Spying on PFVL is an option, but I would rather wrap PerCacheInboundInvocationHandler (I hope that's the correct way). Note that there's even a TestingUtil.wrapInboundInvocationHandler() helper method.
>>
>> If I am missing the complexity, please elaborate.
>>
>> Radim
>>
>>> I'm keen on trying to find a potential solution using 2), but wondered if you have other ideas.
>>>
>>> Cheers,
>>>
>>> [1] https://gist.github.com/galderz/0bce6dce16de018375e43e25c0cf3913
>>> --
>>> Galder Zamarreño
>>> Infinispan, Red Hat
>>>
>>
>>
>> --
>> Radim Vansa <[hidden email]>
>> JBoss Performance Team
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...