[infinispan-dev] State transfer-related terms

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[infinispan-dev] State transfer-related terms

Radim Vansa
Hi,

I've started (again) working on ISPN-5021 [1], and I'd like to get some
common agreement on few terms. So below I summarize my understanding (or
misunderstanding) of these, please state you opinion, thinking a bit
more generally.

State transfer: whole process beginning with some ST-related event =
node being detected to crash/sending join or leave request, and ending
when this event is processed. When another event happens, the current ST
can either be finished or canceled and then *another* ST can begin.
State transfer is a cluster-wide process, though it cannot be started
and ended absolutely simultaneously.

Rebalance: one phase of ST, when the data transfer occurs.

Data rehash: this is a bit painful point: we have DataRehashEvent where
the name suggest that it is related rather to rebalance, but currently
it fires when CacheTopology.getPendingCH() == null (that means when ST
is complete), and the event itself also looks more like it should be
fired at the end of state transfer. If we have something more to do
after the rebalance, I am not sure how useful is firing that just
because all data has been transferred (but for example before old data
has been wiped out). Should I add another StateTransferEvent event (and
appropriate listeners)? That would break the compatibility with tightly
related implementations...

WDYT?

Radim

[1] https://issues.jboss.org/browse/ISPN-5021


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] State transfer-related terms

William Burns-3


On Thu, Jan 19, 2017 at 11:13 AM Radim Vansa <[hidden email]> wrote:
Hi,

I've started (again) working on ISPN-5021 [1], and I'd like to get some
common agreement on few terms. So below I summarize my understanding (or
misunderstanding) of these, please state you opinion, thinking a bit
more generally.

State transfer: whole process beginning with some ST-related event =
node being detected to crash/sending join or leave request, and ending
when this event is processed. When another event happens, the current ST
can either be finished or canceled and then *another* ST can begin.
State transfer is a cluster-wide process, though it cannot be started
and ended absolutely simultaneously.

Rebalance: one phase of ST, when the data transfer occurs.

Data rehash: this is a bit painful point: we have DataRehashEvent where
the name suggest that it is related rather to rebalance, but currently
it fires when CacheTopology.getPendingCH() == null (that means when ST
is complete), and the event itself also looks more like it should be
fired at the end of state transfer. If we have something more to do
after the rebalance, I am not sure how useful is firing that just
because all data has been transferred (but for example before old data
has been wiped out). Should I add another StateTransferEvent event (and
appropriate listeners)? That would break the compatibility with tightly
related implementations...

The stream retry mechanism currently uses DataRehashedEvent. It is beneficial but not required for it to be notified immediately after all entries have been transferred but before any have been removed. This shortens the window for when a stream operation is retried since it has to be sure that all entries for a given segment are present, but we don't care if entries for a non owned segment are present. But the good thing is that the code shouldn't require much work at all to be changed if we needed to.

I am personally not keen on having another event that is like DataRehashedEvent, but only signifies entries were removed. It seems a bit confusing. Is there any usage you were thinking that required the old entries to be removed that could benefit from the new event/listeners?
 

WDYT?

Radim

[1] https://issues.jboss.org/browse/ISPN-5021


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] State transfer-related terms

Radim Vansa
On 01/19/2017 11:14 PM, William Burns wrote:

>
>
> On Thu, Jan 19, 2017 at 11:13 AM Radim Vansa <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi,
>
>     I've started (again) working on ISPN-5021 [1], and I'd like to get
>     some
>     common agreement on few terms. So below I summarize my
>     understanding (or
>     misunderstanding) of these, please state you opinion, thinking a bit
>     more generally.
>
>     State transfer: whole process beginning with some ST-related event =
>     node being detected to crash/sending join or leave request, and ending
>     when this event is processed. When another event happens, the
>     current ST
>     can either be finished or canceled and then *another* ST can begin.
>     State transfer is a cluster-wide process, though it cannot be started
>     and ended absolutely simultaneously.
>
>     Rebalance: one phase of ST, when the data transfer occurs.
>
>     Data rehash: this is a bit painful point: we have DataRehashEvent
>     where
>     the name suggest that it is related rather to rebalance, but currently
>     it fires when CacheTopology.getPendingCH() == null (that means when ST
>     is complete), and the event itself also looks more like it should be
>     fired at the end of state transfer. If we have something more to do
>     after the rebalance, I am not sure how useful is firing that just
>     because all data has been transferred (but for example before old data
>     has been wiped out). Should I add another StateTransferEvent event
>     (and
>     appropriate listeners)? That would break the compatibility with
>     tightly
>     related implementations...
>
>
> The stream retry mechanism currently uses DataRehashedEvent. It is
> beneficial but not required for it to be notified immediately after
> all entries have been transferred but before any have been removed.
> This shortens the window for when a stream operation is retried since
> it has to be sure that all entries for a given segment are present,
> but we don't care if entries for a non owned segment are present. But
> the good thing is that the code shouldn't require much work at all to
> be changed if we needed to.
>
> I am personally not keen on having another event that is like
> DataRehashedEvent, but only signifies entries were removed. It seems a
> bit confusing. Is there any usage you were thinking that required the
> old entries to be removed that could benefit from the new event/listeners?

Okay, so from streams POV, DataRehashedEvent should mark rebalance, and
be fired once all incoming segments arrive. The implementation does not
rely on pendingCH being null when the event is fired, right? I like such
definition as "data rehash" sounds like moving data around.

I can see StateTransferEvent being useful when downscaling the cluster,
and you want to remove one node at a time (as the current leave is not
too graceful and calling cacheManager.stop() does not wait until ST
completes).

R.

>
>     WDYT?
>
>     Radim
>
>     [1] https://issues.jboss.org/browse/ISPN-5021
>
>
>     --
>     Radim Vansa <[hidden email] <mailto:[hidden email]>>
>     JBoss Performance Team
>
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] State transfer-related terms

Dan Berindei
In reply to this post by Radim Vansa
Oops, I thought I had replied to this...


On Thu, Jan 19, 2017 at 6:11 PM, Radim Vansa <[hidden email]> wrote:

> Hi,
>
> I've started (again) working on ISPN-5021 [1], and I'd like to get some
> common agreement on few terms. So below I summarize my understanding (or
> misunderstanding) of these, please state you opinion, thinking a bit
> more generally.
>
> State transfer: whole process beginning with some ST-related event =
> node being detected to crash/sending join or leave request, and ending
> when this event is processed. When another event happens, the current ST
> can either be finished or canceled and then *another* ST can begin.
> State transfer is a cluster-wide process, though it cannot be started
> and ended absolutely simultaneously.
>
> Rebalance: one phase of ST, when the data transfer occurs.
>

I see this exactly the opposite way: state transfer is the actual
transfer of the data, and the rebalance is the whole process of adding
a "pending" consistent hash, starting the state transfer, confirming
the transfer of state, and finally replacing the "current" consistent
hash with the pending one.

> Data rehash: this is a bit painful point: we have DataRehashEvent where
> the name suggest that it is related rather to rebalance, but currently
> it fires when CacheTopology.getPendingCH() == null (that means when ST
> is complete), and the event itself also looks more like it should be
> fired at the end of state transfer. If we have something more to do
> after the rebalance, I am not sure how useful is firing that just
> because all data has been transferred (but for example before old data
> has been wiped out). Should I add another StateTransferEvent event (and
> appropriate listeners)? That would break the compatibility with tightly
> related implementations...
>

The problem I was trying to solve with the "rebalance" name in 5.2 is
that before we had two different concepts:

1. State transfer applied only to replicated caches, and it meant
transferring state from the coordinator to a joiner.
2. Rehashing meant computing a new consistent hash based on the
JGroups view, receiving state from old owners based on the consistent
hash of the old JGroups view, and removing state no longer owned in
the new consistent hash. The coordinator didn't have a special role.

Rebalancing was meant to be an extension of rehashing to cover both
distributed and replicated caches (now with replicated caches also
being able to receive state from multiple members in parallel). Some
of the components (StateTransferManager) and events (DataRehashEvent)
kept their names, however.

> WDYT?
>
> Radim
>
> [1] https://issues.jboss.org/browse/ISPN-5021
>
>
> --
> Radim Vansa <[hidden email]>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...