Quantcast

[infinispan-dev] Cache migration (rolling upgrades, dump/restore, etc)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[infinispan-dev] Cache migration (rolling upgrades, dump/restore, etc)

Tristan Tarrant-2
Hey all,

Infinispan has historically had two ways of performing live migration
between two clusters, via Hot Rod and via REST. We do not currently
provide an offline migration, although we do have a cache store
migration tool.

Gustavo has recently made several changes to the Hot Rod implementation
which have improved it greatly. The REST implementation is still not
robust enough, but I think we can abandon it and just focus on the Hot
Rod one ever for servers using REST.

The following is a list of stuff, mostly compiled by Gustavo, that needs
to be done to make everything smooth and robust:

1) Need a way to automate client redirection to the new cluster. I've
often referred to this as L4 client intelligence, which can also be used
for server-assisted cross-site failover.
2) Need a way to "rollback" the process in case of failures during the
migration: redirecting the clients back to the original cluster without
data loss. This would use the above L4 strategy.
3) Expose metrics and progress
4) Expose a way to cancel the process
5) Expose a container-wide migration process which can be applied to all
caches instead of one cache at a time.
6) The migration process should also take care of automatically
configuring the endpoints / remote cache stores at the beginning of the
process and removing any changes at the end.
6) Provide a future-proof format for the entries
7) Implement dump and restore capabilities which can export the contents
of a cluster to a file (compressed, encrypted, etc) or a collection of
files (one per cache).

Anything else ?

Tristan

--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Cache migration (rolling upgrades, dump/restore, etc)

Vojtech Juranek
On středa 17. května 2017 16:56:25 CEST Tristan Tarrant wrote:
> 2) Need a way to "rollback" the process in case of failures during the
> migration: redirecting the clients back to the original cluster without
> data loss. This would use the above L4 strategy.

it's not only about redirecting clients - IIRC newly created entries on target
cluster are not propagated back to source cluster during rolling upgrade, so
we need also somehow sync these new data back to source cluster during the
rollback to avoid data losses. Same applies to "cancel process" feature
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Cache migration (rolling upgrades, dump/restore, etc)

Gustavo Fernandes-2
On Fri, May 19, 2017 at 11:50 AM, Vojtech Juranek <[hidden email]> wrote:
On středa 17. května 2017 16:56:25 CEST Tristan Tarrant wrote:
> 2) Need a way to "rollback" the process in case of failures during the
> migration: redirecting the clients back to the original cluster without
> data loss. This would use the above L4 strategy.

it's not only about redirecting clients - IIRC newly created entries on target
cluster are not propagated back to source cluster during rolling upgrade,

After the latest changes, new entries written to the target cluster are supposed to propagated back to the source [1].

Did you find any issue with it?
Thanks,
Gustavo

 
so
we need also somehow sync these new data back to source cluster during the
rollback to avoid data losses. Same applies to "cancel process" feature
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Cache migration (rolling upgrades, dump/restore, etc)

Wolf Fink
In reply to this post by Vojtech Juranek
+1 for Vojtech

yes the client's need to moved to the new cluster in one shot current, that was discussed  before.
And it makes the migration because most of the customers are not able to make that happen.
So there is a small possibility of inconsistence if clients connect to the old server update entries until the new server already migrated it.

I see two options
1)
source server need to propagate active to target on update
2)
with the new L4 strategy all clients are moved automatically to the target. So the source is not updated.
I only see a small possibility for this to happen during switch
- a client might still have a request to the source until other clients are moved to target and already accessed the key
- a new client connects with old properties, here we need to ensure that the first request is redirected to the target and not update the source

On Fri, May 19, 2017 at 12:50 PM, Vojtech Juranek <[hidden email]> wrote:
On středa 17. května 2017 16:56:25 CEST Tristan Tarrant wrote:
> 2) Need a way to "rollback" the process in case of failures during the
> migration: redirecting the clients back to the original cluster without
> data loss. This would use the above L4 strategy.

it's not only about redirecting clients - IIRC newly created entries on target
cluster are not propagated back to source cluster during rolling upgrade, so
we need also somehow sync these new data back to source cluster during the
rollback to avoid data losses. Same applies to "cancel process" feature
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Cache migration (rolling upgrades, dump/restore, etc)

Vojtech Juranek
In reply to this post by Gustavo Fernandes-2
On pátek 19. května 2017 13:05:00 CEST Gustavo Fernandes wrote:
> After the latest changes, new entries written to the target cluster are
> supposed to propagated back to the source [1].

I thought that when the rolling upgrade is in progress, entries written to
target cluster are not written to source cluster. If this is incorrect, what
actually does L70 in DistCacheWriterInterceptor [1]?

Thanks

[1] https://github.com/infinispan/infinispan/blob/master/core/src/main/java/
org/infinispan/interceptors/impl/DistCacheWriterInterceptor.java#L70

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Cache migration (rolling upgrades, dump/restore, etc)

Gustavo Fernandes-2
On Mon, May 22, 2017 at 7:45 AM, Vojtech Juranek <[hidden email]> wrote:
On pátek 19. května 2017 13:05:00 CEST Gustavo Fernandes wrote:
> After the latest changes, new entries written to the target cluster are
> supposed to propagated back to the source [1].

I thought that when the rolling upgrade is in progress, entries written to
target cluster are not written to source cluster.

During a RU, there are two agents writing data to the target cluster: the user and the RU process itself.

Since the remote store in the target cluster is not supposed to be used in read-only mode anymore, all
data changes by the user in the target cluster during rolling upgrade are propagate remotely to the source.

OTOH, data written by the Rolling Upgrader itself is not written back to the source, since it is merely doing the migration.

If this is incorrect, what
actually does L70 in DistCacheWriterInterceptor [1]?


This line basically says: if data is being written by the rolling upgrade process itself, write it to the stores (excluding the remoteStore), regardless if the command is conditional or not.

 
Thanks

[1] https://github.com/infinispan/infinispan/blob/master/core/src/main/java/
org/infinispan/interceptors/impl/DistCacheWriterInterceptor.java#L70


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Cache migration (rolling upgrades, dump/restore, etc)

Vojtech Juranek
> This line basically says: if data is being written by the rolling upgrade
> process itself, write it to the stores (excluding the remoteStore),
> regardless if the command is conditional or not.

ok, make sense to me now, thanks for explanation!
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Cache migration (rolling upgrades, dump/restore, etc)

Martin Gencur
In reply to this post by Wolf Fink
Hi Wolf,
that was exactly my thought. Clients redirected to the target cluster do not get updates written by other clients in the source cluster during the rolling upgrade process. It is because the clients in target cluster won't read the data through the remote cache store if they already have the requested key in the local memory. Is there a BZ/JIRA for this?

Martin

On 19.5.2017 13:17, Wolf Fink wrote:
+1 for Vojtech

yes the client's need to moved to the new cluster in one shot current, that was discussed  before.
And it makes the migration because most of the customers are not able to make that happen.
So there is a small possibility of inconsistence if clients connect to the old server update entries until the new server already migrated it.

I see two options
1)
source server need to propagate active to target on update
2)
with the new L4 strategy all clients are moved automatically to the target. So the source is not updated.
I only see a small possibility for this to happen during switch
- a client might still have a request to the source until other clients are moved to target and already accessed the key
- a new client connects with old properties, here we need to ensure that the first request is redirected to the target and not update the source

On Fri, May 19, 2017 at 12:50 PM, Vojtech Juranek <[hidden email]> wrote:
On středa 17. května 2017 16:56:25 CEST Tristan Tarrant wrote:
> 2) Need a way to "rollback" the process in case of failures during the
> migration: redirecting the clients back to the original cluster without
> data loss. This would use the above L4 strategy.

it's not only about redirecting clients - IIRC newly created entries on target
cluster are not propagated back to source cluster during rolling upgrade, so
we need also somehow sync these new data back to source cluster during the
rollback to avoid data losses. Same applies to "cancel process" feature
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...