[infinispan-dev] Kubernetes/OpenShift Rolling updates and configuration changes

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[infinispan-dev] Kubernetes/OpenShift Rolling updates and configuration changes

Sebastian Laskawiec
Hey!

I've been thinking about potential use of Kubernetes/OpenShift (OpenShift = Kubernetes + additional features) Rolling Update mechanism for updating configuration of Hot Rod servers. You might find some more information about the rolling updates here [1][2] but putting it simply, Kubernetes replaces nodes in the cluster one at a time. What's worth mentioning, Kubernetes ensures that the newly created replica is fully operational before taking down another one.

There are two things that make me scratching my head...

#1 - What type of configuration changes can we introduce using rolling updates?

I'm pretty sure introducing a new cache definition won't do any harm. But what if we change a cache type from Distributed to Replicated? Do you have any idea which configuration changes are safe and which are not? Could come up with such list?

#2 - How to prevent loosing data during the rolling update process?

In Kubernetes we have a mechanism called lifecycle hooks [3] (we can invoke a script during container startup/shutdown). The problem with shutdown script is that it's time constrained (if it won't end up within certain amount of time, Kubernetes will simply kill the container). Fortunately this time is configurable.

The idea to prevent from loosing data would be to invoke (enquque and wait for finish) state transfer process triggered by the shutdown hook (with timeout set to maximum value). If for some reason this won't work (e.g. a user has so much data that migrating it this way would take ages), there is a backup plan - Infinispan Rolling Upgrades [4].

What do you think about this? 

Thanks
Sebastian


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Kubernetes/OpenShift Rolling updates and configuration changes

Tristan Tarrant-2
On 14/07/16 12:17, Sebastian Laskawiec wrote:

> Hey!
>
> I've been thinking about potential use of Kubernetes/OpenShift
> (OpenShift = Kubernetes + additional features) Rolling Update
> mechanism for updating configuration of Hot Rod servers. You might
> find some more information about the rolling updates here [1][2] but
> putting it simply, Kubernetes replaces nodes in the cluster one at a
> time. What's worth mentioning, Kubernetes ensures that the newly
> created replica is fully operational before taking down another one.
>
> There are two things that make me scratching my head...
>
> #1 - What type of configuration changes can we introduce using rolling
> updates?
>
> I'm pretty sure introducing a new cache definition won't do any harm.
> But what if we change a cache type from Distributed to Replicated? Do
> you have any idea which configuration changes are safe and which are
> not? Could come up with such list?
Very few changes are safe, but obviously this would need to be verified
on a per-attribute basis. All of the attributes which can be changed at
runtime (timeouts, eviction size) are safe.

>
> #2 - How to prevent loosing data during the rolling update process?
I believe you want to write losing :)

> In Kubernetes we have a mechanism called lifecycle hooks [3] (we can
> invoke a script during container startup/shutdown). The problem with
> shutdown script is that it's time constrained (if it won't end up
> within certain amount of time, Kubernetes will simply kill the
> container). Fortunately this time is configurable.
>
> The idea to prevent from loosing data would be to invoke (enquque and
> wait for finish) state transfer process triggered by the shutdown hook
> (with timeout set to maximum value). If for some reason this won't
> work (e.g. a user has so much data that migrating it this way would
> take ages), there is a backup plan - Infinispan Rolling Upgrades [4].
The thing that concerns me here is the amount of churn involved: the
safest bet for us is that the net topology doesn't change, i.e. you end
up with the exact number of nodes you started with and they are replaced
one by one in a way that the replacement assumes the identity of the
replaced (both as persistent uuid, owned segments and data in a
persistent store). Other types could be supported but they will
definitely have a level of risk.
Also we don't have any guarantees that a newer version will be able to
cluster with an older one...

Tristan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Kubernetes/OpenShift Rolling updates and configuration changes

Emmanuel Bernard
Considering very few options can be changed at runtime safely, should we
rather focus of a strategy where we start a new grid and populate it
with the old grid before flipping the proxy to the new one?

On Mon 2016-07-18 17:12, Tristan Tarrant wrote:

> On 14/07/16 12:17, Sebastian Laskawiec wrote:
> > Hey!
> >
> > I've been thinking about potential use of Kubernetes/OpenShift
> > (OpenShift = Kubernetes + additional features) Rolling Update
> > mechanism for updating configuration of Hot Rod servers. You might
> > find some more information about the rolling updates here [1][2] but
> > putting it simply, Kubernetes replaces nodes in the cluster one at a
> > time. What's worth mentioning, Kubernetes ensures that the newly
> > created replica is fully operational before taking down another one.
> >
> > There are two things that make me scratching my head...
> >
> > #1 - What type of configuration changes can we introduce using rolling
> > updates?
> >
> > I'm pretty sure introducing a new cache definition won't do any harm.
> > But what if we change a cache type from Distributed to Replicated? Do
> > you have any idea which configuration changes are safe and which are
> > not? Could come up with such list?
> Very few changes are safe, but obviously this would need to be verified
> on a per-attribute basis. All of the attributes which can be changed at
> runtime (timeouts, eviction size) are safe.
>
> >
> > #2 - How to prevent loosing data during the rolling update process?
> I believe you want to write losing :)
> > In Kubernetes we have a mechanism called lifecycle hooks [3] (we can
> > invoke a script during container startup/shutdown). The problem with
> > shutdown script is that it's time constrained (if it won't end up
> > within certain amount of time, Kubernetes will simply kill the
> > container). Fortunately this time is configurable.
> >
> > The idea to prevent from loosing data would be to invoke (enquque and
> > wait for finish) state transfer process triggered by the shutdown hook
> > (with timeout set to maximum value). If for some reason this won't
> > work (e.g. a user has so much data that migrating it this way would
> > take ages), there is a backup plan - Infinispan Rolling Upgrades [4].
> The thing that concerns me here is the amount of churn involved: the
> safest bet for us is that the net topology doesn't change, i.e. you end
> up with the exact number of nodes you started with and they are replaced
> one by one in a way that the replacement assumes the identity of the
> replaced (both as persistent uuid, owned segments and data in a
> persistent store). Other types could be supported but they will
> definitely have a level of risk.
> Also we don't have any guarantees that a newer version will be able to
> cluster with an older one...
>
> Tristan
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Kubernetes/OpenShift Rolling updates and configuration changes

Sebastian Laskawiec
Hey Tristan, Emmanuel!

Comments inlined.

Thanks
Sebastian

On Tue, Jul 19, 2016 at 10:08 AM, Emmanuel Bernard <[hidden email]> wrote:
Considering very few options can be changed at runtime safely, should we
rather focus of a strategy where we start a new grid and populate it
with the old grid before flipping the proxy to the new one?

+1, that's exactly what the Infinispan Rolling Upgrade does.
 

On Mon 2016-07-18 17:12, Tristan Tarrant wrote:
> On 14/07/16 12:17, Sebastian Laskawiec wrote:
> > Hey!
> >
> > I've been thinking about potential use of Kubernetes/OpenShift
> > (OpenShift = Kubernetes + additional features) Rolling Update
> > mechanism for updating configuration of Hot Rod servers. You might
> > find some more information about the rolling updates here [1][2] but
> > putting it simply, Kubernetes replaces nodes in the cluster one at a
> > time. What's worth mentioning, Kubernetes ensures that the newly
> > created replica is fully operational before taking down another one.
> >
> > There are two things that make me scratching my head...
> >
> > #1 - What type of configuration changes can we introduce using rolling
> > updates?
> >
> > I'm pretty sure introducing a new cache definition won't do any harm.
> > But what if we change a cache type from Distributed to Replicated? Do
> > you have any idea which configuration changes are safe and which are
> > not? Could come up with such list?
> Very few changes are safe, but obviously this would need to be verified
> on a per-attribute basis. All of the attributes which can be changed at
> runtime (timeouts, eviction size) are safe.
>
> >
> > #2 - How to prevent loosing data during the rolling update process?
> I believe you want to write losing :)

Good one :)
 
> > In Kubernetes we have a mechanism called lifecycle hooks [3] (we can
> > invoke a script during container startup/shutdown). The problem with
> > shutdown script is that it's time constrained (if it won't end up
> > within certain amount of time, Kubernetes will simply kill the
> > container). Fortunately this time is configurable.
> >
> > The idea to prevent from loosing data would be to invoke (enquque and
> > wait for finish) state transfer process triggered by the shutdown hook
> > (with timeout set to maximum value). If for some reason this won't
> > work (e.g. a user has so much data that migrating it this way would
> > take ages), there is a backup plan - Infinispan Rolling Upgrades [4].
> The thing that concerns me here is the amount of churn involved: the
> safest bet for us is that the net topology doesn't change, i.e. you end
> up with the exact number of nodes you started with

Yes, Kubernetes Rolling Update works this way. The number of nodes at the end of the process is equal to the number you started with. 
 
and they are replaced
> one by one in a way that the replacement assumes the identity of the
> replaced (both as persistent uuid, owned segments and data in a
> persistent store). 
Other types could be supported but they will
> definitely have a level of risk.
> Also we don't have any guarantees that a newer version will be able to
> cluster with an older one...

I'm not sure we can ensure the same identity of the replaced node. If we consider configuration changes, a user can change anything...

I think I'm convinced that the Infinispan Rolling Upgrade procedure is the only proper solution at this stage. Other ways (although much simpler) must be treated as - 'do it at your own risk'.
 
>
> Tristan
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Kubernetes/OpenShift Rolling updates and configuration changes

Sanne Grinovero-3
Just wondering, we're looking into how to compare configuration
settings just to try validate the user isn't attempting an insane
upgrade right?

Or is there an actual need to want to change Infinispan version AND
switch the essential grid configuration settings at the same time?
(that seems quite insane to me, and unnecessary as one could do it in
two steps..)

Thanks


On 19 July 2016 at 10:06, Sebastian Laskawiec <[hidden email]> wrote:

> Hey Tristan, Emmanuel!
>
> Comments inlined.
>
> Thanks
> Sebastian
>
> On Tue, Jul 19, 2016 at 10:08 AM, Emmanuel Bernard <[hidden email]>
> wrote:
>>
>> Considering very few options can be changed at runtime safely, should we
>> rather focus of a strategy where we start a new grid and populate it
>> with the old grid before flipping the proxy to the new one?
>
>
> +1, that's exactly what the Infinispan Rolling Upgrade does.
>
>>
>>
>> On Mon 2016-07-18 17:12, Tristan Tarrant wrote:
>> > On 14/07/16 12:17, Sebastian Laskawiec wrote:
>> > > Hey!
>> > >
>> > > I've been thinking about potential use of Kubernetes/OpenShift
>> > > (OpenShift = Kubernetes + additional features) Rolling Update
>> > > mechanism for updating configuration of Hot Rod servers. You might
>> > > find some more information about the rolling updates here [1][2] but
>> > > putting it simply, Kubernetes replaces nodes in the cluster one at a
>> > > time. What's worth mentioning, Kubernetes ensures that the newly
>> > > created replica is fully operational before taking down another one.
>> > >
>> > > There are two things that make me scratching my head...
>> > >
>> > > #1 - What type of configuration changes can we introduce using rolling
>> > > updates?
>> > >
>> > > I'm pretty sure introducing a new cache definition won't do any harm.
>> > > But what if we change a cache type from Distributed to Replicated? Do
>> > > you have any idea which configuration changes are safe and which are
>> > > not? Could come up with such list?
>> > Very few changes are safe, but obviously this would need to be verified
>> > on a per-attribute basis. All of the attributes which can be changed at
>> > runtime (timeouts, eviction size) are safe.
>> >
>> > >
>> > > #2 - How to prevent loosing data during the rolling update process?
>> > I believe you want to write losing :)
>
>
> Good one :)
>
>>
>> > > In Kubernetes we have a mechanism called lifecycle hooks [3] (we can
>> > > invoke a script during container startup/shutdown). The problem with
>> > > shutdown script is that it's time constrained (if it won't end up
>> > > within certain amount of time, Kubernetes will simply kill the
>> > > container). Fortunately this time is configurable.
>> > >
>> > > The idea to prevent from loosing data would be to invoke (enquque and
>> > > wait for finish) state transfer process triggered by the shutdown hook
>> > > (with timeout set to maximum value). If for some reason this won't
>> > > work (e.g. a user has so much data that migrating it this way would
>> > > take ages), there is a backup plan - Infinispan Rolling Upgrades [4].
>> > The thing that concerns me here is the amount of churn involved: the
>> > safest bet for us is that the net topology doesn't change, i.e. you end
>> > up with the exact number of nodes you started with
>
>
> Yes, Kubernetes Rolling Update works this way. The number of nodes at the
> end of the process is equal to the number you started with.
>
>>
>> and they are replaced
>> > one by one in a way that the replacement assumes the identity of the
>> > replaced (both as persistent uuid, owned segments and data in a
>> > persistent store).
>>
>> Other types could be supported but they will
>> > definitely have a level of risk.
>> > Also we don't have any guarantees that a newer version will be able to
>> > cluster with an older one...
>
>
> I'm not sure we can ensure the same identity of the replaced node. If we
> consider configuration changes, a user can change anything...
>
> I think I'm convinced that the Infinispan Rolling Upgrade procedure is the
> only proper solution at this stage. Other ways (although much simpler) must
> be treated as - 'do it at your own risk'.
>
>>
>> >
>> > Tristan
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > [hidden email]
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Kubernetes/OpenShift Rolling updates and configuration changes

Sebastian Laskawiec
Hey Sanne!

I would treat both those things (upgrading Infinispan from version X to Y and changing configuration) separate. 

Thanks
Sebastian

On Tue, Jul 19, 2016 at 1:06 PM, Sanne Grinovero <[hidden email]> wrote:
Just wondering, we're looking into how to compare configuration
settings just to try validate the user isn't attempting an insane
upgrade right?

Or is there an actual need to want to change Infinispan version AND
switch the essential grid configuration settings at the same time?
(that seems quite insane to me, and unnecessary as one could do it in
two steps..)

Thanks


On 19 July 2016 at 10:06, Sebastian Laskawiec <[hidden email]> wrote:
> Hey Tristan, Emmanuel!
>
> Comments inlined.
>
> Thanks
> Sebastian
>
> On Tue, Jul 19, 2016 at 10:08 AM, Emmanuel Bernard <[hidden email]>
> wrote:
>>
>> Considering very few options can be changed at runtime safely, should we
>> rather focus of a strategy where we start a new grid and populate it
>> with the old grid before flipping the proxy to the new one?
>
>
> +1, that's exactly what the Infinispan Rolling Upgrade does.
>
>>
>>
>> On Mon 2016-07-18 17:12, Tristan Tarrant wrote:
>> > On 14/07/16 12:17, Sebastian Laskawiec wrote:
>> > > Hey!
>> > >
>> > > I've been thinking about potential use of Kubernetes/OpenShift
>> > > (OpenShift = Kubernetes + additional features) Rolling Update
>> > > mechanism for updating configuration of Hot Rod servers. You might
>> > > find some more information about the rolling updates here [1][2] but
>> > > putting it simply, Kubernetes replaces nodes in the cluster one at a
>> > > time. What's worth mentioning, Kubernetes ensures that the newly
>> > > created replica is fully operational before taking down another one.
>> > >
>> > > There are two things that make me scratching my head...
>> > >
>> > > #1 - What type of configuration changes can we introduce using rolling
>> > > updates?
>> > >
>> > > I'm pretty sure introducing a new cache definition won't do any harm.
>> > > But what if we change a cache type from Distributed to Replicated? Do
>> > > you have any idea which configuration changes are safe and which are
>> > > not? Could come up with such list?
>> > Very few changes are safe, but obviously this would need to be verified
>> > on a per-attribute basis. All of the attributes which can be changed at
>> > runtime (timeouts, eviction size) are safe.
>> >
>> > >
>> > > #2 - How to prevent loosing data during the rolling update process?
>> > I believe you want to write losing :)
>
>
> Good one :)
>
>>
>> > > In Kubernetes we have a mechanism called lifecycle hooks [3] (we can
>> > > invoke a script during container startup/shutdown). The problem with
>> > > shutdown script is that it's time constrained (if it won't end up
>> > > within certain amount of time, Kubernetes will simply kill the
>> > > container). Fortunately this time is configurable.
>> > >
>> > > The idea to prevent from loosing data would be to invoke (enquque and
>> > > wait for finish) state transfer process triggered by the shutdown hook
>> > > (with timeout set to maximum value). If for some reason this won't
>> > > work (e.g. a user has so much data that migrating it this way would
>> > > take ages), there is a backup plan - Infinispan Rolling Upgrades [4].
>> > The thing that concerns me here is the amount of churn involved: the
>> > safest bet for us is that the net topology doesn't change, i.e. you end
>> > up with the exact number of nodes you started with
>
>
> Yes, Kubernetes Rolling Update works this way. The number of nodes at the
> end of the process is equal to the number you started with.
>
>>
>> and they are replaced
>> > one by one in a way that the replacement assumes the identity of the
>> > replaced (both as persistent uuid, owned segments and data in a
>> > persistent store).
>>
>> Other types could be supported but they will
>> > definitely have a level of risk.
>> > Also we don't have any guarantees that a newer version will be able to
>> > cluster with an older one...
>
>
> I'm not sure we can ensure the same identity of the replaced node. If we
> consider configuration changes, a user can change anything...
>
> I think I'm convinced that the Infinispan Rolling Upgrade procedure is the
> only proper solution at this stage. Other ways (although much simpler) must
> be treated as - 'do it at your own risk'.
>
>>
>> >
>> > Tristan
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > [hidden email]
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...