[infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Sebastian Laskawiec
Hey!

I've been playing with Kubernetes PetSets [1] for a while and I'd like to share some thoughts. Before I dig in, let me give you some PetSets highlights:
  • PetSets are alpha resources for managing stateful apps in Kubernetes 1.3 (and OpenShift Origin 1.3).
  • Since this is an alpha resource, there are no guarantees about backwards compatibility. Alpha resources can also be disabled in some public cloud providers (you can control which API versions are accessible [2]).
  • PetSets allows starting pods in sequence (not relevant for us, but this is a killer feature for master-slave systems).
  • Each Pod has it's own unique entry in DNS, which makes discovery very simple (I'll dig into that a bit later)
  • Volumes are always mounted to the same Pods, which is very important in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades [3]).
Thoughts and ideas after spending some time playing with this feature:
  • PetSets make discovery a lot easier. It's a combination of two things - Headless Services [4] which create multiple A records in DNS and predictable host names. Each Pod has it's own unique DNS entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5]. Here's an example of an Infinispan PetSet deployed on my local cluster [6]. As you can see we have all domain names and IPs from a single DNS query.
  • Maybe we could perform discovery using this mechanism? I'm aware of DNS discovery implemented in KUBE_PING [7][8] but the code looks trivial [9] so maybe it should be implement inside JGroups? @Bela - WDYT?
  • PetSets do not integrate well with OpenShift 'new-app' command. In other words, our users will need to use provided yaml (or json) files to create Infinispan cluster. It's not a show-stopper but it's a bit less convenient than 'oc new-app'.
  • Since PetSets are alpha resources they need to be considered as secondary way to deploy Infinispan on Kubernetes and OpenShift.
  • Finally, the persistent volumes - since a Pod always gets the same volume, it would be safe to use any file-based cache store.
If you'd like to play with PetSets on your local environment, here are necessary yaml files [10].

Thanks
Sebastian


[2] For checking which APIs are accessible, use 'kubectl api-versions'

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Bela Ban
Hi Sebastian

the usual restrictions apply: if DNS discovery depends on external libs,
then it should be hosted in jgroups-extras, otherwise we can add it to
JGroups itself.

On 19/08/16 11:00, Sebastian Laskawiec wrote:

> Hey!
>
> I've been playing with Kubernetes PetSets [1] for a while and I'd like
> to share some thoughts. Before I dig in, let me give you some PetSets
> highlights:
>
>   * PetSets are alpha resources for managing stateful apps in Kubernetes
>     1.3 (and OpenShift Origin 1.3).
>   * Since this is an alpha resource, there are no guarantees about
>     backwards compatibility. Alpha resources can also be disabled in
>     some public cloud providers (you can control which API versions are
>     accessible [2]).
>   * PetSets allows starting pods in sequence (not relevant for us, but
>     this is a killer feature for master-slave systems).
>   * Each Pod has it's own unique entry in DNS, which makes discovery
>     very simple (I'll dig into that a bit later)
>   * Volumes are always mounted to the same Pods, which is very important
>     in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades
>     [3]).
>
> Thoughts and ideas after spending some time playing with this feature:
>
>   * PetSets make discovery a lot easier. It's a combination of two
>     things - Headless Services [4] which create multiple A records in
>     DNS and predictable host names. Each Pod has it's own unique DNS
>     entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5].
>     Here's an example of an Infinispan PetSet deployed on my local
>     cluster [6]. As you can see we have all domain names and IPs from a
>     single DNS query.
>   * Maybe we could perform discovery using this mechanism? I'm aware of
>     DNS discovery implemented in KUBE_PING [7][8] but the code looks
>     trivial [9] so maybe it should be implement inside JGroups? @Bela -
>     WDYT?
>   * PetSets do not integrate well with OpenShift 'new-app' command. In
>     other words, our users will need to use provided yaml (or json)
>     files to create Infinispan cluster. It's not a show-stopper but it's
>     a bit less convenient than 'oc new-app'.
>   * Since PetSets are alpha resources they need to be considered as
>     secondary way to deploy Infinispan on Kubernetes and OpenShift.
>   * Finally, the persistent volumes - since a Pod always gets the same
>     volume, it would be safe to use any file-based cache store.
>
> If you'd like to play with PetSets on your local environment, here are
> necessary yaml files [10].
>
> Thanks
> Sebastian
>
>
> [1] http://kubernetes.io/docs/user-guide/petset/
> [2] For checking which APIs are accessible, use 'kubectl api-versions'
> [3]
> http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
> [4] http://kubernetes.io/docs/user-guide/services/#headless-services
> [5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
> [6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
> [7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns
> [8] https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
> [9] http://stackoverflow.com/a/12405896/562699
> [10] You might need to adjust ImageStream.
> https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0

--
Bela Ban, JGroups lead (http://www.jgroups.org)

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Sebastian Laskawiec
Hey Bela!

No no, the resolution can be done with pure JDK.

Thanks
Sebastian

On Fri, Aug 19, 2016 at 11:18 AM, Bela Ban <[hidden email]> wrote:
Hi Sebastian

the usual restrictions apply: if DNS discovery depends on external libs, then it should be hosted in jgroups-extras, otherwise we can add it to JGroups itself.

On 19/08/16 11:00, Sebastian Laskawiec wrote:
Hey!

I've been playing with Kubernetes PetSets [1] for a while and I'd like
to share some thoughts. Before I dig in, let me give you some PetSets
highlights:

  * PetSets are alpha resources for managing stateful apps in Kubernetes
    1.3 (and OpenShift Origin 1.3).
  * Since this is an alpha resource, there are no guarantees about
    backwards compatibility. Alpha resources can also be disabled in
    some public cloud providers (you can control which API versions are
    accessible [2]).
  * PetSets allows starting pods in sequence (not relevant for us, but
    this is a killer feature for master-slave systems).
  * Each Pod has it's own unique entry in DNS, which makes discovery
    very simple (I'll dig into that a bit later)
  * Volumes are always mounted to the same Pods, which is very important
    in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades
    [3]).

Thoughts and ideas after spending some time playing with this feature:

  * PetSets make discovery a lot easier. It's a combination of two
    things - Headless Services [4] which create multiple A records in
    DNS and predictable host names. Each Pod has it's own unique DNS
    entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5].
    Here's an example of an Infinispan PetSet deployed on my local
    cluster [6]. As you can see we have all domain names and IPs from a
    single DNS query.
  * Maybe we could perform discovery using this mechanism? I'm aware of
    DNS discovery implemented in KUBE_PING [7][8] but the code looks
    trivial [9] so maybe it should be implement inside JGroups? @Bela -
    WDYT?
  * PetSets do not integrate well with OpenShift 'new-app' command. In
    other words, our users will need to use provided yaml (or json)
    files to create Infinispan cluster. It's not a show-stopper but it's
    a bit less convenient than 'oc new-app'.
  * Since PetSets are alpha resources they need to be considered as
    secondary way to deploy Infinispan on Kubernetes and OpenShift.
  * Finally, the persistent volumes - since a Pod always gets the same
    volume, it would be safe to use any file-based cache store.

If you'd like to play with PetSets on your local environment, here are
necessary yaml files [10].

Thanks
Sebastian


[1] http://kubernetes.io/docs/user-guide/petset/
[2] For checking which APIs are accessible, use 'kubectl api-versions'
[3]
http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
[4] http://kubernetes.io/docs/user-guide/services/#headless-services
[5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
[6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
[7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns
[8] https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
[9] http://stackoverflow.com/a/12405896/562699
[10] You might need to adjust ImageStream.
https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0

--
Bela Ban, JGroups lead (http://www.jgroups.org)



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Gustavo Fernandes-2
In reply to this post by Bela Ban
On Fri, Aug 19, 2016 at 10:18 AM, Bela Ban <[hidden email]> wrote:
Hi Sebastian

the usual restrictions apply: if DNS discovery depends on external libs,
then it should be hosted in jgroups-extras, otherwise we can add it to
JGroups itself.


Bela, you forgot to mention an extra restriction: even if it has no external deps, it should
not be too big in terms of LOC (what is the threshold?)  ;)

Gustavo
 

On 19/08/16 11:00, Sebastian Laskawiec wrote:
> Hey!
>
> I've been playing with Kubernetes PetSets [1] for a while and I'd like
> to share some thoughts. Before I dig in, let me give you some PetSets
> highlights:
>
>   * PetSets are alpha resources for managing stateful apps in Kubernetes
>     1.3 (and OpenShift Origin 1.3).
>   * Since this is an alpha resource, there are no guarantees about
>     backwards compatibility. Alpha resources can also be disabled in
>     some public cloud providers (you can control which API versions are
>     accessible [2]).
>   * PetSets allows starting pods in sequence (not relevant for us, but
>     this is a killer feature for master-slave systems).
>   * Each Pod has it's own unique entry in DNS, which makes discovery
>     very simple (I'll dig into that a bit later)
>   * Volumes are always mounted to the same Pods, which is very important
>     in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades
>     [3]).
>
> Thoughts and ideas after spending some time playing with this feature:
>
>   * PetSets make discovery a lot easier. It's a combination of two
>     things - Headless Services [4] which create multiple A records in
>     DNS and predictable host names. Each Pod has it's own unique DNS
>     entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5].
>     Here's an example of an Infinispan PetSet deployed on my local
>     cluster [6]. As you can see we have all domain names and IPs from a
>     single DNS query.
>   * Maybe we could perform discovery using this mechanism? I'm aware of
>     DNS discovery implemented in KUBE_PING [7][8] but the code looks
>     trivial [9] so maybe it should be implement inside JGroups? @Bela -
>     WDYT?
>   * PetSets do not integrate well with OpenShift 'new-app' command. In
>     other words, our users will need to use provided yaml (or json)
>     files to create Infinispan cluster. It's not a show-stopper but it's
>     a bit less convenient than 'oc new-app'.
>   * Since PetSets are alpha resources they need to be considered as
>     secondary way to deploy Infinispan on Kubernetes and OpenShift.
>   * Finally, the persistent volumes - since a Pod always gets the same
>     volume, it would be safe to use any file-based cache store.
>
> If you'd like to play with PetSets on your local environment, here are
> necessary yaml files [10].
>
> Thanks
> Sebastian
>
>
> [1] http://kubernetes.io/docs/user-guide/petset/
> [2] For checking which APIs are accessible, use 'kubectl api-versions'
> [3]
> http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
> [4] http://kubernetes.io/docs/user-guide/services/#headless-services
> [5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
> [6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
> [7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns
> [8] https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
> [9] http://stackoverflow.com/a/12405896/562699
> [10] You might need to adjust ImageStream.
> https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0

--
Bela Ban, JGroups lead (http://www.jgroups.org)

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Bela Ban


On 19/08/16 11:27, Gustavo Fernandes wrote:

> On Fri, Aug 19, 2016 at 10:18 AM, Bela Ban <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Sebastian
>
>     the usual restrictions apply: if DNS discovery depends on external libs,
>     then it should be hosted in jgroups-extras, otherwise we can add it to
>     JGroups itself.
>
>
>
> Bela, you forgot to mention an extra restriction: even if it has no
> external deps, it should
> not be too big in terms of LOC (what is the threshold?)  ;)


You mean because people copy&paste the deps into one class? Yes, that
happened before but since the resulting class was 17K, I rejected it.
The largest class in JGroups (Util) has 4K, so that's a ballpark figure.

However, with jgroups-extras, we now have the possibility of adding deps.


> Gustavo
>
>
>     On 19/08/16 11:00, Sebastian Laskawiec wrote:
>     > Hey!
>     >
>     > I've been playing with Kubernetes PetSets [1] for a while and I'd like
>     > to share some thoughts. Before I dig in, let me give you some PetSets
>     > highlights:
>     >
>      >   * PetSets are alpha resources for managing stateful apps in
>     Kubernetes
>     >     1.3 (and OpenShift Origin 1.3).
>      >   * Since this is an alpha resource, there are no guarantees about
>     >     backwards compatibility. Alpha resources can also be disabled in
>     >     some public cloud providers (you can control which API versions are
>     >     accessible [2]).
>      >   * PetSets allows starting pods in sequence (not relevant for
>     us, but
>     >     this is a killer feature for master-slave systems).
>      >   * Each Pod has it's own unique entry in DNS, which makes discovery
>     >     very simple (I'll dig into that a bit later)
>      >   * Volumes are always mounted to the same Pods, which is very
>     important
>     >     in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades
>     >     [3]).
>     >
>     > Thoughts and ideas after spending some time playing with this feature:
>     >
>      >   * PetSets make discovery a lot easier. It's a combination of two
>     >     things - Headless Services [4] which create multiple A records in
>     >     DNS and predictable host names. Each Pod has it's own unique DNS
>     >     entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5].
>     >     Here's an example of an Infinispan PetSet deployed on my local
>     >     cluster [6]. As you can see we have all domain names and IPs from a
>     >     single DNS query.
>      >   * Maybe we could perform discovery using this mechanism? I'm
>     aware of
>     >     DNS discovery implemented in KUBE_PING [7][8] but the code looks
>     >     trivial [9] so maybe it should be implement inside JGroups? @Bela -
>     >     WDYT?
>      >   * PetSets do not integrate well with OpenShift 'new-app'
>     command. In
>     >     other words, our users will need to use provided yaml (or json)
>     >     files to create Infinispan cluster. It's not a show-stopper but it's
>     >     a bit less convenient than 'oc new-app'.
>      >   * Since PetSets are alpha resources they need to be considered as
>     >     secondary way to deploy Infinispan on Kubernetes and OpenShift.
>      >   * Finally, the persistent volumes - since a Pod always gets the
>     same
>      >     volume, it would be safe to use any file-based cache store.
>      >
>      > If you'd like to play with PetSets on your local environment,
>     here are
>      > necessary yaml files [10].
>      >
>      > Thanks
>      > Sebastian
>      >
>      >
>      > [1] http://kubernetes.io/docs/user-guide/petset/
>     <http://kubernetes.io/docs/user-guide/petset/>
>      > [2] For checking which APIs are accessible, use 'kubectl
>     api-versions'
>      > [3]
>      >
>     http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
>     <http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter>
>      > [4]
>     http://kubernetes.io/docs/user-guide/services/#headless-services
>     <http://kubernetes.io/docs/user-guide/services/#headless-services>
>      > [5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
>     <http://kubernetes.io/docs/user-guide/petset/#peer-discovery>
>      > [6]
>     https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
>     <https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676>
>      > [7]
>     https://github.com/jboss-openshift/openshift-ping/tree/master/dns
>     <https://github.com/jboss-openshift/openshift-ping/tree/master/dns>
>      > [8]
>     https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
>     <https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns>
>      > [9] http://stackoverflow.com/a/12405896/562699
>     <http://stackoverflow.com/a/12405896/562699>
>      > [10] You might need to adjust ImageStream.
>      > https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0
>     <https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0>
>
>     --
>     Bela Ban, JGroups lead (http://www.jgroups.org)
>
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>     <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

--
Bela Ban, JGroups lead (http://www.jgroups.org)

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Bela Ban
In reply to this post by Sebastian Laskawiec
If we add a DNS discovery protocol, it would only be another discovery
protocol among many, and customers can choose which one to use.
I'm also thinking of adding the ability to JGroups to use multiple
discovery protocols in the same stack and combine their result sets into
one. Not sure though if it makes sense to use KUBE_PING and DNS in the
same stack...

On 20/08/16 00:04, Rob Cernich wrote:

> A couple of things...
>
> re. volumes:
> We also need to consider the mounting behavior for scale down scenarios
> and for overage scenarios when doing upgrades.  For the latter,
> OpenShift can spin up pods of the new version before the older version
> pods have terminated.  This may mean that some volumes from the old pods
> are orphaned.  We did see this when testing A-MQ during upgrades.  With
> a single pod, the upgrade process caused the new version to have a new
> mount and the original mount was left orphaned (another upgrade would
> cause the newer pod to pick up the orphaned mount, leaving the new mount
> orphaned).  I believe we worked around this by specifying an overage of
> 0% during upgrades.  This ensured the new pods would pick up the volumes
> left behind by the old pods.  (Actually, we were using subdirectories in
> the mount, since all pods shared the same volume.)
>
> re. dns:
> DNS should work fine as-is, but there are a couple things that you need
> to consider.
> 1. Service endpoints are only available in DNS after the pod becomes
> ready (SVC records on the service name).  Because infinispan attaches
> itself to the cluster, this meant pods were all started as cluster of
> one, then merged once they noticed the other pods.  This had a
> significant impact on startup.  Since then, OpenShift has added the
> ability to query the endpoints associated with a service as soon as the
> pod is created, which would allow initialization to work correctly.  To
> make this work, we'd have to change the form of the DNS query to pick up
> the service endpoints (I forget the naming scheme).
>
> Another thing to keep in mind is that looking up pods by labels allows
> any pod with the specified label to be added to the cluster.  I'm not
> sure of a use case for this, but it would allow other deployments to be
> included in the cluster.  (You could also argue that the service is the
> authority for this and any pod with said label would be added as a
> service endpoint, thus achieving the same behavior...probably more
> simply too.)
>
> Lastly, DNS was a little flaky when we first implemented this, which was
> part of the reason we went straight to kubernetes.  Users were using
> dnsmasq with wildcards that worked well for routes, but ended up routing
> services to the router ip instead of pod ip.  Needless to say, there
> were a lot of complications trying to use DNS and debug user problems
> with service resolution.
>
> Hope that helps,
> Rob
>
> ------------------------------------------------------------------------
>
>     Hey Bela!
>
>     No no, the resolution can be done with pure JDK.
>
>     Thanks
>     Sebastian
>
>     On Fri, Aug 19, 2016 at 11:18 AM, Bela Ban <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>         Hi Sebastian
>
>         the usual restrictions apply: if DNS discovery depends on
>         external libs, then it should be hosted in jgroups-extras,
>         otherwise we can add it to JGroups itself.
>
>         On 19/08/16 11:00, Sebastian Laskawiec wrote:
>
>             Hey!
>
>             I've been playing with Kubernetes PetSets [1] for a while
>             and I'd like
>             to share some thoughts. Before I dig in, let me give you
>             some PetSets
>             highlights:
>
>                * PetSets are alpha resources for managing stateful apps
>             in Kubernetes
>                  1.3 (and OpenShift Origin 1.3).
>                * Since this is an alpha resource, there are no
>             guarantees about
>                  backwards compatibility. Alpha resources can also be
>             disabled in
>                  some public cloud providers (you can control which API
>             versions are
>                  accessible [2]).
>                * PetSets allows starting pods in sequence (not relevant
>             for us, but
>                  this is a killer feature for master-slave systems).
>                * Each Pod has it's own unique entry in DNS, which makes
>             discovery
>                  very simple (I'll dig into that a bit later)
>                * Volumes are always mounted to the same Pods, which is
>             very important
>                  in Cache Store scenarios when we restart pods (e.g.
>             Rolling Upgrades
>                  [3]).
>
>             Thoughts and ideas after spending some time playing with
>             this feature:
>
>                * PetSets make discovery a lot easier. It's a combination
>             of two
>                  things - Headless Services [4] which create multiple A
>             records in
>                  DNS and predictable host names. Each Pod has it's own
>             unique DNS
>                  entry following pattern:
>             {PetSetName}-{PodIndex}.{ServiceName} [5].
>                  Here's an example of an Infinispan PetSet deployed on
>             my local
>                  cluster [6]. As you can see we have all domain names
>             and IPs from a
>                  single DNS query.
>                * Maybe we could perform discovery using this mechanism?
>             I'm aware of
>                  DNS discovery implemented in KUBE_PING [7][8] but the
>             code looks
>                  trivial [9] so maybe it should be implement inside
>             JGroups? @Bela -
>                  WDYT?
>                * PetSets do not integrate well with OpenShift 'new-app'
>             command. In
>                  other words, our users will need to use provided yaml
>             (or json)
>                  files to create Infinispan cluster. It's not a
>             show-stopper but it's
>                  a bit less convenient than 'oc new-app'.
>                * Since PetSets are alpha resources they need to be
>             considered as
>                  secondary way to deploy Infinispan on Kubernetes and
>             OpenShift.
>                * Finally, the persistent volumes - since a Pod always
>             gets the same
>                  volume, it would be safe to use any file-based cache store.
>
>             If you'd like to play with PetSets on your local
>             environment, here are
>             necessary yaml files [10].
>
>             Thanks
>             Sebastian
>
>
>             [1] http://kubernetes.io/docs/user-guide/petset/
>             [2] For checking which APIs are accessible, use 'kubectl
>             api-versions'
>             [3]
>             http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
>             [4]
>             http://kubernetes.io/docs/user-guide/services/#headless-services
>             [5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
>             [6]
>             https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
>             [7]
>             https://github.com/jboss-openshift/openshift-ping/tree/master/dns
>             [8]
>             https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
>             [9] http://stackoverflow.com/a/12405896/562699
>             [10] You might need to adjust ImageStream.
>             https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0
>
>
>         --
>         Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>

--
Bela Ban, JGroups lead (http://www.jgroups.org)

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Sebastian Laskawiec
In reply to this post by Sebastian Laskawiec
Hey Rob!

Thanks a lot for clarification!

More comments inlined.

Thanks
Sebastian

On Sat, Aug 20, 2016 at 12:04 AM, Rob Cernich <[hidden email]> wrote:
A couple of things...

re. volumes:
We also need to consider the mounting behavior for scale down scenarios and for overage scenarios when doing upgrades.  For the latter, OpenShift can spin up pods of the new version before the older version pods have terminated.  This may mean that some volumes from the old pods are orphaned.  We did see this when testing A-MQ during upgrades.  With a single pod, the upgrade process caused the new version to have a new mount and the original mount was left orphaned (another upgrade would cause the newer pod to pick up the orphaned mount, leaving the new mount orphaned).  I believe we worked around this by specifying an overage of 0% during upgrades.  This ensured the new pods would pick up the volumes left behind by the old pods.  (Actually, we were using subdirectories in the mount, since all pods shared the same volume.)


I think PetSets try to address this kind of problems. According to the manual page [11], the storage is linked to the Pod ordinal and hostaname and should be stable. 

 
re. dns:
DNS should work fine as-is, but there are a couple things that you need to consider.
1. Service endpoints are only available in DNS after the pod becomes ready (SVC records on the service name).  Because infinispan attaches itself to the cluster, this meant pods were all started as cluster of one, then merged once they noticed the other pods.  This had a significant impact on startup.  Since then, OpenShift has added the ability to query the endpoints associated with a service as soon as the pod is created, which would allow initialization to work correctly.  To make this work, we'd have to change the form of the DNS query to pick up the service endpoints (I forget the naming scheme).

Yes, I agree. Adding nodes one after another will have significant impact on cluster startup time. However it should be safe to query the cluster (and even put data) during rebalance. So I would say, if a node is up, and cluster is not damaged - we should treat it as ready. 

NB - I proposed a HealthCheck API to Infinispan 9 (currently under development) [12][13]. The overall cluster health can be in one of 3 statuses - GREEN (everything is fine), YELLOW (rebalance in progress), RED (cluster not healthy). Kubernetes/OpenShift readiness probe should check if the status is GREEN or YELLOW. The HealthCheck API is attached to the WF management API so you can query it with CURL or using ispn_cli.sh script.

 
Another thing to keep in mind is that looking up pods by labels allows any pod with the specified label to be added to the cluster.  I'm not sure of a use case for this, but it would allow other deployments to be included in the cluster.  (You could also argue that the service is the authority for this and any pod with said label would be added as a service endpoint, thus achieving the same behavior...probably more simply too.)

I think this is a scenario when someone might try to attach Infinispan in library mode (a dependency in WAR file for example) to the Hot Rod cluster. Gustavo answered question like this a while ago [14].

 
Lastly, DNS was a little flaky when we first implemented this, which was part of the reason we went straight to kubernetes.  Users were using dnsmasq with wildcards that worked well for routes, but ended up routing services to the router ip instead of pod ip.  Needless to say, there were a lot of complications trying to use DNS and debug user problems with service resolution.

I think a governing headless service [15] is required here (PetSets require a service but considering how Infinispan works, it should be a headless service in my opinion). 

 


Hope that helps,
Rob


Hey Bela!

No no, the resolution can be done with pure JDK.

Thanks
Sebastian

On Fri, Aug 19, 2016 at 11:18 AM, Bela Ban <[hidden email]> wrote:
Hi Sebastian

the usual restrictions apply: if DNS discovery depends on external libs, then it should be hosted in jgroups-extras, otherwise we can add it to JGroups itself.

On 19/08/16 11:00, Sebastian Laskawiec wrote:
Hey!

I've been playing with Kubernetes PetSets [1] for a while and I'd like
to share some thoughts. Before I dig in, let me give you some PetSets
highlights:

  * PetSets are alpha resources for managing stateful apps in Kubernetes
    1.3 (and OpenShift Origin 1.3).
  * Since this is an alpha resource, there are no guarantees about
    backwards compatibility. Alpha resources can also be disabled in
    some public cloud providers (you can control which API versions are
    accessible [2]).
  * PetSets allows starting pods in sequence (not relevant for us, but
    this is a killer feature for master-slave systems).
  * Each Pod has it's own unique entry in DNS, which makes discovery
    very simple (I'll dig into that a bit later)
  * Volumes are always mounted to the same Pods, which is very important
    in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades
    [3]).

Thoughts and ideas after spending some time playing with this feature:

  * PetSets make discovery a lot easier. It's a combination of two
    things - Headless Services [4] which create multiple A records in
    DNS and predictable host names. Each Pod has it's own unique DNS
    entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5].
    Here's an example of an Infinispan PetSet deployed on my local
    cluster [6]. As you can see we have all domain names and IPs from a
    single DNS query.
  * Maybe we could perform discovery using this mechanism? I'm aware of
    DNS discovery implemented in KUBE_PING [7][8] but the code looks
    trivial [9] so maybe it should be implement inside JGroups? @Bela -
    WDYT?
  * PetSets do not integrate well with OpenShift 'new-app' command. In
    other words, our users will need to use provided yaml (or json)
    files to create Infinispan cluster. It's not a show-stopper but it's
    a bit less convenient than 'oc new-app'.
  * Since PetSets are alpha resources they need to be considered as
    secondary way to deploy Infinispan on Kubernetes and OpenShift.
  * Finally, the persistent volumes - since a Pod always gets the same
    volume, it would be safe to use any file-based cache store.

If you'd like to play with PetSets on your local environment, here are
necessary yaml files [10].

Thanks
Sebastian


[1] http://kubernetes.io/docs/user-guide/petset/
[2] For checking which APIs are accessible, use 'kubectl api-versions'
[3]
http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
[4] http://kubernetes.io/docs/user-guide/services/#headless-services
[5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
[6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
[7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns
[8] https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
[9] http://stackoverflow.com/a/12405896/562699
[10] You might need to adjust ImageStream.
https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0

--
Bela Ban, JGroups lead (http://www.jgroups.org)





_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Sebastian Laskawiec
In reply to this post by Bela Ban
+1, I think using multiple discovery protocols could be a good idea. 

I think it makes sense to use KUBE_PING and DNS together. The same configuration could have been used for both PetSets (which use DNS) and normal Kubernetes/OpenShift deployments (which use KUBE_PING). But that is something that should be tested carefully. 

Thanks
Sebastian

On Sat, Aug 20, 2016 at 9:19 AM, Bela Ban <[hidden email]> wrote:
If we add a DNS discovery protocol, it would only be another discovery protocol among many, and customers can choose which one to use.
I'm also thinking of adding the ability to JGroups to use multiple discovery protocols in the same stack and combine their result sets into one. Not sure though if it makes sense to use KUBE_PING and DNS in the same stack...


On 20/08/16 00:04, Rob Cernich wrote:
A couple of things...

re. volumes:
We also need to consider the mounting behavior for scale down scenarios
and for overage scenarios when doing upgrades.  For the latter,
OpenShift can spin up pods of the new version before the older version
pods have terminated.  This may mean that some volumes from the old pods
are orphaned.  We did see this when testing A-MQ during upgrades.  With
a single pod, the upgrade process caused the new version to have a new
mount and the original mount was left orphaned (another upgrade would
cause the newer pod to pick up the orphaned mount, leaving the new mount
orphaned).  I believe we worked around this by specifying an overage of
0% during upgrades.  This ensured the new pods would pick up the volumes
left behind by the old pods.  (Actually, we were using subdirectories in
the mount, since all pods shared the same volume.)

re. dns:
DNS should work fine as-is, but there are a couple things that you need
to consider.
1. Service endpoints are only available in DNS after the pod becomes
ready (SVC records on the service name).  Because infinispan attaches
itself to the cluster, this meant pods were all started as cluster of
one, then merged once they noticed the other pods.  This had a
significant impact on startup.  Since then, OpenShift has added the
ability to query the endpoints associated with a service as soon as the
pod is created, which would allow initialization to work correctly.  To
make this work, we'd have to change the form of the DNS query to pick up
the service endpoints (I forget the naming scheme).

Another thing to keep in mind is that looking up pods by labels allows
any pod with the specified label to be added to the cluster.  I'm not
sure of a use case for this, but it would allow other deployments to be
included in the cluster.  (You could also argue that the service is the
authority for this and any pod with said label would be added as a
service endpoint, thus achieving the same behavior...probably more
simply too.)

Lastly, DNS was a little flaky when we first implemented this, which was
part of the reason we went straight to kubernetes.  Users were using
dnsmasq with wildcards that worked well for routes, but ended up routing
services to the router ip instead of pod ip.  Needless to say, there
were a lot of complications trying to use DNS and debug user problems
with service resolution.

Hope that helps,
Rob

------------------------------------------------------------------------

    Hey Bela!

    No no, the resolution can be done with pure JDK.

    Thanks
    Sebastian

    On Fri, Aug 19, 2016 at 11:18 AM, Bela Ban <[hidden email]
    <mailto:[hidden email]>> wrote:

        Hi Sebastian

        the usual restrictions apply: if DNS discovery depends on
        external libs, then it should be hosted in jgroups-extras,
        otherwise we can add it to JGroups itself.

        On 19/08/16 11:00, Sebastian Laskawiec wrote:

            Hey!

            I've been playing with Kubernetes PetSets [1] for a while
            and I'd like
            to share some thoughts. Before I dig in, let me give you
            some PetSets
            highlights:

               * PetSets are alpha resources for managing stateful apps
            in Kubernetes
                 1.3 (and OpenShift Origin 1.3).
               * Since this is an alpha resource, there are no
            guarantees about
                 backwards compatibility. Alpha resources can also be
            disabled in
                 some public cloud providers (you can control which API
            versions are
                 accessible [2]).
               * PetSets allows starting pods in sequence (not relevant
            for us, but
                 this is a killer feature for master-slave systems).
               * Each Pod has it's own unique entry in DNS, which makes
            discovery
                 very simple (I'll dig into that a bit later)
               * Volumes are always mounted to the same Pods, which is
            very important
                 in Cache Store scenarios when we restart pods (e.g.
            Rolling Upgrades
                 [3]).

            Thoughts and ideas after spending some time playing with
            this feature:

               * PetSets make discovery a lot easier. It's a combination
            of two
                 things - Headless Services [4] which create multiple A
            records in
                 DNS and predictable host names. Each Pod has it's own
            unique DNS
                 entry following pattern:
            {PetSetName}-{PodIndex}.{ServiceName} [5].
                 Here's an example of an Infinispan PetSet deployed on
            my local
                 cluster [6]. As you can see we have all domain names
            and IPs from a
                 single DNS query.
               * Maybe we could perform discovery using this mechanism?
            I'm aware of
                 DNS discovery implemented in KUBE_PING [7][8] but the
            code looks
                 trivial [9] so maybe it should be implement inside
            JGroups? @Bela -
                 WDYT?
               * PetSets do not integrate well with OpenShift 'new-app'
            command. In
                 other words, our users will need to use provided yaml
            (or json)
                 files to create Infinispan cluster. It's not a
            show-stopper but it's
                 a bit less convenient than 'oc new-app'.
               * Since PetSets are alpha resources they need to be
            considered as
                 secondary way to deploy Infinispan on Kubernetes and
            OpenShift.
               * Finally, the persistent volumes - since a Pod always
            gets the same
                 volume, it would be safe to use any file-based cache store.

            If you'd like to play with PetSets on your local
            environment, here are
            necessary yaml files [10].

            Thanks
            Sebastian


            [1] http://kubernetes.io/docs/user-guide/petset/
            [2] For checking which APIs are accessible, use 'kubectl
            api-versions'
            [3]
            http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
            [4]
            http://kubernetes.io/docs/user-guide/services/#headless-services
            [5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
            [6]
            https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
            [7]
            https://github.com/jboss-openshift/openshift-ping/tree/master/dns
            [8]
            https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
            [9] http://stackoverflow.com/a/12405896/562699
            [10] You might need to adjust ImageStream.
            https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0


        --
        Bela Ban, JGroups lead (http://www.jgroups.org)




--
Bela Ban, JGroups lead (http://www.jgroups.org)



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Sebastian Laskawiec
In reply to this post by Sebastian Laskawiec
Hey Kevin!

The timing for looking into PetSets is perfect I think. Kubernetes upstream folks are thinking about Sticky IPs [16] (which are not essential for Infinispan but other projects might be really interested in this) and I asked some time ago about exposing PetSets to the outside world [17] (which is essential for accessing the cluster using Hot Rod client). 

Please keep me in the loop. I'm really interested in this.

Thanks
Sebastian



On Sun, Aug 21, 2016 at 7:05 PM, Kevin Conner <[hidden email]> wrote:
Apologies, I’ve been travelling to London for the JDG meeting.

On 19 Aug 2016, at 10:00, Sebastian Laskawiec <[hidden email]> wrote:
> I've been playing with Kubernetes PetSets [1] for a while and I'd like to share some thoughts. Before I dig in, let me give you some PetSets highlights:
>       • PetSets are alpha resources for managing stateful apps in Kubernetes 1.3 (and OpenShift Origin 1.3).
>       • Since this is an alpha resource, there are no guarantees about backwards compatibility. Alpha resources can also be disabled in some public cloud providers (you can control which API versions are accessible [2]).
>       • PetSets allows starting pods in sequence (not relevant for us, but this is a killer feature for master-slave systems).
>       • Each Pod has it's own unique entry in DNS, which makes discovery very simple (I'll dig into that a bit later)
>       • Volumes are always mounted to the same Pods, which is very important in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades [3]).
> Thoughts and ideas after spending some time playing with this feature:
>       • PetSets make discovery a lot easier. It's a combination of two things - Headless Services [4] which create multiple A records in DNS and predictable host names. Each Pod has it's own unique DNS entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5]. Here's an example of an Infinispan PetSet deployed on my local cluster [6]. As you can see we have all domain names and IPs from a single DNS query.
>       • Maybe we could perform discovery using this mechanism? I'm aware of DNS discovery implemented in KUBE_PING [7][8] but the code looks trivial [9] so maybe it should be implement inside JGroups? @Bela - WDYT?
>       • PetSets do not integrate well with OpenShift 'new-app' command. In other words, our users will need to use provided yaml (or json) files to create Infinispan cluster. It's not a show-stopper but it's a bit less convenient than 'oc new-app'.
>       • Since PetSets are alpha resources they need to be considered as secondary way to deploy Infinispan on Kubernetes and OpenShift.
>       • Finally, the persistent volumes - since a Pod always gets the same volume, it would be safe to use any file-based cache store.
> If you'd like to play with PetSets on your local environment, here are necessary yaml files [10].

PetSets are still in extremely early stages and I am part of a working group that has recently formed to cover the introduction of this within the products, including driving any necessary changes back upstream.  There will be Cloud Enablement involvement in this effort and any adoption of petsets will likely go through the existing project.

        Kev

--
JBoss by Red Hat



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Alan Field
In reply to this post by Sebastian Laskawiec
A little late to the discussion but if all of the pods share the same set of volumes, couldn't you also use the FILE_PING discovery protocol between the pods?

Thanks,
Alan


From: "Sebastian Laskawiec" <[hidden email]>
To: "infinispan -Dev List" <[hidden email]>, "Bela Ban" <[hidden email]>, "Kevin Conner" <[hidden email]>, "Rob Cernich" <[hidden email]>
Sent: Friday, August 19, 2016 5:00:12 AM
Subject: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Hey!

I've been playing with Kubernetes PetSets [1] for a while and I'd like to share some thoughts. Before I dig in, let me give you some PetSets highlights:
  • PetSets are alpha resources for managing stateful apps in Kubernetes 1.3 (and OpenShift Origin 1.3).
  • Since this is an alpha resource, there are no guarantees about backwards compatibility. Alpha resources can also be disabled in some public cloud providers (you can control which API versions are accessible [2]).
  • PetSets allows starting pods in sequence (not relevant for us, but this is a killer feature for master-slave systems).
  • Each Pod has it's own unique entry in DNS, which makes discovery very simple (I'll dig into that a bit later)
  • Volumes are always mounted to the same Pods, which is very important in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades [3]).
Thoughts and ideas after spending some time playing with this feature:
  • PetSets make discovery a lot easier. It's a combination of two things - Headless Services [4] which create multiple A records in DNS and predictable host names. Each Pod has it's own unique DNS entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5]. Here's an example of an Infinispan PetSet deployed on my local cluster [6]. As you can see we have all domain names and IPs from a single DNS query.
  • Maybe we could perform discovery using this mechanism? I'm aware of DNS discovery implemented in KUBE_PING [7][8] but the code looks trivial [9] so maybe it should be implement inside JGroups? @Bela - WDYT?
  • PetSets do not integrate well with OpenShift 'new-app' command. In other words, our users will need to use provided yaml (or json) files to create Infinispan cluster. It's not a show-stopper but it's a bit less convenient than 'oc new-app'.
  • Since PetSets are alpha resources they need to be considered as secondary way to deploy Infinispan on Kubernetes and OpenShift.
  • Finally, the persistent volumes - since a Pod always gets the same volume, it would be safe to use any file-based cache store.
If you'd like to play with PetSets on your local environment, here are necessary yaml files [10].

Thanks
Sebastian


[2] For checking which APIs are accessible, use 'kubectl api-versions'

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Sebastian Laskawiec
In reply to this post by Sebastian Laskawiec
Thanks Rob! That clarifies a lot!

On Mon, Aug 22, 2016 at 4:02 PM, Rob Cernich <[hidden email]> wrote:



Hey Rob!

Thanks a lot for clarification!

More comments inlined.

Thanks
Sebastian

On Sat, Aug 20, 2016 at 12:04 AM, Rob Cernich <[hidden email]> wrote:
A couple of things...

re. volumes:
We also need to consider the mounting behavior for scale down scenarios and for overage scenarios when doing upgrades.  For the latter, OpenShift can spin up pods of the new version before the older version pods have terminated.  This may mean that some volumes from the old pods are orphaned.  We did see this when testing A-MQ during upgrades.  With a single pod, the upgrade process caused the new version to have a new mount and the original mount was left orphaned (another upgrade would cause the newer pod to pick up the orphaned mount, leaving the new mount orphaned).  I believe we worked around this by specifying an overage of 0% during upgrades.  This ensured the new pods would pick up the volumes left behind by the old pods.  (Actually, we were using subdirectories in the mount, since all pods shared the same volume.)


I think PetSets try to address this kind of problems. According to the manual page [11], the storage is linked to the Pod ordinal and hostaname and should be stable. 


re. dns:
DNS should work fine as-is, but there are a couple things that you need to consider.
1. Service endpoints are only available in DNS after the pod becomes ready (SVC records on the service name).  Because infinispan attaches itself to the cluster, this meant pods were all started as cluster of one, then merged once they noticed the other pods.  This had a significant impact on startup.  Since then, OpenShift has added the ability to query the endpoints associated with a service as soon as the pod is created, which would allow initialization to work correctly.  To make this work, we'd have to change the form of the DNS query to pick up the service endpoints (I forget the naming scheme).

Yes, I agree. Adding nodes one after another will have significant impact on cluster startup time. However it should be safe to query the cluster (and even put data) during rebalance. So I would say, if a node is up, and cluster is not damaged - we should treat it as ready. 
To be clear, the issue was around the lifecycle and interaction with the readiness probe.  OpenShift/Kubernetes only add the pod to the service once it's "ready."  Our readiness probe defines ready as startup is complete (i.e. x of y services started).  The issue with this is that as pods come up, they only see other pods that are ready when they initialize, so if you start with an initial cluster size of five, the new nodes don't see any of the other nodes until they finish startup and refresh the list of nodes, after which they all have to merge with each other.  This had a significant impact on performance.  Since then, a feature has been added which allows you to query DNS for all endpoints regardless of ready state.  This allows nodes to see the other nodes before they become ready, which allows for a more natural cluster formation.  To reiterate, the impact on startup was significant, especially when scaling up under load.

NB - I proposed a HealthCheck API to Infinispan 9 (currently under development) [12][13]. The overall cluster health can be in one of 3 statuses - GREEN (everything is fine), YELLOW (rebalance in progress), RED (cluster not healthy). Kubernetes/OpenShift readiness probe should check if the status is GREEN or YELLOW. The HealthCheck API is attached to the WF management API so you can query it with CURL or using ispn_cli.sh script.


Another thing to keep in mind is that looking up pods by labels allows any pod with the specified label to be added to the cluster.  I'm not sure of a use case for this, but it would allow other deployments to be included in the cluster.  (You could also argue that the service is the authority for this and any pod with said label would be added as a service endpoint, thus achieving the same behavior...probably more simply too.)

I think this is a scenario when someone might try to attach Infinispan in library mode (a dependency in WAR file for example) to the Hot Rod cluster. Gustavo answered question like this a while ago [14].

Lastly, DNS was a little flaky when we first implemented this, which was part of the reason we went straight to kubernetes.  Users were using dnsmasq with wildcards that worked well for routes, but ended up routing services to the router ip instead of pod ip.  Needless to say, there were a lot of complications trying to use DNS and debug user problems with service resolution.

I think a governing headless service [15] is required here (PetSets require a service but considering how Infinispan works, it should be a headless service in my opinion). 



Hope that helps,
Rob


Hey Bela!

No no, the resolution can be done with pure JDK.

Thanks
Sebastian

On Fri, Aug 19, 2016 at 11:18 AM, Bela Ban <[hidden email]> wrote:
Hi Sebastian

the usual restrictions apply: if DNS discovery depends on external libs, then it should be hosted in jgroups-extras, otherwise we can add it to JGroups itself.

On 19/08/16 11:00, Sebastian Laskawiec wrote:
Hey!

I've been playing with Kubernetes PetSets [1] for a while and I'd like
to share some thoughts. Before I dig in, let me give you some PetSets
highlights:

  * PetSets are alpha resources for managing stateful apps in Kubernetes
    1.3 (and OpenShift Origin 1.3).
  * Since this is an alpha resource, there are no guarantees about
    backwards compatibility. Alpha resources can also be disabled in
    some public cloud providers (you can control which API versions are
    accessible [2]).
  * PetSets allows starting pods in sequence (not relevant for us, but
    this is a killer feature for master-slave systems).
  * Each Pod has it's own unique entry in DNS, which makes discovery
    very simple (I'll dig into that a bit later)
  * Volumes are always mounted to the same Pods, which is very important
    in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades
    [3]).

Thoughts and ideas after spending some time playing with this feature:

  * PetSets make discovery a lot easier. It's a combination of two
    things - Headless Services [4] which create multiple A records in
    DNS and predictable host names. Each Pod has it's own unique DNS
    entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5].
    Here's an example of an Infinispan PetSet deployed on my local
    cluster [6]. As you can see we have all domain names and IPs from a
    single DNS query.
  * Maybe we could perform discovery using this mechanism? I'm aware of
    DNS discovery implemented in KUBE_PING [7][8] but the code looks
    trivial [9] so maybe it should be implement inside JGroups? @Bela -
    WDYT?
  * PetSets do not integrate well with OpenShift 'new-app' command. In
    other words, our users will need to use provided yaml (or json)
    files to create Infinispan cluster. It's not a show-stopper but it's
    a bit less convenient than 'oc new-app'.
  * Since PetSets are alpha resources they need to be considered as
    secondary way to deploy Infinispan on Kubernetes and OpenShift.
  * Finally, the persistent volumes - since a Pod always gets the same
    volume, it would be safe to use any file-based cache store.

If you'd like to play with PetSets on your local environment, here are
necessary yaml files [10].

Thanks
Sebastian


[1] http://kubernetes.io/docs/user-guide/petset/
[2] For checking which APIs are accessible, use 'kubectl api-versions'
[3]
http://infinispan.org/docs/stable/user_guide/user_guide.html#_Rolling_chapter
[4] http://kubernetes.io/docs/user-guide/services/#headless-services
[5] http://kubernetes.io/docs/user-guide/petset/#peer-discovery
[6] https://gist.github.com/slaskawi/0866e63a39276f8ab66376229716a676
[7] https://github.com/jboss-openshift/openshift-ping/tree/master/dns
[8] https://github.com/jgroups-extras/jgroups-kubernetes/tree/master/dns
[9] http://stackoverflow.com/a/12405896/562699
[10] You might need to adjust ImageStream.
https://gist.github.com/slaskawi/7cffb5588dabb770f654557579c5f2d0

--
Bela Ban, JGroups lead (http://www.jgroups.org)







_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets

Sebastian Laskawiec
In reply to this post by Alan Field
Hey Alan!

Yeah, that could work :) I'll be doing some experiments with volumes shortly so I'll definitely check that!

Thanks
Sebastian

On Mon, Aug 22, 2016 at 8:14 PM, Alan Field <[hidden email]> wrote:
A little late to the discussion but if all of the pods share the same set of volumes, couldn't you also use the FILE_PING discovery protocol between the pods?

Thanks,
Alan


From: "Sebastian Laskawiec" <[hidden email]>
To: "infinispan -Dev List" <[hidden email]>, "Bela Ban" <[hidden email]>, "Kevin Conner" <[hidden email]>, "Rob Cernich" <[hidden email]>
Sent: Friday, August 19, 2016 5:00:12 AM
Subject: [infinispan-dev] Infinispan and OpenShift/Kubernetes PetSets


Hey!

I've been playing with Kubernetes PetSets [1] for a while and I'd like to share some thoughts. Before I dig in, let me give you some PetSets highlights:
  • PetSets are alpha resources for managing stateful apps in Kubernetes 1.3 (and OpenShift Origin 1.3).
  • Since this is an alpha resource, there are no guarantees about backwards compatibility. Alpha resources can also be disabled in some public cloud providers (you can control which API versions are accessible [2]).
  • PetSets allows starting pods in sequence (not relevant for us, but this is a killer feature for master-slave systems).
  • Each Pod has it's own unique entry in DNS, which makes discovery very simple (I'll dig into that a bit later)
  • Volumes are always mounted to the same Pods, which is very important in Cache Store scenarios when we restart pods (e.g. Rolling Upgrades [3]).
Thoughts and ideas after spending some time playing with this feature:
  • PetSets make discovery a lot easier. It's a combination of two things - Headless Services [4] which create multiple A records in DNS and predictable host names. Each Pod has it's own unique DNS entry following pattern: {PetSetName}-{PodIndex}.{ServiceName} [5]. Here's an example of an Infinispan PetSet deployed on my local cluster [6]. As you can see we have all domain names and IPs from a single DNS query.
  • Maybe we could perform discovery using this mechanism? I'm aware of DNS discovery implemented in KUBE_PING [7][8] but the code looks trivial [9] so maybe it should be implement inside JGroups? @Bela - WDYT?
  • PetSets do not integrate well with OpenShift 'new-app' command. In other words, our users will need to use provided yaml (or json) files to create Infinispan cluster. It's not a show-stopper but it's a bit less convenient than 'oc new-app'.
  • Since PetSets are alpha resources they need to be considered as secondary way to deploy Infinispan on Kubernetes and OpenShift.
  • Finally, the persistent volumes - since a Pod always gets the same volume, it would be safe to use any file-based cache store.
If you'd like to play with PetSets on your local environment, here are necessary yaml files [10].

Thanks
Sebastian


[2] For checking which APIs are accessible, use 'kubectl api-versions'

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...