Quantcast

[infinispan-dev] Health check use cases

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[infinispan-dev] Health check use cases

Sebastian Laskawiec
Dear Community,

I'd like to ask you for help. I'm currently sketching a design for a REST health check endpoint for Infinispan and I'm trying to imagine possible use cases. 

Could you please give me a hand and tell me what functionalities are important for you? Would you like to be able to check status per-cache or maybe a red (not healthy), green (healthy), yellow (healthy, rebalance in progress) cluster status is sufficient? What kind of information do you expect to be there?

Thanks
Sebastian

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Andrea Cosentino
Hi Sebastian,

This type of feature can be very useful for liveness and readiness probes in a Kubernetes cluster [1].

Maybe you can think at check status per-cache but also at whole server level.


Thanks!
 
--
Andrea Cosentino 
----------------------------------
Apache Camel PMC Member
Apache Karaf Committer
Apache Servicemix Committer
Twitter: @oscerd2
Github: oscerd


On Tuesday, July 26, 2016 7:11 AM, Sebastian Laskawiec <[hidden email]> wrote:


Dear Community,

I'd like to ask you for help. I'm currently sketching a design for a REST health check endpoint for Infinispan and I'm trying to imagine possible use cases. 

Could you please give me a hand and tell me what functionalities are important for you? Would you like to be able to check status per-cache or maybe a red (not healthy), green (healthy), yellow (healthy, rebalance in progress) cluster status is sufficient? What kind of information do you expect to be there?

Thanks
Sebastian

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Vojtech Juranek
In reply to this post by Sebastian Laskawiec
On Tuesday 26 July 2016 07:10:16 Sebastian Laskawiec wrote:
> I'm currently sketching a design for a REST
> health check endpoint for Infinispan

if it's not too broad, I'd include also various information about the cluster
- e.g. number of machines in the cluster, recent exceptions in the log (or
dump of N lines of log) etc. If would be useful at least for testing purposes
so that we won't have to gather various information via JMX and CLI
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Tristan Tarrant-2
In reply to this post by Sebastian Laskawiec
On 26/07/16 07:10, Sebastian Laskawiec wrote:
> Dear Community,
>
> I'd like to ask you for help. I'm currently sketching a design for a
> REST health check endpoint for Infinispan and I'm trying to imagine
> possible use cases.
The health-check should be implemented as an MBean initially, with the
ability to expose it via alternative implementations later. The server
RESTful endpoint should be registered with the management interface via
a special handler.
A cache and cachemanager's health is determined by a combination of
parameters and we probably should allow for a user-pluggable checker. We
already expose a number of statuses already, although obviously this
would be an aggregate.

>
> Could you please give me a hand and tell me what functionalities are
> important for you? Would you like to be able to check status per-cache
> or maybe a red (not healthy), green (healthy), yellow (healthy,
> rebalance in progress) cluster status is sufficient? What kind of
> information do you expect to be there?
I wouldn't want this to be overly complex: a simple OK, KO should be
sufficient. Additional detail may be optionally present, but not a
requirement.


Tristan

--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Sebastian Laskawiec
In reply to this post by Andrea Cosentino
Hey Andrea!

Exactly! One of the most important use cases is Kubernetes. 

I also absolutely agree - per cache and per cache manager level sounds reasonable.

Thanks
Sebastian

On Tue, Jul 26, 2016 at 8:25 AM, Andrea Cosentino <[hidden email]> wrote:
Hi Sebastian,

This type of feature can be very useful for liveness and readiness probes in a Kubernetes cluster [1].

Maybe you can think at check status per-cache but also at whole server level.


Thanks!
 
--
Andrea Cosentino 
----------------------------------
Apache Camel PMC Member
Apache Karaf Committer
Apache Servicemix Committer
Twitter: @oscerd2
Github: oscerd


On Tuesday, July 26, 2016 7:11 AM, Sebastian Laskawiec <[hidden email]> wrote:


Dear Community,

I'd like to ask you for help. I'm currently sketching a design for a REST health check endpoint for Infinispan and I'm trying to imagine possible use cases. 

Could you please give me a hand and tell me what functionalities are important for you? Would you like to be able to check status per-cache or maybe a red (not healthy), green (healthy), yellow (healthy, rebalance in progress) cluster status is sufficient? What kind of information do you expect to be there?

Thanks
Sebastian

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Sebastian Laskawiec
In reply to this post by Vojtech Juranek
Hey Vojtech!

JMX and CLI integration sounds very interesting. I also like the idea of exposing log and exception dump. 

Thanks a lot for the input!
Sebastian

On Tue, Jul 26, 2016 at 9:00 AM, Vojtech Juranek <[hidden email]> wrote:
On Tuesday 26 July 2016 07:10:16 Sebastian Laskawiec wrote:
> I'm currently sketching a design for a REST
> health check endpoint for Infinispan

if it's not too broad, I'd include also various information about the cluster
- e.g. number of machines in the cluster, recent exceptions in the log (or
dump of N lines of log) etc. If would be useful at least for testing purposes
so that we won't have to gather various information via JMX and CLI
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Wolf Fink
Do we expose historical data for the cluster view. Often it is important to see whether there are view changes, rebalancing and unexpected leave/merge events where nodes are kicked by JGroups.
Having special entries for controlled view change and sudden view changes might be good

On Tue, Jul 26, 2016 at 10:06 AM, Sebastian Laskawiec <[hidden email]> wrote:
Hey Vojtech!

JMX and CLI integration sounds very interesting. I also like the idea of exposing log and exception dump. 

Thanks a lot for the input!
Sebastian

On Tue, Jul 26, 2016 at 9:00 AM, Vojtech Juranek <[hidden email]> wrote:
On Tuesday 26 July 2016 07:10:16 Sebastian Laskawiec wrote:
> I'm currently sketching a design for a REST
> health check endpoint for Infinispan

if it's not too broad, I'd include also various information about the cluster
- e.g. number of machines in the cluster, recent exceptions in the log (or
dump of N lines of log) etc. If would be useful at least for testing purposes
so that we won't have to gather various information via JMX and CLI
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Sebastian Laskawiec
In reply to this post by Tristan Tarrant-2
Hey Tristan!

Comments inlined.

Thanks
Sebastian

On Tue, Jul 26, 2016 at 9:50 AM, Tristan Tarrant <[hidden email]> wrote:
On 26/07/16 07:10, Sebastian Laskawiec wrote:
> Dear Community,
>
> I'd like to ask you for help. I'm currently sketching a design for a
> REST health check endpoint for Infinispan and I'm trying to imagine
> possible use cases.
The health-check should be implemented as an MBean initially, with the
ability to expose it via alternative implementations later. The server
RESTful endpoint should be registered with the management interface via
a special handler.

Yes, I think it's a good idea. We could even use tools like Jolokia [1] to expose MBeans through REST interface (it can be added to standalone.conf to the bootstrap classpath). Alternatively we could use JDK embedded HTTP Server [2].

The only restriction that comes into my mind is that we shouldn't allow duplicated domains when exposing MBeans through REST interface. Otherwise it will be very hard to construct proper paths for the endpoint.

 
A cache and cachemanager's health is determined by a combination of
parameters and we probably should allow for a user-pluggable checker. We
already expose a number of statuses already, although obviously this
would be an aggregate.

Could you please elaborate more on that? How do we expose this information? Are you referring to Infinispan Stats [3]?

I also though about supporting queries somehow. An imaginary example from the top of my head could look like the following:

http://$ISPN/_health?cluster.nodes=3&MyCacheManager.MyCache.status=DEGRADED //<-- This would return 200 OK if we have 3 nodes and given cache is in degraded mode.
http://$ISPN/_health?cluster.nodes=3&MyCacheManager.rebalance=IN_PROGRESS //<-- Checks if we have 3 nodes and rebalance is in proress

 

>
> Could you please give me a hand and tell me what functionalities are
> important for you? Would you like to be able to check status per-cache
> or maybe a red (not healthy), green (healthy), yellow (healthy,
> rebalance in progress) cluster status is sufficient? What kind of
> information do you expect to be there?
I wouldn't want this to be overly complex: a simple OK, KO should be
sufficient. Additional detail may be optionally present, but not a
requirement.

I think we will need at least a 3rd state - yellow or something like this. This would mean that a rebalance is in progress of a node is joining/leaving. In other words - the cluster accepts requests but don't touch the nodes!
 


Tristan

--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Sebastian Laskawiec
In reply to this post by Wolf Fink
Hey Wolf!

Technically it's possible but I'm not sure if we should do this. I think this is a responsibility of monitoring tools (e.g. Splunk, Kibana or even Zabbix).

Thanks
Sebastian

On Tue, Jul 26, 2016 at 10:21 AM, Wolf Fink <[hidden email]> wrote:
Do we expose historical data for the cluster view. Often it is important to see whether there are view changes, rebalancing and unexpected leave/merge events where nodes are kicked by JGroups.
Having special entries for controlled view change and sudden view changes might be good

On Tue, Jul 26, 2016 at 10:06 AM, Sebastian Laskawiec <[hidden email]> wrote:
Hey Vojtech!

JMX and CLI integration sounds very interesting. I also like the idea of exposing log and exception dump. 

Thanks a lot for the input!
Sebastian

On Tue, Jul 26, 2016 at 9:00 AM, Vojtech Juranek <[hidden email]> wrote:
On Tuesday 26 July 2016 07:10:16 Sebastian Laskawiec wrote:
> I'm currently sketching a design for a REST
> health check endpoint for Infinispan

if it's not too broad, I'd include also various information about the cluster
- e.g. number of machines in the cluster, recent exceptions in the log (or
dump of N lines of log) etc. If would be useful at least for testing purposes
so that we won't have to gather various information via JMX and CLI
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Tristan Tarrant-2
In reply to this post by Sebastian Laskawiec
On 26/07/16 10:24, Sebastian Laskawiec wrote:

> Hey Tristan!
>
> Comments inlined.
>
> Thanks
> Sebastian
>
> On Tue, Jul 26, 2016 at 9:50 AM, Tristan Tarrant <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On 26/07/16 07:10, Sebastian Laskawiec wrote:
>     > Dear Community,
>     >
>     > I'd like to ask you for help. I'm currently sketching a design for a
>     > REST health check endpoint for Infinispan and I'm trying to imagine
>     > possible use cases.
>     The health-check should be implemented as an MBean initially, with the
>     ability to expose it via alternative implementations later. The server
>     RESTful endpoint should be registered with the management
>     interface via
>     a special handler.
>
>
> Yes, I think it's a good idea. We could even use tools like Jolokia
> [1] to expose MBeans through REST interface (it can be added to
> standalone.conf to the bootstrap classpath). Alternatively we could
> use JDK embedded HTTP Server [2].

No, for server we would not use Jolokia but rely on the management HTTP
server (the one that handles port 9990 already).

>
>     A cache and cachemanager's health is determined by a combination of
>     parameters and we probably should allow for a user-pluggable
>     checker. We
>     already expose a number of statuses already, although obviously this
>     would be an aggregate.
>
>
> Could you please elaborate more on that? How do we expose this
> information? Are you referring to Infinispan Stats [3]?
>
> I also though about supporting queries somehow. An imaginary example
> from the top of my head could look like the following:
>
> http://$ISPN/_health?cluster.nodes=3&MyCacheManager.MyCache.status=DEGRADED 
> //<-- This would return 200 OK if we have 3 nodes and given cache is
> in degraded mode.
> http://$ISPN/_health?cluster.nodes=3&MyCacheManager.rebalance=IN_PROGRESS 
> //<-- Checks if we have 3 nodes and rebalance is in proress
>
> [3]
> https://github.com/infinispan/infinispan/tree/8.2.x/core/src/main/java/org/infinispan/stats
>
>
>     >
>     > Could you please give me a hand and tell me what functionalities are
>     > important for you? Would you like to be able to check status
>     per-cache
>     > or maybe a red (not healthy), green (healthy), yellow (healthy,
>     > rebalance in progress) cluster status is sufficient? What kind of
>     > information do you expect to be there?
>     I wouldn't want this to be overly complex: a simple OK, KO should be
>     sufficient. Additional detail may be optionally present, but not a
>     requirement.
>
>
> I think we will need at least a 3rd state - yellow or something like
> this. This would mean that a rebalance is in progress of a node is
> joining/leaving. In other words - the cluster accepts requests but
> don't touch the nodes!
Agreed.

Tristan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Sebastian Laskawiec
Just for clarification - if we used the management HTTP server - would it be possible to expose health endpoints in Library mode? I think the library use case might be also very important.

On Tue, Jul 26, 2016 at 10:34 AM, Tristan Tarrant <[hidden email]> wrote:
On 26/07/16 10:24, Sebastian Laskawiec wrote:
> Hey Tristan!
>
> Comments inlined.
>
> Thanks
> Sebastian
>
> On Tue, Jul 26, 2016 at 9:50 AM, Tristan Tarrant <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On 26/07/16 07:10, Sebastian Laskawiec wrote:
>     > Dear Community,
>     >
>     > I'd like to ask you for help. I'm currently sketching a design for a
>     > REST health check endpoint for Infinispan and I'm trying to imagine
>     > possible use cases.
>     The health-check should be implemented as an MBean initially, with the
>     ability to expose it via alternative implementations later. The server
>     RESTful endpoint should be registered with the management
>     interface via
>     a special handler.
>
>
> Yes, I think it's a good idea. We could even use tools like Jolokia
> [1] to expose MBeans through REST interface (it can be added to
> standalone.conf to the bootstrap classpath). Alternatively we could
> use JDK embedded HTTP Server [2].

No, for server we would not use Jolokia but rely on the management HTTP
server (the one that handles port 9990 already).

>
>     A cache and cachemanager's health is determined by a combination of
>     parameters and we probably should allow for a user-pluggable
>     checker. We
>     already expose a number of statuses already, although obviously this
>     would be an aggregate.
>
>
> Could you please elaborate more on that? How do we expose this
> information? Are you referring to Infinispan Stats [3]?
>
> I also though about supporting queries somehow. An imaginary example
> from the top of my head could look like the following:
>
> http://$ISPN/_health?cluster.nodes=3&MyCacheManager.MyCache.status=DEGRADED
> //<-- This would return 200 OK if we have 3 nodes and given cache is
> in degraded mode.
> http://$ISPN/_health?cluster.nodes=3&MyCacheManager.rebalance=IN_PROGRESS
> //<-- Checks if we have 3 nodes and rebalance is in proress
>
> [3]
> https://github.com/infinispan/infinispan/tree/8.2.x/core/src/main/java/org/infinispan/stats
>
>
>     >
>     > Could you please give me a hand and tell me what functionalities are
>     > important for you? Would you like to be able to check status
>     per-cache
>     > or maybe a red (not healthy), green (healthy), yellow (healthy,
>     > rebalance in progress) cluster status is sufficient? What kind of
>     > information do you expect to be there?
>     I wouldn't want this to be overly complex: a simple OK, KO should be
>     sufficient. Additional detail may be optionally present, but not a
>     requirement.
>
>
> I think we will need at least a 3rd state - yellow or something like
> this. This would mean that a rebalance is in progress of a node is
> joining/leaving. In other words - the cluster accepts requests but
> don't touch the nodes!
Agreed.

Tristan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Tristan Tarrant-2
In reply to this post by Sebastian Laskawiec
This, and complex monitoring rules, will be handled by the Hawkular
integration.
That is why for the use-case Sebastian is considering we need to keep it
as simple and as linear as possible.

We code datagrids, not monitoring tools :)

Tristan

On 26/07/16 10:31, Sebastian Laskawiec wrote:

> Hey Wolf!
>
> Technically it's possible but I'm not sure if we should do this. I
> think this is a responsibility of monitoring tools (e.g. Splunk,
> Kibana or even Zabbix).
>
> Thanks
> Sebastian
>
> On Tue, Jul 26, 2016 at 10:21 AM, Wolf Fink <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Do we expose historical data for the cluster view. Often it is
>     important to see whether there are view changes, rebalancing and
>     unexpected leave/merge events where nodes are kicked by JGroups.
>     Having special entries for controlled view change and sudden view
>     changes might be good
>
>     On Tue, Jul 26, 2016 at 10:06 AM, Sebastian Laskawiec
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         Hey Vojtech!
>
>         JMX and CLI integration sounds very interesting. I also like
>         the idea of exposing log and exception dump.
>
>         Thanks a lot for the input!
>         Sebastian
>
>         On Tue, Jul 26, 2016 at 9:00 AM, Vojtech Juranek
>         <[hidden email] <mailto:[hidden email]>> wrote:
>
>             On Tuesday 26 July 2016 07:10:16 Sebastian Laskawiec wrote:
>             > I'm currently sketching a design for a REST
>             > health check endpoint for Infinispan
>
>             if it's not too broad, I'd include also various
>             information about the cluster
>             - e.g. number of machines in the cluster, recent
>             exceptions in the log (or
>             dump of N lines of log) etc. If would be useful at least
>             for testing purposes
>             so that we won't have to gather various information via
>             JMX and CLI
>             _______________________________________________
>             infinispan-dev mailing list
>             [hidden email]
>             <mailto:[hidden email]>
>             https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>         _______________________________________________
>         infinispan-dev mailing list
>         [hidden email]
>         <mailto:[hidden email]>
>         https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Tristan Tarrant-2
In reply to this post by Sebastian Laskawiec
On 26/07/16 10:38, Sebastian Laskawiec wrote:
> Just for clarification - if we used the management HTTP server - would
> it be possible to expose health endpoints in Library mode? I think the
> library use case might be also very important.
>
Library mode can be plain JMX, which is the de-facto standard for Java
application monitoring. It should be up to the environment setup to
tunnel that to whatever is needed.
However, we do plan to support Jolokia for interfacing the management
console for embedded uses.

Tristan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Bela Ban
In reply to this post by Wolf Fink
Note that JGroups diagnostics (probe) exposes operations and attributes
via a simple TCP- or UDP-based text protocol. If you guys come up with a
JSON based format, it should be easy to create a REST endpoint that
implements that format.

On 26/07/16 10:21, Wolf Fink wrote:

> Do we expose historical data for the cluster view. Often it is important
> to see whether there are view changes, rebalancing and unexpected
> leave/merge events where nodes are kicked by JGroups.
> Having special entries for controlled view change and sudden view
> changes might be good
>
> On Tue, Jul 26, 2016 at 10:06 AM, Sebastian Laskawiec
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Hey Vojtech!
>
>     JMX and CLI integration sounds very interesting. I also like the
>     idea of exposing log and exception dump.
>
>     Thanks a lot for the input!
>     Sebastian
>
>     On Tue, Jul 26, 2016 at 9:00 AM, Vojtech Juranek
>     <[hidden email] <mailto:[hidden email]>> wrote:
>
>         On Tuesday 26 July 2016 07:10:16 Sebastian Laskawiec wrote:
>         > I'm currently sketching a design for a REST
>         > health check endpoint for Infinispan
>
>         if it's not too broad, I'd include also various information
>         about the cluster
>         - e.g. number of machines in the cluster, recent exceptions in
>         the log (or
>         dump of N lines of log) etc. If would be useful at least for
>         testing purposes
>         so that we won't have to gather various information via JMX and CLI
>         _______________________________________________
>         infinispan-dev mailing list
>         [hidden email]
>         <mailto:[hidden email]>
>         https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

--
Bela Ban, JGroups lead (http://www.jgroups.org)

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Sebastian Laskawiec
In reply to this post by Tristan Tarrant-2
Hi Tristan!

I've been investigating the possibility of integrating with WF monitoring endpoint and I found this article: [1].

If I understand the Infinispan and Wildfly integration bits correctly - all we need to do is to implement additional resources in [2]. This should make the new API available through CLI as well as through REST [1].

The biggest advantage of this solution is that one implementation solves both REST and CLI use cases at the same time. However there are some drawbacks too - we won't be able to support any custom queries (we are limited only to queries supported by WF bits) and the REST api will be a bit complicated to consume:
If I'm right - we will need to document this feature correctly since those URLs are not very intuitive. WDYT?

Thanks
Sebastian


On Tue, Jul 26, 2016 at 10:34 AM, Tristan Tarrant <[hidden email]> wrote:
On 26/07/16 10:24, Sebastian Laskawiec wrote:
> Hey Tristan!
>
> Comments inlined.
>
> Thanks
> Sebastian
>
> On Tue, Jul 26, 2016 at 9:50 AM, Tristan Tarrant <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On 26/07/16 07:10, Sebastian Laskawiec wrote:
>     > Dear Community,
>     >
>     > I'd like to ask you for help. I'm currently sketching a design for a
>     > REST health check endpoint for Infinispan and I'm trying to imagine
>     > possible use cases.
>     The health-check should be implemented as an MBean initially, with the
>     ability to expose it via alternative implementations later. The server
>     RESTful endpoint should be registered with the management
>     interface via
>     a special handler.
>
>
> Yes, I think it's a good idea. We could even use tools like Jolokia
> [1] to expose MBeans through REST interface (it can be added to
> standalone.conf to the bootstrap classpath). Alternatively we could
> use JDK embedded HTTP Server [2].

No, for server we would not use Jolokia but rely on the management HTTP
server (the one that handles port 9990 already).

>
>     A cache and cachemanager's health is determined by a combination of
>     parameters and we probably should allow for a user-pluggable
>     checker. We
>     already expose a number of statuses already, although obviously this
>     would be an aggregate.
>
>
> Could you please elaborate more on that? How do we expose this
> information? Are you referring to Infinispan Stats [3]?
>
> I also though about supporting queries somehow. An imaginary example
> from the top of my head could look like the following:
>
> http://$ISPN/_health?cluster.nodes=3&MyCacheManager.MyCache.status=DEGRADED
> //<-- This would return 200 OK if we have 3 nodes and given cache is
> in degraded mode.
> http://$ISPN/_health?cluster.nodes=3&MyCacheManager.rebalance=IN_PROGRESS
> //<-- Checks if we have 3 nodes and rebalance is in proress
>
> [3]
> https://github.com/infinispan/infinispan/tree/8.2.x/core/src/main/java/org/infinispan/stats
>
>
>     >
>     > Could you please give me a hand and tell me what functionalities are
>     > important for you? Would you like to be able to check status
>     per-cache
>     > or maybe a red (not healthy), green (healthy), yellow (healthy,
>     > rebalance in progress) cluster status is sufficient? What kind of
>     > information do you expect to be there?
>     I wouldn't want this to be overly complex: a simple OK, KO should be
>     sufficient. Additional detail may be optionally present, but not a
>     requirement.
>
>
> I think we will need at least a 3rd state - yellow or something like
> this. This would mean that a rebalance is in progress of a node is
> joining/leaving. In other words - the cluster accepts requests but
> don't touch the nodes!
Agreed.

Tristan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Health check use cases

Tristan Tarrant-2
We are a bit more complex than either ES and SA :)

First of all this needs to work for both standalone and domain modes, as
well as support multiple cache containers. I believe it is possible to
register an io.undertow.server.HttpHandler in the ManagementHttpServer
to handle additional request types and have convenient aliases for the
queries below.

Tristan

On 28/07/16 14:09, Sebastian Laskawiec wrote:

> Hi Tristan!
>
> I've been investigating the possibility of integrating with WF
> monitoring endpoint and I found this article: [1].
>
> If I understand the Infinispan and Wildfly integration bits correctly
> - all we need to do is to implement additional resources in [2]. This
> should make the new API available through CLI as well as through REST [1].
>
> The biggest advantage of this solution is that one implementation
> solves both REST and CLI use cases at the same time. However there are
> some drawbacks too - we won't be able to support any custom queries
> (we are limited only to queries supported by WF bits) and the REST api
> will be a bit complicated to consume:
>
>   * Imaginary examples based on [1]:
>       o curl
>         http://localhost:9990/management/subsystem=datagrid-infinispan/cache-container=local?operation=health&recursive=true&json.pretty=1
>       o curl
>         http://localhost:9990/management/subsystem=datagrid-infinispan/cache-container=local/local-cache=default?operation=health&json.pretty=1
>   * An example what ElasticSearch does [3]:
>       o curl 'http://localhost:9200/_cluster/health?pretty=true'
>   * An example what Spring Actuator does [4]:
>       o curl 'http://localhost:8080/health'
>       o curl 'http://localhost:8080/metrics'
>
> If I'm right - we will need to document this feature correctly since
> those URLs are not very intuitive. WDYT?
> Thanks
> Sebastian
>
> [1] https://docs.jboss.org/author/display/WFLY10/The+HTTP+management+API
> [2]
> https://github.com/infinispan/infinispan/tree/master/server/integration/infinispan/src/main/java/org/jboss/as/clustering/infinispan/subsystem
> [3]
> https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html
> [4] https://spring.io/guides/gs/actuator-service/
>
> On Tue, Jul 26, 2016 at 10:34 AM, Tristan Tarrant <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On 26/07/16 10:24, Sebastian Laskawiec wrote:
>     > Hey Tristan!
>     >
>     > Comments inlined.
>     >
>     > Thanks
>     > Sebastian
>     >
>     > On Tue, Jul 26, 2016 at 9:50 AM, Tristan Tarrant
>     <[hidden email] <mailto:[hidden email]>
>     > <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >
>     >     On 26/07/16 07:10, Sebastian Laskawiec wrote:
>     >     > Dear Community,
>     >     >
>     >     > I'd like to ask you for help. I'm currently sketching a
>     design for a
>     >     > REST health check endpoint for Infinispan and I'm trying
>     to imagine
>     >     > possible use cases.
>     >     The health-check should be implemented as an MBean
>     initially, with the
>     >     ability to expose it via alternative implementations later.
>     The server
>     >     RESTful endpoint should be registered with the management
>     >     interface via
>     >     a special handler.
>     >
>     >
>     > Yes, I think it's a good idea. We could even use tools like Jolokia
>     > [1] to expose MBeans through REST interface (it can be added to
>     > standalone.conf to the bootstrap classpath). Alternatively we could
>     > use JDK embedded HTTP Server [2].
>
>     No, for server we would not use Jolokia but rely on the management
>     HTTP
>     server (the one that handles port 9990 already).
>
>     >
>     >     A cache and cachemanager's health is determined by a
>     combination of
>     >     parameters and we probably should allow for a user-pluggable
>     >     checker. We
>     >     already expose a number of statuses already, although
>     obviously this
>     >     would be an aggregate.
>     >
>     >
>     > Could you please elaborate more on that? How do we expose this
>     > information? Are you referring to Infinispan Stats [3]?
>     >
>     > I also though about supporting queries somehow. An imaginary example
>     > from the top of my head could look like the following:
>     >
>     >
>     http://$ISPN/_health?cluster.nodes=3&MyCacheManager.MyCache.status=DEGRADED
>     > //<-- This would return 200 OK if we have 3 nodes and given cache is
>     > in degraded mode.
>     >
>     http://$ISPN/_health?cluster.nodes=3&MyCacheManager.rebalance=IN_PROGRESS
>     > //<-- Checks if we have 3 nodes and rebalance is in proress
>     >
>     > [3]
>     >
>     https://github.com/infinispan/infinispan/tree/8.2.x/core/src/main/java/org/infinispan/stats
>     >
>     >
>     >     >
>     >     > Could you please give me a hand and tell me what
>     functionalities are
>     >     > important for you? Would you like to be able to check status
>     >     per-cache
>     >     > or maybe a red (not healthy), green (healthy), yellow
>     (healthy,
>     >     > rebalance in progress) cluster status is sufficient? What
>     kind of
>     >     > information do you expect to be there?
>     >     I wouldn't want this to be overly complex: a simple OK, KO
>     should be
>     >     sufficient. Additional detail may be optionally present, but
>     not a
>     >     requirement.
>     >
>     >
>     > I think we will need at least a 3rd state - yellow or something like
>     > this. This would mean that a rebalance is in progress of a node is
>     > joining/leaving. In other words - the cluster accepts requests but
>     > don't touch the nodes!
>     Agreed.
>
>     Tristan
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...