Quantcast

Re: [infinispan-dev] XAResource.isSameRM

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Mircea Markus
FYI, a discussion I have with Jonathan around recovery support from TM

On 5 Jan 2011, at 14:43, Jonathan Halliday wrote:

> On 01/05/2011 02:18 PM, Mircea Markus wrote:
>
>> I don't know how the TM recovery process picks up the XAResource instance on which to call XAResource.recover, but I imagine it expects this method to return all the prepared(or heuristic completed) transactions from the _whole transaction branch_, i.e. from the entire cluster.
>
> all from the logical RM, which you happen to implement as a cluster, yes.
>
>> I'm asking this  because right now there's no way for a node to know all the prepared transaction in the entire cluster. This is doable but would involve an broadcast to query the cluster, which might be costly (time and bandwidth).
>
> right. not to mention it should, strictly speaking, block or fail if any node is unreachable, which kinda sucks from an availability perspective.
>
> Keep in mind the global list of known in-doubt tx is a point in time snapshot anyhow - in an active system it's out of date as soon as another running tx is prepared. So, you can serve one that's slightly stale without too much risk. Not that periodic broadcast of the in-doubt list between nodes is necessarily better than doing it on-demand in response to a recovery call, but at least it's O(1) rather than O(number of clients calling recover).  The (mild) problem we've seen in the past is where a large cluster of app server nodes i.e. tx managers, is started more or less simultaniously, the RMs get a storm of recovery requests every two minutes. If the impl of that is expensive and not cached it can cause performance spikes in the RM.
>
> That said, keep in mind a recovery pass is (in the default config) only every two minutes and run on a background thread. It's not something you want to worry about performance optimization of in the initial implementation. Do it right rather than fast. Optimize it only when users scream.
>
>> On the other hand I imagine this call is performed asynchronously and doesn't impact TM performance in managing ongoing transactions, so it might not be that bad after all.
>
> correct
>
>> Another approach would be to to consider each node as an transaction branch.
>
> you mean a logically separate resource manager, yes. You are basically talking about not doing interposition in the driver/server but rather relying on the transaction manager to handle multiple resources. It may make your implementation simpler but probably less performant on the critical path (transaction commit) vs. recovery.
>
>
>> The advantage here is that recovery can be easily implemented, as the TM recovery would ask all the nodes that were registered for prepared transactions
>
> 'registered'? you plan of having the config on each transaction manager contain a list of all cluster nodes? That's not admin friendly.
>
>> , and no cluster broadcast would be required when XAResource.recover is called.
>
> the broadcast is automatic. Maintaining the list of known nodes in a config file is not. No contest.
>
>> Considering this approach, do you see any drawbacks compared with the other one?
>> E.g. each node being a branch might involve multiple RPC between remote TM and XAResource on each node (v.s. one in prev example).
>
> yeah, as mentioned above its a non-interposed model rather than one where your driver/server is doing interposition. Favour the one that makes the commit path fast, even though it makes recovery an utter pain.
>
> Jonathan.
>
> -
> ------------------------------------------------------------
> Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
> Registered in UK and Wales under Company Registration No. 3798903  Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland)


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Mircea Markus

On 5 Jan 2011, at 14:51, Mircea Markus wrote:

> FYI, a discussion I have with Jonathan around recovery support from TM
>
> On 5 Jan 2011, at 14:43, Jonathan Halliday wrote:
>> On 01/05/2011 02:18 PM, Mircea Markus wrote:
>>
>>> I don't know how the TM recovery process picks up the XAResource instance on which to call XAResource.recover, but I imagine it expects this method to return all the prepared(or heuristic completed) transactions from the _whole transaction branch_, i.e. from the entire cluster.
>>
>> all from the logical RM, which you happen to implement as a cluster, yes.
>>
>>> I'm asking this  because right now there's no way for a node to know all the prepared transaction in the entire cluster. This is doable but would involve an broadcast to query the cluster, which might be costly (time and bandwidth).
>>
>> right. not to mention it should, strictly speaking, block or fail if any node is unreachable, which kinda sucks from an availability perspective.
So if a node does not respond to the broadcast, it is incorrect to return the prepared transactions received from the other nodes? (is this because the TM expects to receive some tx that it knows for sure to be prepared?) Or would a "best effort" be "good enough"? (e.g. I broadcast the query and return all the results received in 1 sec)  
>>
>> Keep in mind the global list of known in-doubt tx is a point in time snapshot anyhow - in an active system it's out of date as soon as another running tx is prepared. So, you can serve one that's slightly stale without too much risk. Not that periodic broadcast of the in-doubt list between nodes is necessarily better than doing it on-demand in response to a recovery call, but at least it's O(1) rather than O(number of clients calling recover).  
Interesting. I would expect a single (periodic) XAResource.recover call per cluster(assuming the cluster is the RM) / transaction manager. Am I wrong?
>> The (mild) problem we've seen in the past is where a large cluster of app server nodes i.e. tx managers, is started more or less simultaniously, the RMs get a storm of recovery requests every two minutes. If the impl of that is expensive and not cached it can cause performance spikes in the RM.
Good to know this, definitely  an approach to consider.
>>
>> That said, keep in mind a recovery pass is (in the default config) only every two minutes and run on a background thread. It's not something you want to worry about performance optimization of in the initial implementation. Do it right rather than fast. Optimize it only when users scream.
+1. AS is one of the use cases for ISPN though, but e.g. if a TM uses Hotrod to interact with the cluster(when HR will support tx)  then there will only be one recovery call/cluster.  
>>
>>> On the other hand I imagine this call is performed asynchronously and doesn't impact TM performance in managing ongoing transactions, so it might not be that bad after all.
>>
>> correct
>>
>>> Another approach would be to to consider each node as an transaction branch.
>>
>> you mean a logically separate resource manager, yes. You are basically talking about not doing interposition in the driver/server but rather relying on the transaction manager to handle multiple resources. It may make your implementation simpler but probably less performant on the critical path (transaction commit) vs. recovery.
I don't know exactly how many RPC's happen in this approach, with TM handling multiple resources. I imagine the TM would do a XAResource.prepare for each of the nodes involved. In this XAResource.prepare call I would have to implement the logic of going remotely to each involved node. Then the same for XAResource.commit. Is that so? (If so then this is pretty much what we already do in ISPN when it comes to commit/rollback).
On of the advantages of allowing TM to handle each individual node is that we can benefit from some nice TM features like read-only optimisation or 1PC for single participants (these are to be implemented anyway in ISPN).
>>
>>
>>> The advantage here is that recovery can be easily implemented, as the TM recovery would ask all the nodes that were registered for prepared transactions
>>
>> 'registered'?
wrong word - I ment enlisted

>> you plan of having the config on each transaction manager contain a list of all cluster nodes? That's not admin friendly.
>>
>>> , and no cluster broadcast would be required when XAResource.recover is called.
>>
>> the broadcast is automatic. Maintaining the list of known nodes in a config file is not. No contest.
>>
>>> Considering this approach, do you see any drawbacks compared with the other one?
>>> E.g. each node being a branch might involve multiple RPC between remote TM and XAResource on each node (v.s. one in prev example).
>>
>> yeah, as mentioned above its a non-interposed model rather than one where your driver/server is doing interposition. Favour the one that makes the commit path fast, even though it makes recovery an utter pain.
+1

>>
>> Jonathan.
>>
>> -
>> ------------------------------------------------------------
>> Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
>> Registered in UK and Wales under Company Registration No. 3798903  Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland)
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Mircea Markus

On 5 Jan 2011, at 17:19, Jonathan Halliday wrote:

> On 01/05/2011 03:42 PM, Mircea Markus wrote:
>>
>> On 5 Jan 2011, at 14:51, Mircea Markus wrote:
>>
>>> FYI, a discussion I have with Jonathan around recovery support from TM
>>>
>>> On 5 Jan 2011, at 14:43, Jonathan Halliday wrote:
>>>> On 01/05/2011 02:18 PM, Mircea Markus wrote:
>>>>
>>>>> I don't know how the TM recovery process picks up the XAResource instance on which to call XAResource.recover, but I imagine it expects this method to return all the prepared(or heuristic completed) transactions from the _whole transaction branch_, i.e. from the entire cluster.
>>>>
>>>> all from the logical RM, which you happen to implement as a cluster, yes.
>>>>
>>>>> I'm asking this  because right now there's no way for a node to know all the prepared transaction in the entire cluster. This is doable but would involve an broadcast to query the cluster, which might be costly (time and bandwidth).
>>>>
>>>> right. not to mention it should, strictly speaking, block or fail if any node is unreachable, which kinda sucks from an availability perspective.
>> So if a node does not respond to the broadcast, it is incorrect to return the prepared transactions received from the other nodes? (is this because the TM expects to receive some tx that it knows for sure to be prepared?) Or would a "best effort" be "good enough"? (e.g. I broadcast the query and return all the results received in 1 sec)
>
> hmm, interesting question.
>
> Keep in mind that the XA spec dates from a time when a typical large clustered RM was 2-3 oracle nodes on the same LAN segment. It simply isn't geared to a world where the number of nodes is so large and widely distributed that the probability of *all* of them being available simultaneously is pretty slim. Likewise the number of transaction managers connected to a resource was assumed to be small, often 1, rather than the large N we see on modern clusters / clouds. As a result, the spec either fails to give guidance on some issues because they weren't significant at the time it was written, or implies/mandates behaviour that is counter productive in modern environments.
>
> Thus IMO some compromises are necessary to make XA usable in the real world, especially at scale. To further complicate matters, these are split across RM and TM, with different vendors having different views on the subject. My advice is geared to the way JBossTS drives XA recovery - other TMs may behave differently and make greater or lesser assumptions about compliance with the letter of the spec. As a result you may find that making your RM work with multiple vendor's TMs requires a) configuration options and b) a lot of painful testing.  Likewise JBossTS contains code paths and config options geared to dealing with bugs or non-compliant behaviour in various vendor's RMs.
>
> Now, on to the specific question: The list returned should, strictly speaking, be complete. There are two problems with that. First, you have to be able to reach all your cluster nodes to build a complete list which, as previously mentioned, is pretty unlikely in a sufficiently large cluster. Your practical strategies are thus as you say: either a) throw an XAException(XAER_RMFAIL) if any node is unreachable within a reasonable timeout and accept that this may mean an unnecessary delay in recovering the subset of tx that are known or b) return a partial list on a best effort basis. The latter approach allows the transaction manager to deal with at least some of the in-doubt tx, which may in turn mean releasing resources/locks in the RM. In general I'd favour that option as having higher practical value in terms of allowing the best possible level of service to be maintained in the face of ongoing failures.
>
> JBossTS will rescan every N minutes (2 by default) and thus you can simply include any newly discovered in-doubt tx as they become known due to e.g. partitioned nodes rejoining the cluster, and the TM will deal with them when they are first seen. Note however that some TMs assume that if they scan an RM and that RM does not subsequently crash, no new in-doubt transactions will occur except from heuristics. Let's gloss over how they can even detect a crash/recover of the RM if the driver masks it with failover or the event happens during a period when the TM makes no call on the driver. Such a TM will perform a recovery scan once at TM startup and not repeat. In such case you may have in-doubt tx from nodes unavailable at that crucial time subsequently sitting around for a prolonged period, tying up precious resources and potentially blocking subsequent updates. Most RM vendors provide some kind of management capability for admins to view and manually force completion of in!
 -doubt tx. command line tool, jmx, web gui, whatever, just so long as it exists.
When a node crashes all the transactions that node owns (i.e. tx which were originated on that node and XAResource instance residing on that node) automatically rollback, so that no resources (locks mainly) are held. The only thing we need to make sure though is that the given transaction ids (the one that heuristically rollback) are returned by theXAResource.recover method - doable in the same way we handle prepares. I imagine that we'll have to keep these XIDs until XAResource.forget(XID) is called, am I right? Is it common/possible for people to use TM _without_ recovery? If so,  this "held heuristic completed TX" functionality should be configurable (enabled/disabled) in order to avoid memory leaks (no recovery means .forget never gets called)    

> Another interesting issue is what constitutes an 'in-doubt' tx. Pretty much all RMs will include heuristically completed tx in the recovery list. Some will include tx branches that have prepared but not yet committed or rolled back. Some will include such only if they have been in the prepared state for greater than some threshold length of time (a few seconds i.e. a couple of order of magnitude longer than a tx would normally be expected to hold that state). There is also the question of when a tx should be removed from the list. The wording of the spec
>
> 'Two consecutive invocation of [recover] that starts from the beginning of the list must return the same list
> of transaction branches unless one of the following takes place:
>       - the transaction manager invokes the commit, forget, prepare, or rollback method for that resource
>       manager, between the two consecutive invocation of the recovery scan
> ...'
>
> seems to imply a single transaction manager.
doesn't this also imply that the prepare-treshold isn't the spec's way? I.e. even though TM doesn't call any method on the RM , the RM returns a new XID in the result of XAResource.recover when the threshold is reached.  
> In cases where more than one TM is connected to the RM, the list clearly can't be considered stable between calls, as that would require prolonged blocking. Thus a reasonable TM should not expect a stable list. However, it's less clear how it should react to items that appear and disappear arbitrarily over consecutive calls to recover.
>
> In the case of JBossTS, it assumes the RM may include tx branches that are actually proceeding normally in the results of a recovery scan. It thus caches the list, sleeps for an interval considered long enough for any normal prepared tx to complete, then rescans and compares the results. Any tx appearing in both scans is considered genuinely in need of recovery. If an RM includes in its recover results a normally proceeding tx branch and the TM does not perform such a backoff, it may, in a clustered environment, rollback a tx branch that another TM will try to commit a split second later, thus resulting in an unnecessary heuristic outcome. The worst scenario I've ever seen was a certain large DB vendor who considered tx to be in-doubt not just from the time they were prepared, but from the moment they were started. ouch.
>
> Naturally most well behaved TMs will have some sense of tx branch ownership and not recover unrecognised tx, but it's one of those delightful gray areas where you may benefit from being a little paranoid.
So each TM would only care to recover the transactions it manages? Sort of makes sense.
> Which, as it happens, gives you a perfect excuse to return a partial list of tx - in terms of the way TM recovery works there is nothing to distinguish a tx that's missing due to node outage from one that is missing due to not yet having reached the in-doubt time threshold.
>
> Of course it does not end with the recovery scan. Let's take the case where a specific tx is identified as in-doubt and returned from the recover call. The TM may then call e.g. commit on it. Processing that commit may involve the driver/server talking to multiple cluster nodes, some of which may not respond. Indeed this is the case during a normal commit too, not just one resulting from a recovery scan. You need to think very carefully about what the result of a failed call should be. A lot of bugs we've seen result from resource managers using incorrect XAException codes or transaction managers misinterpreting them. Be aware of the semantics of specific error codes and be as concise as possible when throwing errors.
Good to know! Something to discuss/review on our next meeting.

>
>> I would expect a single (periodic) XAResource.recover call per cluster(assuming the cluster is the RM) / transaction manager. Am I wrong?
>
> Only in so far as you are assuming one logical TM per cluster, with cluster here meaning JBossAS cluster as distinct from a possible separate infinispan RM cluster.
>
> In practice although a group of JBossAS servers may be clustering e.g. web sessions or EJBs, they don't cluster TMs. Each JBossAS instance is a logically separate and independently functioning TM and each is responsible for running its own recovery. Note that there ways of clustering the JTS version of the TM, but we don't advertise or support them and as a result to the best of my knowledge no end user actually runs with such configuration. Likewise for running a single out of process recovery subsystem which is shared by multiple TMs/JBossAS nodes.
>
>>>> you mean a logically separate resource manager, yes. You are basically talking about not doing interposition in the driver/server but rather relying on the transaction manager to handle multiple resources. It may make your implementation simpler but probably less performant on the critical path (transaction commit) vs. recovery.
>> I don't know exactly how many RPC's happen in this approach, with TM handling multiple resources. I imagine the TM would do a XAResource.prepare for each of the nodes involved. In this XAResource.prepare call I would have to implement the logic of going remotely to each involved node. Then the same for XAResource.commit. Is that so? (If so then this is pretty much what we already do in ISPN when it comes to commit/rollback).
>> On of the advantages of allowing TM to handle each individual node is that we can benefit from some nice TM features like read-only optimisation or 1PC for single participants (these are to be implemented anyway in ISPN).
>
> Although the RPC topology can matter, particularly where WAN hops are involved, the critical difference is actually where the logging responsibility lies.
>
> If the TM has more than one XAResource enlisted, it has to write to disk between the prepare and commit phases. So where it has got e.g. two infinispan nodes as separate XAResources, the tx is bottlenecked on disk I/O even though the RM is entirely in (network distributed) RAM.
Right! I was looking in the wrong place.
>
> Where the clustered RM presents itself as a single resource, the TM won't necessarily log, due to automatic one phase commit optimisation.
>
> However...
>
> In that case your driver/server has to guarantee correct behaviour in the event of node failure during the commit call. In other words, it has to do its own persistence if the state updates span more than one cluster node. In the case of infinispan that's likely to be via. additional RPCs for replication rather than by a disk write as in most database cluster RMs.
>
> In short, you're trading disk I/O in the TM against additional network I/O in the clustered RM. All else being equal I think having the RM do the work will perform better at scale, but that's just a guess.
+1
> You more or less need the code for that either way, as even where a single infinispan node is a logically separate RM, it still needs to talk to its cluster mates (or the disk I guess) for persistence or it can't guarantee durability of the tx.
Thanks Jonathan!
>
> Jonathan.
>
> --
> ------------------------------------------------------------
> Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
> Registered in UK and Wales under Company Registration No. 3798903  Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland)


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Jonathan Halliday
On 01/06/2011 10:45 AM, Mircea Markus wrote:

> When a node crashes all the transactions that node owns (i.e. tx which were originated on that node and XAResource instance residing on that node) automatically rollback, so that no resources (locks mainly) are held. The only thing we need to make sure though is that the given transaction ids (the one that heuristically rollback) are returned by theXAResource.recover method - doable in the same way we handle prepares. I imagine that we'll have to keep these XIDs until XAResource.forget(XID) is called, am I right?

I was under the impression a node does not own a tx. It may
own a *branch* of that tx. Take the case where node
JBossAS-1 starts a tx, propagates it to node JBossAS-2 and
both JBossAS-1 and JBossAS-2 then simultaneously contact
different nodes of the infinispan cluster in the scope of
that tx. Each node would see a different branch of the same
global tx. You presumably don't want to have to sync the
ownership for each new tx across the cluster? The only way
you could tie an entire tx to a single infinispan node is
e.g. consistent hashing to force the decision of which node
the driver connects to based on the tx context.

An XAResource does not reside in an infinispan node
(although there may be something equivalent holding tx state
on the server side) - it's a client/driver side construct.
Given that the driver does transparent failover / load
balancing and such, the XAResource can't be said to belong
to a specific infinispan node unless you throw away some of
the clustering availability advantages. It's really a
question of where you are going to put the clustering
intelligence - in a smart client side driver or in a server
side component that acts as a kind of routing proxy.

With your 'rollback tx branch on node crash' model you are
failing to provide ACID semantics for the cluster as a
whole. You can abort them before the prepare stage, but post
prepare it's not an option as you'll piss off clients who
expect the cluster to behave correctly as long as a majority
of its nodes survive. I'm not saying it's flat out wrong,
just that it needs to be very clearly documented in order to
avoid getting whined at by disgruntled users. My
understanding is you're pitching infinispan not as a
volatile cache, but an in-memory data grid. In that model
the node does not own the tx, the cluster owns the tx and is
responsible for masking node failures from the client.
Rollback of prepared tx on node failure is therefore not an
option - some part of the tx state may already have been
committed by surviving nodes and you'll get inconsistencies.
You need to replicate enough information to avoid that,
otherwise the client app is going to have to explicitly
provide logic to do the reconciliation, which sucks.

> Is it common/possible for people to use TM _without_ recovery? If so,  this "held heuristic completed TX" functionality should be configurable (enabled/disabled) in order to avoid memory leaks (no recovery means .forget never gets called)

It is not common. That said, JBossTS has for similar reasons
got a 'give up after N hours' config option which will
eventually abandon tx that have not recovered. It's off
(i.e. never give up) by default but a small number of users
find it handy. Most just use the admin tooling to manually
clean up the small number of unrecoverable situations - it's
safer in most cases.

>> Another interesting issue is what constitutes an 'in-doubt' tx. Pretty much all RMs will include heuristically completed tx in the recovery list. Some will include tx branches that have prepared but not yet committed or rolled back. Some will include such only if they have been in the prepared state for greater than some threshold length of time (a few seconds i.e. a couple of order of magnitude longer than a tx would normally be expected to hold that state). There is also the question of when a tx should be removed from the list. The wording of the spec
>>
>> 'Two consecutive invocation of [recover] that starts from the beginning of the list must return the same list
>> of transaction branches unless one of the following takes place:
>>        - the transaction manager invokes the commit, forget, prepare, or rollback method for that resource
>>        manager, between the two consecutive invocation of the recovery scan
>> ...'
>>
>> seems to imply a single transaction manager.

> doesn't this also imply that the prepare-treshold isn't the spec's way? I.e. even though TM doesn't call any method on the RM , the RM returns a new XID in the result of XAResource.recover when the threshold is reached.

The definition of what constitutes an 'in-doubt' tx for
purposes of inclusion in the recovery list is not well
defined by the spec.

In a world where there is only one TM driving the RM and
that TM performs recovery before starting up and running new
tx, no new tx will be added to the recovery list. In the
real world new ones are being added continuously as the
system is always under load.

The concern is more around the stability of the list with
respect to not *removing* things except in response to TM
activity. i.e. if a node goes away, should you return a
cached snapshot of its tx for list stability, or exclude
them? both options carry risks.


Jonathan.

--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111
Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No.
3798903  Directors: Michael Cunningham (USA), Charlie Peters
(USA), Matt Parsons (USA) and Brendan Lane (Ireland)
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Mircea Markus

On 6 Jan 2011, at 11:48, Jonathan Halliday wrote:

On 01/06/2011 10:45 AM, Mircea Markus wrote:

When a node crashes all the transactions that node owns (i.e. tx which were originated on that node and XAResource instance residing on that node) automatically rollback, so that no resources (locks mainly) are held. The only thing we need to make sure though is that the given transaction ids (the one that heuristically rollback) are returned by theXAResource.recover method - doable in the same way we handle prepares. I imagine that we'll have to keep these XIDs until XAResource.forget(XID) is called, am I right?

I was under the impression a node does not own a tx. It may own a *branch* of that tx.
Take the case where node JBossAS-1 starts a tx, propagates it to node JBossAS-2 and both JBossAS-1 and JBossAS-2 then simultaneously contact different nodes of the infinispan cluster in the scope of that tx. Each node would see a different branch of the same global tx. You presumably don't want to have to sync the ownership for each new tx across the cluster? The only way you could tie an entire tx to a single infinispan node is e.g. consistent hashing to force the decision of which node the driver connects to based on the tx context.
you are absolutely right, I don't want to sync ownership across the cluster.  

An XAResource does not reside in an infinispan node (although there may be something equivalent holding tx state on the server side) - it's a client/driver side construct.
I see.
At the moment the *only way to transactionally* access a node is by collocating the client and the server in the same VM. Hence the XAResource residing in the ISPN node's side. ISPN 5.0 will take things further by allowing  client/server access through Hotrod:http://community.jboss.org/wiki/TransactionsOverHotRod
Given that the driver does transparent failover / load balancing and such, the XAResource can't be said to belong to a specific infinispan node unless you throw away some of the clustering availability advantages. It's really a question of where you are going to put the clustering intelligence - in a smart client side driver or in a server side component that acts as a kind of routing proxy.
At the moment if a node that is touched by a transaction/branch crashes before transaction commits then all the acquired resources are releases and the transaction is marked for rollback. 

With your 'rollback tx branch on node crash' model you are failing to provide ACID semantics for the cluster as a whole. You can abort them before the prepare stage, but post prepare it's not an option as you'll piss off clients who expect the cluster to behave correctly as long as a majority of its nodes survive. I'm not saying it's flat out wrong, just that it needs to be very clearly documented in order to avoid getting whined at by disgruntled users. My understanding is you're pitching infinispan not as a volatile cache, but an in-memory data grid. In that model the node does not own the tx, the cluster owns the tx and is responsible for masking node failures from the client. Rollback of prepared tx on node failure is therefore not an option - some part of the tx state may already have been committed by surviving nodes and you'll get inconsistencies. You need to replicate enough information to avoid that, otherwise the client app is going to have to explicitly provide logic to do the reconciliation, which sucks.
Good point. At the moment (i.e. client and the node in same VM) this is not an issue because if the node crashes you can safely assume that the client is crashed as well. Something to be considered though, with remote TX over hotrod in mind.     

Is it common/possible for people to use TM _without_ recovery? If so,  this "held heuristic completed TX" functionality should be configurable (enabled/disabled) in order to avoid memory leaks (no recovery means .forget never gets called)

It is not common. That said, JBossTS has for similar reasons got a 'give up after N hours' config option which will eventually abandon tx that have not recovered. It's off (i.e. never give up) by default but a small number of users find it handy. Most just use the admin tooling to manually clean up the small number of unrecoverable situations - it's safer in most cases.
Right, JMX might come handy for that. 

Another interesting issue is what constitutes an 'in-doubt' tx. Pretty much all RMs will include heuristically completed tx in the recovery list. Some will include tx branches that have prepared but not yet committed or rolled back. Some will include such only if they have been in the prepared state for greater than some threshold length of time (a few seconds i.e. a couple of order of magnitude longer than a tx would normally be expected to hold that state). There is also the question of when a tx should be removed from the list. The wording of the spec

'Two consecutive invocation of [recover] that starts from the beginning of the list must return the same list
of transaction branches unless one of the following takes place:
      - the transaction manager invokes the commit, forget, prepare, or rollback method for that resource
      manager, between the two consecutive invocation of the recovery scan
...'

seems to imply a single transaction manager.

doesn't this also imply that the prepare-treshold isn't the spec's way? I.e. even though TM doesn't call any method on the RM , the RM returns a new XID in the result of XAResource.recover when the threshold is reached.

The definition of what constitutes an 'in-doubt' tx for purposes of inclusion in the recovery list is not well defined by the spec.

In a world where there is only one TM driving the RM and that TM performs recovery before starting up and running new tx, no new tx will be added to the recovery list. In the real world new ones are being added continuously as the system is always under load.

The concern is more around the stability of the list with respect to not *removing* things except in response to TM activity. i.e. if a node goes away, should you return a cached snapshot of its tx for list stability, or exclude them? both options carry risks.
Thanks for the clarification.


Jonathan.

--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903  Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland)


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Jonathan Halliday
On 01/06/2011 02:29 PM, Mircea Markus wrote:

> At the moment the *only way to transactionally* access a
> node is by collocating the client and the server in the same
> VM.

So the scope of the transaction is limited to data residing
in that local node? What if I want a single transaction to
span the local node and data in a remote node? Or what about
where the data is all in the local node, but the node has a
remote replica for fault tolerance?

> client and the node in same VM ... if the node crashes you can
> safely assume that the client is crashed as well.

No, you can't. The client has persistent state on disk
courtesy of the transaction manager log and will restore
that after the crash. It's not volatile. The transaction
updates may also have spanned out to other systems. Your
model only works for infinispan in single node volatile
store configurations. As soon as you cluster or spill to
persistent backing store you are in trouble. That's better
than nothing, but pretty limiting for users. I'm not really
clear what advantage it has over implementing
Synchronization rather than XAResource. Why are you
bothering with 2PC if you're not going to actually provide
ACID semantics?


Jonathan.

--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111
Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No.
3798903  Directors: Michael Cunningham (USA), Charlie Peters
(USA), Matt Parsons (USA) and Brendan Lane (Ireland)
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Mircea Markus

On 6 Jan 2011, at 14:45, Jonathan Halliday wrote:

On 01/06/2011 02:29 PM, Mircea Markus wrote:

At the moment the *only way to transactionally* access a
node is by collocating the client and the server in the same
VM.

So the scope of the transaction is limited to data residing in that local node? What if I want a single transaction to span the local node and data in a remote node?
that's possible. It's just that you have to always interact with the local node that will acquire remote locks remotely on behalf of your transaction.
Or what about where the data is all in the local node, but the node has a remote replica for fault tolerance?
The same scenario as above. Attached is a diagram depicting how ISPN handle tx under the hood(TM, User and NodeA are in the same VM)

client and the node in same VM ... if the node crashes you can
safely assume that the client is crashed as well.

No, you can't. The client has persistent state on disk courtesy of the transaction manager log and will restore that after the crash. It's not volatile. The transaction updates may also have spanned out to other systems.
Wouldn't that be solved if the XID for the heuristic rollback transaction is returned by the RM(ISPN cluster) in the XAResource.recover method? 
Might be wrong but I see two possibilities:
a) node goes down before TM issued prepare
     - when TM resurrects and calls XAResource.recover it receives the given XID, realises that there's an heuristic decision (because it didn't call prepare) and take some action (rollbacks other participants, notify sys admin?). 
b) node goes down after TM issues prepare
     - when TM issues a commit it receives an XAException (perhaps XA_HEURRB)  and again it is aware of the heuristic outcome 
Your model only works for infinispan in single node volatile store configurations. As soon as you cluster or spill to persistent backing store you are in trouble. That's better than nothing, but pretty limiting for users. I'm not really clear what advantage it has over implementing Synchronization rather than XAResource. Why are you bothering with 2PC if you're not going to actually provide ACID semantics?
The plan is to support ACID semantics.

Jonathan.

--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903  Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland)


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

PastedGraphic-2.pdf (75K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Jonathan Halliday
On 01/06/2011 05:43 PM, Mircea Markus wrote:

>
> On 6 Jan 2011, at 14:45, Jonathan Halliday wrote:
>
>> On 01/06/2011 02:29 PM, Mircea Markus wrote:
>>
>>> At the moment the *only way to transactionally* access a
>>> node is by collocating the client and the server in the same
>>> VM.
>>
>> So the scope of the transaction is limited to data
>> residing in that local node? What if I want a single
>> transaction to span the local node and data in a remote node?
> that's possible. It's just that you have to always interact
> with the local node that will acquire remote locks remotely
> on behalf of your transaction.

ok, so the cluster intelligence is in the local node rather
than the client, not that there is any significant
distinction for now as they are co-located.

> a) node goes down before TM issued prepare
> - when TM resurrects and calls XAResource.recover it
> receives the given XID, realises that there's an heuristic
> decision (because it didn't call prepare) and take some
> action (rollbacks other participants, notify sys admin?).

That's not a heuristic decision. A RM is perfectly entitled
to throw away any tx state up until prepare. Under the
presumed abort doctrine it simply throws an error from
prepare and the tx aborts cleanly. Recovery is not involved
- it applies only to tx that have reached prepare.

> b) node goes down after TM issues prepare
> - when TM issues a commit it receives an XAException
> (perhaps XA_HEURRB) and again it is aware of the heuristic
> outcome

Returning cleanly from a prepare is a promise by the RM to
successfully apply any subsequent commit. You're not in a
position to make such a promise unless your state is fault
tolerant, as a node crash would otherwise leave you with
inconsistent state.

It's not as simple as saying you'd rollback - what if you
prepare, get told to commit, apply remote changes (step
4.2.1), then crash before applying local changes (4.2.2)?
You can't report that as a rollback - you applied some of
the updates. You have to include the tx in the recovery list
as heuristic hazard (unless NodeA will transparently
repopulate with the committed data, in which case you can
mask the failure or report heuristic commit), but how to
even detect the heuristic at recovery time? NodeA has no
persistent record of the tx and NodeB thinks it completed
cleanly and has cleaned up its tx record to avoid leaking.
Where is the data that tells you you've got a problem?

Or have a more sophisticated scenario where there is an
additional NodeC, thus requiring multiple 'apply remote
changes' calls. Are those atomic across the cluster? If
there is a possibility that NodeB will apply the update but
NodeC won't, or NodeA will crash after issuing a call to B
but before C, you can wind up with inconsistent state in the
surviving B and C. Alternatively, what if A survives but C
crashes whilst applying changes that B has already
sucessfully applied? That's not necessarily a recovery
situation as far as the TM is concerned, but it may be from
your perspective as you'll need to detect and (ideally) fix
or (as a last resort) report the inconsistent data.

A lot of your behaviour is going to depend on what it means
for a node to recover after a crash. If it simply comes up
empty and expects to be repopulated from an external source,
as with a normal cache, then your relation to the XAResource
of that external source is critical. On the other hand if
your cluster node is itself fault tolerant through
replication, then you need to think carefully about how the
RM functionality ties into that replication - basically the
tx state information is not local to the node where the
XAResource resides, but must be replicated in the same
manner as the other data in that node and that replication
must be synchronous at certain state transitions in the tx
lifecycle - it's logging recovery information through RPC
rather than disk write. Really interesting things are going
to happen if a single transaction spans data that is a mix
of cache copy of data stored persistently in an XA database
and data for which infinispan is the definitive, fault
tolerant repository.

To make a cluster appear to the outside world as a single
logical entity for transaction purposes, you're pretty much
going to wind up doing interposition. That means you're
implementing not only an RM but substantial chunks of what
amounts to a TM too. Have fun.

Jonathan.

--
------------------------------------------------------------
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111
Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No.
3798903  Directors: Michael Cunningham (USA), Charlie Peters
(USA), Matt Parsons (USA) and Brendan Lane (Ireland)
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Mark Little
+1


On 6 Jan 2011, at 19:01, Jonathan Halliday wrote:

> On 01/06/2011 05:43 PM, Mircea Markus wrote:
>>
>> On 6 Jan 2011, at 14:45, Jonathan Halliday wrote:
>>
>>> On 01/06/2011 02:29 PM, Mircea Markus wrote:
>>>
>>>> At the moment the *only way to transactionally* access a
>>>> node is by collocating the client and the server in the same
>>>> VM.
>>>
>>> So the scope of the transaction is limited to data
>>> residing in that local node? What if I want a single
>>> transaction to span the local node and data in a remote node?
>> that's possible. It's just that you have to always interact
>> with the local node that will acquire remote locks remotely
>> on behalf of your transaction.
>
> ok, so the cluster intelligence is in the local node rather
> than the client, not that there is any significant
> distinction for now as they are co-located.
>
>> a) node goes down before TM issued prepare
>> - when TM resurrects and calls XAResource.recover it
>> receives the given XID, realises that there's an heuristic
>> decision (because it didn't call prepare) and take some
>> action (rollbacks other participants, notify sys admin?).
>
> That's not a heuristic decision. A RM is perfectly entitled
> to throw away any tx state up until prepare. Under the
> presumed abort doctrine it simply throws an error from
> prepare and the tx aborts cleanly. Recovery is not involved
> - it applies only to tx that have reached prepare.
>
>> b) node goes down after TM issues prepare
>> - when TM issues a commit it receives an XAException
>> (perhaps XA_HEURRB) and again it is aware of the heuristic
>> outcome
>
> Returning cleanly from a prepare is a promise by the RM to
> successfully apply any subsequent commit. You're not in a
> position to make such a promise unless your state is fault
> tolerant, as a node crash would otherwise leave you with
> inconsistent state.
>
> It's not as simple as saying you'd rollback - what if you
> prepare, get told to commit, apply remote changes (step
> 4.2.1), then crash before applying local changes (4.2.2)?
> You can't report that as a rollback - you applied some of
> the updates. You have to include the tx in the recovery list
> as heuristic hazard (unless NodeA will transparently
> repopulate with the committed data, in which case you can
> mask the failure or report heuristic commit), but how to
> even detect the heuristic at recovery time? NodeA has no
> persistent record of the tx and NodeB thinks it completed
> cleanly and has cleaned up its tx record to avoid leaking.
> Where is the data that tells you you've got a problem?
>
> Or have a more sophisticated scenario where there is an
> additional NodeC, thus requiring multiple 'apply remote
> changes' calls. Are those atomic across the cluster? If
> there is a possibility that NodeB will apply the update but
> NodeC won't, or NodeA will crash after issuing a call to B
> but before C, you can wind up with inconsistent state in the
> surviving B and C. Alternatively, what if A survives but C
> crashes whilst applying changes that B has already
> sucessfully applied? That's not necessarily a recovery
> situation as far as the TM is concerned, but it may be from
> your perspective as you'll need to detect and (ideally) fix
> or (as a last resort) report the inconsistent data.
>
> A lot of your behaviour is going to depend on what it means
> for a node to recover after a crash. If it simply comes up
> empty and expects to be repopulated from an external source,
> as with a normal cache, then your relation to the XAResource
> of that external source is critical. On the other hand if
> your cluster node is itself fault tolerant through
> replication, then you need to think carefully about how the
> RM functionality ties into that replication - basically the
> tx state information is not local to the node where the
> XAResource resides, but must be replicated in the same
> manner as the other data in that node and that replication
> must be synchronous at certain state transitions in the tx
> lifecycle - it's logging recovery information through RPC
> rather than disk write. Really interesting things are going
> to happen if a single transaction spans data that is a mix
> of cache copy of data stored persistently in an XA database
> and data for which infinispan is the definitive, fault
> tolerant repository.
>
> To make a cluster appear to the outside world as a single
> logical entity for transaction purposes, you're pretty much
> going to wind up doing interposition. That means you're
> implementing not only an RM but substantial chunks of what
> amounts to a TM too. Have fun.
>
> Jonathan.
>
> --
> ------------------------------------------------------------
> Registered Address: Red Hat UK Ltd, Amberley Place, 107-111
> Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
> Registered in UK and Wales under Company Registration No.
> 3798903  Directors: Michael Cunningham (USA), Charlie Peters
> (USA), Matt Parsons (USA) and Brendan Lane (Ireland)
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

---
Mark Little
[hidden email]

JBoss, by Red Hat
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland).





_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Mark Little
In reply to this post by Mircea Markus

On 6 Jan 2011, at 10:45, Mircea Markus wrote:

>
> On 5 Jan 2011, at 17:19, Jonathan Halliday wrote:
>
>> On 01/05/2011 03:42 PM, Mircea Markus wrote:
>>>
>>> On 5 Jan 2011, at 14:51, Mircea Markus wrote:
>>>
>>>> FYI, a discussion I have with Jonathan around recovery support from TM
>>>>
>>>> On 5 Jan 2011, at 14:43, Jonathan Halliday wrote:
>>>>> On 01/05/2011 02:18 PM, Mircea Markus wrote:
>>>>>
>>>>>> I don't know how the TM recovery process picks up the XAResource instance on which to call XAResource.recover, but I imagine it expects this method to return all the prepared(or heuristic completed) transactions from the _whole transaction branch_, i.e. from the entire cluster.
>>>>>
>>>>> all from the logical RM, which you happen to implement as a cluster, yes.
>>>>>
>>>>>> I'm asking this  because right now there's no way for a node to know all the prepared transaction in the entire cluster. This is doable but would involve an broadcast to query the cluster, which might be costly (time and bandwidth).
>>>>>
>>>>> right. not to mention it should, strictly speaking, block or fail if any node is unreachable, which kinda sucks from an availability perspective.
>>> So if a node does not respond to the broadcast, it is incorrect to return the prepared transactions received from the other nodes? (is this because the TM expects to receive some tx that it knows for sure to be prepared?) Or would a "best effort" be "good enough"? (e.g. I broadcast the query and return all the results received in 1 sec)
>>
>> hmm, interesting question.
>>
>> Keep in mind that the XA spec dates from a time when a typical large clustered RM was 2-3 oracle nodes on the same LAN segment. It simply isn't geared to a world where the number of nodes is so large and widely distributed that the probability of *all* of them being available simultaneously is pretty slim. Likewise the number of transaction managers connected to a resource was assumed to be small, often 1, rather than the large N we see on modern clusters / clouds. As a result, the spec either fails to give guidance on some issues because they weren't significant at the time it was written, or implies/mandates behaviour that is counter productive in modern environments.
>>
>> Thus IMO some compromises are necessary to make XA usable in the real world, especially at scale. To further complicate matters, these are split across RM and TM, with different vendors having different views on the subject. My advice is geared to the way JBossTS drives XA recovery - other TMs may behave differently and make greater or lesser assumptions about compliance with the letter of the spec. As a result you may find that making your RM work with multiple vendor's TMs requires a) configuration options and b) a lot of painful testing.  Likewise JBossTS contains code paths and config options geared to dealing with bugs or non-compliant behaviour in various vendor's RMs.
>>
>> Now, on to the specific question: The list returned should, strictly speaking, be complete. There are two problems with that. First, you have to be able to reach all your cluster nodes to build a complete list which, as previously mentioned, is pretty unlikely in a sufficiently large cluster. Your practical strategies are thus as you say: either a) throw an XAException(XAER_RMFAIL) if any node is unreachable within a reasonable timeout and accept that this may mean an unnecessary delay in recovering the subset of tx that are known or b) return a partial list on a best effort basis. The latter approach allows the transaction manager to deal with at least some of the in-doubt tx, which may in turn mean releasing resources/locks in the RM. In general I'd favour that option as having higher practical value in terms of allowing the best possible level of service to be maintained in the face of ongoing failures.

+1

In fact on some mainframe implementations of CICS, for example, it behaves exactly like this.

>>
>> JBossTS will rescan every N minutes (2 by default) and thus you can simply include any newly discovered in-doubt tx as they become known due to e.g. partitioned nodes rejoining the cluster, and the TM will deal with them when they are first seen. Note however that some TMs assume that if they scan an RM and that RM does not subsequently crash, no new in-doubt transactions will occur except from heuristics. Let's gloss over how they can even detect a crash/recover of the RM if the driver masks it with failover or the event happens during a period when the TM makes no call on the driver. Such a TM will perform a recovery scan once at TM startup and not repeat. In such case you may have in-doubt tx from nodes unavailable at that crucial time subsequently sitting around for a prolonged period, tying up precious resources and potentially blocking subsequent updates. Most RM vendors provide some kind of management capability for admins to view and manually force completion of i!
 n!
> -doubt tx. command line tool, jmx, web gui, whatever, just so long as it exists.
> When a node crashes all the transactions that node owns (i.e. tx which were originated on that node and XAResource instance residing on that node) automatically rollback, so that no resources (locks mainly) are held. The only thing we need to make sure though is that the given transaction ids (the one that heuristically rollback) are returned by theXAResource.recover method - doable in the same way we handle prepares. I imagine that we'll have to keep these XIDs until XAResource.forget(XID) is called, am I right? Is it common/possible for people to use TM _without_ recovery? If so,  this "held heuristic completed TX" functionality should be configurable (enabled/disabled) in order to avoid memory leaks (no recovery means .forget never gets called)    

If you're using a transaction manager then use it all. Don't futz about and just use this bit or that bit and still say you're using transactions ;-)

Mark.

---
Mark Little
[hidden email]

JBoss, by Red Hat
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland).





_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] XAResource.isSameRM

Mark Little
In reply to this post by Jonathan Halliday

On 6 Jan 2011, at 11:48, Jonathan Halliday wrote:

>
>> Is it common/possible for people to use TM _without_ recovery? If so,  this "held heuristic completed TX" functionality should be configurable (enabled/disabled) in order to avoid memory leaks (no recovery means .forget never gets called)
>
> It is not common. That said, JBossTS has for similar reasons
> got a 'give up after N hours' config option which will
> eventually abandon tx that have not recovered. It's off
> (i.e. never give up) by default but a small number of users
> find it handy. Most just use the admin tooling to manually
> clean up the small number of unrecoverable situations - it's
> safer in most cases.

That option is there because sometimes machines don't recover. The assumption behind transactions is that failed machines do eventually recover, but the realities are sometimes slightly different. Or sys admins decide to recover a machine's state on a different machine, with a different IP address, and expect things (e.g., the TM) to be omniscient and figure it all out ;-)

Mark.

---
Mark Little
[hidden email]

JBoss, by Red Hat
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom.
Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland).





_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...