[infinispan-dev] Protobuf metadata cache and x-site

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] Protobuf metadata cache and x-site

Galder Zamarreno
Hi,

We have an issue with protobuf metadata cache.

If you run in a multi-site scenario, protobuf metadata information does not travel across sites by default.

Being an internal cache, is it possible to somehow override/reconfigure it so that cross-site configuration can be added in standalone.xml?

We're currently running a periodic job that checks if the metadata is present and if not present add it. So, we have a workaround for it, but it'd be not very user friendly for end users.

Thoughts?

Cheers,
Galder

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Protobuf metadata cache and x-site

Pedro Ruivo-2


On 12-04-2018 15:49, Galder Zamarreno wrote:
> Hi,
>
> We have an issue with protobuf metadata cache.
>
> If you run in a multi-site scenario, protobuf metadata information does
> not travel across sites by default.
>
> Being an internal cache, is it possible to somehow override/reconfigure
> it so that cross-site configuration can be added in standalone.xml?

No :( since it is an internal cache, its configuration can't be changed.

>
> We're currently running a periodic job that checks if the metadata is
> present and if not present add it. So, we have a workaround for it, but
> it'd be not very user friendly for end users.
>
> Thoughts?

Unfortunately none... it is the first time an internal cache needs to do
some x-site.

>
> Cheers,
> Galder
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Protobuf metadata cache and x-site

Galder Zamarreno
Ok, we do need to find a better way to deal with this.


On Thu, Apr 12, 2018 at 5:56 PM Pedro Ruivo <[hidden email]> wrote:


On 12-04-2018 15:49, Galder Zamarreno wrote:
> Hi,
>
> We have an issue with protobuf metadata cache.
>
> If you run in a multi-site scenario, protobuf metadata information does
> not travel across sites by default.
>
> Being an internal cache, is it possible to somehow override/reconfigure
> it so that cross-site configuration can be added in standalone.xml?

No :( since it is an internal cache, its configuration can't be changed.

>
> We're currently running a periodic job that checks if the metadata is
> present and if not present add it. So, we have a workaround for it, but
> it'd be not very user friendly for end users.
>
> Thoughts?

Unfortunately none... it is the first time an internal cache needs to do
some x-site.

>
> Cheers,
> Galder
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Protobuf metadata cache and x-site

Pedro Ruivo-2
Wouldn't be better to assume the protobuf cache doesn't fit the internal
cache use case? :)

On 12-04-2018 17:21, Galder Zamarreno wrote:

> Ok, we do need to find a better way to deal with this.
>
> JIRA: https://issues.jboss.org/browse/ISPN-9074
>
> On Thu, Apr 12, 2018 at 5:56 PM Pedro Ruivo <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>
>     On 12-04-2018 15:49, Galder Zamarreno wrote:
>      > Hi,
>      >
>      > We have an issue with protobuf metadata cache.
>      >
>      > If you run in a multi-site scenario, protobuf metadata
>     information does
>      > not travel across sites by default.
>      >
>      > Being an internal cache, is it possible to somehow
>     override/reconfigure
>      > it so that cross-site configuration can be added in standalone.xml?
>
>     No :( since it is an internal cache, its configuration can't be changed.
>
>      >
>      > We're currently running a periodic job that checks if the
>     metadata is
>      > present and if not present add it. So, we have a workaround for
>     it, but
>      > it'd be not very user friendly for end users.
>      >
>      > Thoughts?
>
>     Unfortunately none... it is the first time an internal cache needs
>     to do
>     some x-site.
>
>      >
>      > Cheers,
>      > Galder
>      >
>      >
>      > _______________________________________________
>      > infinispan-dev mailing list
>      > [hidden email]
>     <mailto:[hidden email]>
>      > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>      >
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Protobuf metadata cache and x-site

Tristan Tarrant-2
It is definitely an internal cache. Because of this, automatically
backing it up to a remote site might not be such a good idea.

Backups are enabled per-cache, and therefore just blindly replicating
the schema cache to the other site is not a good idea.

I think that we need a cache-manager-level backup setting that does the
right thing.

Tristan

On 4/12/18 7:01 PM, Pedro Ruivo wrote:

> Wouldn't be better to assume the protobuf cache doesn't fit the internal
> cache use case? :)
>
> On 12-04-2018 17:21, Galder Zamarreno wrote:
>> Ok, we do need to find a better way to deal with this.
>>
>> JIRA: https://issues.jboss.org/browse/ISPN-9074
>>
>> On Thu, Apr 12, 2018 at 5:56 PM Pedro Ruivo <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>
>>
>>      On 12-04-2018 15:49, Galder Zamarreno wrote:
>>       > Hi,
>>       >
>>       > We have an issue with protobuf metadata cache.
>>       >
>>       > If you run in a multi-site scenario, protobuf metadata
>>      information does
>>       > not travel across sites by default.
>>       >
>>       > Being an internal cache, is it possible to somehow
>>      override/reconfigure
>>       > it so that cross-site configuration can be added in standalone.xml?
>>
>>      No :( since it is an internal cache, its configuration can't be changed.
>>
>>       >
>>       > We're currently running a periodic job that checks if the
>>      metadata is
>>       > present and if not present add it. So, we have a workaround for
>>      it, but
>>       > it'd be not very user friendly for end users.
>>       >
>>       > Thoughts?
>>
>>      Unfortunately none... it is the first time an internal cache needs
>>      to do
>>      some x-site.
>>
>>       >
>>       > Cheers,
>>       > Galder
>>       >
>>       >
>>       > _______________________________________________
>>       > infinispan-dev mailing list
>>       > [hidden email]
>>      <mailto:[hidden email]>
>>       > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>       >
>>      _______________________________________________
>>      infinispan-dev mailing list
>>      [hidden email] <mailto:[hidden email]>
>>      https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

--
Tristan Tarrant
Infinispan Lead and Data Grid Architect
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Protobuf metadata cache and x-site

Adrian Nistor
Backing up caches with protobuf payload to a remote site will not work
if they are indexed, unless the remote site already has the schema for
the types in question, or else indexing will fail. If the cache is not
indexed it matters less.

So the replication of protobuf metadata cache has to be arranged somehow
before any other data is replicated. Manual replication is indeed PITA.

I remember in the very early version of remote query the protobuf
metadata cache configuration was created programatically on startup
unless a manually defined configuration with that name was found,
already provided by the user. In that case the user's config was used.
This approach had the benefit of allowing the user to gain control if
needed. But can also lead to gloom and doom. Was that too bad to do it
again :)))?

Adrian

On 04/12/2018 10:27 PM, Tristan Tarrant wrote:

> It is definitely an internal cache. Because of this, automatically
> backing it up to a remote site might not be such a good idea.
>
> Backups are enabled per-cache, and therefore just blindly replicating
> the schema cache to the other site is not a good idea.
>
> I think that we need a cache-manager-level backup setting that does the
> right thing.
>
> Tristan
>
> On 4/12/18 7:01 PM, Pedro Ruivo wrote:
>> Wouldn't be better to assume the protobuf cache doesn't fit the internal
>> cache use case? :)
>>
>> On 12-04-2018 17:21, Galder Zamarreno wrote:
>>> Ok, we do need to find a better way to deal with this.
>>>
>>> JIRA: https://issues.jboss.org/browse/ISPN-9074
>>>
>>> On Thu, Apr 12, 2018 at 5:56 PM Pedro Ruivo <[hidden email]
>>> <mailto:[hidden email]>> wrote:
>>>
>>>
>>>
>>>       On 12-04-2018 15:49, Galder Zamarreno wrote:
>>>        > Hi,
>>>        >
>>>        > We have an issue with protobuf metadata cache.
>>>        >
>>>        > If you run in a multi-site scenario, protobuf metadata
>>>       information does
>>>        > not travel across sites by default.
>>>        >
>>>        > Being an internal cache, is it possible to somehow
>>>       override/reconfigure
>>>        > it so that cross-site configuration can be added in standalone.xml?
>>>
>>>       No :( since it is an internal cache, its configuration can't be changed.
>>>
>>>        >
>>>        > We're currently running a periodic job that checks if the
>>>       metadata is
>>>        > present and if not present add it. So, we have a workaround for
>>>       it, but
>>>        > it'd be not very user friendly for end users.
>>>        >
>>>        > Thoughts?
>>>
>>>       Unfortunately none... it is the first time an internal cache needs
>>>       to do
>>>       some x-site.
>>>
>>>        >
>>>        > Cheers,
>>>        > Galder
>>>        >
>>>        >
>>>        > _______________________________________________
>>>        > infinispan-dev mailing list
>>>        > [hidden email]
>>>       <mailto:[hidden email]>
>>>        > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>        >
>>>       _______________________________________________
>>>       infinispan-dev mailing list
>>>       [hidden email] <mailto:[hidden email]>
>>>       https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Protobuf metadata cache and x-site

Tristan Tarrant-2
I think we can certainly make it additive, especially now that we have
configuration templates in place: the user supplies a base template, and
the internal cache logic override with what is needed so that broken
configs are less probable (but still possible). Alternatively, instead
of overriding, we just check that it matches the requirements.

Tristan

On 4/12/18 10:10 PM, Adrian Nistor wrote:

> Backing up caches with protobuf payload to a remote site will not work
> if they are indexed, unless the remote site already has the schema for
> the types in question, or else indexing will fail. If the cache is not
> indexed it matters less.
>
> So the replication of protobuf metadata cache has to be arranged somehow
> before any other data is replicated. Manual replication is indeed PITA.
>
> I remember in the very early version of remote query the protobuf
> metadata cache configuration was created programatically on startup
> unless a manually defined configuration with that name was found,
> already provided by the user. In that case the user's config was used.
> This approach had the benefit of allowing the user to gain control if
> needed. But can also lead to gloom and doom. Was that too bad to do it
> again :)))?
>
> Adrian
>
> On 04/12/2018 10:27 PM, Tristan Tarrant wrote:
>> It is definitely an internal cache. Because of this, automatically
>> backing it up to a remote site might not be such a good idea.
>>
>> Backups are enabled per-cache, and therefore just blindly replicating
>> the schema cache to the other site is not a good idea.
>>
>> I think that we need a cache-manager-level backup setting that does the
>> right thing.
>>
>> Tristan
>>
>> On 4/12/18 7:01 PM, Pedro Ruivo wrote:
>>> Wouldn't be better to assume the protobuf cache doesn't fit the internal
>>> cache use case? :)
>>>
>>> On 12-04-2018 17:21, Galder Zamarreno wrote:
>>>> Ok, we do need to find a better way to deal with this.
>>>>
>>>> JIRA: https://issues.jboss.org/browse/ISPN-9074
>>>>
>>>> On Thu, Apr 12, 2018 at 5:56 PM Pedro Ruivo <[hidden email]
>>>> <mailto:[hidden email]>> wrote:
>>>>
>>>>
>>>>
>>>>       On 12-04-2018 15:49, Galder Zamarreno wrote:
>>>>        > Hi,
>>>>        >
>>>>        > We have an issue with protobuf metadata cache.
>>>>        >
>>>>        > If you run in a multi-site scenario, protobuf metadata
>>>>       information does
>>>>        > not travel across sites by default.
>>>>        >
>>>>        > Being an internal cache, is it possible to somehow
>>>>       override/reconfigure
>>>>        > it so that cross-site configuration can be added in
>>>> standalone.xml?
>>>>
>>>>       No :( since it is an internal cache, its configuration can't
>>>> be changed.
>>>>
>>>>        >
>>>>        > We're currently running a periodic job that checks if the
>>>>       metadata is
>>>>        > present and if not present add it. So, we have a workaround
>>>> for
>>>>       it, but
>>>>        > it'd be not very user friendly for end users.
>>>>        >
>>>>        > Thoughts?
>>>>
>>>>       Unfortunately none... it is the first time an internal cache
>>>> needs
>>>>       to do
>>>>       some x-site.
>>>>
>>>>        >
>>>>        > Cheers,
>>>>        > Galder
>>>>        >
>>>>        >
>>>>        > _______________________________________________
>>>>        > infinispan-dev mailing list
>>>>        > [hidden email]
>>>>       <mailto:[hidden email]>
>>>>        > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>        >
>>>>       _______________________________________________
>>>>       infinispan-dev mailing list
>>>>       [hidden email]
>>>> <mailto:[hidden email]>
>>>>       https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>

--
Tristan Tarrant
Infinispan Lead and Data Grid Architect
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Protobuf metadata cache and x-site

Tristan Tarrant-2
We also need: backup priority for internal caches as well as conflict
resolution for backups to avoid broken data replicating in the wrong
direction.

Tristan

On 4/12/18 10:13 PM, Tristan Tarrant wrote:

> I think we can certainly make it additive, especially now that we have
> configuration templates in place: the user supplies a base template, and
> the internal cache logic override with what is needed so that broken
> configs are less probable (but still possible). Alternatively, instead
> of overriding, we just check that it matches the requirements.
>
> Tristan
>
> On 4/12/18 10:10 PM, Adrian Nistor wrote:
>> Backing up caches with protobuf payload to a remote site will not work
>> if they are indexed, unless the remote site already has the schema for
>> the types in question, or else indexing will fail. If the cache is not
>> indexed it matters less.
>>
>> So the replication of protobuf metadata cache has to be arranged
>> somehow before any other data is replicated. Manual replication is
>> indeed PITA.
>>
>> I remember in the very early version of remote query the protobuf
>> metadata cache configuration was created programatically on startup
>> unless a manually defined configuration with that name was found,
>> already provided by the user. In that case the user's config was used.
>> This approach had the benefit of allowing the user to gain control if
>> needed. But can also lead to gloom and doom. Was that too bad to do it
>> again :)))?
>>
>> Adrian
>>
>> On 04/12/2018 10:27 PM, Tristan Tarrant wrote:
>>> It is definitely an internal cache. Because of this, automatically
>>> backing it up to a remote site might not be such a good idea.
>>>
>>> Backups are enabled per-cache, and therefore just blindly replicating
>>> the schema cache to the other site is not a good idea.
>>>
>>> I think that we need a cache-manager-level backup setting that does the
>>> right thing.
>>>
>>> Tristan
>>>
>>> On 4/12/18 7:01 PM, Pedro Ruivo wrote:
>>>> Wouldn't be better to assume the protobuf cache doesn't fit the
>>>> internal
>>>> cache use case? :)
>>>>
>>>> On 12-04-2018 17:21, Galder Zamarreno wrote:
>>>>> Ok, we do need to find a better way to deal with this.
>>>>>
>>>>> JIRA: https://issues.jboss.org/browse/ISPN-9074
>>>>>
>>>>> On Thu, Apr 12, 2018 at 5:56 PM Pedro Ruivo <[hidden email]
>>>>> <mailto:[hidden email]>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>       On 12-04-2018 15:49, Galder Zamarreno wrote:
>>>>>        > Hi,
>>>>>        >
>>>>>        > We have an issue with protobuf metadata cache.
>>>>>        >
>>>>>        > If you run in a multi-site scenario, protobuf metadata
>>>>>       information does
>>>>>        > not travel across sites by default.
>>>>>        >
>>>>>        > Being an internal cache, is it possible to somehow
>>>>>       override/reconfigure
>>>>>        > it so that cross-site configuration can be added in
>>>>> standalone.xml?
>>>>>
>>>>>       No :( since it is an internal cache, its configuration can't
>>>>> be changed.
>>>>>
>>>>>        >
>>>>>        > We're currently running a periodic job that checks if the
>>>>>       metadata is
>>>>>        > present and if not present add it. So, we have a
>>>>> workaround for
>>>>>       it, but
>>>>>        > it'd be not very user friendly for end users.
>>>>>        >
>>>>>        > Thoughts?
>>>>>
>>>>>       Unfortunately none... it is the first time an internal cache
>>>>> needs
>>>>>       to do
>>>>>       some x-site.
>>>>>
>>>>>        >
>>>>>        > Cheers,
>>>>>        > Galder
>>>>>        >
>>>>>        >
>>>>>        > _______________________________________________
>>>>>        > infinispan-dev mailing list
>>>>>        > [hidden email]
>>>>>       <mailto:[hidden email]>
>>>>>        > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>        >
>>>>>       _______________________________________________
>>>>>       infinispan-dev mailing list
>>>>>       [hidden email]
>>>>> <mailto:[hidden email]>
>>>>>       https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> [hidden email]
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>
>

--
Tristan Tarrant
Infinispan Lead and Data Grid Architect
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev