Expected Behavior
The online-store latency remains the same and doesn't grow over the time. We assume that amount of data in redis is not growing.
Current Behavior
The median latency growth over the past 30 days is on the chart above.
Steps to reproduce
We use a custom aiohttp python service to relay requests to the online-store, where we invoke the native feast client. The service is completely stateless, and I don't expect it to be the source of the problem.
here is our feast config:
config = RepoConfig(
project='feast',
registry='gs://our-bucket/feast/registry_file_3.db',
provider='gcp',
online_store=RedisOnlineStoreConfig(
connection_string=get_redis_connection_string(),
redis_type='redis',
key_ttl_seconds=REDIS_TTL_SECONDS,
),
offline_store=BigQueryOfflineStoreConfig(
dataset='feature_store',
project_id='project-id',
),
entity_key_serialization_version=2,
)
as you see, we use the default registry cache TTL (=600).
Specifications
- Version: 0.28.0
- Platform: amd64
- Subsystem: debian 10
Possible Solution
We noticed that changing the path to file-based registry (i.e. effectively re-creating it) eliminates the latency growth and it back to normal (today's chart):
Therefore, a solution might be related to fixing the registry caching mechanism.
Let me know if further details are needed!
Expected Behavior
The online-store latency remains the same and doesn't grow over the time. We assume that amount of data in redis is not growing.
Current Behavior
The median latency growth over the past 30 days is on the chart above.
Steps to reproduce
We use a custom aiohttp python service to relay requests to the online-store, where we invoke the native feast client. The service is completely stateless, and I don't expect it to be the source of the problem.
here is our feast config:
as you see, we use the default registry cache TTL (=600).
Specifications
Possible Solution
We noticed that changing the path to file-based registry (i.e. effectively re-creating it) eliminates the latency growth and it back to normal (today's chart):
Therefore, a solution might be related to fixing the registry caching mechanism.
Let me know if further details are needed!