Sorry, something went wrong.
| _, err := r.getRegistryProto() | ||
| if err != nil { | ||
| if _, ok := r.registryStore.(*FileRegistryStore); ok { // S3에는 굳이 연동할 필요 없어 보임. 오히려 이로 인해 정상적이던 s3 내의 레지스트리 파일이 초기화된 파일로 덮어쓰기 될 수도 있지 않음? | ||
| if _, ok := r.registryStore.(*FileRegistryStore); ok { |
There was a problem hiding this comment.
I didn't apply if statement in case that registry store is S3RegistryStore. because this initialized empty registry store is likely to overwrite normal registry file in s3 storage. Of course, this case is when the err of r.getRegistryProto() method is the network error not abnormal file in s3 storage.
Sorry, something went wrong.
There was a problem hiding this comment.
would you mind adding a test for this please?
Sorry, something went wrong.
|
@franciscojavierarceo thanks for review. But I think about how to write test-code for s3 registry store. As you konw, for testing s3 registry, any s3 bucket needs to exist. Of course, in my localhost test, my own s3 bucket exists but in the test of this public repository, I thins that a single s3 bucket will be public.. it is right? Or, how to make mock s3 bucket? Can you advise to me about how to add test-code for s3 registry store? |
Sorry, something went wrong.
|
@franciscojavierarceo thanks for review. But I think about how to write test-code for s3 registry store. As you konw, for testing s3 registry, any s3 bucket needs to exist. Of course, in my localhost test, my own s3 bucket exists but in the test of this public repository, I thins that a single s3 bucket will be public.. it is right? Or, how to make mock s3 bucket? Can you advise to me about how to add test-code for s3 registry store? Hi @iamcodingcat , thank you for this feature. I know add unit test/integration test here is difficult. by any chance, can you post any results/logs from your local run here as a reference for us? So far, the only concern from my side is the authentication and the credential handling for accessing the S3. |
Sorry, something went wrong.
|
import (
"bytes"
"context"
"errors"
"io"
"testing"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
"github.com/feast-dev/feast/go/internal/feast/registry"
"github.com/stretchr/testify/assert"
)
// Define a mock S3 client
type MockS3Client struct {
GetObjectFn func(ctx context.Context, input *s3.GetObjectInput) (*s3.GetObjectOutput, error)
DeleteObjectFn func(ctx context.Context, input *s3.DeleteObjectInput) (*s3.DeleteObjectOutput, error)
}
func (m *MockS3Client) GetObject(ctx context.Context, input *s3.GetObjectInput) (*s3.GetObjectOutput, error) {
if m.GetObjectFn != nil {
return m.GetObjectFn(ctx, input)
}
return nil, errors.New("not implemented")
}
func (m *MockS3Client) DeleteObject(ctx context.Context, input *s3.DeleteObjectInput) (*s3.DeleteObjectOutput, error) {
if m.DeleteObjectFn != nil {
return m.DeleteObjectFn(ctx, input)
}
return nil, errors.New("not implemented")
}
func NewMockS3RegistryStore(mockClient *MockS3Client, filePath string) *registry.S3RegistryStore {
return ®istry.S3RegistryStore{
filePath: filePath,
s3Client: mockClient,
}
}
|
Sorry, something went wrong.
|
Hi @iamcodingcat , thank you for this feature. I know add unit test/integration test here is difficult. by any chance, can you post any results/logs from your local run here as a reference for us? Hi. @shuchu thanks for review. I attached my console std output after running ./feast --type=http command connected to my s3 registry store. |
Sorry, something went wrong.
|
@franciscojavierarceo thanks for your guide. I try to add test-code and if I have a question, I will ask for it. |
Sorry, something went wrong.
|
@shuchu @franciscojavierarceo {"level":"error","error":"operation error S3: GetObject, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name.","time":"2025-05-09T05:01:28Z","message":"Registry refresh Failed"} At present, when this error occurs, the server is still running not panic. Of course, when that error occurs, it seems that the health check endpoint (/health) also does not return a normal response. So even if the Go feature server is deployed in a Kubernetes environment as a Deployment, the readinessProbe would fail, and traffic likely wouldn’t be routed to the pods with the error during a rolling update. Still, would it be better to explicitly panic in order to clearly indicate the error? |
Sorry, something went wrong.
| ifeq ($(PB_ARCH), aarch64) | ||
| PB_ARCH=aarch_64 | ||
| endif |
There was a problem hiding this comment.
In linux based arm64 architecture platform, output of uname -m command is aarch64. If this if-statement doesn't exist, PB_ARCH value is aarch64 and this cause error when downloading protobuf release download. Because arm architecture name of protobuf release file name is aarch_64 not aarch64. please refer below link
Sorry, something went wrong.
There was a problem hiding this comment.
@shuchu @franciscojavierarceo Hi! I add test-code for s3 registry store with mocking s3 client. I ask for you to review it.
Sorry, something went wrong.
| }) | ||
| if err != nil { | ||
| return nil, err | ||
| panic(err) |
There was a problem hiding this comment.
If I don't apply panic, a failed test case seems to normal.
Sorry, something went wrong.
| }) | ||
| if err != nil { | ||
| return err | ||
| panic(err) |
There was a problem hiding this comment.
If I don't apply panic, a failed test case seems to normal.
Sorry, something went wrong.
|
@shuchu @franciscojavierarceo In the go feature server, I encountered synchronization issue using redis as online-store. pleease check this issue This is automatically resolved.. I am confused at... anyway it is normal. you ignore this comment! |
Sorry, something went wrong.
|
@iamcodingcat can you fix the DCO? you just have to click button in UI. |
Sorry, something went wrong.
|
@franciscojavierarceo thnaks for review approval. I fixed DCO. I request your review again :) |
Sorry, something went wrong.
|
Hi @iamcodingcat it looks like this PR broke the master branch, would you mind reopening it so I can trigger the integration tests? So sorry about that 😢 |
Sorry, something went wrong.
|
@franciscojavierarceo Sure. I reopened #5352 . please check it :) |
Sorry, something went wrong.
What this PR does / why we need it:
This pull request integrates S3 storage as a registry store supported by the Go Feature Server, which is currently in alpha. At the moment, the Go Feature Server only supports a registry store based on the local file system. To make it suitable for production use in a service environment, we developed support for a registry store based on AWS S3 — a widely adopted and popular cloud storage solution.
I developed below things
*For your reference, the commit username younghun-jo-ilevit-com is also me.
Which issue(s) this PR fixes:
No. it is just for feature enhancement.
Misc