[Update 10/31/2024] This Go feature server code is updated from the Expedia Group's forked Feast branch (https://github.com/ExpediaGroup/feast.git) on 10/22/2024. Thanks the engineers of the Expedia Groups who contributed and improved the Go Feature Server.
To build and run the Go Feature Server locally, create a feature_store.yaml file with necessary configurations and run below commands:
go build -o feast-go ./go/main.go
# start the http server (metrics on port 9090 by default)
./feast-go --type=http --port=8080 --metrics-port=9090
# or start the gRPC server
#./feast-go --type=grpc --port=[your-choice] --metrics-port=9091
The server exposes Prometheus metrics at the /metrics endpoint on a dedicated port (default :9090).
- HTTP Mode: Metrics server runs on port 9090 (configurable via -metrics-port).
- gRPC Mode: Metrics server runs on port 9090 (configurable via -metrics-port).
Key metrics include:
- http_request_duration_seconds: Histogram of response latency.
- http_requests_total: Counter of HTTP requests by status, method, and path.
- Standard Go and Process metrics.
A /health endpoint is available on the main application port (default :8080) for readiness probes.
The OS level env variable ENABLE_OTEL_TRACING=="true"/"false" (string type) is used to enable/disable this service (with Tracing only).
You can also configure the service name using OTEL_SERVICE_NAME env variable (defaults to "FeastGoFeatureServer").
The default exporter URL is "http://localhost:4318". The default schema of sending data to collector is HTTP. Please refer the following two docs about the configuration of the OTEL exporter:
- https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/
- https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp#WithEndpoint
List of files have OTEL observability code
- internal/feast/transformation/transformation.go
- internal/feast/onlinestore/redisonlinestore.go
- internal/feast/server/grpc_server.go
- internal/feast/server/http_server.go
- internal/feast/server/server_commons.go
- internal/feast/featurestore.go
Example monitoring infra setup
- docker compose file to setup Prometheus, Jaeger, and OTEL-collector.
services:
prometheus:
image: prom/prometheus
volumes:
- ./prometheus.yaml:/etc/prometheus/prometheus.yaml
ports:
- 9090:9090 # web UI http://localhost:9090
jaeger:
image: jaegertracing/all-in-one:latest
ports:
- 16686:16686 # Web UI: http://localhost:16686
- 14268:14268 # http based receiver
- 14250:14250 # gRPC based receiver
otel-collector:
image: otel/opentelemetry-collector-contrib
volumes:
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
ports:
#- 1888:1888 # pprof extension
- 8888:8888 # Prometheus metrics exposed by the Collector
- 8889:8889 # Prometheus exporter metrics
#- 13133:13133 # health_check extension
#- 4317:4317 # OTLP gRPC receiver
- 4318:4318 # OTLP http receiver
- 55679:55679 # zpages extension. check http://localhost:55679/debug/tracez
depends_on:
- jaeger
- prometheus
- OTEL collector configure file.
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
exporters:
prometheus:
endpoint: 0.0.0.0:8889
namespace: feast-go
otlp/jaeger: # this is a gRPC based exporter. use "otelhttp" for http based exporter.
endpoint: jaeger:4317
tls:
insecure: true
extensions:
zpages:
endpoint: 0.0.0.0:55679
processors:
batch:
service:
extensions: [zpages]
pipelines:
metrics:
receivers: [otlp]
exporters: [prometheus]
traces:
receivers: [otlp]
exporters: [otlp/jaeger]
- Prometheus config.
#https://github.com/prometheus/prometheus/blob/release-3.6/config/testdata/conf.good.yml
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 1m
scrape_timeout: 30s # Increase this if needed
static_configs:
# Check the IP address of or Docker host network.
# Refer: https://stackoverflow.com/questions/48546124/what-is-the-linux-equivalent-of-host-docker-internal
- targets: ['172.17.0.1:8888'] # Replace with the Collector's IP and port
- job_name: 'otel-collected'
scrape_interval: 1m
scrape_timeout: 30s # Increase this if needed
static_configs:
- targets: ['172.17.0.1:8889'] # Replace with the Collector's IP and port
- Jaeger config file is not used in this setup.