Sorry, something went wrong.
There was a problem hiding this comment.
Very nice!! 🚀🚀🚀
Sorry, something went wrong.
| def get_feature_transformation(self) -> Optional[Transformation]: | ||
| if not self.udf: | ||
| return None | ||
| if self.mode == TransformationMode.pandas or self.mode == "pandas": |
There was a problem hiding this comment.
Probably can just do a dictionary mapping
Sorry, something went wrong.
There was a problem hiding this comment.
sure will add that
Sorry, something went wrong.
| udf_string: str = "", | ||
| tags: Optional[Dict[str, str]] = None, | ||
| description: str = "", | ||
| owner: str = "", |
There was a problem hiding this comment.
Probably can add the singleton parameter too
Sorry, something went wrong.
|
|
||
|
|
||
| class TransformationMode(Enum): | ||
| PYTHON = "python" |
There was a problem hiding this comment.
Nice
Sorry, something went wrong.
| self, | ||
| *, | ||
| name: str, | ||
| mode: Union[TransformationMode, str], |
There was a problem hiding this comment.
Nice
Sorry, something went wrong.
| class TransformationMode(Enum): | ||
| PYTHON = "python" | ||
| PANDAS = "pandas" | ||
| spark = "spark" |
There was a problem hiding this comment.
| spark = "spark" | |
| SPARK = "spark" |
Sorry, something went wrong.
|
Do you think we can scope one type of batch transformation as our MVP? If so, which one would you feel most comfortable doing? |
Sorry, something went wrong.
|
Do you think we can scope one type of batch transformation as our MVP? If so, which one would you feel most comfortable doing? yep, maybe Spark SQL. that should be easy to implement, and we can get it live soon. |
Sorry, something went wrong.
|
Do you think we can scope one type of batch transformation as our MVP? If so, which one would you feel most comfortable doing? yep, maybe Spark SQL. that should be easy to implement, and we can get it live soon. LFG! |
Sorry, something went wrong.
|
FYI @HaoXuAI a very selfish goal I have is to use a Spark batch transformation to handle scaling embedding documents. So in an ideal world we could do something like: def create_embeddings(partitionData):
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
for row in partitionData:
document = str(row.document)
inputs = tokenizer(document, padding=True, truncation=True, return_tensors="pt", max_length=512)
result = model(**inputs)
embeddings = result.last_hidden_state[:, 0, :].cpu().detach().numpy()
lst = embeddings.flatten().tolist()
yield [row.id, lst, "", "{}", None]
And embeddings = dataset_df.rdd.mapPartitions(create_embeddings)
Used in a batch feature view and an on demand feature view (on write) to support offline and online processing of docs for RAG. The key thing here is that we'd be able to really scale RAG offline embedding and make the transition seamless to online. This example is from the Pinecone docs here: https://docs.pinecone.io/integrations/databricks#3-create-the-vector-embeddings |
Sorry, something went wrong.
|
FYI @HaoXuAI a very selfish goal I have is to use a Spark batch transformation to handling scaling embedding documents. So in an ideal world we could do something like: def create_embeddings(partitionData):
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
for row in partitionData:
document = str(row.document)
inputs = tokenizer(document, padding=True, truncation=True, return_tensors="pt", max_length=512)
result = model(**inputs)
embeddings = result.last_hidden_state[:, 0, :].cpu().detach().numpy()
lst = embeddings.flatten().tolist()
yield [row.id, lst, "", "{}", None]
And embeddings = dataset_df.rdd.mapPartitions(create_embeddings)
Used in a batch feature view and an on demand feature view (on write) to support offline and online processing of docs for RAG. The key thing here is that we'd be able to really scale RAG offline embedding and make the transition seamless to online. This example is from the Pinecone docs here: https://docs.pinecone.io/integrations/databricks#3-create-the-vector-embeddings This seems to be a really good example of the BatchFeatureView + transformation |
Sorry, something went wrong.
|
PR diverged, cherrypick at #5181 |
Sorry, something went wrong.
What this PR does / why we need it:
Created a Transformation interface. it still works with the current pandas_transformation, python_transformation etc.
The next step is refactor the BatchMaterializationEngine to make it works for both Materialization and Transformation.
Which issue(s) this PR fixes:
#4584
#4277 (comment)
#4696
Misc