Which version of the Amazon Review Dataset was adopted?

[wylapp]

Dear authors,

There are 3 versions (2014, 2018, 2023) of the Amazon Review Dataset, but I found that your paper didn't mention which version was adopted.
Could you please provide more detailed information about which version was adopted?

Thanks!

[kCMI113]

I'm not the author, but based on my understanding, the 2014 version does not contain the Fashion category, so the dataset used is likely either the 2018 or 2023 version. Based on the data_process.py file, the dataset is separated into amazon and new_amazon, which suggests that amazon likely refers to the 2018 version, while new_amazon corresponds to the 2023 version.

[wylapp]

Thanks for the information!
That's not sounds like a good news. The Beauty subset of 2018 version only contains 991 users and 85 items (after preprocessing), conducting experiments on such a small dataset makes the experimental results very brittle.
I'll do some further investigation.

[ChenJazzyBoss]

@kCMI113 Hello, could you please tell me how your replication is going? Currently, I'm facing several issues:
Firstly, I'm only using the Yelp dataset. I've noticed that the shape of the LLM embeddings provided by the author is (11383, 1536). The embeddings provided by the author are not the full set. In comparison, the LLM embeddings I replicated and generated have a shape of (139918, 1536). Here, 11383 refers to the quantity, and 1536 refers to the dimension.
Secondly, it's impossible to run the code using the author's LLM embeddings. An error related to the quantity will occur during the process. However, after I used the full set of LLM embeddings, the result was extremely high.
I sincerely hope to know about your replication situation. Thank you so much!

[shikainan-hub]

@kCMI113 Hello, could you please tell me how your replication is going? Currently, I'm facing several issues: Firstly, I'm only using the Yelp dataset. I've noticed that the shape of the LLM embeddings provided by the author is (11383, 1536). The embeddings provided by the author are not the full set. In comparison, the LLM embeddings I replicated and generated have a shape of (139918, 1536). Here, 11383 refers to the quantity, and 1536 refers to the dimension. Secondly, it's impossible to run the code using the author's LLM embeddings. An error related to the quantity will occur during the process. However, after I used the full set of LLM embeddings, the result was extremely high. I sincerely hope to know about your replication situation. Thank you so much!

Hi there,

I noticed you’ve been working with the author’s Yelp setup as well. Could you tell me where you obtained the precomputed LLM embeddings (the 11,383×1536 matrix)? I can only find instructions to generate embeddings myself, and when I do—training on the full set—the evaluation metrics come out unrealistically high.

[ChenJazzyBoss]

Hello, the author has provided an embedded download link: Link as follows. Currently, the situation I'm encountering is similar to yours, and the final indicator result is very high. I haven't found the problem yet. @shikainan-hub

[kCMI113]

@kCMI113 Hello, could you please tell me how your replication is going? Currently, I'm facing several issues: Firstly, I'm only using the Yelp dataset. I've noticed that the shape of the LLM embeddings provided by the author is (11383, 1536). The embeddings provided by the author are not the full set. In comparison, the LLM embeddings I replicated and generated have a shape of (139918, 1536). Here, 11383 refers to the quantity, and 1536 refers to the dimension. Secondly, it's impossible to run the code using the author's LLM embeddings. An error related to the quantity will occur during the process. However, after I used the full set of LLM embeddings, the result was extremely high. I sincerely hope to know about your replication situation. Thank you so much!

Currently, I'm working with the Amazon Fashion dataset, but after running scft/avg.bash, all the llmemb values in convert.ipynb turned out to be NaN.
There were some version mismatches (like Python, torch, CUDA, etc.), so I made a few changes—switched to llama2-7b-hf, removed DeepSpeed, and so on.
To figure out what went wrong, I’m gonna try testing codes on the Yelp dataset now.
I’ll share the results once I have them!

[kCMI113]

@ChenJazzyBoss

Hello, as previously shared, I encountered NaN values for all LLM embeddings in fashion dataset — and now I'm seeing the same issue with the Yelp dataset as well. I suspect this may be due to some modifications I made.

That aside, since I haven't modified any data-related code, I re-ran the data processing script for the Beauty and Fashion datasets to compare the number of items with those in the provided llm-embedding.pkl files. As you previously mentioned for the Yelp dataset, there is also a mismatch in item counts for both the 2018 amazon Fashion and Beauty datasets. Additionally, the data format of the 2023 Amazon dataset does not align with the current processing code.

I think the author should take a look at this issue and explain it.

[ChenJazzyBoss]

@kCMI113 Hello! After careful adjustments, I have split and modified the code in experiments/yelp/scft/avg.bash into two distinct parts: embedding generation and model training. Additionally, I optimized data/yelp/item_prompt.ipynb to enable the generation of longer prompts. The embedding generation process has been completed, and the final run is now underway. However, I noticed that the model metrics are unusually high. Based on this, I suspect there might be an issue with the dataset splitting strategy, potentially causing data leakage where training and test data are cross-contaminated. Thoroughly checking this will require substantial time and effort to validate the dataset partitioning logic, ensure the independence of data across different stages, and debug related code to confirm the integrity of the data split.

[kCMI113]

@ChenJazzyBoss

Could you tell me your environment setting? I'm running on an A100 GPU. When I use CUDA 10.2 + torch 1.12.9, it gives a torch version error. So I used higher versions of Python, torch, and other dependencies. However, this caused some code errors when I used the original code.
Additionally, when I use DeepSpeed in a bash script with the meta-llama/Llama-2-7b-hf or huggyllama/llama-7b models, it gives an error saying it can't find adapter_config.json. So, I modified the bash script to remove DeepSpeed and run it in the format of python main_llm.py --args ~.
I think these changes may be causing NaN values in my LLM-embedding results.
If you could share your settings (Python, torch, CUDA versions, and the LLaMA repository you used), it would be very helpful for me.
Thank you! :)

[ChenJazzyBoss]

@kCMI113 I didn't encounter the problem of not being able to find adapter_config.json. However, this might be due to the fact that the deepseek library is not installed. The following is the environment I used when executing the rat step. But it should be noted that since I am using a 3090 graphics card, compared with an A100 graphics card, some errors may occur, and you need to distinguish carefully. In addition, this is a virtual environment I used during the reproduction process. Some of the packages required during the scft step may not be in this environment, and you should pay attention to it.

requirements_llmemb.txt.txt

[kCMI113]

@ChenJazzyBoss I'll debug my code again based on your information. Thanks for your reply!

[ChenJazzyBoss]

@kCMI113 The virtual environment uploaded earlier was specifically prepared for the rat step. Due to package dependency conflicts, we had to create this environment separately.
requirements_llmemb.txt.txt

[WZY-Z]

I also encountered abnormally high evaluation metrics when using the provided LLMEmb embedding model. Have you identified where the issue lies?

@kCMI113 The virtual environment uploaded earlier was specifically prepared for the rat step. Due to package dependency conflicts, we had to create this environment separately. requirements_llmemb.txt.txt

[ChenJazzyBoss]

@WZY-Z Due to certain factors, the replication work of the article could not be continued. Regarding the previously mentioned abnormal indicator issue, the initial speculation was that it might be caused by data leakage. However, during the subsequent preliminary investigation, it was found that the existing clues are insufficient to conclusively prove the hypothesis of data leakage. Therefore, it is currently impossible to determine whether the abnormal indicators are caused by data leakage.

[shikainan-hub]

@ChenJazzyBoss you can read another paper LLMESR，which is the same as this paper，this paper contains dataset info