Release 1.0.0: Changed to local model storage (#208)

* Refactoring/#146 use new model save functionality (#186)
* switched use to huggingface transfer save pretrained version
* changed to load local model
* removed download sample model fixture because of duplication
* #147: Removed huggingface token from model loading (#203)
* removed token_conn everywhere except model download and tests
* removed token_conn from relevant tests
* Documentation improvements (#199)
* Prepared release 1.0.0 (#206)

Co-authored-by: Torsten Kilias <torsten.kilias@exasol.com>

Author: MarleneKress79789

README.md

@@ -3,6 +3,9 @@
 An Exasol extension to use state-of-the-art pretrained machine learning models 
 via the [transformers api](https://github.com/huggingface/transformers).
 
+This Extension is build and tested for Linux OS, and does not have Windows or MacOS support. 
+It might work regardless but proceed at your own risk.
+
 
 ## Table of Contents
 

buildspec.yml

@@ -22,6 +22,5 @@ phases:
       - echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USER" --password-stdin
   build:
     commands:
-      - poetry run nox -s unit_tests
       - poetry run nox -s start_database
       - poetry run nox -s integration_tests

doc/changes/changelog.md

@@ -1,5 +1,6 @@
 # Changelog
 
+* [1.0.0](changes_1.0.0.md)
 * [0.10.0](changes_0.10.0.md)
 * [0.9.2](changes_0.9.2.md)
 * [0.9.1](changes_0.9.1.md)

doc/changes/changes_0.10.0.md

@@ -20,4 +20,3 @@ Deploying SLC under Windows, releasing to PyPi.
 
 ### Security 
 
-

doc/changes/changes_0.10.1.md

@@ -0,0 +1,46 @@
+# Transformers Extension 1.0.0, 2024-04-12
+
+Code name: Local model loading
+
+
+## Summary
+
+In this release, we integrated a new model loading functionality which means downloaded models will now be saved 
+in the BucketFS. This means, the Prediction UDFs do not connect to the internet to look for model updates. 
+There are also documentation updates, and we updated cryptography to >= 42.0.4.
+
+### Breaking API changes
+
+The change in the model loading functionality means the API for the Prediction UDFs has changed. 
+The 'token_conn' parameter was removed from the UDF calls. You can now call the UDFs 
+as follows (Example case for the filling mask udf):
+
+```sql
+SELECT TE_FILLING_MASK_UDF(
+    device_id,
+    bucketfs_conn,
+    sub_dir,
+    model_name,
+    text_data,
+    top_k
+)
+```
+
+
+### Features
+
+- #205: Added vagrant setup
+- #146: Integrated new download and load functions using save_pretrained
+
+### Documentation
+
+- #133: Improved user and developer documentation with additional information
+
+### Refactorings
+
+- #147: Removed token_con from Prediction UDFs
+
+### Security 
+
+ - Updated cryptography to >= 42.0.4
+

doc/changes/changes_1.0.0.md

@@ -27,6 +27,12 @@ and install it as follows:
 pip install <path/wheel-filename.whl> --extra-index-url https://download.pytorch.org/whl/cpu
 ```
 
+### Check wheel installation
+
+The wheel should be installed in `transformers-extension/dist`. After updating and building a new release 
+there may be multiple wheels installed here. This leads to problems, so check and delete the old wheels if necessary.
+You may also need to check 
+`transformers-extension/language_container/exasol_transformers_extension_container/flavor_base/release/dist` for the same reason.
 
 ### Run Tests
 All unit and integration tests can be run within the Poetry environment created 
@@ -42,6 +48,8 @@ Start a test database and run integration tests:
       poetry run nox -s integration_tests
 ```
 
+You can find more information regarding the tests in the [Tests](#tests) section below
+
 ## Add Transformer Tasks
 In the transformers-extension library, the 8 most popular NLP tasks provided by 
 [Transformers API](https://huggingface.co/docs/transformers/index) have already 
@@ -126,3 +134,21 @@ Currently, the CodeBuild project is managed manually and is triggered with a web
 For this our aws-ci user is added to this Repository. The webhook can be configured in the AWS CodeBuild 
 project directly.
 The CodeBuild project also uses our DockerHub user for the build. For this it has access to the AWS SecretsManager.
+
+
+#### 3. Release download test
+
+After you do a Release on the project, you may want to trigger the [SLC Download Test](https://github.com/exasol/transformers-extension/blob/8f57d1f0ca3f95a2d3edc9b84e8dd779aa6093d8/tests/integration_tests/with_db/deployment/test_language_container_deployer_cli.py#L117)
+to make sure the new SLC is uploaded and correctly named. 
+**This is especially important if the naming convention of the SLC was changed!**
+* testfile: [tests/integration_tests/with_db/deployment/test_language_container_deployer_cli.py](../../tests/integration_tests/with_db/deployment/test_language_container_deployer_cli.py)
+* test_name: test test_language_container_deployer_cli_by_downloading_container
+
+Also, during a release, the version object should be updated from time to time,
+so we actually use the correct SLC for the test. Do this 
+[here](https://github.com/exasol/transformers-extension/blob/8f57d1f0ca3f95a2d3edc9b84e8dd779aa6093d8/tests/integration_tests/with_db/deployment/test_language_container_deployer_cli.py#L128)
+
+## Good to know
+
+* Hugging Face models consist of 2 parts, the model and the Tokenizer. 
+Most of our functions deal with both parts