Skip to content

Migration workflow 1.2x

Jump to bottom
Peter Weber edited this page Jul 14, 2026 · 33 revisions

v1.30.0

Alembic (Invenio)

uv run invenio alembic upgrade

API Harvester

# verify "Archives audiovisuelles" location pid
uv run invenio reroils api-harvester add-source VS-MEMO --url https://archives.memovs.ch/docs/api/ --classname rero_ils.modules.api_harvester.memovs.api.ApiMemovs --code memovs --update --setting location_pid=1222 --setting item_type_pid=67

uv run invenio reroils api-harvester harvest -n VS-MEMO -v -f 1990-01-01 2>&1 |tee /network/nfs/data_ils/ils/test/logs/memovs.log

v1.28.0 --> v1.29.0

Permissions

from rero_ils.modules.cli.fixtures import load_role_policies
from rero_ils.modules.cli.fixtures import load_system_role_policies
cfg ={   "org-access": [
    "pro_full_permissions",
    "pro_library_administrator"
  ]}
load_role_policies(cfg)

organisation

uv run invenio rero es index update-mapping

Operation logs

from invenio_search import current_search, current_search_client
# update operation logs templates
[p for p in current_search.put_templates()]

# update operation logs mapping for 2026
res = current_search_client.indices.get_template('operation_logs')
mapping = res['operation_logs']['mappings']
current_search_client.indices.put_mapping(mapping, 'operation_logs-2026')

Alembic (Invenio)

INVENIO_SQLALCHEMY_DATABASE_URI="$INVENIO_SQLALCHEMY_DATABASE_URI?options=-c%20statement_timeout%3D0" uv run invenio alembic upgrade

uv run invenio alembic stamp f9843093f686
INVENIO_SQLALCHEMY_DATABASE_URI="$INVENIO_SQLALCHEMY_DATABASE_URI?options=-c%20statement_timeout%3D0" uv run invenio alembic upgrade

uv run invenio alembic stamp 66db9c49c699
INVENIO_SQLALCHEMY_DATABASE_URI="$INVENIO_SQLALCHEMY_DATABASE_URI?options=-c%20statement_timeout%3D0" uv run invenio alembic upgrade

load slogan, html block, placeholders for the 3 organisations in the data (github.com/rero/rero-ils/pull/4112)

from logging import getLogger
from rero_ils.modules.organisations.api import Organisation

LOGGER = getLogger("migration")

PLACEHOLDER = [
    {"language": "en", "value": "What are you looking for?"},
    {"language": "fr", "value": "Que cherchez-vous ?"},
    {"language": "de", "value": "Was suchen Sie?"},
    {"language": "it", "value": "Cosa stai cercando?"},
]

HOMEPAGE_BY_CODE = {
    "bulle": {
        "placeholder": PLACEHOLDER,
        "slogan": [
            {"language": "en", "value": "Reading Opens Worlds"},
            {"language": "fr", "value": "Lire pour s'ouvrir sur le monde"},
            {"language": "de", "value": "Lesen, um sich der Welt zu öffnen"},
            {"language": "it", "value": "Leggere per aprirsi al mondo"},
        ],
        "blocks": {
            "left": [
                {
                    "language": "en",
                    "value": (
                        "<h5>Bulle Library Catalogue</h5>"
                        "<p>This interface gives you access to a list of documents"
                        " from the following three libraries:</p>"
                        '<ul class="pl-3">'
                        "<li>Bulle Library</li>"
                        "<li>Lec'Tour, La Tour-de-Trême school library</li>"
                        "<li>Libellule, La Léchère school library</li>"
                        "</ul>"
                        "<p>Although the interface allows requests for items on shelf,"
                        " only requests for borrowed items will be considered.</p>"
                    ),
                },
                {
                    "language": "fr",
                    "value": (
                        "<h5>Catalogue de la Bibliothèque de Bulle</h5>"
                        "<p>Cette interface donne accès à la liste des documents"
                        " des trois bibliothèques suivantes:</p>"
                        '<ul class="pl-3">'
                        "<li>La Bibliothèque de Bulle</li>"
                        "<li>Lec'Tour, la bibliothèque scolaire de la Tour-de-Trême</li>"
                        "<li>Libellule, la bibliothèque scolaire de La Léchère</li>"
                        "</ul>"
                        "<p>Bien que l'interface autorise les demandes sur des documents"
                        " disponibles en rayon, seules les demandes sur des documents"
                        " empruntés seront prises en compte.</p>"
                    ),
                },
                {
                    "language": "de",
                    "value": (
                        "<h5>Online-Katalog der Bibliothek Bulle</h5>"
                        "<p>Diese Webseite ermöglicht den Zugriff auf die Liste der"
                        " Dokumente der folgenden drei Bibliotheken:</p>"
                        '<ul class="pl-3">'
                        "<li>Die Bibliothek Bulle</li>"
                        "<li>Lec'Tour, die Schulbibliothek La Tour-de-Trême</li>"
                        "<li>Libellule, die Schulbibliothek La Léchère</li>"
                        "</ul>"
                        "<p>Obwohl die Schnittstelle Bestellungen für im Regal"
                        " vorhandene Dokumente erlaubt, werden nur Bestellungen"
                        " für ausgeliehene Dokumente berücksichtigt.</p>"
                    ),
                },
                {
                    "language": "it",
                    "value": (
                        "<h5>Catalogo della Biblioteca di Bulle</h5>"
                        "<p>Questa interfaccia dà accesso all'elenco dei documenti"
                        " delle tre seguenti biblioteche:</p>"
                        '<ul class="pl-3">'
                        "<li>La Biblioteca di Bulle</li>"
                        "<li>Lec'Tour, la biblioteca scolastica di La Tour-de-Trême</li>"
                        "<li>Libellule, la biblioteca scolastica di La Léchère</li>"
                        "</ul>"
                        "<p>Anche se l'interfaccia permette richieste di documenti"
                        " disponibili sullo scaffale, solo le richieste di documenti"
                        " presi in prestito saranno considerate.</p>"
                    ),
                },
            ],
            "center": [
                {
                    "language": "en",
                    "value": (
                        '<p class="mb-3"><strong>When you log in for the'
                        " first time, please reset your password using your e-mail"
                        ' (function "Forgot password"). Registrants without an'
                        " e-mail address can contact the lending desk.</strong></p>"
                        "<h5>Services and opening hours</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/">Bulle Library</a></li>'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/lectour-la-tour-de-treme/">Lec\'Tour</a></li>'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/libellule-dardens/">Libellule</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "fr",
                    "value": (
                        '<p class="mb-3"><strong>Lors de votre première'
                        " connexion, merci de redéfinir votre mot de passe au moyen"
                        ' de votre e-mail (fonction "Mot de passe oublié"). Les'
                        " personnes inscrites sans e-mail peuvent s'adresser au"
                        " bureau de prêt.</strong></p>"
                        "<h5>Services et horaires</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/">Bibliothèque de Bulle</a></li>'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/lectour-la-tour-de-treme/">Lec\'Tour</a></li>'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/libellule-dardens/">Libellule</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "de",
                    "value": (
                        '<p class="mb-3"><strong>Wenn Sie sich zum ersten'
                        " Mal einloggen, setzen Sie bitte Ihr Passwort mit Hilfe Ihrer"
                        ' E-Mail zurück (Funktion "Passwort vergessen"). Registrierte'
                        " Benutzer ohne E-Mail-Adresse können sich an die Ausleihtheke"
                        " wenden.</strong></p>"
                        "<h5>Dienstleistungen und Öffnungszeiten</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/">Bibliothek Bulle</a></li>'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/lectour-la-tour-de-treme/">Lec\'Tour</a></li>'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/libellule-dardens/">Libellule</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "it",
                    "value": (
                        '<p class="mb-3"><strong>Quando accedi per la prima'
                        " volta, reimposta la tua password usando la tua e-mail"
                        ' (funzione "Password dimenticata"). Gli utenti registrati'
                        " senza un indirizzo e-mail possono contattare il banco"
                        " prestiti.</strong></p>"
                        "<h5>Servizi e orari</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/">Biblioteca di Bulle</a></li>'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/lectour-la-tour-de-treme/">Lec\'Tour</a></li>'
                        '<li><a href="https://musee-gruerien.ch/bibliotheque-de-bulle/libellule-dardens/">Libellule</a></li>'
                        "</ul>"
                    ),
                },
            ],
            "right": [
                {
                    "language": "en",
                    "value": (
                        "<h5>BCU Fribourg Digital services</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.fr.ch/bcu/fri-emedia">Fri-eMedia</a></li>'
                        "</ul>"
                        "<h5>SLSP Network Catalogue</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=en">Swisscovery</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "fr",
                    "value": (
                        "<h5>Offre numérique de la BCU Fribourg</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.fr.ch/bcu/fri-emedia">Fri-eMedia</a></li>'
                        "</ul>"
                        "<h5>Catalogue du réseau SLSP</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=fr">Swisscovery</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "de",
                    "value": (
                        "<h5>Digitales Angebot der BKU Freiburg</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.fr.ch/bcu/fri-emedia">Fri-eMedia</a></li>'
                        "</ul>"
                        "<h5>SLSP-Netzwerk-Online-Katalog</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=de">Swisscovery</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "it",
                    "value": (
                        "<h5>Offerta digitale della BCU di Friburgo</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.fr.ch/bcu/fri-emedia">Fri-eMedia</a></li>'
                        "</ul>"
                        "<h5>Catalogo della rete SLSP</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=it">Swisscovery</a></li>'
                        "</ul>"
                    ),
                },
            ],
        },
    },
    "rbnj": {
        "placeholder": PLACEHOLDER,
        "slogan": [
            {"language": "fr", "value": "Nouveau catalogue des bibliothèques de lecture publique et patrimoniales"},
            {"language": "en", "value": "New catalogue of public and heritage libraries"},
            {"language": "de", "value": "Neuer Katalog der öffentlichen und patrimonialen Bibliotheken"},
            {"language": "it", "value": "Nuovo catalogo delle biblioteche di lettura pubblica e patrimoniali"},
        ],
        "blocks": {
            "left": [
                {
                    "language": "fr",
                    "value": (
                        "<h5>Le nouveau catalogue</h5>"
                        "<p>Le nouveau catalogue RERO ILS donne accès aux documents"
                        " et ressources disponibles dans les <b>bibliothèques de"
                        " lecture publique</b> et les <b>bibliothèques patrimoniales</b>.</p>"
                    ),
                },
                {
                    "language": "en",
                    "value": (
                        "<h5>The new catalogue</h5>"
                        "<p>The new RERO ILS catalogue gives access to the documents"
                        " and resources available in <b>public libraries</b>"
                        " and <b>heritage libraries</b>.</p>"
                    ),
                },
                {
                    "language": "de",
                    "value": (
                        "<h5>Der neue Katalog</h5>"
                        "<p>Der neue Katalog RERO ILS bietet Zugang zu den Dokumenten"
                        " und Ressourcen, die in den <b>öffentlichen Bibliotheken</b>"
                        " und den <b>patrimonialen Bibliotheken</b> verfügbar sind.</p>"
                    ),
                },
                {
                    "language": "it",
                    "value": (
                        "<h5>Il nuovo catalogo</h5>"
                        "<p>Il nuovo catalogo RERO ILS dà accesso ai documenti"
                        " e alle risorse disponibili nelle <b>biblioteche di"
                        " lettura pubblica</b> e nelle <b>biblioteche patrimoniali</b>.</p>"
                    ),
                },
            ],
            "center": [
                {
                    "language": "fr",
                    "value": (
                        "<h5>Que devient le RBNJ ?</h5>"
                        "Le réseau des bibliothèques neuchâteloises et jurassiennes continue d'exister."
                        " Il est désormais structuré par catégories de bibliothèques:"
                        '<ul class="list-unstyled mt-2">'
                        '<li><b>Bibliothèques publiques et patrimoniales (<a href="/">RERO+</a>):</b>'
                        '<ul class="pl-3">'
                        "<li>Bibliothèque publique de Neuchâtel</li>"
                        "<li>Bibliothèques de la Ville de La Chaux-de-Fonds</li>"
                        "<li>Bibliothèque de la Ville du Locle</li>"
                        "<li>Bibliothèque cantonale jurassienne</li>"
                        '<li><a href="https://www.rero.ch/about/clients/clients-rero-ils#rbnj">Autres bibliothèques</a></li>'
                        "</ul></li>"
                        '<li class="mt-2"><b>Bibliothèques des universités et hautes écoles'
                        ' (<a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=fr">Swisscovery</a>):</b>'
                        '<ul class="pl-3">'
                        "<li>Université de Neuchâtel</li>"
                        "<li>HE-Arc</li>"
                        "<li>HEP − BEJUNE</li>"
                        "<li>IRDP − Conférence intercantonale de l'instruction publique</li>"
                        "<li>CIES − Centre international d'étude du sport</li>"
                        "</ul></li>"
                        "</ul>"
                    ),
                },
                {
                    "language": "en",
                    "value": (
                        "<h5>What is happening to the RBNJ?</h5>"
                        "The network of Neuchâtel and Jura libraries continues to exist."
                        " It is now organised by library category:"
                        '<ul class="list-unstyled mt-2">'
                        '<li><b>Public and heritage libraries (<a href="/">RERO+</a>):</b>'
                        '<ul class="pl-3">'
                        "<li>Bibliothèque publique de Neuchâtel</li>"
                        "<li>Bibliothèques de la Ville de La Chaux-de-Fonds</li>"
                        "<li>Bibliothèque de la Ville du Locle</li>"
                        "<li>Bibliothèque cantonale jurassienne</li>"
                        '<li><a href="https://www.rero.ch/about/clients/clients-rero-ils#rbnj">Other libraries</a></li>'
                        "</ul></li>"
                        '<li class="mt-2"><b>University and higher education libraries'
                        ' (<a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=en">Swisscovery</a>):</b>'
                        '<ul class="pl-3">'
                        "<li>Université de Neuchâtel</li>"
                        "<li>HE-Arc</li>"
                        "<li>HEP − BEJUNE</li>"
                        "<li>IRDP − Conférence intercantonale de l'instruction publique</li>"
                        "<li>CIES − Centre international d'étude du sport</li>"
                        "</ul></li>"
                        "</ul>"
                    ),
                },
                {
                    "language": "de",
                    "value": (
                        "<h5>Was wird aus dem RBNJ?</h5>"
                        "Das Netzwerk der Neuenburger und Jurassischen Bibliotheken besteht weiterhin."
                        " Es ist neu nach Bibliothekskategorien gegliedert:"
                        '<ul class="list-unstyled mt-2">'
                        '<li><b>Öffentliche und patrimoniale Bibliotheken (<a href="/">RERO+</a>):</b>'
                        '<ul class="pl-3">'
                        "<li>Bibliothèque publique de Neuchâtel</li>"
                        "<li>Bibliothèques de la Ville de La Chaux-de-Fonds</li>"
                        "<li>Bibliothèque de la Ville du Locle</li>"
                        "<li>Bibliothèque cantonale jurassienne</li>"
                        '<li><a href="https://www.rero.ch/about/clients/clients-rero-ils#rbnj">Weitere Bibliotheken</a></li>'
                        "</ul></li>"
                        '<li class="mt-2"><b>Bibliotheken der Universitäten und Hochschulen'
                        ' (<a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=de">Swisscovery</a>):</b>'
                        '<ul class="pl-3">'
                        "<li>Université de Neuchâtel</li>"
                        "<li>HE-Arc</li>"
                        "<li>HEP − BEJUNE</li>"
                        "<li>IRDP − Conférence intercantonale de l'instruction publique</li>"
                        "<li>CIES − Centre international d'étude du sport</li>"
                        "</ul></li>"
                        "</ul>"
                    ),
                },
                {
                    "language": "it",
                    "value": (
                        "<h5>Cosa ne è del RBNJ?</h5>"
                        "La rete delle biblioteche di Neuchâtel e del Giura continua a esistere."
                        " Ora è strutturata per categorie di biblioteche:"
                        '<ul class="list-unstyled mt-2">'
                        '<li><b>Biblioteche pubbliche e patrimoniali (<a href="/">RERO+</a>):</b>'
                        '<ul class="pl-3">'
                        "<li>Bibliothèque publique de Neuchâtel</li>"
                        "<li>Bibliothèques de la Ville de La Chaux-de-Fonds</li>"
                        "<li>Bibliothèque de la Ville du Locle</li>"
                        "<li>Bibliothèque cantonale jurassienne</li>"
                        '<li><a href="https://www.rero.ch/about/clients/clients-rero-ils#rbnj">Altre biblioteche</a></li>'
                        "</ul></li>"
                        '<li class="mt-2"><b>Biblioteche delle università e delle scuole universitarie'
                        ' (<a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=it">Swisscovery</a>):</b>'
                        '<ul class="pl-3">'
                        "<li>Université de Neuchâtel</li>"
                        "<li>HE-Arc</li>"
                        "<li>HEP − BEJUNE</li>"
                        "<li>IRDP − Conférence intercantonale de l'instruction publique</li>"
                        "<li>CIES − Centre international d'étude du sport</li>"
                        "</ul></li>"
                        "</ul>"
                    ),
                },
            ],
            "right": [
                {
                    "language": "fr",
                    "value": (
                        "<h5>Principaux catalogues romands</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="/">Tout RERO</a></li>'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=fr">Swisscovery (universités et hautes écoles)</a></li>'
                        '<li><a href="https://renouvaud1.primo.exlibrisgroup.com/discovery/search?vid=41BCULAUSA_LIB:VU2&lang=fr">RenouVaud (Vaud)</a></li>'
                        '<li><a href="https://www.helveticat.ch/discovery/search?vid=41SNL_51_INST:helveticat&lang=fr">Helveticat (Bibliothèque nationale suisse)</a></li>'
                        '<li><a href="https://catalogue-bm.geneve.ch/">Bibliothèque de la Ville de Genève</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "en",
                    "value": (
                        "<h5>Main catalogues of French-speaking Switzerland</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="/">All RERO</a></li>'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=en">Swisscovery (universities and higher education)</a></li>'
                        '<li><a href="https://renouvaud1.primo.exlibrisgroup.com/discovery/search?vid=41BCULAUSA_LIB:VU2&lang=en">RenouVaud (Vaud)</a></li>'
                        '<li><a href="https://www.helveticat.ch/discovery/search?vid=41SNL_51_INST:helveticat&lang=en">Helveticat (Swiss National Library)</a></li>'
                        '<li><a href="https://catalogue-bm.geneve.ch/">Bibliothèque de la Ville de Genève</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "de",
                    "value": (
                        "<h5>Wichtigste Westschweizer Kataloge</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="/">RERO gesamt</a></li>'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=de">Swisscovery (Universitäten und Hochschulen)</a></li>'
                        '<li><a href="https://renouvaud1.primo.exlibrisgroup.com/discovery/search?vid=41BCULAUSA_LIB:VU2&lang=de">RenouVaud (Waadt)</a></li>'
                        '<li><a href="https://www.helveticat.ch/discovery/search?vid=41SNL_51_INST:helveticat&lang=de">Helveticat (Schweizerische Nationalbibliothek)</a></li>'
                        '<li><a href="https://catalogue-bm.geneve.ch/">Bibliothèque de la Ville de Genève</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "it",
                    "value": (
                        "<h5>Principali cataloghi della Svizzera romanda</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="/">Tutto RERO</a></li>'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=it">Swisscovery (università e scuole universitarie)</a></li>'
                        '<li><a href="https://renouvaud1.primo.exlibrisgroup.com/discovery/search?vid=41BCULAUSA_LIB:VU2&lang=it">RenouVaud (Vaud)</a></li>'
                        '<li><a href="https://www.helveticat.ch/discovery/search?vid=41SNL_51_INST:helveticat&lang=it">Helveticat (Biblioteca nazionale svizzera)</a></li>'
                        '<li><a href="https://catalogue-bm.geneve.ch/">Bibliothèque de la Ville de Genève</a></li>'
                        "</ul>"
                    ),
                },
            ],
        },
    },
    "vs": {
        "placeholder": PLACEHOLDER,
        "blocks": {
            "left": [
                {
                    "language": "fr",
                    "value": (
                        "<h5>Services</h5>"
                        '<ul class="list-unstyled">'
                        "<li>Prêt entre bibliothèques (pour les documents hors Valais)"
                        "<ul>"
                        '<li><a href="/vs/ill_requests/new">Formulaire</a></li>'
                        '<li><a href="https://www.mediatheque.ch/data/documents/infos_pratiques/PEBpourREROILSVS_franais.pdf">Tarifs</a></li>'
                        "</ul></li>"
                        '<li><a href="https://www.rero.ch/about/clients/clients-rero-ils#vs">Partenaires RERO+ VALAIS</a></li>'
                        "</ul>"
                        "<h5>e-Offres</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.mediatheque.ch/fr/ressources-ligne-70.html">Ressources en ligne</a></li>'
                        '<li><a href="https://www.mediatheque.ch/fr/journaux-articles-88.html">Presse en ligne</a></li>'
                        '<li><a href="https://www.mediatheque.ch/fr/ressources-ligne-autres-documents-252.html">Ressources numériques pédagogiques</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "en",
                    "value": (
                        "<h5>Services</h5>"
                        '<ul class="list-unstyled">'
                        "<li>Interlibrary loan (for documents outside Valais)"
                        "<ul>"
                        '<li><a href="/vs/ill_requests/new">Form</a></li>'
                        '<li><a href="https://www.mediatheque.ch/data/documents/infos_pratiques/PEBpourREROILSVS_franais.pdf">Rates</a></li>'
                        "</ul></li>"
                        '<li><a href="https://www.rero.ch/about/clients/clients-rero-ils#vs">RERO+ VALAIS partners</a></li>'
                        "</ul>"
                        "<h5>e-Offers</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.mediatheque.ch/fr/ressources-ligne-70.html">Online resources</a></li>'
                        '<li><a href="https://www.mediatheque.ch/fr/journaux-articles-88.html">Online press</a></li>'
                        '<li><a href="https://www.mediatheque.ch/fr/ressources-ligne-autres-documents-252.html">Educational digital resources</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "de",
                    "value": (
                        "<h5>Dienstleistungen</h5>"
                        '<ul class="list-unstyled">'
                        "<li>Fernleihbestellung (für Dokumente ausserhalb des Wallis)"
                        "<ul>"
                        '<li><a href="/vs/ill_requests/new">Formular</a></li>'
                        '<li><a href="https://www.mediatheque.ch/data/documents/infos_pratiques/PEBpourREROILSVS_allemand.pdf">Gebühren</a></li>'
                        "</ul></li>"
                        '<li><a href="https://www.rero.ch/about/clients/clients-rero-ils#vs">Partner RERO+ Wallis</a></li>'
                        "</ul>"
                        "<h5>e-Medienangebote</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.mediatheque.ch/de/online-medienangebot-70.html">e-Ressourcen</a></li>'
                        '<li><a href="https://www.mediatheque.ch/de/digitalisierte-zeitungen-88.html">Digitalisierte Zeitungen</a></li>'
                        '<li><a href="https://www.mediatheque.ch/de/online-ressourcen-andere-dokumente-252.html">Pädagogische e-Ressourcen</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "it",
                    "value": (
                        "<h5>Servizi</h5>"
                        '<ul class="list-unstyled">'
                        "<li>Prestito interbibliotecario (per i documenti fuori dal Vallese)"
                        "<ul>"
                        '<li><a href="/vs/ill_requests/new">Modulo</a></li>'
                        '<li><a href="https://www.mediatheque.ch/data/documents/infos_pratiques/PEBpourREROILSVS_franais.pdf">Tariffe</a></li>'
                        "</ul></li>"
                        '<li><a href="https://www.rero.ch/about/clients/clients-rero-ils#vs">Partner RERO+ VALLESE</a></li>'
                        "</ul>"
                        "<h5>e-Offerte</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.mediatheque.ch/fr/ressources-ligne-70.html">Risorse online</a></li>'
                        '<li><a href="https://www.mediatheque.ch/fr/journaux-articles-88.html">Stampa online</a></li>'
                        '<li><a href="https://www.mediatheque.ch/fr/ressources-ligne-autres-documents-252.html">Risorse digitali didattiche</a></li>'
                        "</ul>"
                    ),
                },
            ],
            "center": [
                {
                    "language": "fr",
                    "value": '<h4 class="text-center">Bienvenue dans le catalogue RERO+ Valais</h4><h4 class="text-center">Mille et une connaissances à portée de clic</h4><p class="mt-3 text-center"><a href="/help/public/">Aide</a></p>',
                },
                {
                    "language": "en",
                    "value": '<h4 class="text-center">Welcome to the RERO+ Valais catalogue</h4><h4 class="text-center">A thousand and one pieces of knowledge a click away</h4><p class="mt-3 text-center"><a href="/help/public/">Help</a></p>',
                },
                {
                    "language": "de",
                    "value": '<h4 class="text-center">Willkommen im Katalog RERO+ Wallis</h4><h4 class="text-center">Unendliches Wissen nur einen Klick entfernt</h4><p class="mt-3 text-center"><a href="/help/public/">Hilfe</a></p>',
                },
                {
                    "language": "it",
                    "value": '<h4 class="text-center">Benvenuti nel catalogo RERO+ Vallese</h4><h4 class="text-center">Mille e una conoscenza a portata di clic</h4><p class="mt-3 text-center"><a href="/help/public/">Aiuto</a></p>',
                },
            ],
            "right": [
                {
                    "language": "fr",
                    "value": (
                        "<h5>Catalogues collectifs</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="/">Tout Rero+</a></li>'
                        '<li><a href="https://www.bibliovalais-cbv.biz/NetBiblio/">Bibliothèques valaisannes</a></li>'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=fr">Swisscovery</a></li>'
                        "</ul>"
                        "<h5>Autres catalogues</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.vallesiana.ch/#!search">Vallesiana</a></li>'
                        '<li><a href="https://renouvaud1.primo.exlibrisgroup.com/discovery/search?vid=41BCULAUSA_LIB:VU2&lang=fr">Renouvaud</a></li>'
                        '<li><a href="https://helveticall.ch/discovery/search?vid=41SNL_51_INST:helveticall&lang=fr">HelveticAll</a></li>'
                        '<li><a href="https://www.e-rara.ch/?lang=fr">E-rara.ch</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "en",
                    "value": (
                        "<h5>Union catalogues</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="/">All Rero+</a></li>'
                        '<li><a href="https://www.bibliovalais-cbv.biz/NetBiblio/">Valais libraries</a></li>'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=en">Swisscovery</a></li>'
                        "</ul>"
                        "<h5>Other catalogues</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.vallesiana.ch/#!search">Vallesiana</a></li>'
                        '<li><a href="https://renouvaud1.primo.exlibrisgroup.com/discovery/search?vid=41BCULAUSA_LIB:VU2&lang=en">Renouvaud</a></li>'
                        '<li><a href="https://helveticall.ch/discovery/search?vid=41SNL_51_INST:helveticall&lang=en">HelveticAll</a></li>'
                        '<li><a href="https://www.e-rara.ch/?lang=en">E-rara.ch</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "de",
                    "value": (
                        "<h5>Gesamtkataloge</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="/">RERO+ global</a></li>'
                        '<li><a href="https://www.bibliovalais-cbv.biz/NetBiblio/">Walliser Bibliotheken</a></li>'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=de">Swisscovery</a></li>'
                        "</ul>"
                        "<h5>Weitere Kataloge</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.vallesiana.ch/kulturerbe/digitale-kulturgut-wallis-3.html#!search">Vallesiana</a></li>'
                        '<li><a href="https://renouvaud1.primo.exlibrisgroup.com/discovery/search?vid=41BCULAUSA_LIB:VU2&lang=de">Renouvaud</a></li>'
                        '<li><a href="https://helveticall.ch/discovery/search?vid=41SNL_51_INST:helveticall&lang=de">HelveticAll</a></li>'
                        '<li><a href="https://www.e-rara.ch/doc/home?lang=de">E-rara.ch</a></li>'
                        "</ul>"
                    ),
                },
                {
                    "language": "it",
                    "value": (
                        "<h5>Cataloghi collettivi</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="/">Tutto Rero+</a></li>'
                        '<li><a href="https://www.bibliovalais-cbv.biz/NetBiblio/">Biblioteche vallesane</a></li>'
                        '<li><a href="https://swisscovery.ch/discovery/search?vid=41SLSP_NETWORK:VU1_UNION&lang=it">Swisscovery</a></li>'
                        "</ul>"
                        "<h5>Altri cataloghi</h5>"
                        '<ul class="list-unstyled">'
                        '<li><a href="https://www.vallesiana.ch/#!search">Vallesiana</a></li>'
                        '<li><a href="https://renouvaud1.primo.exlibrisgroup.com/discovery/search?vid=41BCULAUSA_LIB:VU2&lang=it">Renouvaud</a></li>'
                        '<li><a href="https://helveticall.ch/discovery/search?vid=41SNL_51_INST:helveticall&lang=it">HelveticAll</a></li>'
                        '<li><a href="https://www.e-rara.ch/?lang=it">E-rara.ch</a></li>'
                        "</ul>"
                    ),
                },
            ],
        },
    },
}

for uuid in Organisation.get_all_ids():
    org = Organisation.get_record(uuid)
    code = org.get("code")
    if code not in HOMEPAGE_BY_CODE:
        continue
    if org.get("homepage"):
        print(f"org#{org.pid} [{code}] already has homepage, skipping")
        continue
    print(f"Setting homepage for org#{org.pid} [{code}]")
    org["homepage"] = HOMEPAGE_BY_CODE[code]
    org.update(org, dbcommit=True, reindex=True)

print("Done.")

enable new task

uv run invenio reroils scheduler enable-tasks -n celery.delete-inactive-patrons -v

Zabbix

Update Zabbix monitoring Templates ES --> Search

v1.27.0 --> v1.28.0

Config

docker-service.yml

cache: image: redis:8

mq: image: rabbitmq:4.2-management

Alembic

uv run invenio rero es index update-mapping

Reindex

from rero_invenio_base.modules.tasks import run_on_worker
from rero_invenio_base.modules.utils import chunk
from rero_ils.modules.items.api import Item, ItemsSearch

code = '''
def reindex(_ids):
    from rero_ils.modules.items.api import Item
    n = 0
    errors = []
    for _id in _ids:
        try:
            item = Item.get_record(_id)
            item.reindex()
            n += 1
        except Exception as e:
            print('error', e)
            errors.append(_id)
    return (n, errors)
'''

parallel = 6
# all items
count = 0
query = ItemsSearch().filter("exists", field="issue")
print(f"Issues: {query.count()} no sort_key: {query.exclude("exists", field="issue.sort_key").count()}")
ids = [hit.meta.id for hit in query.source().scan()]
for c in chunk([str(val) for val in ids], len(ids) // parallel):
    count += 1
    res = run_on_worker.delay(code, "reindex", c)
    print("items", count, len(c), res)

print(f"Issues: {query.count()} no sort_key: {query.exclude("exists", field="issue.sort_key").count()}")

Thumbnails

Correct local files thumbnails.

from rero_ils.modules.documents.api import Document, DocumentsSearch

print(app.config['RERO_INVENIO_THUMBNAILS_URL'])


ISBNS = [
    "3573310006279",
    "9782081382671",
    "9782883413177",
    "9782940532131",
    "9782970037385",
    "9791092253016",
    "9791092253047",
    "9791092253108",
    "9791092253122",
    "9791092253177",
    "9791095066439",
    "9791096537303",
    "9781545292655",
    "9782809651522",
    "9782931008515",
    "9782970037347",
    "9788412488661",
    "9791092253023",
    "9791092253085",
    "9791092253115",
    "9791092253160",
    "9791092253184",
    "9791095316206",
    "9791097542122"
]

query = DocumentsSearch().filter("term", identifiedBy__type="bf:Isbn").filter("terms", identifiedBy__value=ISBNS)

print(f"---> ES: {query.count()}, ISBNS: {len(ISBNS)}")
for hit in query.source("pid").scan():
    if doc := Document.get_record_by_pid(hit["pid"]):
        print(doc.pid, doc["electronicLocator"])
        new_electronic_locators = []
        for electronic_locator in doc["electronicLocator"]:
            if electronic_locator.get("content") == "coverImage" and electronic_locator.get("type") == "relatedResource":
                isbn = electronic_locator.get("url", "").split("/")[-1].split(".")[0]
                if isbn not in ISBNS:
                    new_electronic_locators.append(electronic_locator)
        if new_electronic_locators:
            doc["electronicLocator"] = new_electronic_locators
        else:
            doc.pop("electronicLocator", None)
        doc.update(data=doc, dbcommit=True, reindex=True)
        doc = Document.get_record_by_pid(hit["pid"])
        print(">>", doc.pid, doc["electronicLocator"])

v1.26.0 --> v1.27.0

CSS

INVENIO_RERO_ILS_THEME_ORGANISATION_CSS_ENDPOINT=/static/themes/css/

Alembic

uv run invenio alembic upgrade
uv run invenio rero es index update-mapping

Remote Entity

index_name=`uv run invenio rero es index info -i ent | grep remote`
echo $index_name
uv run invenio rero es index move remote_entities $index_name remote_entities-remote_entity-v0.0.1-20260128 -v
uv run invenio index delete $index_name --yes-i-know

Notifications

index_name=`uv run invenio rero es index info -i notif`
echo $index_name
uv run invenio rero es index move notifications $index_name notifications-notification-v0.0.1-20260128 -v
uv run invenio index delete $index_name --yes-i-know

Patrons

index_name=`uv run invenio rero es index info -i patrons`
echo $index_name
uv run invenio rero es index move patrons $index_name patrons-patron-v0.0.1-20260128 -v
uv run invenio index delete $index_name --yes-i-know

Operations Logs

from invenio_search import current_search
# update operation logs templates
[p for p in current_search.put_templates()]

from invenio_search import current_search_client
def reindex(source, destination):
    res = current_search_client.reindex(
        body=dict(
            source=dict(
                index=source
            ),
            dest=dict(
                index=destination,
                version_type='external_gte'
            )
        ),
        wait_for_completion=False
    )
    return res['task']

def index_in_new(indices):
    tasks = []
    body = {
        "settings": {
        "number_of_shards": "8",
        "number_of_replicas": "1",
        "max_result_window": "100000"
    }}
    for index_name in indices:
        print(index_name)
        new_index_name = f'{index_name}-new'
        current_search_client.indices.create(new_index_name, body=body)
        tasks.append(reindex(index_name, new_index_name))
    return tasks

def remove_old(indices):
    for index_name in indices:
        current_search_client.indices.delete(index_name)
        print(f'{index_name} has been deleted')

def rename_to_old(indices):
    tasks = []
    body = {
        "settings": {
        "number_of_shards": "8",
        "number_of_replicas": "1",
        "max_result_window": "100000"
    }}
    for index_name in indices:
        print(index_name)
        new_index_name = f'{index_name}-new'
        current_search_client.indices.create(index_name, body=body)
        tasks.append(reindex(new_index_name, index_name))
    return tasks

def check_status_tasks(tasks):
    """Check status of current tasks."""
    for t in tasks:
        task = current_search_client.tasks.get(t)
        if not task.get('completed'):
            total = task['task']['status']['total']
            created = task['task']['status']['created']
            print(task['task']['description'], "%.2f" % (created/total * 100), "%")

# Execute one line after the other
# Get the list of the operation logs indices
indices = list(current_search_client.indices.get_alias('operation_logs').keys())
tasks = index_in_new(indices)
# Check for completion
check_status_tasks(tasks)
remove_old(indices)
tasks = rename_to_old(indices)
check_status_tasks(tasks)
remove_old([i+'-new' for i in indices])

Holdings: remove patterns for irregular frequencies https://github.com/rero/rero-ils/pull/3983 

from rero_ils.modules.holdings.api import Holding, HoldingsSearch

query = HoldingsSearch().filter("term", patterns__frequency="rdafr:1016").source(False)
uuids = [hit.meta.id for hit in query.scan()]
for idx, uuid in enumerate(uuids, 1):
    record = Holding.get_record(uuid)
    record.pop("patterns", None)
    record.update(record, dbcommit=True, commit=True, reindex=True)

v1.25.0 --> v1.26.0

Alembic

uv run invenio alembic stamp 428b919be0ea
uv run invenio alembic upgrade
uv run invenio rero es index update-mapping

correct docker scripts

change poetry -> uv change rero_ils.schedulers.RedisScheduler -> rero_ils.schedulers.DatabaseScheduler

correct invenio constants

INVENIO_APP_ALLOWED_HOSTS: None -> INVENIO_TRUSTED_HOSTS: '["_", "localhost", "10.240.0.100", "bib.rero.ch"]'
CELERY_BEAT_DBURI

Update nginx server config

To allow PDF preview: https://github.com/rero/rero-ils/pull/3933/files

Rename environment variables

According to this PR: https://github.com/rero/rero-ils/pull/3940 RERO_IMPORT_CACHE >> RERO_ILS_IMPORT_CACHE

Add reminder_counter to existing notif operation_logs

Update the mapping of all operation log indices.

from invenio_search import current_search_client
res = current_search_client.indices.get_template('operation_logs')
mapping = res['operation_logs']['mappings']

for op_index in current_search_client.indices.get_alias('operation_logs').keys():
    print(f'Updating mapping for index: {op_index}')
    current_search_client.indices.put_mapping(mapping, op_index)

Populate the reminder counter (7h)

from elasticsearch_dsl import Q
from invenio_search.api import RecordsSearch

from rero_ils.modules.notifications.api import Notification
from rero_ils.modules.notifications.logs.api import NotificationOperationLog
from rero_ils.modules.operation_logs.api import OperationLogsSearch

query = (
    OperationLogsSearch()
    .filter("term", notification__type="overdue")
    .filter("bool", must_not=[Q("exists", field="notification.reminder_counter")])
)
pids = [hit.pid for hit in query.source("pid").scan()]
total = len(pids)
print(f"Upgrade notif operation logs :: {total}")
errors = 0
for idx, pid in enumerate(pids, 1):
    record = NotificationOperationLog.get_record(pid)
    notification_pid = record["record"]["value"]
    if notification := Notification.get_record_by_pid(notification_pid):
        print(f"{idx}/{total}, {pid}", end="\r")
        try:
            record["notification"]["reminder_counter"] = notification["context"].get("reminder_counter")
            NotificationOperationLog.update(pid, record["date"], record)
        except Exception as err:
            print(f"{idx:<10} {pid} {err}")
            errors += 1
    else:
        print(f"{idx:<10} {pid} notif not found {notification_pid}")
        errors += 1
OperationLogsSearch.flush_and_refresh()
print(f"Changed: {idx} Errors: {errors}")

Reindex documents and migrate the holdings field

uv run invenio reroils index update_mapping
uv run invenio index create -b rero_ils/modules/documents/mappings/v7/documents/document-v0.0.1.json documents-document-v0.0.1-20251007

execute on the es browser extension (40')

POST /_reindex
{
  "source": {
    "index": "documents"
  },
  "dest": {
    "index": "documents-document-v0.0.1-20251007"
  },
  "script": {
    "source": "ctx._source.nested_holdings = ctx._source.holdings"
  }
}

once it is finished (some minutes)

index_name=`uv run invenio rero es index info -i documents`
echo $index_name
uv run invenio rero es index switch $index_name documents-document-v0.0.1-20251007
uv run invenio index delete $index_name --yes-i-know

v1.24.1 --> v1.25.0

copy static files

docker-compose exec -u root web-ui bash -c "rm -fr /invenio/instance/static/*"
docker-compose exec -u root web-ui bash -c "cp -r /invenio/var/instance/static/* /invenio/instance/static/."

Alembic

poetry run invenio alembic upgrade
poetry run invenio rero es index update-mapping

anonymize recipients' field in operation_logs

# from rero_ils.modules.operation_logs.api import OperationLog, OperationLogsSearch

# query = OperationLogsSearch()\
#     .filter("term", record__type="notif")\
#     .filter("term", loan__patron__name="anonymized")

# print(f"Found {query.count()} logs to anonymize")

# ids = [hit.meta.id for hit in query.scan()]
# for id in ids:
#     if op_log := OperationLog.get_record(id):
#         if recipients := op_log["notification"].get("recipients"):
#             op_log["notification"]["recipients"] = ["anonymized"]
#             OperationLog.update(id, op_log["date"],op_log)

from invenio_search import current_search_client

query="loan.patron.name:anonymized AND record.type:notif NOT notification.recipients:anonymized"
body = {
    "script": {
        "source": "ctx._source.notification.recipients = ['anonymized']",
        "lang": "painless",
    },
}
print(
    current_search_client.count(q=query, index="operation_logs")
)
# res = current_search_client.update_by_query(body=body, index="operation_logs", q=query, conflicts='proceed', wait_for_completion=False)
res = current_search_client.update_by_query(body=body, index="operation_logs", q=query, wait_for_completion=False)

current_search_client.tasks.get(res['task'])

Reindex ALL documents

index_name=`poetry run invenio rero es index info -i documents`
echo $index_name
poetry run invenio rero es index move documents $index_name documents-document-v0.0.1-20250520 -v
poetry run invenio index delete $index_name --yes-i-know

Needed because of call_number.raw: https://github.com/rero/rero-ils/commit/25b114d92f872ca4b89e997b72b1316052029464

Acquisitions

import mock
from invenio_db import db

from rero_ils.modules.acquisition.acq_order_lines.api import AcqOrderLine
from rero_ils.modules.acquisition.acq_orders.api import AcqOrder
from rero_ils.modules.acquisition.acq_orders.models import AcqOrderStatus
from rero_ils.modules.acquisition.acq_receipts.api import AcqReceipt, AcqReceiptsSearch

def migrate(cls, fields=[]):
    total = cls.count()
    ids = cls.get_all_ids()
    n = 1
    for idx, _id in enumerate(ids, 1):
        rec = cls.get_record(_id)
        changed = False
        for field in fields:
            if rec.get(field):
                del rec[field]
                changed = True
        if changed:
            rec.commit()
            n += 1
        if n % 100 == 0:
           db.session.commit()
        print(f'{idx}/{total}', end='\r')
    db.session.commit()
    print(f'{n} records changed')

def migrate_order(cls, fields=[]):
    total = cls.count()
    ids = cls.get_all_ids()
    n = 1
    for idx, _id in enumerate(ids, 1):
        rec = cls.get_record(_id)
        for field in fields:
            if rec.get(field):
                del rec[field]
        try:
            line = next(rec.get_order_lines())
            if line.get('order_date'):
                rec['order_date'] = line['order_date']
        except StopIteration:
            pass
        rec.commit()
        n += 1
        if n % 100 == 0:
           db.session.commit()
        print(f'{idx}/{total}', end='\r')
    db.session.commit()
    print(f'{n} records changed')

migrate_order(AcqOrder, ['type'])

with mock.patch(
    "rero_ils.modules.acquisition.acq_orders.api.AcqOrder.get_status_by_pid",
    mock.MagicMock(return_value=AcqOrderStatus.ORDERED),
):
    migrate(AcqOrderLine, ['exchange_rate', 'order_date', 'receipt_date'])

with mock.patch(
    "rero_ils.modules.acquisition.acq_orders.api.AcqOrder.get_status_by_pid",
    mock.MagicMock(return_value=AcqOrderStatus.ORDERED),
):
    migrate(AcqReceipt, ['exchange_rate'])

# remove unused es index
from invenio_search import current_search, current_search_client
current_search_client.indices.delete(list(current_search_client.indices.get_alias('acq_invoices').keys())[0])

# reindex order lines in worker
from rero_invenio_base.modules.tasks import run_on_worker
from rero_invenio_base.modules.utils import chunk
from rero_ils.modules.acquisition.acq_order_lines.api import AcqOrderLinesSearch

code = '''
def reindex(_ids):
    from rero_ils.modules.acquisition.acq_order_lines.api import AcqOrderLine
    n = 0
    errors = []
    for _id in _ids:
        try:
            doc = AcqOrderLine.get_record(_id)
            doc.reindex()
            n += 1
        except Exception as e:
            print('error', e)
            errors.append(_id)
    return (n, errors)
'''
parallel = 6
count = 0
query = AcqOrderLinesSearch()
ids = [hit.meta.id for hit in query.source().scan()]
for c in chunk([str(val) for val in ids], len(ids) // parallel):
    count += 1
    res = run_on_worker.delay(code, 'reindex', c)
    print('order lines', count, len(c), res)

# fix data
def remove_recept_lines(pid):
    rec = AcqReceipt.get_record_by_pid(pid)
    for l in rec.get_receipt_lines():
        if not l.order_line:
            with mock.patch("rero_ils.modules.acquisition.acq_receipt_lines.api.AcqReceiptLine.can_delete", [True, '']):
                l.delete(dbcommit=True)
                current_search_client.delete_by_query(q=f'pid:{l.pid}', index="acq_receipt_lines", body={})
pids = [
   '11',
   '1218',
    '1457',
    '1644',
    '45',
    '616',
    '676',
    '770',
    '780',
    '940'
]
for pid in pids:
    print(f'process: {pid}', end='\r')
    remove_recept_lines(pid)

code = '''
def reindex(_ids):
    from rero_ils.modules.acquisition.acq_receipts.api import AcqReceipt
    n = 0
    errors = []
    for _id in _ids:
        try:
            doc = AcqReceipt.get_record(_id)
            doc.reindex()
            n += 1
        except Exception as e:
            print('error', e)
            errors.append(_id)
    return (n, errors)
'''
parallel = 6
count = 0
query = AcqReceiptsSearch()
ids = [hit.meta.id for hit in query.source().scan()]
for c in chunk([str(val) for val in ids], len(ids) // parallel):
    count += 1
    res = run_on_worker.delay(code, 'reindex', c)
    print('receipts', count, len(c), res)

Recreate indices

index_name=`poetry run invenio rero es index info -i acq_orders`
echo $index_name
poetry run invenio rero es index move acq_orders $index_name acq_orders-acq_order-v0.0.1-20250520 -v
poetry run invenio index delete $index_name --yes-i-know

index_name=`poetry run invenio rero es index info -i acq_order_lines`
echo $index_name
poetry run invenio rero es index move acq_order_lines $index_name acq_order_lines-acq_order_line-v0.0.1-20250520 -v
poetry run invenio index delete $index_name --yes-i-know

index_name=`poetry run invenio rero es index info -i acq_receipts`
echo $index_name
poetry run invenio rero es index move acq_receipts $index_name acq_receipts-acq_receipt-v0.0.1-20250520 -v
poetry run invenio index delete $index_name --yes-i-know

EBOOKS

Delete ebooks harvest schedule

poetry run invenio reroils scheduler enable-tasks -n ebooks-harvester -d -v
from rero_ils.schedulers import current_scheduler

current_scheduler.remove('ebooks-harvester')

Delete all old ebooks

import click

from invenio_db import db
from invenio_pidstore.models import PersistentIdentifier
from rero_ils.modules.documents.api import Document, DocumentsSearch
from rero_ils.modules.holdings.api import Holding, HoldingsSearch

# delete all ebibliomedia holdings
for etype in "ebibliomedia", "mv-cantook":
    es_query = (
        HoldingsSearch()
        .filter("term", electronic_location__source=etype)
        .filter('term', holdings_type="electronic")
    )
    count = es_query.count()
    pids = [hit.pid for hit in es_query]
    res = es_query.delete()
    click.secho(f"Delete Holdings in ES {count} {etype} {res}")
    HoldingsSearch.flush_and_refresh()


    count = len(pids)
    click.secho(f"Delete Holdings pid's in PidStore {count}")
    for idx, pid in enumerate(pids, 1):
        identifier = PersistentIdentifier.get('hold', pid)
        print(f"{idx:>10}/{count} delete: {identifier}")
        db.session.delete(identifier)
        if idx % 1000 == 0:
            click.secho(f"{idx:>10}/{count} COMMIT", fg='blue')
            db.session.commit()
    db.session.commit()

    db_query = (
        Holding.model_cls.query
        .filter(Holding.model_cls.json['electronic_location'][0]['source'].as_string() == etype)
        .filter(Holding.model_cls.json['holdings_type'].as_string() == "electronic")
    )
    count = db_query.count()
    res = db_query.delete(synchronize_session=False)
    db.session.commit()
    click.echo(f"Delete Holdings in DB {count} {etype} {res}")

# delete all ebibliomedia documents
query = DocumentsSearch().filter("term", identifiedBy__source="cantook")
print(f"Cantook found: {query.count()}")

pids = [hit.pid for hit in query.source('pid').scan()]
count = len(pids)
for idx, pid in enumerate(pids, 1):
    if doc := Document.get_record_by_pid(pid):
        try:
            doc["harvested"] = False
            doc.delete(dbcommit=False, delindex=True)
            print(f"{idx:>10}/{count} doc:{pid} DELETE", end="\r")
        except Exception as err:
            reasons_not_to_delete = doc.reasons_not_to_delete()
            click.secho(f"{idx:>10}/{count} doc:{pid} {reasons_not_to_delete} ERROR: {err}", fg='red')
    else:
        click.secho(f"{idx:>10}/{count} doc:{pid} DOCUMENT NOT FOUND!", fg='yellow')
    if idx % 1000 == 0:
        click.secho(f"{idx:>10}/{count} COMMIT", fg='blue')
        db.session.commit()
db.session.commit()

INIT API config and harvest new ebooks

poetry run invenio reroils api-harvester init-config ./data/apisources.yml

for NAME in "VS-CANTOOK" "NJ-CANTOOK"
do
  poetry run invenio reroils api-harvester set-last-run ${NAME} -d 1900-01-01
  poetry run invenio reroils api-harvester harvest -n ${NAME} -v
done
poetry run invenio reroils scheduler enable-tasks -n harvest-vs-cantook -v
poetry run invenio reroils scheduler enable-tasks -n harvest-nj-cantook -v

delete old OAI config

from invenio_oaiharvester.utils import get_oaiharvest_object
from invenio_db import db

obj = get_oaiharvest_object('ebooks')
db.session.delete(obj)
db.session.commit()

--> v1.24.1-fix-20250210 Fix trimester number of active patrons

import arrow
from dateutil.relativedelta import relativedelta
from datetime import datetime

from rero_ils.modules.stats.api.pricing import StatsForPricing
from rero_ils.modules.stats.api.api import Stat, StatsSearch
from invenio_db import db

# select billing stats for wrong trimesters
search = StatsSearch()\
    .filter('terms', pid=["1263", "1647", "2056", "2480", "3264"])

ids = [hit.meta.id for hit in search.source().scan()]
for id_ in sorted(ids):
    stat = Stat.get_record(id_)
    print(f"Stat: {stat.pid}")
    to_date = arrow.Arrow.fromdatetime(stat.created - relativedelta(days=1))
    compute = StatsForPricing(to_date=to_date)
    new_values = []
    for val in stat.get('values', []):
        lib_pid = val['library']['pid']
        # replace number of active patrons value
        new_number_of_active_patrons = compute.number_of_active_patrons(lib_pid)
        print(
            f"    lib: {lib_pid:<4}"
            f" activ ptrn: {val['number_of_active_patrons']:<6}"
            f" new active: {new_number_of_active_patrons}"
        )
        new_values.append(val)
        new_values[-1]["number_of_active_patrons"] = new_number_of_active_patrons
        db.session.close()
    stat = Stat.get_record(id_)
    stat['values'] = new_values
    stat.update(stat, commit=True, dbcommit=True, reindex=True)
    print(f"Finished: {stat.pid}")
print("Finished")

v1.23.0 --> v1.24.0

Migration index creation

  • add a PYTHONPATH='<path>/rero-ils-migration' variable in the docker-service.yaml file.
poetry run invenio reroils migrations index init 
# create the bramois migration project
poetry run invenio reroils migrations create bramois 64 -d " Migration données bibliothèque de Bramois" "rero_ils_migrations.converter_2024_bramois.BramoisConverter"
# convert the data to JSON
poetry run invenio reroils migrations data load bramois <path>/bramois.xml 
# run the dedup
poetry run invenio reroils migrations data dedup bramois

Migration permission

# update permissions
from rero_ils.modules.cli.fixtures import load_role_policies
from rero_ils.modules.cli.fixtures import load_system_role_policies
cfg = {
    "mig-search": [
    "pro_full_permissions",
    "pro_library_administrator",
    "pro_catalog_manager"
  ],
  "mig-read": [
    "pro_full_permissions",
    "pro_library_administrator",
    "pro_catalog_manager"
  ],
  "mig-access": [
    "pro_full_permissions",
    "pro_library_administrator",
    "pro_catalog_manager"
  ],
  "mig-update": [
    "pro_full_permissions",
    "pro_library_administrator",
    "pro_catalog_manager"
  ]
}
load_role_policies(cfg)

change Bibliomedia harvested

from rero_ils.modules.documents.api import Document, DocumentsSearch

query = (
    DocumentsSearch()
    .filter("exists", field="adminMetadata")
    .filter("match", adminMetadata__note="Bibliomedia")
    .exclude("term", harvested=True)
)
doc_pids = [hit.pid for hit in query.source('pid').scan()]
for pid in doc_pids:
    if doc := Document.get_record_by_pid(pid):
        change = False
        admin_metadata = doc.get('adminMetadata')
        new_note = []
        changed = False
        # print('---->', admin_metadata.get('note', []))
        for note in admin_metadata.get('note', []):
            if note.startswith('!!! Attention : notice Bibliomedia, ne pas se raccrocher !!!'):
                print(doc.pid, admin_metadata['note'])
                doc['harvested'] = True
                change = True
        if change:
            doc.update(data=doc, dbcommit=True, reindex=True)

Sync remote entities bf:Topic, bf:Temproral, bf:Place

from rero_ils.modules.entities.remote_entities.api import RemoteEntity
from rero_ils.modules.entities.remote_entities.sync import SyncEntity

sync = SyncEntity()
query = RemoteEntitiesSearch().filter("terms", type=["bf:Topic", "bf:Temporal", "bf:Place"])
count = query.count()
for idx, hit in enumerate(query.source("pid").scan(), 1):
    res = sync.sync_record(hit.pid)
    print(f"{idx:>7}/{count} {hit.pid} {res}")

Reindex Documents with remote entities

from rero_invenio_base.modules.tasks import run_on_worker
from rero_invenio_base.modules.utils import chunk
from rero_ils.modules.documents.api import Document, DocumentsSearch

code = '''
def reindex(_ids):
    from rero_ils.modules.documents.api import Document
    n = 0
    errors = []
    for _id in _ids:
        try:
            doc = Document.get_record(_id)
            doc.reindex()
            n += 1
        except Exception as e:
            print('error', e)
            errors.append(_id)
    return (n, errors)
'''

parallel = 7
# subjects
count = 0
query = DocumentsSearch().filter("exists", field="subjects.entity.pid")
ids = [hit.meta.id for hit in query.source().scan()]
for c in chunk([str(val) for val in ids], len(ids) // parallel):
    count += 1
    res = run_on_worker.delay(code, 'reindex', c)
    print('subjects', count, len(c), res)
# genreForm
count = 0
query = DocumentsSearch().filter("exists", field="genreForm.entity.pid")
ids = [hit.meta.id for hit in query.source().scan()]
for c in chunk([str(val) for val in ids], len(ids) // parallel):
    count += 1
    res = run_on_worker.delay(code, 'reindex', c)
    print('genreForm', count, len(c), res)

# from rero_ils.modules.documents.api import Document, DocumentsSearch
#
# for entity_type in ["subjects", "genreForm"]:
#     query = DocumentsSearch().filter("exists", field=f"{entity_type}.entity.pid")
#     count = query.count()
#     for idx, hit in enumerate(query.source("pid").scan(), 1):
#         if doc := Document.get_record_by_pid(hit.pid):
#             print(f"{idx:>7}/{count} {entity_type} {doc.pid}")
#             doc.reindex()

v1.22.1 --> v1.23.0

sync files

rsync -avr /network/nfs/files_prod/* /network/nfs/files/  --exclude=lost+found/

Clean RABBITMQ

# after docker compose down
sudo rm -rf /data/ils/prod/mq/*

copy static files

docker-compose exec -u root web-ui bash -c "cp -r /invenio/var/instance/static/* /invenio/instance/static/."

Alembic

poetry run invenio alembic upgrade

Update mapping

poetry run invenio rero es index update-mapping

Reindex Remote entities (~1h30)

from rero_ils.modules.entities.remote_entities.api import \
    RemoteEntitiesIndexer, RemoteEntity
from rero_ils.modules.tasks import process_bulk_queue

entities_ids = RemoteEntity.get_all_ids()
RemoteEntitiesIndexer().bulk_index(entities_ids)
process_bulk_queue.apply_async()

Reindex Operation Logs (<1h)

from invenio_search import current_search
# update operation logs templates
[p for p in current_search.put_templates()]

from invenio_search import current_search_client
def reindex(source, destination):
    res = current_search_client.reindex(
        body=dict(
            source=dict(
                index=source
            ),
            dest=dict(
                index=destination,
                version_type='external_gte'
            )
        ),
        wait_for_completion=False
    )
    return res['task']

def index_in_new(indices):
    tasks = []
    body = {
        "settings": {
        "number_of_shards": "8",
        "number_of_replicas": "1",
        "max_result_window": "100000"
    }}
    for index_name in indices:
        print(index_name)
        new_index_name = f'{index_name}-new'
        current_search_client.indices.create(new_index_name, body=body)
        tasks.append(reindex(index_name, new_index_name))
    return tasks

def remove_old(indices):
    for index_name in indices:
        current_search_client.indices.delete(index_name)
        print(f'{index_name} has been deleted')

def rename_to_old(indices):
    tasks = []
    body = {
        "settings": {
        "number_of_shards": "8",
        "number_of_replicas": "1",
        "max_result_window": "100000"
    }}
    for index_name in indices:
        print(index_name)
        new_index_name = f'{index_name}-new'
        current_search_client.indices.create(index_name, body=body)
        tasks.append(reindex(new_index_name, index_name))
    return tasks

# Execute one line after the other
# Get the list of the operation logs indices
indices = ['operation_logs-2024']
tasks = index_in_new(indices)
# Check for completion
[current_search_client.tasks.get(t).get('completed') for t in tasks]
remove_old(indices)
tasks = rename_to_old(indices)
[current_search_client.tasks.get(t).get('completed') for t in tasks]
remove_old([i+'-new' for i in indices])

Reharvest ebooks

poetry run invenio reroils oaiharvester harvest -n ebooks -q -f 1990-01-01

Enable task

poetry run invenio reroils scheduler enable_tasks -n automatic_renewal

v1.21.0 --> v1.22.1

Requirement

# create new frontend image with the new configuration
docker-compose build frontend
docker-compose stop selfcheck scheduler worker frontend

copy static files

docker-compose exec -u root web-ui bash -c "cp -r /invenio/var/instance/static/* /invenio/instance/static/."

Babeltheque

Set RERO_ILS_APP_BABELTHEQUE_ENABLED_VIEWS = ['vs']

Update mapping

Files

Set INVENIO_RERO_ILS_FILES_FOLDER = /network/nfs/files

# create new tables
poetry run invenio db create
# create new indices
poetry run invenio rero es index update-mapping
poetry run invenio index create  -b rero_ils/modules/files/mappings/v7/files/record-v1.0.0.json files-record-v1.0.0-20240521

# init storage
poetry run invenio files location create --default default /network/nfs/files
from invenio_search import current_search, current_search_client
current_search_client.indices.put_alias('files-record-v1.0.0-20240521', 'files')
current_search_client.indices.put_alias('files-record-v1.0.0-20240521', 'files-record-v1.0.0')

# update permissions
from rero_ils.modules.cli.fixtures import load_role_policies
from rero_ils.modules.cli.fixtures import load_system_role_policies
cfg = {
  "file-create": [
    "pro_full_permissions",
    "pro_catalog_manager",
    "pro_library_administrator"
  ],
  "file-update": [
    "pro_full_permissions",
    "pro_catalog_manager",
    "pro_library_administrator"
  ],
  "file-delete": [
    "pro_full_permissions",
    "pro_catalog_manager",
    "pro_library_administrator"
  ]
}
load_role_policies(cfg)

sys_cfg = {
  "file-search": [
    "any_user"
  ],
  "file-read": [
    "any_user"
  ],
  }
load_system_role_policies(sys_cfg)

Records

index_name=`poetry run invenio rero es index info -i records`
echo $index_name
poetry run invenio rero es index move records $index_name records-record-v1.0.0-20240521 -v
poetry run invenio index delete $index_name
from invenio_search import current_search, current_search_client
current_search_client.indices.put_alias('records-record-v1.0.0-20240521', 'records-record-v1.0.0')

Documents (20')

index_name=`poetry run invenio rero es index info -i documents`
echo $index_name
poetry run invenio rero es index move documents $index_name documents-document-v0.0.1-20240521 -v
poetry run invenio index delete $index_name

Fiction (4' + 13' + 15')

from sqlalchemy import func
from invenio_db import db
from rero_ils.modules.documents.api import Document, DocumentsSearch
from invenio_search import current_search_client
from time import sleep

def change(query, fiction_statement, delay=5):
    """change DB and ES."""
    print(f'Get ids ({query.count()}) ...')
    ids = [hit.meta.id for hit in query.source(False).scan()]

    print('Update DB ...', end=' ')
    count = Document.model_cls.query \
        .filter(Document.model_cls.id.in_(ids)) \
        .update(
            {
                'json': func.jsonb_set(
                    Document.model_cls.json,
                    '{fiction_statement}',
                    f'"{fiction_statement}"'
                ),
                'version_id': Document.model_cls.version_id + 1
            },
            synchronize_session=False
        )
    print(count)
    db.session.commit()

    body = query.to_dict()
    body.update(
        {"script": {
            "source": f"ctx._source['fiction_statement'] = '{fiction_statement}'"
        }}
    )

    info = current_search_client.update_by_query(
        index='documents',
        body=body,
        wait_for_completion=False
    )
    task_id = info['task']
    print(f'Update ES ... task id: "{task_id}"')
    task = current_search_client.tasks.get(task_id)
    print(
        f'updated: {task["task"]["status"]["updated"]} '
        f'conflicts: {task["task"]["status"]["version_conflicts"]}',
        end='\r'
    )
    while not task['completed']:
        sleep(delay)
        task = current_search_client.tasks.get(task_id)
        print(
            f'updated: {task["task"]["status"]["updated"]} '
            f'conflicts: {task["task"]["status"]["version_conflicts"]}',
            end='\r'
        )
    print()
    return task['response']


# if errors
failures = response.get('failures', [])
print(f'Correct errors: {len(failures)}')
for idx, failure in enumerate(failures, 1):
    id_ = failure.get('id')
    print(idx, id_, end='\r')
    current_search_client.delete(
        index='documents',
        id=id_,
        refresh=True
    )
print('Sleep: 60                              ')
sleep(60)
for idx, failure in enumerate(failures, 1):
    id_ = failure.get('id')
    print(idx, id_, end='\r')
    doc = Document.get_record(id_)
    try:
        doc.reindex()
    except Exception as err:
        print(idx, id_, err)
# if errors


FICTIONS_TERMS = ['Fictions', 'Films de fiction']
# Fiction
query = DocumentsSearch() \
    .filter('terms', facet_genre_form_en=FICTIONS_TERMS)
response = change(query, 'fiction')
# test
fiction_count = DocumentsSearch() \
    .filter('term', fiction_statement='fiction') \
    .count()
print(f'term: {query.count()} statement: {fiction_count}')

# Non fiction
query = DocumentsSearch() \
    .exclude('term', harvested=True) \
    .exclude('terms', facet_genre_form_en=FICTIONS_TERMS) \
    .filter('exists', field='subjects')
response = change(query, 'non_fiction')
# test
non_fiction_count = DocumentsSearch() \
    .filter('term', fiction_statement='non_fiction') \
    .count()
print(f'term: {query.count()} statement: {non_fiction_count}')

# Unspecified
query = DocumentsSearch() \
    .exclude('exists', field='fiction_statement')
db.session.close()
response = change(query, 'unspecified')
# test
count = DocumentsSearch().count() - fiction_count - non_fiction_count
unspecified_count = DocumentsSearch() \
    .filter('term', fiction_statement='unspecified') \
    .count()
print(f'{count} statement: {unspecified_count}')

Remove legacy fields (20')

Delete field legacy_circulation_rules from all items (https://github.com/rero/rero-ils/pull/3671).

from rero_ils.modules.items.api import ItemsSearch, Item
from invenio_search import current_search_client
from invenio_db import db
from time import sleep
from elasticsearch_dsl import Q


query = ItemsSearch() \
    .filter(
        Q('exists', field='legacy_circulation_rules')
    )

print(f'Get ids ({query.count()}) ...')
ids = [hit.meta.id for hit in query.source().scan()]

print('Change DB ... ', end='')
count = Item.model_cls.query \
    .filter(Item.model_cls.id.in_(ids)) \
    .update(
        {
            Item.model_cls.json: (
                Item.model_cls.json - 'legacy_circulation_rules'
            ),
            'version_id': Item.model_cls.version_id + 1
        },
        synchronize_session=False
    )
db.session.commit()
print(count)

print('Change ES ...')
body = query.to_dict()
body.update({"script" : "ctx._source.remove(\"legacy_circulation_rules\")"}, )

info = current_search_client.update_by_query(
    index='items',
    body=body,
    wait_for_completion=False
)
task_id = info['task']
print(f'Update ES ... task id: "{task_id}"')
task = current_search_client.tasks.get(task_id)
print(
    f'updated: {task["task"]["status"]["updated"]} '
    f'conflicts: {task["task"]["status"]["version_conflicts"]}',
    end='\r'
)
while not task['completed']:
    sleep(5)
    task = current_search_client.tasks.get(task_id)
    print(
        f'updated: {task["task"]["status"]["updated"]} '
        f'conflicts: {task["task"]["status"]["version_conflicts"]}',
        end='\r'
    )
print()

Alembic fiction

poetry run invenio alembic upgrade

poetry run invenio alembic stamp 2e97565eba72
poetry run invenio alembic upgrade
poetry run invenio alembic stamp 8ae99b034410
poetry run invenio alembic upgrade
poetry run invenio alembic stamp a29271fd78f8
poetry run invenio alembic upgrade

Update SLM

Add files* to slm_daily_all.json.

poetry run invenio rero es slm put daily slm_daily_all.json

v1.20.0 --> v1.21.0

Reindexing

Flask2

Flask-Wiki

  • add a INVENIO_WIKI_INDEX_DIR in the configmap of you producution environment.

migrate invenio-userprofle to user.user_profile

from invenio_db import db
from rero_ils.modules.patrons.api import Patron
db.session.execute("DELETE from alembic_version where version_num = 'c25ef2c50ffa'")
db.session.commit()
# run poetry run invenio alembic upgrade
# INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
# INFO  [alembic.runtime.migration] Will assume transactional DDL.
# INFO  [alembic.runtime.migration] Running upgrade 04480be1593e -> 842a62b56e60, Change FK AccountsRole to string (downgrade recipe).
# INFO  [alembic.runtime.migration] Running upgrade e12419831262 -> 999dcbd19ace, Add versioning information to models.
# INFO  [alembic.runtime.migration] Running upgrade 999dcbd19ace -> dfbdf43a3e96, Separate login info from user table.
# INFO  [alembic.runtime.migration] Running upgrade dfbdf43a3e96 -> 62efc52773d4, Create UserIdentity table.
# INFO  [alembic.runtime.migration] Running upgrade 62efc52773d4 -> eb9743315a9d, Add user profile and preferences as JSON fields to the User table.
# INFO  [alembic.runtime.migration] Running upgrade eb9743315a9d -> f2522cdd5fcd, Change AccountsRole primary key to string.
# INFO  [alembic.runtime.migration] Running upgrade f2522cdd5fcd, 842a62b56e60 -> f9843093f686, Change FK AccountsRole to string (upgrade recipe).
# INFO  [alembic.runtime.migration] Running upgrade f9843093f686 -> 037afe10e9ff, Add user moderation fields.
# INFO  [alembic.runtime.migration] Running upgrade bff1f190b9bd -> aaa265b0afa6, Move UserIdentity to accounts.
# INFO  [alembic.runtime.migration] Running upgrade  -> 759d47cbdba7, Create oaiserver branch.
# INFO  [alembic.runtime.migration] Running upgrade 759d47cbdba7 -> e655021de0de, Create oiaserver tables.
# INFO  [alembic.runtime.migration] Running upgrade e655021de0de -> 5d25c1981985, Add system_created field.
total = db.session.execute('SELECT COUNT(*) from userprofiles_userprofile').all()[0][0]
errors = {}
for idx, prof in enumerate(db.session.execute('SELECT * from userprofiles_userprofile').all(), 1):
    print(f'{idx}/{total}', end='\r')
    prof = {k: v for k, v in dict(prof).items() if v}
    user_id = prof.pop('user_id')
    user = Patron._get_user_by_user_id(user_id)
    if user.username:
        continue
    prof.pop('displayname', None)
    try:
        if birth_date := prof.get('birth_date'):
            prof['birth_date'] = birth_date.strftime('%Y-%m-%d')
        if username := prof.pop('username', None):
            user.username = username
        user.user_profile = prof
        db.session.merge(user)
        if idx % 100 == 0:
            db.session.commit()
    except Exception as e:
        errors[user_id] = e

db.session.commit()
if errors:
    print('errors:', errors)
else:
  db.session.execute('DROP TABLE userprofiles_userprofile')

Mapping

Elasticsearch

poetry run invenio rero es index update-mapping
index_name=`poetry run invenio rero es index info -i vendors`
echo $index_name
poetry run invenio rero es index move vendors $index_name vendors-vendor-v0.0.1-20240206 -v
poetry run invenio rero es index update-mapping
poetry run invenio index delete $index_name

index_name=`poetry run invenio rero es index info -i acq_orders`
echo $index_name
poetry run invenio rero es index move acq_orders $index_name acq_orders-acq_order-v0.0.1-20240206 -v
poetry run invenio rero es index update-mapping
poetry run invenio index delete $index_name

index_name=`poetry run invenio rero es index info -i documents`
echo $index_name
poetry run invenio rero es index move documents $index_name documents-document-v0.0.1-20240206 -v
poetry run invenio rero es index update-mapping
poetry run invenio index delete $index_name

Documents partOf

from rero_ils.modules.documents.api import Document, DocumentsSearch
from invenio_db import db

query = DocumentsSearch().filter('exists', field='partOf.numbering')
total = query.count()
for idx, hit in enumerate(query.source('pid').scan(), 1):
    print(f'{idx}/{total} {hit.pid}', end='\r')
    if doc := Document.get_record_by_pid(hit.pid):
        part_of = doc['partOf']
        for part_of in doc['partOf']:
            for numbering in part_of.get('numbering', []):
                if 'volume' in numbering:
                     numbering['volume'] = str(numbering['volume'])
                if 'issue' in numbering:
                     numbering['issue'] = str(numbering['issue'])
        # print(part_of)
        # TODO: make it faster
        # doc.update(data=doc, dbcommit=True, reindex=True)
        doc.model.json = doc
        db.session.merge(doc.model)
        doc.reindex()
        if idx % 100 == 0:
            db.session.commit()
db.session.commit()

—— rero.ch | Twitter | Gitter —— RERO+ catalogue | RERO ILS test version | RERO DOC | RERO MEF ——

Clone this wiki locally