Merge pull request #52 from seanboyce/dev

Natively handle ISBN search

Author: seanboyce

CHANGELOG.md

@@ -1,6 +1,17 @@
 # New Features in this Version:
+1. Federated search! You can link up ubiblio instances, and search each other's books (but not edit them).
+2. An advanced search page. For now, it just handles federated searches.
+3. Google Books API returns HTTP 429 errors when no API key is specified. The library used by ubiblio could not correctly set Google Books API keys. So I wrote functions to access (Google Books, OpenLibrary, Wikipedia) so ubiblio no longer relies on external libraries for this. 
+4. Google books will be skipped on ISBN auto-add unless you specify a Google books API key (free) by setting the environment variable GB_API="YOUR-API-KEY". I'll re-enable it without API key if it starts working again.
 
-1. Database backup can now handle files, and the backu/restore UI lets you manage that.
+
+# Notes
+
+The application can be used just fine from the Kindle Paperwhite (and probably most other ebook readers with web browsers), because it is very simple. I find navigating / searching my ebooks gets impractically slow on-device. Ubiblio is much faster, and additionally lets me store all my ebooks elsewhere. No need to ever worry about device storage again! Incidentally, for better display on ebook displays, I've made some text in bold (ebook file management). 
+
+# New Features in last Version:
+
+1. Database backup can now handle files, and the backup/restore UI lets you manage that.
 2. If ISBN auto-adder failes to find a book, try other sources. This can take a little longer, but makes adding books faster overall.
 3. Clean up book details template -- infrequently used (admin) actions are in an accordion element. Remove that useless 'Back' button.
 4. Categorize action buttons on book details page
@@ -10,28 +21,9 @@
 8. Admin actions should be hidden on all pages for non-admin users (let me know if I've missed one)
 9. Added a cursed branch, for deploying on RISCV64. Still no easter eggs, though.
 
-# Notes
-
-The application can be used just fine from the Kindle Paperwhite (and probably most other ebook readers with web browsers), because it is very simple. I find navigating / searching my ebooks gets impractically slow on-device. Ubiblio is much faster, and additionally lets me store all my ebooks elsewhere. No need to ever worry about device storage again! Incidentally, for better display on ebook displays, I've made some text in bold (ebook file management). 
-
-# New Features in last Version:
-
-1. Database import / export / restore from a GUI (using the backup script still works though).
-2. Database versioning and automatic upgrade (it will back up first).
-3. A library config menu for admins, where you can manage backups and other things.
-4. You can now judge books by their cover! Image support for up to 16 images per book (enable in config menu -- or if you don't want it, leave it disabled)
-5. A luxurious two custom fields you can enable and name at your leisure (from the config menu -- just give them a name to enable)
-6. A view for withdrawn books that makes more sense -- you can see who withdrew a book.
-7. You can export your book list as CSV in the backup management interface. For example if you just want to use ubiblio for data entry.
-8. If you import a book list from ubiblio, is will ADD the books to the database instead of replacing the current database. For example, this lets you merge different libraries. 
-9. New books no longer default to science fiction! Slightly inconvenient for time travelers, better for the rest of you.
-10. Various minor UI improvements.
-11. Still zero easter eggs!
-
 # Notes:
 
-1. Be advised that any files you add (book images, backups) are not included in the database backup presently. These are stored in /static/bookImages and /export respectively. In the next major release, files will be backed up and compressed too. For now, you must manually back up these folders. 
 
-2. Database upgrade is a little sketchy in this first major update. I haven't been able to break it in testing though. Note that ubiblio will back up your database before attempting to upgrade -- if you encounter a bug with the upgrade process, please reach out. 
+1. Database upgrade is a little sketchy. I haven't been able to break it in testing though. Note that ubiblio will back up your database before attempting to upgrade -- if you encounter a bug with the upgrade process, please reach out. 
 
 

SETUP.md

@@ -62,7 +62,8 @@ There are a number of environment variables you can set to slightly change the b
 | USER_PASSWORD | Password to create | No | - |
 | USE_REDIS | Use Redis for DDOS protection? | No | False |
 | REDIS_URI | External Redis URI | No | - |
-| LANGUAGE | Set the language | No | -|
+| LANGUAGE | Set the language | No | - |
+| GB_API | Google Books API key | No | - |
 
 For example, to set up an admin user at first launch, you'll need to run a command like below (replacing username and password as you see fit):
 ```bash
@@ -72,6 +73,19 @@ Then visit /user-setup and the account will be created. It will return you to lo
 
 To set the language to French, launch with LANGUAGE=FR
 
+### Notes on Google Books API Keys
+
+Google Books used to work without API keys. Presently, it does not (reach out if this changes!). To get a free API key for this service, follow the instructions here:
+https://developers.google.com/books/docs/v1/getting_started
+
+It's reasonable (but not required) that you restrict the API key to Google Books only. Note that you should keep the API key secret even though the docs at Google say you don't have to -- looks to me like they've had some issues with this recently.
+
+If you don't specify a Google Books API key, ubiblio will simply skip using this service (it's a good data source though!).
+
+Example (note the API key below is not valid):
+```bash
+env GB_API=AIzaSyAElqyY5RB1EedRnk01vEwkpfhU6aZdrjU uvicorn ubiblio.main:app --host 0.0.0.0 --port 8000 --reload 
+```
 ## Launching
 
 There are two supported ways to launch ubiblio -- uvicorn and gunicorn. Both are ridiculously overpowered for such a simple application, but asking why we need such power is a question for philosophers and cowards.

requirements.txt

@@ -1,6 +1,5 @@
 fastapi==0.115.4
 fastapi_limiter==0.1.6
-isbnlib==3.10.14
 passlib==1.7.4
 pydantic==1.10.2
 PyMySQL==1.1.1
@@ -15,3 +14,4 @@ python-multipart==0.0.5
 aiofiles==24.1.0
 pillow==9.0.1
 uuid==1.30
+requests==2.32.2

ubiblio/main.py

@@ -23,7 +23,6 @@
 from fastapi_limiter.depends import RateLimiter
 import json
 from datetime import datetime
-from isbnlib import *
 from .vars import *
 import sqlite3
 import csv
@@ -33,7 +32,9 @@
 import shutil
 from ecdsa import SigningKey, VerifyingKey, SECP256k1, BadSignatureError
 from hashlib import sha256
- 
+import requests
+import time
+
 console = Console()
 CHUNK_SIZE = 1024 * 1024 #for uploads
 
@@ -486,6 +487,76 @@ def searchbookTitle(request: Request, user: schemas.User = Depends(get_current_u
     except Exception as e:
         db.close()
         return "An error has occured."
+
+# --------------------------------------------------------------------------
+# Book fetch functions, as an alternative to isbnlib
+# --------------------------------------------------------------------------
+def goobMeta(isbn,key):
+    url ="https://www.googleapis.com/books/v1/volumes?q=+isbn:"+str(isbn)+"&key="+str(key)
+    response = requests.get(url)
+    if response.ok:
+        rawBook = json.loads(response.text)["items"][0]["volumeInfo"]
+        book={}
+        book["Title"]=rawBook["title"]
+        book["Author"]=rawBook["authors"][0]
+        try:
+            book["Summary"] = rawBook["description"]
+        except:
+            book["Summary"] = ""
+        return book, 200
+    else:
+        return {},response.status_code
+def openLibMeta(isbn):
+    url = "https://openlibrary.org/isbn/" + str(isbn) + ".json"
+    headers = {
+    "User-Agent": 'ubiblio_bot/1.0 (https://github.com/seanboyce/ubiblio;)',
+    "Accept-Encoding": 'gzip'
+} # In case we somehow cause an issue for them, they can contact me
+    response = requests.get(url,headers=headers)
+    if response.ok:
+        rawBook = json.loads(response.text)
+        book={}
+        book["Title"]=rawBook["title"]
+        authorURL = str(rawBook["authors"][0]["key"])
+        url = "https://openlibrary.org" + authorURL + ".json"
+        time.sleep(1) #Obey openlibrary ratelimit
+        response = requests.get(url)
+        if response.ok:
+            book["Author"] = json.loads(response.text)["personal_name"]
+        else:
+            print(response.status_code)
+        try:
+            book["Summary"] = rawBook["description"]["value"]
+        except:
+            book["Summary"] = ""
+        return book, 200
+    else:
+        return {},response.status_code
+        
+def openWikiMeta(isbn):
+    url = "https://en.wikipedia.org/api/rest_v1/data/citation/mediawiki/" + str(isbn)
+    headers = {
+    "User-Agent": 'ubiblio_bot/1.0 (https://github.com/seanboyce/ubiblio;)',
+    "Accept-Encoding": 'gzip'
+    } # In case we somehow cause an issue for them, they can contact me
+    response = requests.get(url,headers=headers)
+    if response.ok:
+        rawBook = json.loads(response.text)[0]
+        book={}
+        book["Title"]=rawBook["title"]
+        rawAuthor = rawBook["author"][0]
+        author = ""
+        for i in rawAuthor:
+            author = author + i + " "
+        author = author.strip()
+        book["Author"] = author
+        book["Summary"] = "" # Summary is always empty on wikipedia
+        return book, 200
+    else:
+        return {}, response.status_code
+
+
+
 # --------------------------------------------------------------------------
 # ISBN autoadd
 # --------------------------------------------------------------------------
@@ -495,36 +566,42 @@ def new_isbn(isbn, method, response: Response, request: Request, user: schemas.U
     try:
         if user.isAdmin == True:
             book = {}
-            try:
-                book = meta(isbn,service='goob')
-                print(book)
-            except Exception as e:
-                print("Google Books API failed with response: " )
-                print(e)
+            isbn = isbn.strip()
+            if len(GOOGLE_BOOKS_API_KEY)>0:
+                try:
+                    book,response = goobMeta(isbn, GOOGLE_BOOKS_API_KEY)
+                    if response != 200:
+                        print("Google Books API failed with response: " )
+                        print(response)
+                except Exception as e:
+                    print("Google Books API failed with response: " )
+                    print(response)
+            else:
+                pass #No google API key defined, so skip google books. Otherwise it's just going to return HTTP 429 error
             try:
                 if len(book)==0:
-                    book = meta(isbn,service="openl")
+                    book,response = openLibMeta(isbn)
+                    if response != 200:
+                        print("Open Library API failed with response: " )
+                        print(response)
                 else: pass
             except Exception as e:
                 print("Open Library API failed with response: " )
                 print(e)
             try:
                 if len(book)==0:
-                    book = meta(isbn,service='wiki')
+                    book,response = openWikiMeta(isbn)
+                    if response != 200:
+                        print("Wikipedia API failed with response: " )
+                        print(response)
                 else: pass
             except Exception as e:
                 print("Wikipedia API failed with response: " )
                 print(e)
             if len(book)==0:
                 raise LookupError(f"Book with isbn {isbn} not found!")
-            title = book["Title"]
-            author = book["Authors"][0]
-            try:
-                summary = desc(isbn)
-                summary = summary.replace('\n', ' ')
-            except: summary=""
             addISBN = [0]
-            book = schemas.BookCreate(title=title, author=author, summary=summary, ISBN=isbn)
+            book = schemas.BookCreate(title=book["Title"], author=book["Author"], summary=book["Summary"], ISBN=isbn)
             db = SessionLocal()
             config = crud.getConfig(db)
             db.close()
@@ -1503,7 +1580,7 @@ async def refreshVkey(body: bytes = Depends(get_body), user: schemas.User = Depe
 # --------------------------------------------------------------------------
 
 @app.post("/stats", dependencies=[get_rate_limiter(times=2, seconds=2)], response_class=HTMLResponse)
-async def refreshVkey(body: bytes = Depends(get_body), user: schemas.User = Depends(get_current_user_from_token)):
+async def statsp(body: bytes = Depends(get_body), user: schemas.User = Depends(get_current_user_from_token)):
     if not user.isAdmin == True:
         return "You are not authorized to access the library stats page, only Admins can do this."
     body = json.loads(body)
@@ -1523,7 +1600,7 @@ def vdir(obj):
 
 
 @app.get("/stats", dependencies=[get_rate_limiter(times=2, seconds=2)], response_class=HTMLResponse)
-async def refreshVkey(request: Request, user: schemas.User = Depends(get_current_user_from_token)):
+async def statsg(request: Request, user: schemas.User = Depends(get_current_user_from_token)):
     if not user.isAdmin == True:
         return "You are not authorized to access the library stats page, only Admins can do this."
     try:

ubiblio/vars.py

@@ -9,7 +9,7 @@
 REDIS_URL = os.environ.get("REDIS_URI", "redis://localhost")
 
 LANGUAGE = os.environ.get("LANGUAGE", "")
-
+GOOGLE_BOOKS_API_KEY = os.environ.get("GB_API", "")
 TOKEN_TTL = int(os.environ.get("ACCESS_TOKEN_EXPIRE_MINUTES", "120"))
 
 CREATE_ADMIN_USER = os.environ.get("CREATE_ADMIN_USER", "false").lower() == "true"