Feat: TARS refactor (#2)

* init commit

* tests for tars

* cleanup

* remove 404 errors

* Feat init commit update (#3)

* feat: uses async client

* fix: return limit

---------

Co-authored-by: jtyoung84 <104453205+jtyoung84@users.noreply.github.com>

Author: mekhlakapoor

workflow_templates/tag_and_publish.yml .github/workflows/tag_and_publish.ymlworkflow_templates/tag_and_publish.yml renamed to .github/workflows/tag_and_publish.yml

@@ -1,21 +1,17 @@
-# aind-service-template
-A template repo that can be used for rapid development of FastAPI services.
+# aind-tars-service
+REST service to connect to TARS and return information.
 
 ## Usage
 
-### Initialize a repo
+### Create the server.
 
-- Click the green "Use This Template" button
-- Once the repo is created, please wait several seconds for the github action to initialize the files. You can check the status in the "Actions" tab.
-- Update the main README and *-server/README files if desired
-- It's recommended to update the repo settings to require Pull Requests and Reviews.
+- The server code is developed in the aind-tars-service-server package.
+- On a push to main, a docker image will be built and published to GitHub's container registry.
+- You can then run the server in a docker container or k8s pod.
+- More details can be found in the [service README](aind-tars-service-server/README.md) file
 
-### Auto-run tests, publish docker image, and create client code
+### Auto-generated client code.
 
-- In the repo settings, add `svc-aindscicomp` as an admin to the list of Collaborators.
-- Move the yaml files from `workflow_templates` to `.github/workflows/`
-- When a PR is opened, the server code will be tested.
-- When a PR is merged into main:
-  - The tag will be incremented
-  - The server code will be published as a docker image
-  - The client code will be autogenerated from the openapi specification and published to PyPI
+- The client code is autogenerated using an openapi generator.
+- On a push to main, a python library will be built and published to PyPI.
+- The client can then be pip installed as `pip install aind-tars-service-client`

workflow_templates/test_server.yml .github/workflows/test_server.ymlworkflow_templates/test_server.yml renamed to .github/workflows/test_server.yml

@@ -1,11 +1,78 @@
-# aind-service-template-server
+# aind-tars-service-server
 
 [![License](https://img.shields.io/badge/license-MIT-brightgreen)](LICENSE)
 ![Code Style](https://img.shields.io/badge/code%20style-black-black)
-[![semantic-release: angular](https://img.shields.io/badge/semantic--release-angular-e10079?logo=semantic-release)](https://github.com/semantic-release/semantic-release)
-![Interrogate](https://img.shields.io/badge/interrogate-100.0%25-brightgreen)
+	@@ -7,5 +7,74 @@
 ![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen?logo=codecov)
 ![Python](https://img.shields.io/badge/python->=3.10-blue?logo=python)
 
-REST service to retrieve data from an backend.
+REST service to retrieve data from TARS.
 
+## Local Development
+
+Requires docker to build and run package locally.
+
+- Create a file called env/webapp.env with appropriate env variables.
+- Run `docker build -t aind-tars-service-server-local:latest .`
+- Run `docker run -p 5000:80 --env-file=env/webapp.env aind-tars-service-server-local:latest`
+- Service will be available at `http://localhost:5000`
+- Check docs at `http://localhost:5000/docs`
+
+### Linters and testing
+
+There are several libraries used to run linters, check documentation, and run
+ tests.
+
+- Please test your changes using the **coverage** library, which will run the
+ tests and log a coverage report:
+
+```
+coverage run -m pytest && coverage report
+```
+
+- Use **interrogate** to check that modules, methods, etc. have been documented
+ thoroughly:
+
+```
+interrogate .
+```
+
+- Use **flake8** to check that code is up to standards
+ (no unused imports, etc.):
+```
+flake8 .
+```
+
+- Use **black** to automatically format the code into PEP standards:
+```
+black .
+```
+
+- Use **isort** to automatically sort import statements:
+```
+isort .
+```
+
+### Pull requests
+
+For internal members, please create a branch. For external members, please fork
+ the repo and open a pull request from the fork. We'll primarily use
+ [Angular](https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit)
+  style for commit messages. Roughly, they should follow the pattern:
+```
+<type>(<scope>): <short summary>
+```
+
+where scope (optional) describes the packages affected by the code changes and
+type (mandatory) is one of:
+
+- **build**: Changes that affect the build system or external dependencies
+ (example scopes: pyproject.toml, setup.py)
+- **ci**: Changes to our CI configuration files and scripts
+ (examples: .github/workflows/ci.yml)
+- **docs**: Documentation only changes
+- **feat**: A new feature
+- **fix**: A bug fix
+- **perf**: A code change that improves performance
+- **refactor**: A code change that neither fixes a bug nor adds a feature
+- **test**: Adding missing tests or correcting existing tests

README.md

@@ -17,10 +17,12 @@ readme = "README.md"
 dynamic = ["version"]
 
 dependencies = [
-    'requests-toolbelt',  # Used for example. Remove if not needed.
     'pydantic>=2.0',
     'pydantic-settings>=2.0',
     'fastapi[standard]>=0.114.0',
+    'azure-identity>=1.15.0',
+    'fastapi-cache2[redis]>=0.2.2',
+    'httpx'
 ]
 
 [project.optional-dependencies]
@@ -90,5 +92,9 @@ fail-under = 100
 asyncio_mode="auto"
 asyncio_default_fixture_loop_scope="function"
 env = [
-    "MYENV_HOST=example"
+    "TARS_TENANT_ID=tenant_id",
+    "TARS_CLIENT_ID=client_id",
+    "TARS_CLIENT_SECRET=client_secret",
+    "TARS_SCOPE=http://example.com/scope",
+    "TARS_RESOURCE=http://example.com/resource"
 ]

aind-tars-service-server/README.md

@@ -1,18 +1,62 @@
-"""Module for settings to connect to backend"""
+"""Module for settings to connect to TARS backend"""
 
-from pydantic import Field
+from typing import Optional
+from urllib.parse import urljoin
+
+from pydantic import Field, HttpUrl, RedisDsn, SecretStr
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
 
 class Settings(BaseSettings):
-    """
-    ### Settings needed to connect to a database or website.
-    We will just connect to an example website.
-    """
-
-    model_config = SettingsConfigDict(env_prefix="MYENV_")
-    host: str = Field(
-        ...,
-        title="Host",
-        description="Host address of example.com",
+    """Settings needed to connect to TARS."""
+
+    # noinspection SpellCheckingInspection
+    model_config = SettingsConfigDict(env_prefix="TARS_")
+    tenant_id: str = Field(..., description="Azure tenant id.")
+    client_id: str = Field(
+        ..., description="Client ID of the service account accessing resource."
     )
+    client_secret: SecretStr = Field(
+        ..., description="Secret used to access the account."
+    )
+    scope: str = Field(..., description="Scope")
+    resource: HttpUrl = Field(..., description="Resource")
+    redis_url: Optional[RedisDsn] = Field(default=None)
+
+    @property
+    def viral_prep_lots_url(self) -> HttpUrl:
+        """URL for ViralPrepLots"""
+        return HttpUrl.build(
+            scheme=self.resource.scheme,
+            host=self.resource.host,
+            path=urljoin(
+                f"{self.resource.path.lstrip('/')}/", "api/v1/ViralPrepLots"
+            ).lstrip("/"),
+        )
+
+    @property
+    def viruses_url(self) -> HttpUrl:
+        """URL for Viruses"""
+        return HttpUrl.build(
+            scheme=self.resource.scheme,
+            host=self.resource.host,
+            path=urljoin(
+                f"{self.resource.path.lstrip('/')}/", "api/v1/Viruses"
+            ).lstrip("/"),
+        )
+
+    @property
+    def molecules_url(self) -> HttpUrl:
+        """URL for Molecules"""
+        return HttpUrl.build(
+            scheme=self.resource.scheme,
+            host=self.resource.host,
+            path=urljoin(
+                f"{self.resource.path.lstrip('/')}/", "api/v1/Molecules"
+            ).lstrip("/"),
+        )
+
+
+def get_settings() -> Settings:
+    """Utility method to return a settings object."""
+    return Settings()

aind-tars-service-server/pyproject.toml

@@ -1,40 +1,60 @@
-"""Module to retrieve data from a backend using a session object"""
+"""Module to retrieve data from TARS using session object"""
 
-import logging
+from copy import deepcopy
+from typing import Any, Dict, List, Optional
 
-from requests_toolbelt.sessions import BaseUrlSession
-
-from aind_tars_service_server.models import Content
+from httpx import AsyncClient
 
 
 class SessionHandler:
-    """Handle session object to get data"""
+    """Handle pulling data from Sharepoint lists"""
 
-    def __init__(self, session: BaseUrlSession):
-        """Class constructor"""
-        self.session = session
+    def __init__(self, client: AsyncClient):
+        """
+        Class constructor.
+        Parameters
+        ----------
+        client : AsyncClient
+        """
+        self.client = client
 
-    def get_info(self, example_arg: str) -> Content:
+    async def get_data(
+        self, url: str, params: Optional[Dict[str, str]], limit: int = 0
+    ) -> List[Dict[str, Any]]:
         """
-        Get information from a backend. An example argument is added.
+        Returns data from TARS.
 
         Parameters
         ----------
-        example_arg : str
+        url : str
+        params : Dict[str, str] | None
+        limit : int
+          Limit number of items returned. Set to 0 to return all.
 
         Returns
         -------
-        str
-          Contents of a webpage.
+        List[Dict[str, Any]]
 
         """
 
-        logging.debug(f"Sending request for {example_arg}")
-        response = self.session.get("")
+        params_copy = {"page": "0"} if params is None else deepcopy(params)
+        response = await self.client.get(url=url, params=params_copy)
         response.raise_for_status()
-        logging.debug(f"Received response for {example_arg}")
-        text = response.text
-        if example_arg == "length":
-            return Content(info=str(len(text)), arg=example_arg)
-        else:
-            return Content(info=text, arg=example_arg)
+        all_data = []
+        more_pages = response.json()["morePages"]
+        total_count = response.json()["totalCount"]
+        page_num = response.json()["page"]
+        all_data.extend(response.json()["data"])
+        number_of_records = len(all_data)
+        max_count = total_count if limit == 0 else min(total_count, limit)
+        while more_pages and number_of_records < max_count:
+            params_copy["page"] = page_num + 1
+            response = await self.client.get(url=url, params=params_copy)
+            response.raise_for_status()
+            more_pages = response.json()["morePages"]
+            page_num = response.json()["page"]
+            all_data.extend(response.json()["data"])
+            number_of_records = len(all_data)
+        if limit > 0:
+            all_data = all_data[0:limit]
+        return all_data

aind-tars-service-server/src/aind_tars_service_server/configs.py

@@ -2,31 +2,54 @@
 
 import logging
 import os
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
 
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.routing import APIRoute
+from fastapi_cache import FastAPICache
+from fastapi_cache.backends.inmemory import InMemoryBackend
+from fastapi_cache.backends.redis import RedisBackend
+
+# from redis import asyncio as aioredis  # noqa
+from redis.asyncio import from_url  # noqa
 
 from aind_tars_service_server import __version__ as service_version
+from aind_tars_service_server.configs import get_settings
 from aind_tars_service_server.route import router
 
 # The log level can be set by adding an environment variable before launch.
 log_level = os.getenv("LOG_LEVEL", "INFO")
 logging.basicConfig(level=log_level)
 
 description = """
-## aind-service-template
+## aind-tars-service
 
-Service to pull data from example backend.
+Service to pull data from TARS.
 
 """
 
+
+@asynccontextmanager
+async def lifespan(_: FastAPI) -> AsyncIterator[None]:
+    """Init cache and add to lifespan of app"""
+    settings = get_settings()
+    if settings.redis_url is not None:
+        redis = from_url(settings.redis_url)
+        FastAPICache.init(RedisBackend(redis), prefix="fastapi-cache")
+    else:
+        FastAPICache.init(InMemoryBackend(), prefix="fastapi-cache")
+    yield
+
+
 # noinspection PyTypeChecker
 app = FastAPI(
-    title="aind-service-template",
+    title="aind-tars-service",
     description=description,
-    summary="Serves data from example backend.",
+    summary="Serves data from TARS.",
     version=service_version,
+    lifespan=lifespan,
 )
 
 # noinspection PyTypeChecker