James/overhaul docs

.github/workflows/cypress.yml

@@ -31,7 +31,7 @@ jobs:
           parallel: true
           group: "CI RUN"
           ci-build-id: ${{ github.run_id }}
-          wait-on: "http://localhost:3000, http://localhost:9001"
+          wait-on: "http://localhost:3000"
           wait-on-timeout: 600
           spec: e2e/*.{js,jsx,ts,tsx}
         env:

README.md

@@ -29,25 +29,17 @@ The root folder contains a few loose files associated with the project as a whol
 
 ## Installing local dependencies
 
-`deliberation-empirica` assumes certain dependencies prior to local development. These are:
-
-- Node.js (https://nodejs.org/en/download)
-- Docker (https://www.docker.com/)
-- Empirica (https://docs.empirica.ly/getting-started/setup); the top of the setup page contains a one-line command for installing Empirica.
-
-Running `npm run build` without these dependencies leads to an error (e.g, `docker: command not found`; `empirica: command not found`).
-
-[For Mac users] One easy way of installing Docker is via homebrew (https://docs.brew.sh/Installation). Once homebrew is installed, simply run:
-
-```
-brew install docker --cask
-```
+`deliberation-empirica` assumes a few dependencies:
+
+- Node.js (https://nodejs.org/en/download) is required before running any scripts.
+- Empirica CLI (https://docs.empirica.ly/getting-started/setup) is installed automatically the first time you run `npm run build`, but you can pre-install it manually if you prefer.
+- Docker (https://www.docker.com/) is only required when you want to run the local etherpad helper via `npm run start:etherpad`. The helper script will attempt to install Docker automatically if it is missing, though installing it yourself (e.g., `brew install docker --cask` on macOS) is usually faster.
 
 ## Setting up the local environment
 
-At the root folder of `deliberation-empirica`, the system expects an `.env` file with the following structure:
-
-```
+At the root folder of `deliberation-empirica`, the system expects an `.env` file with the following structure:
+
+```
 DAILY_APIKEY=
 QUALTRICS_API_TOKEN=
 QUALTRICS_DATACENTER=
@@ -59,49 +51,24 @@ TEST_CONTROLS=
 GITHUB_PRIVATE_DATA_OWNER=
 GITHUB_PUBLIC_DATA_OWNER=
 GITHUB_PRIVATE_DATA_REPO=
-GITHUB_PRIVATE_DATA_BRANCH=
-GITHUB_PUBLIC_DATA_REPO=
-GITHUB_PUBLIC_DATA_BRANCH=
-```
-
-Starting the server without the `.env` file will work, but experiments will fail without the proper API keys, GitHub repos, etc.
-
-After installing these dependencies and setting up the local environment, you can proceed to running on dev.
-
-If you would like to run the local environment without the API keys, you can use the `.env` settings below. Note that this disables functionality for video calling and pushing batch data to GitHub.
-
-```
-DAILY_APIKEY=none
-QUALTRICS_API_TOKEN=none
-QUALTRICS_DATACENTER=none
-ETHERPAD_API_KEY=none
-ETHERPAD_BASE_URL=none
-DELIBERATION_MACHINE_USER_TOKEN=none
-EMPIRICA_ADMIN_PW=localpwd
-TEST_CONTROLS=enabled
-GITHUB_PRIVATE_DATA_OWNER=none
-GITHUB_PUBLIC_DATA_OWNER=none
-GITHUB_PRIVATE_DATA_REPO=none
-GITHUB_PRIVATE_DATA_BRANCH=none
-GITHUB_PUBLIC_DATA_REPO=none
-GITHUB_PUBLIC_DATA_BRANCH=none
-```
+GITHUB_PRIVATE_DATA_BRANCH=
+GITHUB_PUBLIC_DATA_REPO=
+GITHUB_PUBLIC_DATA_BRANCH=
+```
+
+Starting the server without the `.env` file will work, but experiments will fail without the proper API keys, GitHub repos, etc.
+
+The repo includes `default.env`, which contains commented placeholder values (`# comment` syntax) that keep the stack runnable for local demos without touching real services. `npm run build` copies this template to `.env` the first time it runs, so edit `.env` (not `default.env`) when you are ready to plug in real secrets.
+
+Running `npm run build` now auto-installs the Empirica CLI (if necessary), initializes `server/` and `client/` dependencies, and creates an `.env` file if it does not exist. The generated `.env` is copied from `default.env`, which contains “safe for local dev” placeholder values. Update any secrets before running experiments that touch real services.
 
 ## Running on dev
 
-The first time you start the environment, you need to build the etherpad container and install any project dependencies. To do this, run:
-
-```bash
-npm run build
-```
-
-Then, whenever you want to start the dev environment, you need to start the empirica server, the testing cdn, and the etherpad instance. To do this, type:
-
-```bash
-npm run start
-```
-
-This runs the `runner.sh` script, which is only run in development.
+1. Run `npm run build` once per checkout to install Empirica, populate `.env` from `default.env` (if needed), and install dependencies for both `server/` and `client/`.
+2. (Optional) If you need the local Etherpad instance—for example, when testing chat features—run `npm run start:etherpad` in a separate terminal. The helper checks for Docker, installs it if missing, builds the custom Etherpad image, and starts the container on `http://localhost:9001`.
+3. Start the main dev environment with `npm run start`. This launches Empirica and the mock CDN. Etherpad is no longer started automatically; only run it when your workflow needs it.
+
+This runs the `runner.sh` script, which is only run in development.
 
 Now that everything is set up, you can visit
 

builder.sh

@@ -1,43 +1,73 @@
 #!/bin/bash
+set -euo pipefail
 
 echo "-------- builder.sh --------"
 
 cwd=$(pwd)
 echo "Base directory $cwd"
 
-mkdir -p ${cwd}/data
-
-echo "Building etherpad"
-cd ${cwd}/etherpad
-
-echo "Installing Docker if needed"
-if ! docker -v &> /dev/null; then
-    if ! curl -fsSL https://get.docker.com -o install-docker.sh; then
-        echo "Failed to download Docker installation script. Exiting."
-        exit 1
-    fi
-    if ! sh install-docker.sh; then
-        echo "Failed to install Docker. Exiting."
-        exit 1
-    fi
-    echo "Docker installed successfully."
-    rm -f install-docker.sh
-fi
-
-echo "Building docker"
-docker buildx build \
-  --platform linux/amd64 \
-  --tag deliberation-etherpad \
-  --file Dockerfile \
-  .
-
-# echo "Installing empirica"
-# cd $cwd
-# curl -fsS https://install.empirica.dev | sh
-
-echo "Installing empirica dependencies"
-cd $cwd/server 
+mkdir -p "${cwd}/data"
+
+ENV_FILE="${cwd}/.env"
+DEFAULT_ENV="${cwd}/default.env"
+
+# Fetches the Empirica CLI if the developer hasn't installed it yet.
+ensure_empirica_cli() {
+  if command -v empirica >/dev/null 2>&1; then
+    echo "Empirica CLI already installed."
+    return
+  fi
+
+  echo "Empirica CLI not found. Installing..."
+  if ! curl -fsS https://install.empirica.dev | sh; then
+    echo "Failed to install Empirica CLI. Exiting."
+    exit 1
+  fi
+}
+
+# Creates .env if missing, preferring the checked-in default.env template.
+ensure_env_file() {
+  if [ -f "${ENV_FILE}" ]; then
+    echo ".env already exists. Skipping creation."
+    return
+  fi
+
+  if [ "${CI:-}" = "true" ] || [ -n "${GITHUB_ACTIONS:-}" ]; then
+    echo "CI environment detected and no .env present; skipping auto-creation so workflow-provided secrets take precedence."
+    return
+  fi
+
+  if [ -f "${DEFAULT_ENV}" ]; then
+    cp "${DEFAULT_ENV}" "${ENV_FILE}"
+    echo "Created .env from default.env. Update it with real secrets before running production workloads."
+  else
+    cat <<'EOF' > "${ENV_FILE}"
+DAILY_APIKEY=none
+QUALTRICS_API_TOKEN=none
+QUALTRICS_DATACENTER=none
+ETHERPAD_API_KEY=none
+ETHERPAD_BASE_URL=none
+DELIBERATION_MACHINE_USER_TOKEN=none
+EMPIRICA_ADMIN_PW=localpwd
+TEST_CONTROLS=enabled
+GITHUB_PRIVATE_DATA_OWNER=none
+GITHUB_PUBLIC_DATA_OWNER=none
+GITHUB_PRIVATE_DATA_REPO=none
+GITHUB_PRIVATE_DATA_BRANCH=none
+GITHUB_PUBLIC_DATA_REPO=none
+GITHUB_PUBLIC_DATA_BRANCH=none
+EOF
+    echo "Created .env with local-safe defaults. Replace these with real values as needed."
+  fi
+}
+
+ensure_env_file
+ensure_empirica_cli
+
+echo "Installing empirica dependencies for server/"
+cd "${cwd}/server"
 empirica npm install
 
-cd $cwd/client 
+echo "Installing empirica dependencies for client/"
+cd "${cwd}/client"
 empirica npm install

cypress/e2e/00_Naked_URL.js

@@ -39,12 +39,6 @@ const configJson = `{
 
 describe("Naked URL", { retries: { runMode: 2, openMode: 0 } }, () => {
   beforeEach(() => {
-    // when the etherpad server restarts, it minifies the javascript it sends
-    // the first time the site is visited. Doing that here means that when
-    // we visit during the test, this has already happened and the test can
-    // proceed without waiting for the minification to complete.
-    cy.visit("http://localhost:9001/p/forceMinify");
-
     // using beforeEach even though there is just one test, so that if we retry the test it will run again
     cy.empiricaClearBatches();
     cy.empiricaCreateCustomBatch(configJson, {});

cypress/e2e/10_Etherpad_Qualtrics.js

@@ -1,6 +1,8 @@
 // You can only open one etherpad document on a page, because etherpad sets a cookie.
 // So, we test the etherpad data pipeline in a separate test file with only one user.
 
+// We're currently skipping this test in CI. If we put it back in, we need to make sure to start the etherpad server there, which we currently don't do.
+
 describe(
   "Etherpad and Qualtrics Test",
   { retries: { runMode: 2, openMode: 0 } },

cypress/fixtures/mockCDN/demo/annotated_demo/README.md

@@ -0,0 +1,25 @@
+# Annotated demo
+
+This folder contains files for an annotated demo experiment.
+
+### treatment file
+
+- `demo.treatments.yaml` describes what gets displayed to whom, when, and describes the full experiment flow from the participants perspective, for all treatments.
+
+### batch config files
+
+- `demo.config.json` is batch configuration items that set up the batch on the prod server.
+- `dev.config.json` is batch configuration for running locally on your own machine.
+
+### prompt and other resources
+
+These resources can technically be placed anywhere, but for ease of management, we organize them as follows:
+
+- `intro/` contains files first displayed during the asynchronous intro sequence (before randomization)
+- `game/` contains files displayed during the synchronous portion
+- `exit/` contains files displayed during the asynchronous exit sequence
+- `topics/` lists the stimuli used across the experiment.
+
+## Using this demo
+
+This demo expects both a republican and democratic participant. Open the experiment in two separate browsers (eaisest) and make sure when prompted for party affiliation in the intro sequence to respond once as a democrat, and in the other window as a republican.

cypress/fixtures/mockCDN/demo/annotated_demo/demo.config.json

@@ -0,0 +1,26 @@
+{
+  "batchName": "annotated_demo",
+  "cdn": "prod",
+  "treatmentFile": "demo/annotated_demo/demo.treatments.yaml",
+  "customIdInstructions": "demo/annotated_demo/intro/describeAnnotatedDemo.md",
+  "platformConsent": "US",
+  "consentAddendum": "none",
+  "checkAudio": true,
+  "checkVideo": true,
+  "introSequence": "demo_intro_sequence",
+  "treatments": ["demo_israelAid", "demo_slaveReparations"],
+  "payoffs": "equal",
+  "knockdowns": "none",
+  "dispatchWait": 1,
+  "launchDate": "immediate",
+  "centralPrereg": false,
+  "preregRepos": [],
+  "dataRepos": [],
+  "videoStorage": "none",
+  "exitCodes": {
+    "complete": "MyCompleteExitCode",
+    "error": "MyErrorExitCode",
+    "lobbyTimeout": "MyTimeoutExitCode",
+    "failedEquipmentCheck": "MyFailedEquipmentCheckExitCode"
+  }
+}