From 8d02b1a9afad4148a46569ff81c0be9f71868437 Mon Sep 17 00:00:00 2001 From: everbarry Date: Wed, 18 Mar 2026 11:15:06 +0100 Subject: [PATCH] add docs --- DEVELOPER_GUIDE.md | 141 +++++++++++++++++++++++++++++++++++++++++++++ README.md | 80 +++++++++++++++++++++++++ USER_GUIDE.md | 89 ++++++++++++++++++++++++++++ dashboard.py | 18 +++++- 4 files changed, 325 insertions(+), 3 deletions(-) create mode 100644 DEVELOPER_GUIDE.md create mode 100644 USER_GUIDE.md diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md new file mode 100644 index 0000000..e6b2cd1 --- /dev/null +++ b/DEVELOPER_GUIDE.md @@ -0,0 +1,141 @@ +# Developer Guide + +Technical reference for developers working on or extending the Wildlife Monitoring Dashboard. + +## Architecture Overview + +The application is a single-file Flask server (`dashboard.py`) with HTML/CSS/JS templates embedded as Python string constants. There is no separate frontend build step. + +``` +Browser <──HTTP──> Flask (dashboard.py) + │ + ├── Model inference (PyTorch, EfficientNet V2-S) + ├── XAI pipeline (ScoreCAM, LIME, nearest neighbours) + ├── In-memory detection state + └── File-based XAI cache (_xai_cache/) +``` + +## Key Components + +### Model & Data Loading (module level) + +On startup, the following are loaded once: + +- **Model**: `EfficientNet_V2_S_Weights` base with a custom 7-class classification head, loaded from `efficientnet_v2_wild_forest_animals.pt`. +- **ScoreCAM**: Targets the last convolutional layer (`model.features[-1]`). +- **LIME explainer**: `LimeImageExplainer` instance. +- **Dataset auto-download**: If `wild-forest-animals-and-person-1/` is missing, the Roboflow SDK downloads it automatically on startup. +- **Datasets**: `test_ds` and `train_ds` using `WildForestAnimalsDataset`, a custom `Dataset` class reading from the Roboflow-exported directory structure with `_classes.csv` label files. +- **Training features**: Pre-extracted feature vectors for all training images (used for nearest neighbour lookup). Stored as a normalised matrix `train_feats` for fast cosine similarity via matrix multiplication. + +### Detection Simulation (`/api/simulate`) + +Each call: +1. Picks a random image from `test_ds` and a random camera. +2. Runs model inference to get predicted class and confidence. +3. Creates a detection dict with a unique 8-char hex ID. +4. Saves the original image to `_xai_cache//original.png`. +5. Kicks off `compute_xai` in a **background thread** (serialised by `_xai_lock` to prevent GPU OOM). +6. Returns the detection as JSON. + +### XAI Pipeline (`compute_xai`) + +Produces all explanation artefacts for a detection and writes them to `_xai_cache//`: + +| File | Method | +|---|---| +| `original.png` | Raw input image (224x224) | +| `chart.png` | Matplotlib probability bar chart (kept for potential offline use) | +| `scorecam.png` | ScoreCAM heatmap overlay | +| `lime1.png` | LIME explanation for top-1 predicted class | +| `lime2.png` | LIME explanation for top-2 predicted class | +| `contrastive.png` | Contrastive LIME (top-1 vs top-2) | +| `nb1.png`, `nb2.png`, `nb3.png` | Three nearest training neighbours | +| `meta.json` | Metadata: predictions, probabilities, LIME class labels, neighbour info | + +The function is **idempotent** — it checks for `meta.json` existence before computing. A `threading.Lock` (`_xai_lock`) ensures only one computation runs at a time to avoid GPU memory exhaustion. + +### Background Pre-computation + +When a detection is simulated, XAI computation starts immediately in a background thread. The `/api/xai/` endpoint waits on a `threading.Event` for that detection. This means: + +- If the user clicks a detection after a few seconds, XAI is likely already cached. +- If they click immediately, the request blocks until computation finishes. +- If somehow no background thread was started, it falls back to synchronous computation. + +### API Endpoints + +| Endpoint | Method | Purpose | +|---|---|---| +| `/` | GET | Home page (map + sidebar) | +| `/det/` | GET | Detection detail page | +| `/cam/` | GET | Camera feed page | +| `/api/simulate` | POST | Simulate a new detection | +| `/api/xai/` | GET | Get XAI results (triggers computation if needed) | +| `/api/detections` | GET | List all detections | +| `/api/verify/` | POST | Verify or correct a detection | +| `/map.webp` | GET | Serve the map image | +| `/xai//` | GET | Serve cached XAI images | + +### HTML Templates + +Three template constants are defined as raw Python f-strings (`r"""..."""`): + +- `HOME_HTML` — fullscreen map, camera markers, sidebar with chart/filters/list, auto-simulation JS loop. +- `DETAIL_HTML` — verification bar, XAI carousel with left/right navigation, HTML/CSS probability chart. +- `CAM_HTML` — responsive grid of detection cards for a single camera. + +Templates use Jinja2 syntax (via Flask's `render_template_string`) and receive context variables like `cameras`, `class_names`, `det`, etc. + +### In-Memory State + +All detection state is held in the `detections` Python list. This means: + +- State is lost on server restart. +- There is no database. +- This is intentional for a demonstration/prototype. + +## Extending the Application + +### Adding a new XAI method + +1. Add the computation logic inside `compute_xai()`, saving the output as a PNG in the `out` directory. +2. Add any relevant metadata to the `meta` dict. +3. Add a new slide entry in the `slides` array in `DETAIL_HTML`'s JavaScript. + +### Adding a new species + +1. Add the class name to `CLASS_NAMES`. +2. Add an emoji to `SPECIES_ICON`. +3. Retrain the model with the new class and update `efficientnet_v2_wild_forest_animals.pt`. + +### Adding or moving cameras + +Edit the `CAMERAS` dict. The `px` and `py` values are percentages relative to the map image dimensions (left/top). + +### Persisting detections + +To add persistence, replace the `detections` list with a database (e.g. SQLite). Key fields per detection: `id`, `idx`, `cam`, `pred`, `conf`, `time`, `verified`, `manual`. The `_xai_cache` directory already provides file-based persistence for XAI artefacts. + +## Dependencies + +Core runtime dependencies (see `pyproject.toml`): + +| Package | Purpose | +|---|---| +| `flask` | Web framework | +| `torch`, `torchvision` | Model inference and image transforms | +| `grad-cam` | ScoreCAM implementation | +| `lime` | LIME explainer | +| `scipy` | Required by LIME internals | +| `matplotlib` | Probability chart generation | +| `Pillow` (via torchvision) | Image I/O | +| `scikit-image` (via lime) | `mark_boundaries` for LIME visualisation | + +## Running in Development + +```bash +uv run python dashboard.py +``` + +Flask runs in debug mode with auto-reload on port 5000. The `_xai_cache/` directory can be deleted at any time to force recomputation of all explanations. diff --git a/README.md b/README.md index e69de29..33904dc 100644 --- a/README.md +++ b/README.md @@ -0,0 +1,80 @@ +# Wildlife Monitoring Dashboard — Nationaal Park De Hoge Veluwe + +A real-time wildlife camera-trap monitoring dashboard powered by an EfficientNet V2-S classifier and multiple Explainable AI (XAI) methods. Built for the 0HM340 Human-AI Interaction assignment. + +The dashboard simulates a live feed of camera-trap detections across five locations in [Nationaal Park De Hoge Veluwe](https://hogeveluwe.nl/), classifying images into seven species (bear, deer, fox, hare, moose, person, wolf) and providing interactive explanations of each prediction. + +## Prerequisites + +- **Python 3.13+** +- **[uv](https://docs.astral.sh/uv/)** package manager +- **CUDA GPU** (optional, falls back to CPU) + +## Quick Start + +### 1. Clone the repository + +```bash +git clone +cd assignment-HAI +``` + +### 2. Obtain the model weights + +The trained model weights are not included in the repository due to their size. Place the file in the project root: + +| File | Size | Description | +|---|---|---| +| `efficientnet_v2_wild_forest_animals.pt` | ~78 MB | Fine-tuned EfficientNet V2-S weights | + +The model was fine-tuned in `final.ipynb`. + +The **dataset** (`wild-forest-animals-and-person-1/`) is downloaded automatically from [Roboflow](https://roboflow.com/) on first launch if not already present on disk. + +### 3. Install dependencies + +```bash +uv sync +``` + +### 4. Run the dashboard + +```bash +uv run python dashboard.py +``` + +The server starts at **http://localhost:5000**. + +## Project Structure + +``` +assignment-HAI/ +├── dashboard.py # Flask application (main entry point) +├── final.ipynb # Training, evaluation, and XAI notebook +├── map.webp # Park map background image +├── pyproject.toml # Project metadata and dependencies +├── uv.lock # Locked dependency versions +├── .python-version # Python version pin (3.13) +├── .gitignore +├── README.md # This file +├── USER_GUIDE.md # End-user documentation +├── DEVELOPER_GUIDE.md # Developer documentation +├── efficientnet_v2_wild_forest_animals.pt # Model weights (not in git) +└── wild-forest-animals-and-person-1/ # Dataset (not in git) + ├── train/ + ├── test/ + └── valid/ +``` + +## XAI Methods + +| Method | What it shows | +|---|---| +| **ScoreCAM** | Gradient-free saliency heatmap highlighting regions the model attends to | +| **LIME** | Superpixel-based explanations for the top-2 predicted classes | +| **Contrastive LIME** | Highlights which regions distinguish the top-1 prediction from the top-2 | +| **Nearest Neighbours** | Three training images most similar in feature space (cosine similarity) | + +## License + +Academic project — 0HM340 Human-AI Interaction, TU/e. diff --git a/USER_GUIDE.md b/USER_GUIDE.md new file mode 100644 index 0000000..a7f01d6 --- /dev/null +++ b/USER_GUIDE.md @@ -0,0 +1,89 @@ +# User Guide + +This guide explains how to use the Wildlife Monitoring Dashboard as an end user (e.g. a park ranger). + +## Starting the Dashboard + +After setup (see [README.md](README.md)), run: + +```bash +uv run python dashboard.py +``` + +Open **http://localhost:5000** in your browser. + +## Home Page — Live Map + +The home page shows a fullscreen map of Nationaal Park De Hoge Veluwe with five camera locations marked as blue dots. Each dot represents a physical camera trap deployed in the park. + +### Live Camera Feed + +The cameras continuously capture images and send them to the dashboard for classification. When a new image arrives from a camera: + +- The AI model classifies the captured animal in real time. +- A **toast notification** appears at the top of the screen with the detected species, camera location, and confidence level. +- The **camera marker pulses red** briefly to indicate activity at that location. +- The detection is added to the **sidebar**. + +### Wolf Warnings + +When a wolf is detected, a **pulsing red circle** appears around the camera location on the map as an urgent alert. The warning automatically fades after 10 seconds. This gives rangers immediate awareness of wolf activity in the park. + +### Sidebar + +Click the toggle button (top-right) to open or close the sidebar. It contains: + +- **Detection count** — total detections received during the current session. +- **Species bar chart** — shows how many of each species have been detected. Click any species bar to overlay a heatmap on the map showing where that species has been spotted across the park. Click again to dismiss. This helps identify which areas of the park are frequented by specific animals. +- **Filter buttons** — filter the detection list by verification status: + - **All** — show everything + - **Unverified** — detections not yet reviewed by a ranger + - **Verified** — confirmed correct by a ranger + - **Corrected** — detections where a ranger changed the predicted class +- **Detection list** — scrollable list of all detections, newest first. Click any detection to view its detail page. + +### Camera Pages + +Click a **camera marker** on the map to view that camera's feed page, which shows a grid of all recent detections from that location with thumbnails, predicted class, confidence, and verification status. Click any card to go to the detection detail page. + +## Detection Detail Page + +Accessible by clicking a detection from the sidebar or a camera page. + +### Verification Bar + +At the top, a verification bar lets you review the AI's classification: + +- **Correct** — confirms the model's prediction is accurate. +- **Wrong** — opens a dropdown to select the true species. The detection is then marked as "manually corrected" and the predicted class is updated. + +Once verified, the bar shows the outcome (verified correct or manually corrected). Verification status is visible throughout the dashboard (sidebar badges, camera page cards). + +### XAI Carousel + +The main area shows a single image viewer with left/right navigation (arrow buttons or keyboard arrow keys). These visualisations help you understand *why* the AI made its prediction: + +| Slide | Description | +|---|---| +| **Original Image** | The raw camera-trap capture | +| **ScoreCAM** | Heatmap overlay showing which regions of the image the model focused on | +| **LIME (top-1 class)** | Superpixel explanation for the most likely class — green regions support the prediction, red regions oppose it | +| **LIME (top-2 class)** | Same explanation for the second most likely class | +| **Contrastive** | Blue regions support the top-1 class over the top-2; red regions support the top-2 over the top-1 | +| **Nearest Neighbour 1–3** | The three most similar images from the training set, with species label and cosine similarity score | + +Navigation dots at the bottom let you jump to any slide directly. + +### Probability Chart + +On the right side (always visible), a bar chart shows the model's predicted probability for each of the seven classes. The predicted class is highlighted in orange. + +Below the chart, the predicted class, confidence percentage, and perplexity are displayed. Lower perplexity means the model is more certain. + +## Interpreting XAI Results + +- **ScoreCAM**: If the highlighted region corresponds to the animal in the image, the model is likely using relevant features. If it highlights background or irrelevant areas, the prediction may be unreliable. +- **LIME**: Green superpixels are evidence *for* the class, red superpixels are evidence *against*. A good prediction should show green on the animal. +- **Contrastive**: Helps understand *why* the model chose one class over another. Useful when the top two predictions are close in confidence. +- **Nearest Neighbours**: If the neighbours are visually similar and of the same species, the model's internal representation is coherent. Mismatched neighbours may indicate confusion. +- **Perplexity**: A value close to 1.0 means the model is very confident. Higher values (e.g. 3–7) indicate uncertainty across multiple classes. diff --git a/dashboard.py b/dashboard.py index 2d4e310..7d1ab65 100644 --- a/dashboard.py +++ b/dashboard.py @@ -100,6 +100,17 @@ _eval_tf = transforms.Compose([ transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), ]) +# ── dataset auto-download ────────────────────────────────────────────────── + +if not DATASET_DIR.exists(): + print("Dataset not found -- downloading from Roboflow ...") + from roboflow import Roboflow + rf = Roboflow(api_key="VCZWezdoCHQz7juipBdt") + project = rf.workspace("forestanimals").project("wild-forest-animals-and-person") + project.version(1).download("multiclass") + if not DATASET_DIR.exists(): + raise RuntimeError(f"Download finished but {DATASET_DIR} not found.") + # ── model ───────────────────────────────────────────────────────────────────── print("Loading model and data …") @@ -486,13 +497,14 @@ body{font-family:system-ui,-apple-system,sans-serif;overflow:hidden;height:100vh .heatmap-dot{ position:absolute;border-radius:50%;pointer-events:none;z-index:4; transform:translate(-50%,-50%); - background:radial-gradient(circle,rgba(251,191,36,.55) 0%,rgba(251,191,36,.15) 60%,transparent 100%); + background:radial-gradient(circle,rgba(251,191,36,.75) 0%,rgba(251,191,36,.35) 50%,rgba(251,191,36,.08) 80%,transparent 100%); + border:2px solid rgba(251,191,36,.6); animation:heatFade .5s ease; } .heatmap-label{ position:absolute;transform:translate(-50%,0);z-index:6;pointer-events:none; - color:#fbbf24;font-size:11px;font-weight:700;white-space:nowrap; - text-shadow:0 1px 4px rgba(0,0,0,.7); + color:#fbbf24;font-size:12px;font-weight:700;white-space:nowrap; + text-shadow:0 1px 6px rgba(0,0,0,.9),0 0 3px rgba(0,0,0,.6); } @keyframes heatFade{from{opacity:0;transform:translate(-50%,-50%) scale(.5)}to{opacity:1;transform:translate(-50%,-50%) scale(1)}}