Stable Diffusion: RunPod model catalog - Configurable model catalog (name -> model filename -> download sources, incl. companion files) in the RunPod settings block. - When the ComfyUI URL points at the runpod-lazy proxy, the Model dropdown lists catalog entries instead of querying the backend; selecting one pushes the catalog + active model to the proxy, which downloads exactly that set (re-provisioning a running pod on change). - homelab/: proxy v3 (catalog-driven provisioning, workflow filename matching on wake, /lazy/catalog endpoint).
| @@ -1,23 +1,25 @@ | |||
| 1 | #!/usr/bin/env python3 | 1 | #!/usr/bin/env python3 |
| 2 | """ComfyUI lazy proxy for a volumeless RunPod on-demand pod. | 2 | """ComfyUI lazy proxy for a volumeless RunPod on-demand pod (catalog edition). |
| 3 | 3 | ||
| 4 | SillyTavern's fallback chain points at this proxy as a normal ComfyUI server; | 4 | SillyTavern's fallback chain points at this proxy as a normal ComfyUI server. |
| 5 | the pod downloads its models at boot (boot-models.py in the image), so no | 5 | Models come from a CATALOG pushed by the ST UI: entries of |
| 6 | network volume or region lock is needed. | 6 | {"name": display, "value": model filename, "files": [{"dest": "unet/x.gguf", "url": "..."}]} |
| 7 | 7 | The pod downloads exactly the selected entry's files at boot (boot-models.py | |
| 8 | Control API (CORS-enabled, used by the ST warmup UI and the CLI script): | 8 | reads MODEL_MANIFEST). Changing the active model with a running pod recreates |
| 9 | GET /lazy/status -> {"state": "red"|"orange"|"green", ...} | 9 | it (fresh disk = old model gone, only the new one downloads). |
| 10 | POST /lazy/warmup -> start pod with MODELS=all (idempotent) | 10 | |
| 11 | Control API (CORS-enabled; ST warmup UI + runpod-pod.sh CLI): | ||
| 12 | GET /lazy/status -> {"state": red|orange|green, "model": ..., ...} | ||
| 13 | POST /lazy/warmup -> start pod for the active catalog entry | ||
| 11 | POST /lazy/shutdown -> terminate pod now | 14 | POST /lazy/shutdown -> terminate pod now |
| 12 | POST /lazy/ping -> keepalive (extends idle timer; never starts a pod) | 15 | POST /lazy/ping -> keepalive (extends idle timer; never starts a pod) |
| 16 | POST /lazy/catalog -> {"models": [...], "active": "<value>"} store catalog; | ||
| 17 | re-provision the pod if the active model changed | ||
| 13 | 18 | ||
| 14 | Wake semantics: a ComfyUI /prompt POST arriving with no pod starts one that | 19 | Wake semantics: a ComfyUI /prompt arriving with no ready pod matches the |
| 15 | downloads only the model group the workflow references (flux/qwen/all). | 20 | workflow against catalog filenames and provisions for that entry (falls back |
| 16 | The idle timer lives HERE (NAS side): IDLE_SECONDS after the last activity | 21 | to the active entry). The idle timer lives here (NAS side): IDLE_SECONDS after |
| 17 | signal (comfy traffic or keepalive ping) the pod is terminated - regardless | 22 | the last activity signal the pod is terminated, frontend or no frontend. |
| 18 | of whether any frontend still exists. | ||
| 19 | |||
| 20 | stdlib only; runs on python:3.12-alpine. | ||
| 21 | """ | 23 | """ |
| 22 | import json | 24 | import json |
| 23 | import os | 25 | import os |
| @@ -30,7 +32,7 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer | |||
| 30 | RUNPOD_KEY = os.environ['RUNPOD_KEY'] | 32 | RUNPOD_KEY = os.environ['RUNPOD_KEY'] |
| 31 | LISTEN_PORT = int(os.environ.get('LISTEN_PORT', '8189')) | 33 | LISTEN_PORT = int(os.environ.get('LISTEN_PORT', '8189')) |
| 32 | IDLE_SECONDS = int(os.environ.get('IDLE_SECONDS', '900')) | 34 | IDLE_SECONDS = int(os.environ.get('IDLE_SECONDS', '900')) |
| 33 | DATACENTERS = [d for d in os.environ.get('DATACENTERS', '').split(',') if d] # empty = any | 35 | DATACENTERS = [d for d in os.environ.get('DATACENTERS', '').split(',') if d] |
| 34 | CLOUD_TYPE = os.environ.get('CLOUD_TYPE', 'SECURE') | 36 | CLOUD_TYPE = os.environ.get('CLOUD_TYPE', 'SECURE') |
| 35 | IMAGE = os.environ.get('IMAGE', 'ghcr.io/permissionbrick/comfyui-runpod-worker:latest') | 37 | IMAGE = os.environ.get('IMAGE', 'ghcr.io/permissionbrick/comfyui-runpod-worker:latest') |
| 36 | GPU_TYPES = os.environ.get('GPU_TYPES', 'NVIDIA RTX 6000 Ada Generation,NVIDIA L40S,NVIDIA L40,NVIDIA RTX A6000,NVIDIA A40').split(',') | 38 | GPU_TYPES = os.environ.get('GPU_TYPES', 'NVIDIA RTX 6000 Ada Generation,NVIDIA L40S,NVIDIA L40,NVIDIA RTX A6000,NVIDIA A40').split(',') |
| @@ -39,13 +41,11 @@ START_TIMEOUT = int(os.environ.get('START_TIMEOUT', '1500')) | |||
| 39 | HF_TOKEN = os.environ.get('HF_TOKEN', '') | 41 | HF_TOKEN = os.environ.get('HF_TOKEN', '') |
| 40 | COMFY_ARGS = os.environ.get('COMFY_ARGS', '--listen 0.0.0.0 --port 8188 --use-pytorch-cross-attention') | 42 | COMFY_ARGS = os.environ.get('COMFY_ARGS', '--listen 0.0.0.0 --port 8188 --use-pytorch-cross-attention') |
| 41 | CACHE_FILE = os.environ.get('CACHE_FILE', '/app/system_stats.cache') | 43 | CACHE_FILE = os.environ.get('CACHE_FILE', '/app/system_stats.cache') |
| 42 | 44 | CATALOG_FILE = os.environ.get('CATALOG_FILE', '/app/catalog.json') | |
| 43 | # Model-group detection: filename fragment (lowercased) -> group | ||
| 44 | GROUP_HINTS = {'flux-2-klein': 'flux', 'flux2-klein': 'flux', 'flux2-vae': 'flux', 'qwen-rapid': 'qwen'} | ||
| 45 | 45 | ||
| 46 | state = { | 46 | state = { |
| 47 | 'pod_id': None, # current pod id (may still be booting) | 47 | 'pod_id': None, |
| 48 | 'models': None, # MODELS value the pod was created with | 48 | 'model': None, # catalog `value` the pod was provisioned for |
| 49 | 'phase': 'red', # red | orange | green | 49 | 'phase': 'red', # red | orange | green |
| 50 | 'gpu': None, | 50 | 'gpu': None, |
| 51 | 'since': time.time(), | 51 | 'since': time.time(), |
| @@ -58,6 +58,32 @@ def log(*args): | |||
| 58 | print(time.strftime('%H:%M:%S'), *args, flush=True) | 58 | print(time.strftime('%H:%M:%S'), *args, flush=True) |
| 59 | 59 | ||
| 60 | 60 | ||
| 61 | def load_catalog(): | ||
| 62 | try: | ||
| 63 | with open(CATALOG_FILE) as f: | ||
| 64 | return json.load(f) | ||
| 65 | except Exception: | ||
| 66 | return {'models': [], 'active': None} | ||
| 67 | |||
| 68 | |||
| 69 | def save_catalog(catalog): | ||
| 70 | with open(CATALOG_FILE, 'w') as f: | ||
| 71 | json.dump(catalog, f, indent=2) | ||
| 72 | |||
| 73 | |||
| 74 | def catalog_entry(value): | ||
| 75 | if not value: | ||
| 76 | return None | ||
| 77 | for entry in load_catalog().get('models', []): | ||
| 78 | if entry.get('value') == value: | ||
| 79 | return entry | ||
| 80 | return None | ||
| 81 | |||
| 82 | |||
| 83 | def active_value(): | ||
| 84 | return load_catalog().get('active') | ||
| 85 | |||
| 86 | |||
| 61 | def api(method, path, body=None): | 87 | def api(method, path, body=None): |
| 62 | req = urllib.request.Request( | 88 | req = urllib.request.Request( |
| 63 | f'https://rest.runpod.io/v1{path}', | 89 | f'https://rest.runpod.io/v1{path}', |
| @@ -73,13 +99,20 @@ def find_pod(): | |||
| 73 | try: | 99 | try: |
| 74 | for pod in api('GET', '/pods'): | 100 | for pod in api('GET', '/pods'): |
| 75 | if pod.get('name') == POD_NAME and pod.get('desiredStatus') == 'RUNNING': | 101 | if pod.get('name') == POD_NAME and pod.get('desiredStatus') == 'RUNNING': |
| 76 | return pod['id'], (pod.get('env') or {}).get('MODELS'), pod.get('machine', {}).get('gpuTypeId') | 102 | env = pod.get('env') or {} |
| 103 | return pod['id'], env.get('MODEL_KEY'), pod.get('machine', {}).get('gpuTypeId') | ||
| 77 | except Exception as err: | 104 | except Exception as err: |
| 78 | log('find_pod failed:', err) | 105 | log('find_pod failed:', err) |
| 79 | return None, None, None | 106 | return None, None, None |
| 80 | 107 | ||
| 81 | 108 | ||
| 82 | def create_pod(models): | 109 | def create_pod(value): |
| 110 | entry = catalog_entry(value) | ||
| 111 | env = {'HF_TOKEN': HF_TOKEN, 'MODEL_KEY': value or ''} | ||
| 112 | if entry and entry.get('files'): | ||
| 113 | env['MODEL_MANIFEST'] = json.dumps(entry['files']) | ||
| 114 | else: | ||
| 115 | env['MODELS'] = 'all' # legacy fallback when no catalog is configured | ||
| 83 | body = { | 116 | body = { |
| 84 | 'name': POD_NAME, | 117 | 'name': POD_NAME, |
| 85 | 'imageName': IMAGE, | 118 | 'imageName': IMAGE, |
| @@ -88,13 +121,13 @@ def create_pod(models): | |||
| 88 | 'cloudType': CLOUD_TYPE, | 121 | 'cloudType': CLOUD_TYPE, |
| 89 | 'containerDiskInGb': 80, | 122 | 'containerDiskInGb': 80, |
| 90 | 'ports': ['8188/http'], | 123 | 'ports': ['8188/http'], |
| 91 | 'env': {'MODELS': models, 'HF_TOKEN': HF_TOKEN}, | 124 | 'env': env, |
| 92 | 'dockerStartCmd': ['bash', '-c', f'python3 /boot-models.py && cd /comfyui && exec python main.py {COMFY_ARGS}'], | 125 | 'dockerStartCmd': ['bash', '-c', f'python3 /boot-models.py && cd /comfyui && exec python main.py {COMFY_ARGS}'], |
| 93 | } | 126 | } |
| 94 | if DATACENTERS: | 127 | if DATACENTERS: |
| 95 | body['dataCenterIds'] = DATACENTERS | 128 | body['dataCenterIds'] = DATACENTERS |
| 96 | pod = api('POST', '/pods', body) | 129 | pod = api('POST', '/pods', body) |
| 97 | log(f"pod created: {pod['id']} MODELS={models} {pod.get('machine', {}).get('gpuTypeId')} ${pod.get('costPerHr')}/hr") | 130 | log(f"pod created: {pod['id']} model={value or 'legacy-all'} {pod.get('machine', {}).get('gpuTypeId')} ${pod.get('costPerHr')}/hr") |
| 98 | return pod['id'], pod.get('machine', {}).get('gpuTypeId') | 131 | return pod['id'], pod.get('machine', {}).get('gpuTypeId') |
| 99 | 132 | ||
| 100 | 133 | ||
| @@ -115,35 +148,33 @@ def upstream_ready(pod_id): | |||
| 115 | return False | 148 | return False |
| 116 | 149 | ||
| 117 | 150 | ||
| 118 | def groups_cover(have, need): | 151 | def terminate(pod_id): |
| 119 | if not have: | 152 | try: |
| 120 | return False | 153 | api('DELETE', f'/pods/{pod_id}') |
| 121 | have_set = set(have.split(',')) | 154 | log('pod terminated:', pod_id) |
| 122 | need_set = set(need.split(',')) | 155 | except Exception as err: |
| 123 | return 'all' in have_set or need_set <= have_set | 156 | log('terminate failed:', err) |
| 124 | 157 | ||
| 125 | 158 | ||
| 126 | def ensure_pod(models='all', wait=True): | 159 | def ensure_pod(value=None, wait=True): |
| 127 | """Ensures a pod covering the requested model groups; optionally waits for readiness.""" | 160 | """Ensures a pod provisioned for catalog entry `value` (None = active entry).""" |
| 161 | value = value or active_value() | ||
| 128 | with state['lock']: | 162 | with state['lock']: |
| 129 | pod_id, have, gpu = (state['pod_id'], state['models'], state['gpu']) | 163 | pod_id, have, gpu = (state['pod_id'], state['model'], state['gpu']) |
| 130 | if not pod_id: | 164 | if not pod_id: |
| 131 | pod_id, have, gpu = find_pod() | 165 | pod_id, have, gpu = find_pod() |
| 132 | if pod_id and not groups_cover(have, models): | 166 | if pod_id and value and have != value: |
| 133 | log(f'pod has MODELS={have}, need {models} - recreating with all') | 167 | log(f'pod has model={have}, need {value} - recreating') |
| 134 | try: | 168 | terminate(pod_id) |
| 135 | api('DELETE', f'/pods/{pod_id}') | ||
| 136 | except Exception as err: | ||
| 137 | log('delete failed:', err) | ||
| 138 | pod_id, have, gpu = None, None, None | 169 | pod_id, have, gpu = None, None, None |
| 139 | models = 'all' | ||
| 140 | if not pod_id: | 170 | if not pod_id: |
| 141 | state['phase'] = 'orange' | 171 | state['phase'] = 'orange' |
| 142 | state['since'] = time.time() | 172 | state['since'] = time.time() |
| 143 | last_err = None | 173 | last_err = None |
| 144 | for attempt in range(3): | 174 | for attempt in range(3): |
| 145 | try: | 175 | try: |
| 146 | pod_id, gpu = create_pod(models) | 176 | pod_id, gpu = create_pod(value) |
| 177 | have = value | ||
| 147 | break | 178 | break |
| 148 | except urllib.error.HTTPError as err: | 179 | except urllib.error.HTTPError as err: |
| 149 | last_err = err.read().decode()[:200] | 180 | last_err = err.read().decode()[:200] |
| @@ -152,8 +183,7 @@ def ensure_pod(models='all', wait=True): | |||
| 152 | if not pod_id: | 183 | if not pod_id: |
| 153 | state['phase'] = 'red' | 184 | state['phase'] = 'red' |
| 154 | raise RuntimeError(f'could not create pod: {last_err}') | 185 | raise RuntimeError(f'could not create pod: {last_err}') |
| 155 | have = models | 186 | state['pod_id'], state['model'], state['gpu'] = pod_id, have, gpu |
| 156 | state['pod_id'], state['models'], state['gpu'] = pod_id, have, gpu | ||
| 157 | state['last'] = time.monotonic() | 187 | state['last'] = time.monotonic() |
| 158 | 188 | ||
| 159 | if state['phase'] != 'green': | 189 | if state['phase'] != 'green': |
| @@ -162,12 +192,14 @@ def ensure_pod(models='all', wait=True): | |||
| 162 | return pod_id | 192 | return pod_id |
| 163 | deadline = time.monotonic() + START_TIMEOUT | 193 | deadline = time.monotonic() + START_TIMEOUT |
| 164 | while time.monotonic() < deadline: | 194 | while time.monotonic() < deadline: |
| 195 | # Booting/downloading counts as activity, else the reaper would kill a | ||
| 196 | # warming pod mid-download. | ||
| 197 | state['last'] = time.monotonic() | ||
| 165 | if upstream_ready(pod_id): | 198 | if upstream_ready(pod_id): |
| 166 | if state['phase'] != 'green': | 199 | if state['phase'] != 'green': |
| 167 | state['phase'] = 'green' | 200 | state['phase'] = 'green' |
| 168 | state['since'] = time.time() | 201 | state['since'] = time.time() |
| 169 | log('pod ready:', pod_id) | 202 | log('pod ready:', pod_id) |
| 170 | state['last'] = time.monotonic() | ||
| 171 | return pod_id | 203 | return pod_id |
| 172 | if state['pod_id'] != pod_id: | 204 | if state['pod_id'] != pod_id: |
| 173 | raise RuntimeError('pod replaced while waiting') | 205 | raise RuntimeError('pod replaced while waiting') |
| @@ -175,15 +207,17 @@ def ensure_pod(models='all', wait=True): | |||
| 175 | raise RuntimeError('pod not ready within START_TIMEOUT') | 207 | raise RuntimeError('pod not ready within START_TIMEOUT') |
| 176 | 208 | ||
| 177 | 209 | ||
| 178 | def detect_groups(workflow_bytes): | 210 | def match_workflow_model(workflow_bytes): |
| 211 | """Returns the catalog value whose filename appears in the workflow, else None.""" | ||
| 179 | try: | 212 | try: |
| 180 | text = workflow_bytes.decode('utf-8', 'replace').lower() | 213 | text = workflow_bytes.decode('utf-8', 'replace') |
| 181 | except Exception: | 214 | except Exception: |
| 182 | return 'all' | 215 | return None |
| 183 | found = {group for hint, group in GROUP_HINTS.items() if hint in text} | 216 | for entry in load_catalog().get('models', []): |
| 184 | if len(found) == 1: | 217 | value = entry.get('value') |
| 185 | return found.pop() | 218 | if value and value in text: |
| 186 | return 'all' | 219 | return value |
| 220 | return None | ||
| 187 | 221 | ||
| 188 | 222 | ||
| 189 | def idle_reaper(): | 223 | def idle_reaper(): |
| @@ -194,16 +228,12 @@ def idle_reaper(): | |||
| 194 | pod_id = state['pod_id'] or find_pod()[0] | 228 | pod_id = state['pod_id'] or find_pod()[0] |
| 195 | if pod_id: | 229 | if pod_id: |
| 196 | log(f'idle {int(idle)}s - terminating pod') | 230 | log(f'idle {int(idle)}s - terminating pod') |
| 197 | try: | 231 | terminate(pod_id) |
| 198 | api('DELETE', f'/pods/{pod_id}') | ||
| 199 | except Exception as err: | ||
| 200 | log('terminate failed:', err) | ||
| 201 | if pod_id or state['phase'] != 'red': | 232 | if pod_id or state['phase'] != 'red': |
| 202 | state.update({'pod_id': None, 'models': None, 'gpu': None, 'phase': 'red', 'since': time.time()}) | 233 | state.update({'pod_id': None, 'model': None, 'gpu': None, 'phase': 'red', 'since': time.time()}) |
| 203 | 234 | ||
| 204 | 235 | ||
| 205 | def status_body(): | 236 | def status_body(): |
| 206 | # Refresh green-ness cheaply on demand | ||
| 207 | pod_id = state['pod_id'] | 237 | pod_id = state['pod_id'] |
| 208 | if pod_id and state['phase'] != 'green' and upstream_ready(pod_id): | 238 | if pod_id and state['phase'] != 'green' and upstream_ready(pod_id): |
| 209 | state['phase'] = 'green' | 239 | state['phase'] = 'green' |
| @@ -211,14 +241,15 @@ def status_body(): | |||
| 211 | if not pod_id: | 241 | if not pod_id: |
| 212 | found, have, gpu = find_pod() | 242 | found, have, gpu = find_pod() |
| 213 | if found: | 243 | if found: |
| 214 | state.update({'pod_id': found, 'models': have, 'gpu': gpu}) | 244 | state.update({'pod_id': found, 'model': have, 'gpu': gpu}) |
| 215 | state['phase'] = 'green' if upstream_ready(found) else 'orange' | 245 | state['phase'] = 'green' if upstream_ready(found) else 'orange' |
| 216 | else: | 246 | elif state['phase'] != 'orange': |
| 217 | state['phase'] = 'red' | 247 | state['phase'] = 'red' |
| 218 | return json.dumps({ | 248 | return json.dumps({ |
| 219 | 'state': state['phase'], | 249 | 'state': state['phase'], |
| 220 | 'pod_id': state['pod_id'], | 250 | 'pod_id': state['pod_id'], |
| 221 | 'models': state['models'], | 251 | 'model': state['model'], |
| 252 | 'active': active_value(), | ||
| 222 | 'gpu': state['gpu'], | 253 | 'gpu': state['gpu'], |
| 223 | 'since': state['since'], | 254 | 'since': state['since'], |
| 224 | 'url': pod_url(state['pod_id']) if state['pod_id'] else None, | 255 | 'url': pod_url(state['pod_id']) if state['pod_id'] else None, |
| @@ -258,34 +289,44 @@ class Proxy(BaseHTTPRequestHandler): | |||
| 258 | state['last'] = time.monotonic() | 289 | state['last'] = time.monotonic() |
| 259 | return self._reply(200, b'{"ok": true}') | 290 | return self._reply(200, b'{"ok": true}') |
| 260 | if path == '/lazy/warmup': | 291 | if path == '/lazy/warmup': |
| 261 | threading.Thread(target=self._safe_ensure, args=('all',), daemon=True).start() | 292 | threading.Thread(target=self._safe_ensure, args=(None,), daemon=True).start() |
| 262 | state['phase'] = state['phase'] if state['phase'] == 'green' else 'orange' | 293 | if state['phase'] != 'green': |
| 294 | state['phase'] = 'orange' | ||
| 263 | return self._reply(200, status_body()) | 295 | return self._reply(200, status_body()) |
| 264 | if path == '/lazy/shutdown': | 296 | if path == '/lazy/shutdown': |
| 265 | pod_id = state['pod_id'] or find_pod()[0] | 297 | pod_id = state['pod_id'] or find_pod()[0] |
| 266 | if pod_id: | 298 | if pod_id: |
| 299 | terminate(pod_id) | ||
| 300 | state.update({'pod_id': None, 'model': None, 'gpu': None, 'phase': 'red', 'since': time.time()}) | ||
| 301 | return self._reply(200, status_body()) | ||
| 302 | if path == '/lazy/catalog': | ||
| 267 | try: | 303 | try: |
| 268 | api('DELETE', f'/pods/{pod_id}') | 304 | payload = json.loads(body or b'{}') |
| 269 | except Exception as err: | 305 | catalog = {'models': payload.get('models', []), 'active': payload.get('active')} |
| 270 | return self._reply(500, json.dumps({'error': str(err)}).encode()) | 306 | save_catalog(catalog) |
| 271 | state.update({'pod_id': None, 'models': None, 'gpu': None, 'phase': 'red', 'since': time.time()}) | 307 | log(f"catalog updated: {len(catalog['models'])} models, active={catalog['active']}") |
| 308 | # Re-provision proactively when the active model changed under a live pod. | ||
| 309 | if state['pod_id'] and catalog['active'] and state['model'] != catalog['active']: | ||
| 310 | log('active model changed - re-provisioning pod') | ||
| 311 | threading.Thread(target=self._safe_ensure, args=(catalog['active'],), daemon=True).start() | ||
| 272 | return self._reply(200, status_body()) | 312 | return self._reply(200, status_body()) |
| 313 | except Exception as err: | ||
| 314 | return self._reply(400, json.dumps({'error': str(err)}).encode()) | ||
| 273 | 315 | ||
| 274 | # ---- comfy proxying ---- | 316 | # ---- comfy proxying ---- |
| 275 | state['last'] = time.monotonic() | 317 | state['last'] = time.monotonic() |
| 276 | pod_id = state['pod_id'] | 318 | pod_id = state['pod_id'] |
| 277 | ready = pod_id and upstream_ready(pod_id) | 319 | ready = pod_id and upstream_ready(pod_id) |
| 278 | if not ready and self.command == 'GET' and path == '/system_stats': | 320 | if not ready and self.command == 'GET' and path == '/system_stats': |
| 279 | # Reachability probe while cold: answer from cache; do NOT wake. | ||
| 280 | try: | 321 | try: |
| 281 | with open(CACHE_FILE, 'rb') as f: | 322 | with open(CACHE_FILE, 'rb') as f: |
| 282 | return self._reply(200, f.read()) | 323 | return self._reply(200, f.read()) |
| 283 | except FileNotFoundError: | 324 | except FileNotFoundError: |
| 284 | return self._reply(200, json.dumps({'system': {'comfyui_version': 'cold'}, 'devices': []}).encode()) | 325 | return self._reply(200, json.dumps({'system': {'comfyui_version': 'cold'}, 'devices': []}).encode()) |
| 285 | if not ready: | 326 | if not ready or (self.command == 'POST' and path == '/prompt'): |
| 286 | models = detect_groups(body) if (self.command == 'POST' and path == '/prompt' and body) else 'all' | 327 | value = match_workflow_model(body) if (self.command == 'POST' and path == '/prompt' and body) else None |
| 287 | try: | 328 | try: |
| 288 | pod_id = ensure_pod(models) | 329 | pod_id = ensure_pod(value) |
| 289 | except Exception as err: | 330 | except Exception as err: |
| 290 | log('ensure_pod failed:', err) | 331 | log('ensure_pod failed:', err) |
| 291 | return self._reply(502, json.dumps({'error': str(err)}).encode()) | 332 | return self._reply(502, json.dumps({'error': str(err)}).encode()) |
| @@ -304,11 +345,11 @@ class Proxy(BaseHTTPRequestHandler): | |||
| 304 | finally: | 345 | finally: |
| 305 | state['last'] = time.monotonic() | 346 | state['last'] = time.monotonic() |
| 306 | 347 | ||
| 307 | def _safe_ensure(self, models): | 348 | def _safe_ensure(self, value): |
| 308 | try: | 349 | try: |
| 309 | ensure_pod(models) | 350 | ensure_pod(value) |
| 310 | except Exception as err: | 351 | except Exception as err: |
| 311 | log('warmup failed:', err) | 352 | log('provision failed:', err) |
| 312 | 353 | ||
| 313 | do_GET = _handle | 354 | do_GET = _handle |
| 314 | do_POST = _handle | 355 | do_POST = _handle |
| @@ -317,7 +358,7 @@ class Proxy(BaseHTTPRequestHandler): | |||
| 317 | def main(): | 358 | def main(): |
| 318 | threading.Thread(target=idle_reaper, daemon=True).start() | 359 | threading.Thread(target=idle_reaper, daemon=True).start() |
| 319 | server = ThreadingHTTPServer(('0.0.0.0', LISTEN_PORT), Proxy) | 360 | server = ThreadingHTTPServer(('0.0.0.0', LISTEN_PORT), Proxy) |
| 320 | log(f'runpod-lazy v2 on :{LISTEN_PORT} (pod "{POD_NAME}", DCs {DATACENTERS or "any"}, idle {IDLE_SECONDS}s)') | 361 | log(f'runpod-lazy v3 (catalog) on :{LISTEN_PORT} (pod "{POD_NAME}", DCs {DATACENTERS or "any"}, idle {IDLE_SECONDS}s)') |
| 321 | server.serve_forever() | 362 | server.serve_forever() |
| 322 | 363 | ||
| 323 | 364 | ||
| @@ -376,6 +376,9 @@ const defaultSettings = { | |||
| 376 | // RunPod lazy-pod proxy base URL ('' = feature off) | 376 | // RunPod lazy-pod proxy base URL ('' = feature off) |
| 377 | runpod_lazy_url: '', | 377 | runpod_lazy_url: '', |
| 378 | 378 | ||
| 379 | // RunPod model catalog ({ name, value, downloads } entries) | ||
| 380 | runpod_models: [], | ||
| 381 | |||
| 379 | // Dedicated LLM connection profile for image-prompt generation ('' = use active model) | 382 | // Dedicated LLM connection profile for image-prompt generation ('' = use active model) |
| 380 | prompt_generation_profile: '', | 383 | prompt_generation_profile: '', |
| 381 | 384 | ||
| @@ -396,6 +399,7 @@ const PRESET_EXCLUDE_KEYS = [ | |||
| 396 | 'ref_images_enabled', | 399 | 'ref_images_enabled', |
| 397 | 'ref_images', | 400 | 'ref_images', |
| 398 | 'runpod_lazy_url', | 401 | 'runpod_lazy_url', |
| 402 | 'runpod_models', | ||
| 399 | // The image-prompt LLM profile is independent of the image backend, so it must | 403 | // The image-prompt LLM profile is independent of the image backend, so it must |
| 400 | // never be captured/swapped by image-generation presets or the fallback retry. | 404 | // never be captured/swapped by image-generation presets or the fallback retry. |
| 401 | 'prompt_generation_profile', | 405 | 'prompt_generation_profile', |
| @@ -713,6 +717,10 @@ async function loadSettings() { | |||
| 713 | extension_settings.sd.ref_images = []; | 717 | extension_settings.sd.ref_images = []; |
| 714 | } | 718 | } |
| 715 | 719 | ||
| 720 | if (!Array.isArray(extension_settings.sd.runpod_models)) { | ||
| 721 | extension_settings.sd.runpod_models = []; | ||
| 722 | } | ||
| 723 | |||
| 716 | if (!Array.isArray(extension_settings.sd.custom_entries)) { | 724 | if (!Array.isArray(extension_settings.sd.custom_entries)) { |
| 717 | extension_settings.sd.custom_entries = []; | 725 | extension_settings.sd.custom_entries = []; |
| 718 | } | 726 | } |
| @@ -787,7 +795,9 @@ async function loadSettings() { | |||
| 787 | $('#sd_runpod_lazy_url').val(extension_settings.sd.runpod_lazy_url ?? ''); | 795 | $('#sd_runpod_lazy_url').val(extension_settings.sd.runpod_lazy_url ?? ''); |
| 788 | renderPresetChain(); | 796 | renderPresetChain(); |
| 789 | renderRefImages(); | 797 | renderRefImages(); |
| 798 | renderRunpodModels(); | ||
| 790 | setupRunpodLoops(); | 799 | setupRunpodLoops(); |
| 800 | pushRunpodCatalog(); | ||
| 791 | 801 | ||
| 792 | for (const style of extension_settings.sd.styles) { | 802 | for (const style of extension_settings.sd.styles) { |
| 793 | const option = document.createElement('option'); | 803 | const option = document.createElement('option'); |
| @@ -1594,6 +1604,122 @@ function onRunpodLazyUrlInput() { | |||
| 1594 | setupRunpodLoops(); | 1604 | setupRunpodLoops(); |
| 1595 | } | 1605 | } |
| 1596 | 1606 | ||
| 1607 | /** | ||
| 1608 | * Whether a comfy URL points at the runpod-lazy proxy. | ||
| 1609 | * @param {string} url ComfyUI URL to test. | ||
| 1610 | * @returns {boolean} True when it equals the configured proxy URL. | ||
| 1611 | */ | ||
| 1612 | function isRunpodProxyUrl(url) { | ||
| 1613 | const lazy = getRunpodLazyUrl(); | ||
| 1614 | return !!lazy && String(url ?? '').trim().replace(/\/$/, '') === lazy; | ||
| 1615 | } | ||
| 1616 | |||
| 1617 | /** | ||
| 1618 | * Returns the configured RunPod model catalog entries that have a filename. | ||
| 1619 | * @returns {{name: string, value: string, downloads: string}[]} Catalog entries. | ||
| 1620 | */ | ||
| 1621 | function getRunpodCatalog() { | ||
| 1622 | const models = Array.isArray(extension_settings.sd.runpod_models) ? extension_settings.sd.runpod_models : []; | ||
| 1623 | return models.filter(m => m && String(m.value ?? '').trim()); | ||
| 1624 | } | ||
| 1625 | |||
| 1626 | /** | ||
| 1627 | * Parses a catalog entry's download lines ("dest-subpath url" per line). | ||
| 1628 | * @param {string} text Multiline downloads definition. | ||
| 1629 | * @returns {{dest: string, url: string}[]} Parsed file list. | ||
| 1630 | */ | ||
| 1631 | function parseRunpodFiles(text) { | ||
| 1632 | return String(text ?? '').split('\n') | ||
| 1633 | .map(line => line.trim()) | ||
| 1634 | .filter(Boolean) | ||
| 1635 | .map(line => { | ||
| 1636 | const space = line.indexOf(' '); | ||
| 1637 | return space > 0 ? { dest: line.slice(0, space).trim(), url: line.slice(space + 1).trim() } : null; | ||
| 1638 | }) | ||
| 1639 | .filter(x => x && x.dest && x.url); | ||
| 1640 | } | ||
| 1641 | |||
| 1642 | /** | ||
| 1643 | * The model filename saved for whichever config uses the proxy: the live | ||
| 1644 | * settings if they point at it, else the chain entry that does. | ||
| 1645 | * @returns {string|null} Active catalog value. | ||
| 1646 | */ | ||
| 1647 | function getRunpodActiveModel() { | ||
| 1648 | if (isRunpodProxyUrl(extension_settings.sd.comfy_url)) { | ||
| 1649 | return extension_settings.sd.model || null; | ||
| 1650 | } | ||
| 1651 | const chain = Array.isArray(extension_settings.sd.settings_preset_chain) ? extension_settings.sd.settings_preset_chain : []; | ||
| 1652 | const entry = chain.find(e => isRunpodProxyUrl(e?.preset?.comfy_url)); | ||
| 1653 | return entry?.preset?.model ?? null; | ||
| 1654 | } | ||
| 1655 | |||
| 1656 | /** Pushes the model catalog + active selection to the proxy (fire-and-forget). */ | ||
| 1657 | async function pushRunpodCatalog() { | ||
| 1658 | const url = getRunpodLazyUrl(); | ||
| 1659 | if (!url) { | ||
| 1660 | return; | ||
| 1661 | } | ||
| 1662 | const models = getRunpodCatalog().map(m => ({ | ||
| 1663 | name: m.name || m.value, | ||
| 1664 | value: m.value, | ||
| 1665 | files: parseRunpodFiles(m.downloads), | ||
| 1666 | })); | ||
| 1667 | try { | ||
| 1668 | await fetch(`${url}/lazy/catalog`, { | ||
| 1669 | method: 'POST', | ||
| 1670 | headers: { 'Content-Type': 'application/json' }, | ||
| 1671 | signal: AbortSignal.timeout(8000), | ||
| 1672 | body: JSON.stringify({ models, active: getRunpodActiveModel() }), | ||
| 1673 | }); | ||
| 1674 | } catch (error) { | ||
| 1675 | console.warn('SD: runpod catalog push failed', error); | ||
| 1676 | } | ||
| 1677 | } | ||
| 1678 | |||
| 1679 | /** | ||
| 1680 | * Rebuilds the RunPod model catalog editor in the settings UI. | ||
| 1681 | */ | ||
| 1682 | function renderRunpodModels() { | ||
| 1683 | const container = $('#sd_runpod_models_list'); | ||
| 1684 | if (!container.length) { | ||
| 1685 | return; | ||
| 1686 | } | ||
| 1687 | container.empty(); | ||
| 1688 | const models = Array.isArray(extension_settings.sd.runpod_models) ? extension_settings.sd.runpod_models : []; | ||
| 1689 | if (models.length === 0) { | ||
| 1690 | container.append($('<small></small>').text('No models configured.')); | ||
| 1691 | return; | ||
| 1692 | } | ||
| 1693 | models.forEach((model, index) => { | ||
| 1694 | const nameInput = $('<input>').addClass('text_pole flex1').attr({ type: 'text', placeholder: 'Display name' }) | ||
| 1695 | .val(model.name || '') | ||
| 1696 | .on('change', function () { model.name = String($(this).val() ?? '').trim(); saveSettingsDebounced(); pushRunpodCatalog(); }); | ||
| 1697 | const valueInput = $('<input>').addClass('text_pole flex1').attr({ type: 'text', placeholder: 'Model filename (used by %model%)' }) | ||
| 1698 | .val(model.value || '') | ||
| 1699 | .on('change', function () { model.value = String($(this).val() ?? '').trim(); saveSettingsDebounced(); pushRunpodCatalog(); }); | ||
| 1700 | const deleteButton = $('<div></div>').addClass('menu_button menu_button_icon fa-solid fa-trash-can') | ||
| 1701 | .attr('title', 'Remove model') | ||
| 1702 | .on('click', () => { models.splice(index, 1); saveSettingsDebounced(); renderRunpodModels(); pushRunpodCatalog(); }); | ||
| 1703 | const downloadsInput = $('<textarea></textarea>').addClass('text_pole textarea_compact') | ||
| 1704 | .attr({ rows: 2, placeholder: 'One file per line: <models-subpath> <url>\ne.g. unet/model.gguf https://huggingface.co/...' }) | ||
| 1705 | .val(model.downloads || '') | ||
| 1706 | .on('change', function () { model.downloads = String($(this).val() ?? ''); saveSettingsDebounced(); pushRunpodCatalog(); }); | ||
| 1707 | container.append( | ||
| 1708 | $('<div></div>').addClass('flexFlowColumn marginTopBot5') | ||
| 1709 | .append($('<div></div>').addClass('flex-container alignItemsCenter').append(nameInput).append(valueInput).append(deleteButton)) | ||
| 1710 | .append(downloadsInput)); | ||
| 1711 | }); | ||
| 1712 | } | ||
| 1713 | |||
| 1714 | function onRunpodModelsAddClick() { | ||
| 1715 | if (!Array.isArray(extension_settings.sd.runpod_models)) { | ||
| 1716 | extension_settings.sd.runpod_models = []; | ||
| 1717 | } | ||
| 1718 | extension_settings.sd.runpod_models.push({ name: '', value: '', downloads: '' }); | ||
| 1719 | saveSettingsDebounced(); | ||
| 1720 | renderRunpodModels(); | ||
| 1721 | } | ||
| 1722 | |||
| 1597 | // #endregion | 1723 | // #endregion |
| 1598 | 1724 | ||
| 1599 | /** | 1725 | /** |
| @@ -2452,6 +2578,12 @@ async function onModelChange() { | |||
| 2452 | extension_settings.sd.model = selectedModel.val(); | 2578 | extension_settings.sd.model = selectedModel.val(); |
| 2453 | saveSettingsDebounced(); | 2579 | saveSettingsDebounced(); |
| 2454 | 2580 | ||
| 2581 | // Selecting a catalog model while pointed at the RunPod proxy kicks off the | ||
| 2582 | // download / re-provision for it right away. | ||
| 2583 | if (isRunpodProxyUrl(extension_settings.sd.comfy_url)) { | ||
| 2584 | pushRunpodCatalog(); | ||
| 2585 | } | ||
| 2586 | |||
| 2455 | if (extension_settings.sd.model && extension_settings.sd.source === sources.electronhub) { | 2587 | if (extension_settings.sd.model && extension_settings.sd.source === sources.electronhub) { |
| 2456 | const cachedModel = selectedModel.data('model'); | 2588 | const cachedModel = selectedModel.data('model'); |
| 2457 | const models = cachedModel ? [cachedModel] : await loadElectronHubModels(); | 2589 | const models = cachedModel ? [cachedModel] : await loadElectronHubModels(); |
| @@ -3502,6 +3634,11 @@ function loadNovelSchedulers() { | |||
| 3502 | } | 3634 | } |
| 3503 | 3635 | ||
| 3504 | async function loadComfyModels() { | 3636 | async function loadComfyModels() { |
| 3637 | if (isRunpodProxyUrl(extension_settings.sd.comfy_url)) { | ||
| 3638 | // The RunPod pod downloads models on demand: list the configured catalog | ||
| 3639 | // instead of asking the (possibly cold) backend what it has on disk. | ||
| 3640 | return getRunpodCatalog().map(m => ({ value: m.value, text: m.name || m.value })); | ||
| 3641 | } | ||
| 3505 | if (extension_settings.sd.comfy_type === comfyTypes.runpod_serverless) { | 3642 | if (extension_settings.sd.comfy_type === comfyTypes.runpod_serverless) { |
| 3506 | $('#sd_runpod_key').toggleClass('success', !!secret_state[SECRET_KEYS.COMFY_RUNPOD]); | 3643 | $('#sd_runpod_key').toggleClass('success', !!secret_state[SECRET_KEYS.COMFY_RUNPOD]); |
| 3507 | return [ | 3644 | return [ |
| @@ -7045,6 +7182,7 @@ export async function init() { | |||
| 7045 | $('#sd_runpod_lazy_url').on('input', onRunpodLazyUrlInput); | 7182 | $('#sd_runpod_lazy_url').on('input', onRunpodLazyUrlInput); |
| 7046 | $('#sd_runpod_warmup').on('click', () => runpodControl('warmup')); | 7183 | $('#sd_runpod_warmup').on('click', () => runpodControl('warmup')); |
| 7047 | $('#sd_runpod_shutdown').on('click', () => runpodControl('shutdown')); | 7184 | $('#sd_runpod_shutdown').on('click', () => runpodControl('shutdown')); |
| 7185 | $('#sd_runpod_models_add').on('click', onRunpodModelsAddClick); | ||
| 7048 | $('#sd_custom_entry_add').on('click', onAddCustomEntryClick); | 7186 | $('#sd_custom_entry_add').on('click', onAddCustomEntryClick); |
| 7049 | $('#sd_custom_entries_list').on('click', '[data-action]', function () { | 7187 | $('#sd_custom_entries_list').on('click', '[data-action]', function () { |
| 7050 | const id = $(this).attr('data-entry-id'); | 7188 | const id = $(this).attr('data-entry-id'); |
| @@ -94,6 +94,15 @@ | |||
| 94 | </a> | 94 | </a> |
| 95 | </div> | 95 | </div> |
| 96 | <small data-i18n="sd_runpod_small">While a SillyTavern tab is open, a keepalive ping keeps the pod alive; it shuts down 15 minutes after the last activity either way. Red = off, orange = starting/downloading models, green = ready.</small> | 96 | <small data-i18n="sd_runpod_small">While a SillyTavern tab is open, a keepalive ping keeps the pod alive; it shuts down 15 minutes after the last activity either way. Red = off, orange = starting/downloading models, green = ready.</small> |
| 97 | <h5 data-i18n="Model catalog">Model catalog</h5> | ||
| 98 | <small data-i18n="sd_runpod_models_small">When the ComfyUI URL points at the proxy, the Model dropdown lists these entries instead of asking the backend. The pod downloads exactly the selected entry's files at boot; changing the selection re-provisions with the new model. Filename = what "%model%" resolves to; downloads = one "subpath url" per line, including companion files (text encoder, VAE, LoRA).</small> | ||
| 99 | <div id="sd_runpod_models_list" class="flex-container flexFlowColumn marginTopBot5"></div> | ||
| 100 | <div class="flex-container marginTopBot5"> | ||
| 101 | <div id="sd_runpod_models_add" class="menu_button menu_button_icon"> | ||
| 102 | <i class="fa-solid fa-plus"></i> | ||
| 103 | <span data-i18n="Add model">Add model</span> | ||
| 104 | </div> | ||
| 105 | </div> | ||
| 97 | <hr> | 106 | <hr> |
| 98 | <label for="sd_source" data-i18n="Source">Source</label> | 107 | <label for="sd_source" data-i18n="Source">Source</label> |
| 99 | <select id="sd_source" class="text_pole"> | 108 | <select id="sd_source" class="text_pole"> |