Stable Diffusion: RunPod model catalog - Configurable model catalog (name -> model filename -> download sources, incl. companion files) in the RunPod settings block. - When the ComfyUI URL points at the runpod-lazy proxy, the Model dropdown lists catalog entries instead of querying the backend; selecting one pushes the catalog + active model to the proxy, which downloads exactly that set (re-provisioning a running pod on change). - homelab/: proxy v3 (catalog-driven provisioning, workflow filename matching on wake, /lazy/catalog endpoint).

630785ad4c5344a76994566c0b7fc880a116fa74

permissionBRICK <40219477+permissionBRICK@users.noreply.github.com>

Parents: 3d710c4ecae3dbfe73452198a003283a8140b5ba

homelab/runpod-lazy-proxy.py+117 -76 Viewed

 #!/usr/bin/env python3
 """ComfyUI lazy proxy for a volumeless RunPod on-demand pod (catalog edition).
 SillyTavern's fallback chain points at this proxy as a normal ComfyUI server;.
 -the pod downloads its models at boot (boot-models.py in the image), so no
 +Models come from a CATALOG pushed by the ST UI: entries of
 -network volume or region lock is needed.
 +  {"name": display, "value": model filename, "files": [{"dest": "unet/x.gguf", "url": "..."}]}
+-
 +The pod downloads exactly the selected entry's files at boot (boot-models.py
 -Control API (CORS-enabled, used by the ST warmup UI and the CLI script):
 +reads MODEL_MANIFEST). Changing the active model with a running pod recreates
 -  GET  /lazy/status    -> {"state": "red"|"orange"|"green", ...}
 +it (fresh disk = old model gone, only the new one downloads).
 -  POST /lazy/warmup    -> start pod with MODELS=all (idempotent)
++
 +Control API (CORS-enabled; ST warmup UI + runpod-pod.sh CLI):
 +  GET  /lazy/status    -> {"state": red|orange|green, "model": ..., ...}
 +  POST /lazy/warmup    -> start pod for the active catalog entry
   POST /lazy/shutdown  -> terminate pod now
   POST /lazy/ping      -> keepalive (extends idle timer; never starts a pod)
 +  POST /lazy/catalog   -> {"models": [...], "active": "<value>"} store catalog;
 +                          re-provision the pod if the active model changed
 Wake semantics: a ComfyUI /prompt POST arriving with no podready startspod onematches thatthe
 -downloads only the model group the workflow references (flux/qwen/all).
 +workflow against catalog filenames and provisions for that entry (falls back
 to the active entry). The idle timer lives HEREhere (NAS side): IDLE_SECONDS after the last activity
 signal (comfy trafficthe orlast keepaliveactivity ping)signal the pod is terminated, -frontend regardlessor no frontend.
 -of whether any frontend still exists.
+-
 -stdlib only; runs on python:3.12-alpine.
 """
 import json
 import os

 RUNPOD_KEY = os.environ['RUNPOD_KEY']
 LISTEN_PORT = int(os.environ.get('LISTEN_PORT', '8189'))
 IDLE_SECONDS = int(os.environ.get('IDLE_SECONDS', '900'))
 DATACENTERS = [d for d in os.environ.get('DATACENTERS', '').split(',') if d]  # empty = any
 CLOUD_TYPE = os.environ.get('CLOUD_TYPE', 'SECURE')
 IMAGE = os.environ.get('IMAGE', 'ghcr.io/permissionbrick/comfyui-runpod-worker:latest')
 GPU_TYPES = os.environ.get('GPU_TYPES', 'NVIDIA RTX 6000 Ada Generation,NVIDIA L40S,NVIDIA L40,NVIDIA RTX A6000,NVIDIA A40').split(',')

 HF_TOKEN = os.environ.get('HF_TOKEN', '')
 COMFY_ARGS = os.environ.get('COMFY_ARGS', '--listen 0.0.0.0 --port 8188 --use-pytorch-cross-attention')
 CACHE_FILE = os.environ.get('CACHE_FILE', '/app/system_stats.cache')
+-
 +CATALOG_FILE = os.environ.get('CATALOG_FILE', '/app/catalog.json')
 -# Model-group detection: filename fragment (lowercased) -> group
 -GROUP_HINTS = {'flux-2-klein': 'flux', 'flux2-klein': 'flux', 'flux2-vae': 'flux', 'qwen-rapid': 'qwen'}
 state = {
 -    'pod_id': None,     # current pod id (may still be booting)
 +    'pod_id': None,
     'modelsmodel': None,           # MODELScatalog `value` the pod was createdprovisioned withfor
     'phase': 'red',     # red | orange | green
     'gpu': None,
     'since': time.time(),

     print(time.strftime('%H:%M:%S'), *args, flush=True)
 +def load_catalog():
 +    try:
 +        with open(CATALOG_FILE) as f:
 +            return json.load(f)
 +    except Exception:
 +        return {'models': [], 'active': None}
++
++
 +def save_catalog(catalog):
 +    with open(CATALOG_FILE, 'w') as f:
 +        json.dump(catalog, f, indent=2)
++
++
 +def catalog_entry(value):
 +    if not value:
 +        return None
 +    for entry in load_catalog().get('models', []):
 +        if entry.get('value') == value:
 +            return entry
 +    return None
++
++
 +def active_value():
 +    return load_catalog().get('active')
++
++
 def api(method, path, body=None):
     req = urllib.request.Request(
         f'https://rest.runpod.io/v1{path}',

     try:
         for pod in api('GET', '/pods'):
             if pod.get('name') == POD_NAME and pod.get('desiredStatus') == 'RUNNING':
 -                return pod['id'], (pod.get('env') or {}).get('MODELS'), pod.get('machine', {}).get('gpuTypeId')
 +                env = pod.get('env') or {}
 +                return pod['id'], env.get('MODEL_KEY'), pod.get('machine', {}).get('gpuTypeId')
     except Exception as err:
         log('find_pod failed:', err)
     return None, None, None
 def create_pod(modelsvalue):
 +    entry = catalog_entry(value)
 +    env = {'HF_TOKEN': HF_TOKEN, 'MODEL_KEY': value or ''}
 +    if entry and entry.get('files'):
 +        env['MODEL_MANIFEST'] = json.dumps(entry['files'])
 +    else:
 +        env['MODELS'] = 'all'  # legacy fallback when no catalog is configured
     body = {
         'name': POD_NAME,
         'imageName': IMAGE,

         'cloudType': CLOUD_TYPE,
         'containerDiskInGb': 80,
         'ports': ['8188/http'],
 -        'env': {'MODELS': models, 'HF_TOKEN': HF_TOKEN},
 +        'env': env,
         'dockerStartCmd': ['bash', '-c', f'python3 /boot-models.py && cd /comfyui && exec python main.py {COMFY_ARGS}'],
+    }
     if DATACENTERS:
         body['dataCenterIds'] = DATACENTERS
     pod = api('POST', '/pods', body)
     log(f"pod created: {pod['id']} MODELSmodel={modelsvalue or 'legacy-all'} {pod.get('machine', {}).get('gpuTypeId')} ${pod.get('costPerHr')}/hr")
     return pod['id'], pod.get('machine', {}).get('gpuTypeId')

     return False
 def groups_coverterminate(have, needpod_id):
 -    if not have:
 +    try:
 -        return False
 +        api('DELETE', f'/pods/{pod_id}')
 -    have_set = set(have.split(','))
 +        log('pod terminated:', pod_id)
 -    need_set = set(need.split(','))
 +    except Exception as err:
 -    return 'all' in have_set or need_set <= have_set
 +        log('terminate failed:', err)
 def ensure_pod(modelsvalue='all'None, wait=True):
     """Ensures a pod coveringprovisioned thefor requestedcatalog modelentry groups;`value` optionally(None waits= foractive readinessentry)."""
 +    value = value or active_value()
     with state['lock']:
         pod_id, have, gpu = (state['pod_id'], state['modelsmodel'], state['gpu'])
         if not pod_id:
             pod_id, have, gpu = find_pod()
         if pod_id and notvalue groups_cover(and have, models)!= value:
             log(f'pod has MODELSmodel={have}, need {modelsvalue} - recreating with all')
 -            try:
 +            terminate(pod_id)
 -                api('DELETE', f'/pods/{pod_id}')
 -            except Exception as err:
 -                log('delete failed:', err)
             pod_id, have, gpu = None, None, None
 -            models = 'all'
         if not pod_id:
             state['phase'] = 'orange'
             state['since'] = time.time()
             last_err = None
             for attempt in range(3):
                 try:
                     pod_id, gpu = create_pod(modelsvalue)
 +                    have = value
                     break
                 except urllib.error.HTTPError as err:
                     last_err = err.read().decode()[:200]

             if not pod_id:
                 state['phase'] = 'red'
                 raise RuntimeError(f'could not create pod: {last_err}')
 -            have = models
 +        state['pod_id'], state['model'], state['gpu'] = pod_id, have, gpu
 -        state['pod_id'], state['models'], state['gpu'] = pod_id, have, gpu
         state['last'] = time.monotonic()
     if state['phase'] != 'green':

         return pod_id
     deadline = time.monotonic() + START_TIMEOUT
     while time.monotonic() < deadline:
 +        # Booting/downloading counts as activity, else the reaper would kill a
 +        # warming pod mid-download.
 +        state['last'] = time.monotonic()
         if upstream_ready(pod_id):
             if state['phase'] != 'green':
                 state['phase'] = 'green'
                 state['since'] = time.time()
                 log('pod ready:', pod_id)
 -            state['last'] = time.monotonic()
             return pod_id
         if state['pod_id'] != pod_id:
             raise RuntimeError('pod replaced while waiting')

     raise RuntimeError('pod not ready within START_TIMEOUT')
 def detect_groupsmatch_workflow_model(workflow_bytes):
 +    """Returns the catalog value whose filename appears in the workflow, else None."""
     try:
         text = workflow_bytes.decode('utf-8', 'replace').lower()
     except Exception:
         return 'all'None
 -    found = {group for hint, group in GROUP_HINTS.items() if hint in text}
 +    for entry in load_catalog().get('models', []):
 -    if len(found) == 1:
 +        value = entry.get('value')
 -        return found.pop()
 +        if value and value in text:
 -    return 'all'
 +            return value
 +    return None
 def idle_reaper():

             pod_id = state['pod_id'] or find_pod()[0]
             if pod_id:
                 log(f'idle {int(idle)}s - terminating pod')
 -                try:
 +                terminate(pod_id)
 -                    api('DELETE', f'/pods/{pod_id}')
 -                except Exception as err:
 -                    log('terminate failed:', err)
             if pod_id or state['phase'] != 'red':
                 state.update({'pod_id': None, 'modelsmodel': None, 'gpu': None, 'phase': 'red', 'since': time.time()})
 def status_body():
 -    # Refresh green-ness cheaply on demand
     pod_id = state['pod_id']
     if pod_id and state['phase'] != 'green' and upstream_ready(pod_id):
         state['phase'] = 'green'

     if not pod_id:
         found, have, gpu = find_pod()
         if found:
             state.update({'pod_id': found, 'modelsmodel': have, 'gpu': gpu})
             state['phase'] = 'green' if upstream_ready(found) else 'orange'
 -        else:
 +        elif state['phase'] != 'orange':
             state['phase'] = 'red'
     return json.dumps({
         'state': state['phase'],
         'pod_id': state['pod_id'],
         'modelsmodel': state['modelsmodel'],
 +        'active': active_value(),
         'gpu': state['gpu'],
         'since': state['since'],
         'url': pod_url(state['pod_id']) if state['pod_id'] else None,

                 state['last'] = time.monotonic()
             return self._reply(200, b'{"ok": true}')
         if path == '/lazy/warmup':
             threading.Thread(target=self._safe_ensure, args=('all'None,), daemon=True).start()
             state['phase'] = state['phase'] if state['phase'] =!= 'green' else 'orange':
 +                state['phase'] = 'orange'
             return self._reply(200, status_body())
         if path == '/lazy/shutdown':
             pod_id = state['pod_id'] or find_pod()[0]
             if pod_id:
 +                terminate(pod_id)
 +            state.update({'pod_id': None, 'model': None, 'gpu': None, 'phase': 'red', 'since': time.time()})
 +            return self._reply(200, status_body())
 +        if path == '/lazy/catalog':
             try:
 -                    api('DELETE', f'/pods/{pod_id}')
 +                payload = json.loads(body or b'{}')
 -                except Exception as err:
 +                catalog = {'models': payload.get('models', []), 'active': payload.get('active')}
 -                    return self._reply(500, json.dumps({'error': str(err)}).encode())
 +                save_catalog(catalog)
 -            state.update({'pod_id': None, 'models': None, 'gpu': None, 'phase': 'red', 'since': time.time()})
 +                log(f"catalog updated: {len(catalog['models'])} models, active={catalog['active']}")
 +                # Re-provision proactively when the active model changed under a live pod.
 +                if state['pod_id'] and catalog['active'] and state['model'] != catalog['active']:
 +                    log('active model changed - re-provisioning pod')
 +                    threading.Thread(target=self._safe_ensure, args=(catalog['active'],), daemon=True).start()
                 return self._reply(200, status_body())
 +            except Exception as err:
 +                return self._reply(400, json.dumps({'error': str(err)}).encode())
         # ---- comfy proxying ----
         state['last'] = time.monotonic()
         pod_id = state['pod_id']
         ready = pod_id and upstream_ready(pod_id)
         if not ready and self.command == 'GET' and path == '/system_stats':
 -            # Reachability probe while cold: answer from cache; do NOT wake.
             try:
                 with open(CACHE_FILE, 'rb') as f:
                     return self._reply(200, f.read())
             except FileNotFoundError:
                 return self._reply(200, json.dumps({'system': {'comfyui_version': 'cold'}, 'devices': []}).encode())
 -        if not ready:
 +        if not ready or (self.command == 'POST' and path == '/prompt'):
             modelsvalue = detect_groupsmatch_workflow_model(body) if (self.command == 'POST' and path == '/prompt' and body) else 'all'None
             try:
                 pod_id = ensure_pod(modelsvalue)
             except Exception as err:
                 log('ensure_pod failed:', err)
                 return self._reply(502, json.dumps({'error': str(err)}).encode())

         finally:
             state['last'] = time.monotonic()
     def _safe_ensure(self, modelsvalue):
         try:
             ensure_pod(modelsvalue)
         except Exception as err:
             log('warmupprovision failed:', err)
     do_GET = _handle
     do_POST = _handle

 def main():
     threading.Thread(target=idle_reaper, daemon=True).start()
     server = ThreadingHTTPServer(('0.0.0.0', LISTEN_PORT), Proxy)
     log(f'runpod-lazy v2v3 (catalog) on :{LISTEN_PORT} (pod "{POD_NAME}", DCs {DATACENTERS or "any"}, idle {IDLE_SECONDS}s)')
     server.serve_forever()

public/scripts/extensions/stable-diffusion/index.js+138 -0 Viewed

     // RunPod lazy-pod proxy base URL ('' = feature off)
     runpod_lazy_url: '',
 +    // RunPod model catalog ({ name, value, downloads } entries)
 +    runpod_models: [],
++
     // Dedicated LLM connection profile for image-prompt generation ('' = use active model)
     prompt_generation_profile: '',

     'ref_images_enabled',
     'ref_images',
     'runpod_lazy_url',
 +    'runpod_models',
     // The image-prompt LLM profile is independent of the image backend, so it must
     // never be captured/swapped by image-generation presets or the fallback retry.
     'prompt_generation_profile',

@@ -713,6 +717,10 @@ async function loadSettings() {
713	717	extension_settings.sd.ref_images = [];
714	718	}
715	719
	720	+ if (!Array.isArray(extension_settings.sd.runpod_models)) {
	721	+ extension_settings.sd.runpod_models = [];
	722	+ }
	723	+
716	724	if (!Array.isArray(extension_settings.sd.custom_entries)) {
717	725	extension_settings.sd.custom_entries = [];
718	726	}

     $('#sd_runpod_lazy_url').val(extension_settings.sd.runpod_lazy_url ?? '');
     renderPresetChain();
     renderRefImages();
 +    renderRunpodModels();
     setupRunpodLoops();
 +    pushRunpodCatalog();
     for (const style of extension_settings.sd.styles) {
         const option = document.createElement('option');

     setupRunpodLoops();
+}
 +/**
 + * Whether a comfy URL points at the runpod-lazy proxy.
 + * @param {string} url ComfyUI URL to test.
 + * @returns {boolean} True when it equals the configured proxy URL.
 + */
 +function isRunpodProxyUrl(url) {
 +    const lazy = getRunpodLazyUrl();
 +    return !!lazy && String(url ?? '').trim().replace(/\/$/, '') === lazy;
 +}
++
 +/**
 + * Returns the configured RunPod model catalog entries that have a filename.
 + * @returns {{name: string, value: string, downloads: string}[]} Catalog entries.
 + */
 +function getRunpodCatalog() {
 +    const models = Array.isArray(extension_settings.sd.runpod_models) ? extension_settings.sd.runpod_models : [];
 +    return models.filter(m => m && String(m.value ?? '').trim());
 +}
++
 +/**
 + * Parses a catalog entry's download lines ("dest-subpath url" per line).
 + * @param {string} text Multiline downloads definition.
 + * @returns {{dest: string, url: string}[]} Parsed file list.
 + */
 +function parseRunpodFiles(text) {
 +    return String(text ?? '').split('\n')
 +        .map(line => line.trim())
 +        .filter(Boolean)
 +        .map(line => {
 +            const space = line.indexOf(' ');
 +            return space > 0 ? { dest: line.slice(0, space).trim(), url: line.slice(space + 1).trim() } : null;
 +        })
 +        .filter(x => x && x.dest && x.url);
 +}
++
 +/**
 + * The model filename saved for whichever config uses the proxy: the live
 + * settings if they point at it, else the chain entry that does.
 + * @returns {string|null} Active catalog value.
 + */
 +function getRunpodActiveModel() {
 +    if (isRunpodProxyUrl(extension_settings.sd.comfy_url)) {
 +        return extension_settings.sd.model || null;
 +    }
 +    const chain = Array.isArray(extension_settings.sd.settings_preset_chain) ? extension_settings.sd.settings_preset_chain : [];
 +    const entry = chain.find(e => isRunpodProxyUrl(e?.preset?.comfy_url));
 +    return entry?.preset?.model ?? null;
 +}
++
 +/** Pushes the model catalog + active selection to the proxy (fire-and-forget). */
 +async function pushRunpodCatalog() {
 +    const url = getRunpodLazyUrl();
 +    if (!url) {
 +        return;
 +    }
 +    const models = getRunpodCatalog().map(m => ({
 +        name: m.name || m.value,
 +        value: m.value,
 +        files: parseRunpodFiles(m.downloads),
 +    }));
 +    try {
 +        await fetch(`${url}/lazy/catalog`, {
 +            method: 'POST',
 +            headers: { 'Content-Type': 'application/json' },
 +            signal: AbortSignal.timeout(8000),
 +            body: JSON.stringify({ models, active: getRunpodActiveModel() }),
 +        });
 +    } catch (error) {
 +        console.warn('SD: runpod catalog push failed', error);
 +    }
 +}
++
 +/**
 + * Rebuilds the RunPod model catalog editor in the settings UI.
 + */
 +function renderRunpodModels() {
 +    const container = $('#sd_runpod_models_list');
 +    if (!container.length) {
 +        return;
 +    }
 +    container.empty();
 +    const models = Array.isArray(extension_settings.sd.runpod_models) ? extension_settings.sd.runpod_models : [];
 +    if (models.length === 0) {
 +        container.append($('<small></small>').text('No models configured.'));
 +        return;
 +    }
 +    models.forEach((model, index) => {
 +        const nameInput = $('<input>').addClass('text_pole flex1').attr({ type: 'text', placeholder: 'Display name' })
 +            .val(model.name || '')
 +            .on('change', function () { model.name = String($(this).val() ?? '').trim(); saveSettingsDebounced(); pushRunpodCatalog(); });
 +        const valueInput = $('<input>').addClass('text_pole flex1').attr({ type: 'text', placeholder: 'Model filename (used by %model%)' })
 +            .val(model.value || '')
 +            .on('change', function () { model.value = String($(this).val() ?? '').trim(); saveSettingsDebounced(); pushRunpodCatalog(); });
 +        const deleteButton = $('<div></div>').addClass('menu_button menu_button_icon fa-solid fa-trash-can')
 +            .attr('title', 'Remove model')
 +            .on('click', () => { models.splice(index, 1); saveSettingsDebounced(); renderRunpodModels(); pushRunpodCatalog(); });
 +        const downloadsInput = $('<textarea></textarea>').addClass('text_pole textarea_compact')
 +            .attr({ rows: 2, placeholder: 'One file per line: <models-subpath> <url>\ne.g. unet/model.gguf https://huggingface.co/...' })
 +            .val(model.downloads || '')
 +            .on('change', function () { model.downloads = String($(this).val() ?? ''); saveSettingsDebounced(); pushRunpodCatalog(); });
 +        container.append(
 +            $('<div></div>').addClass('flexFlowColumn marginTopBot5')
 +                .append($('<div></div>').addClass('flex-container alignItemsCenter').append(nameInput).append(valueInput).append(deleteButton))
 +                .append(downloadsInput));
 +    });
 +}
++
 +function onRunpodModelsAddClick() {
 +    if (!Array.isArray(extension_settings.sd.runpod_models)) {
 +        extension_settings.sd.runpod_models = [];
 +    }
 +    extension_settings.sd.runpod_models.push({ name: '', value: '', downloads: '' });
 +    saveSettingsDebounced();
 +    renderRunpodModels();
 +}
++
 // #endregion
 /**

     extension_settings.sd.model = selectedModel.val();
     saveSettingsDebounced();
 +    // Selecting a catalog model while pointed at the RunPod proxy kicks off the
 +    // download / re-provision for it right away.
 +    if (isRunpodProxyUrl(extension_settings.sd.comfy_url)) {
 +        pushRunpodCatalog();
 +    }
++
     if (extension_settings.sd.model && extension_settings.sd.source === sources.electronhub) {
         const cachedModel = selectedModel.data('model');
         const models = cachedModel ? [cachedModel] : await loadElectronHubModels();

+}
 async function loadComfyModels() {
 +    if (isRunpodProxyUrl(extension_settings.sd.comfy_url)) {
 +        // The RunPod pod downloads models on demand: list the configured catalog
 +        // instead of asking the (possibly cold) backend what it has on disk.
 +        return getRunpodCatalog().map(m => ({ value: m.value, text: m.name || m.value }));
 +    }
     if (extension_settings.sd.comfy_type === comfyTypes.runpod_serverless) {
         $('#sd_runpod_key').toggleClass('success', !!secret_state[SECRET_KEYS.COMFY_RUNPOD]);
         return [

     $('#sd_runpod_lazy_url').on('input', onRunpodLazyUrlInput);
     $('#sd_runpod_warmup').on('click', () => runpodControl('warmup'));
     $('#sd_runpod_shutdown').on('click', () => runpodControl('shutdown'));
 +    $('#sd_runpod_models_add').on('click', onRunpodModelsAddClick);
     $('#sd_custom_entry_add').on('click', onAddCustomEntryClick);
     $('#sd_custom_entries_list').on('click', '[data-action]', function () {
         const id = $(this).attr('data-entry-id');

public/scripts/extensions/stable-diffusion/settings.html+9 -0 Viewed

                 </a>
             </div>
             <small data-i18n="sd_runpod_small">While a SillyTavern tab is open, a keepalive ping keeps the pod alive; it shuts down 15 minutes after the last activity either way. Red = off, orange = starting/downloading models, green = ready.</small>
 +            <h5 data-i18n="Model catalog">Model catalog</h5>
 +            <small data-i18n="sd_runpod_models_small">When the ComfyUI URL points at the proxy, the Model dropdown lists these entries instead of asking the backend. The pod downloads exactly the selected entry's files at boot; changing the selection re-provisions with the new model. Filename = what &quot;%model%&quot; resolves to; downloads = one &quot;subpath url&quot; per line, including companion files (text encoder, VAE, LoRA).</small>
 +            <div id="sd_runpod_models_list" class="flex-container flexFlowColumn marginTopBot5"></div>
 +            <div class="flex-container marginTopBot5">
 +                <div id="sd_runpod_models_add" class="menu_button menu_button_icon">
 +                    <i class="fa-solid fa-plus"></i>
 +                    <span data-i18n="Add model">Add model</span>
 +                </div>
 +            </div>
             <hr>
             <label for="sd_source" data-i18n="Source">Source</label>
             <select id="sd_source" class="text_pole">