CUDA · 2× RTX 3090
Connecting…

📦 Installed Models

Loading…

Serve profiles engine settings come from these recipes only

Pick Agentic or OCR in the sidebar or Server tab and Reload. Per-model JSON overrides are disabled.

⬇️ Model Downloader search Hugging Face

Search for a model to get started.

📥 Downloads

No active downloads.
🛠️ Serve profiles agentic or OCR · reload to switch ▼
Engine settings are fixed per profile. Reload takes a few minutes.
Recipe (read only)

              

                
                
                
                
              

            

          

          

            

              📡 Telemetry
              ▼
            

            

              
Loading…

              
Throughput history

              

                
Generation speed—

                

              

              

                
Time-to-first-token—

                

              

              

                
Queue depth—

                

              

              
System

              

            

          

        

      


      

        

        
Loading…

      


      

        

          

            

              
API keys

              
Each key identifies one user. Clients send
                Authorization: Bearer <key> to chat and to the manager gateway
                (:8893/v1/chat/completions, /v1/completions,
                /v1/models). The engine on
                :8010 still uses the .env API key. Once the first key exists,
                chat and the manager gateway require a valid key.

            

            

              
              
            

          

          

          

            
API usage log

            
Every manager-gateway / web-chat completion is recorded forever: user, model, prompt & generation tokens, speed, duration. View the log here, filter by date, and download CSV or JSONL. Start new usage period zeros per-user counters and empties the log (the previous usage.jsonl is kept as a timestamped backup). API keys are not deleted.

            

              
              
              
              
              
              
              
              
              
            

            

              

                
                  

                    TimeUserModelEndpoint
                    PromptGenPrefill t/sGen t/sDuration
                  

                
                
                  
Set a date range and click View log, or leave dates empty for the latest records.

                
              

            

            

              
              
            

          

        

      


      

        

          
📋 Container Logs
            
              
              
            
          

          
No logs yet.

        

      


      

        

          
📅 Period statistics from the per-call log — every attributed completion carries a timestamp and an owner, so any range can be recomputed

          

            

            

            

              

                
                
                
                
                
              

            

          

          
Loading…

          

            

              
By day

              

                
DateCallsPromptGeneratedTotal

                

            

            

              
By user / API key

              

                
UserCallsPromptGeneratedTotal

                

            

            

              
By model

              

                
ModelCallsPromptGeneratedTotal

                

            

          

          
—

        


        

          
🔢 Engine totals the engine's own /metrics counters · covers ALL traffic to :8010, including calls that bypass the gateway · not sliceable by date · Cached and Requests only began reporting 2026-09-13

          
Loading…

        


        

          
🧠 Prefill cache & HiCache radix/prefix cache + hierarchical cache · live

          

          

            L3 KV cache: —
            
            
            
            A poisoned prefix can live in any tier. Purge L3 clears the on-disk store only; Purge + flush engine also drops the engine's in-memory L1/L2 in seconds (it refuses while requests are in flight); Purge + restart recreates the container (~4 min) if the engine is unresponsive.
          

        

        

          
📈 Token throughput tokens/min · last hour · prefill ■ vs generation ■

          

        

        

          
📊 Per-model totals engine-side counters · all traffic · retired model labels remain here as history

          

            

              
ModelPrefill totalGenerated totalCachedRequestsLive prefill tok/sLive gen tok/sLast activity

              
            

          

        

        

          
👤 Per-user / API key who is using :8893 · live · same keys as the Keys tab

          

            

              
UserRequestsPromptGenTotalShareLast modelLast call

              
            

          

        

        

          
🔴 Live API calls in-flight now, then newest completions · user + endpoint + tokens + speed
            
              
            
          

          
No in-flight manager requests.

          

            

              
                

                  TimeUserAPIModel
                  PromptGenPrefill t/sGen t/sDuration
                

              
              
                
Waiting for gateway / web-chat completions…

              
            

          

          
Only traffic through the manager gateway (:8893/v1) and this UI’s Chat tab is attributed. Direct engine calls on :8010 with the shared API_KEY are not billed per user.

        

      


      

        

          
🛰️ Active requests every request through :8893 and this UI · refreshing every second
            —
          

          

            

              
                

                  StartedElapsedUserAPI keyEndpoint
                  ModelPhasePromptGenReasoning
                  TTFTGen tok/sRID
                

              
              
                
No active requests.

              
            

          

          
A row appears the moment a request enters the gateway and leaves when it finishes (it then shows below). Prompt is only known once the engine reports usage at completion, so it reads “—” while in flight; Phase is prefill until the first token arrives.

        

        

          
✅ Recently finished last 5 minutes

          

            

              
                

                  StartedFinishedDurationUserAPI key
                  EndpointModelPromptGenReasoning
                  TTFTGen tok/sStatus
                

              
              
                
Nothing has finished yet.

              
            

          

          
Token counts here come from the engine’s own usage block, so they are exact — unlike the live counters above, which are counted from stream deltas.

        

      


      

        

          
⚡ Performance Benchmark prefill & generation tokens/s

          

            

            

            

            

          

          

          

        

      


      

        

          
🔌 Integrations

          
Connect coding tools to the manager gateway on port 8893, or directly to the engine on 8010.