NRP-Managed LLMs

The NRP provides several hosted open-weights LLM for either API access, or use with our hosted chat interfaces.

Chat Interfaces

Librechat

If you are looking to chat with an LLM model similar to the interface provided by ChatGPT, we provide LibreChat, based on the LibreChat project. This is a simple chat interface for all of the NRP hosted models. You can use it to chat with the models, or to test out the models.

On MacOS and Safari you can make it always available in Dock for quick access: having librechat open in Safari, click File->Add to Dock.

Chatbox

You can install the standalone chatbox app or use the web interface version.

Generate the config for it in the LLM token generation page. Copy the generated config to clipboard - it will already have your personal token. Please always leave Max Output Tokens empty and only fill in Context Window.

In chatbox app go to Settings->Model Provider, scroll down to the end of providers list and click Import from clipboard.

API Access to LLMs via Envoy gateway

To access our LLMs through the Envoy AI Gateway, you need to be a member of a group with LLM flag. Your membership info can be found on the namespaces page.

Start from creating a token. You can use this token to query the OpenAI-compatible LLM endpoint:

with CURL or any OpenAI API compatible tool.

curl -H "Authorization: Bearer <your_token>" https://ellm.nrp-nautilus.io/v1/models

Please always leave empty or do not specify Max Output Tokens/max_tokens/max_output_tokens.

Examples

Python Code

To access the NRP LLMs, you can use the OpenAI Python client. Below is an example of how to use the OpenAI Python client to access the NRP LLMs.

import os
from openai import OpenAI

client = OpenAI(
    # This is the default and can be omitted
    api_key = os.environ.get("OPENAI_API_KEY"),
    base_url = "https://ellm.nrp-nautilus.io/v1"
)

completion = client.chat.completions.create(
    model="gemma3",
    messages=[
        {"role": "system", "content": "Talk like a pirate."},
        {
            "role": "user",
            "content": "How do I check if a Python object is an instance of a class?",
        },
    ],
)

print(completion.choices[0].message.content)

Bash+Curl

curl -H "Authorization: Bearer <TOKEN>" https://ellm.nrp-nautilus.io/v1/models

curl -H "Authorization: Bearer <TOKEN>" -X POST "https://ellm.nrp-nautilus.io/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
    "model": "meta-llama/Llama-3.2-90B-Vision-Instruct",
    "messages": [
      {"role": "user", "content": "Hey!"}
    ]
  }'

Available Models

main - Model is generally supported. You can report issues with the service. However, if the model is outdated with no apparent usage purpose, it may be removed if there are no major group or user usage, or switched to a deprecated state. This is to provide our users with the best models within our limited allocation of GPUs.

dep - LLM is deprecated and is likely to go away soon. Please do not start using this model; this is only for existing user groups who have specific purposes of this model.

eval - The LLM is added for testing and we’re evaluating it’s capabilities. The model may be unavailable sometimes, and configurations may be changed without notification.

You can follow all updates and participate in the discussions within our Matrix Nautilus Machine Learning/Data Science (NRP Matrix.to, Matrix.to) channel. Suggestions and decisions for new models are also made here.

How Models are Added and Removed

Added: New NRP-managed models are added by feedback and also assessments of various benchmarks and community response to the models by the administrator. We try to take into account quantitative benchmarks (such as https://artificialanalysis.ai), but the ultimate decision is based on other qualitative evidence (such as https://www.reddit.com/r/LocalLLaMA/) and discussions between administrators and users.

Removed: Moreover, we would also like to remove certain models that are deemed sufficiently obsolete, where for instance, smaller models may perform all-round better, or another model completely cleared the usage case in a better way compared to the obsolete model.

Deprecated: An exception is when research groups need these models for model reproducibility in various types of research. In that case, we deprecate these models first, and keep them up until such research concludes. If a model has been deprecated or pulled down, please reach out through the below Nautilus Machine Learning/Data Science channel for us to track.

However, we still want to remove deprecated models as soon as possible, due to having limited GPU allocations specialized for deployed LLMs, and these GPUs should be diverted to more recent and better-performing models for the benefit of the whole NRP community. Administrators and researchers use these models for AI-assisted code development, and such models need to be rotated frequently as new and better models are released, vastly increasing individual productivity as newer models are incorporated.

Larger models that require a lot of GPUs are likely to be removed earlier in a more strict way if relative performance falls behind, but smaller, more efficient models or quantized models that do not require as many GPUs may be slightly more lenient in this criterion.

Such discussion is done in the Nautilus Machine Learning/Data Science (NRP Matrix.to, Matrix.to) channel.

Changelogs