Ask a program for a short button label and it returns “Open File”. Give another program “A cat having a nap” and it finds a sentence about a sleeping kitten. Both examples can now run through local models inside cREXX.
The optional llama.rexx provider connects cREXX to llama.cpp, the engine that runs the models. cREXX is an open-source Rexx compiler and runtime. Here the model executes in the same process as the application: once the software and model files are installed, these calls need no separate inference server or network connection.
Two small examples
The generation example opens a local model file and asks it for a label for a button that opens a text file. With the small SmolLM2 model used for this check, the answer was "Open File".
The central call is short:
config = .llmconfig("llama", args[1])
client = .llm.open(config)
answer = client.generate("Suggest a short label for a button that opens a text file. Reply with the label only.")
args[1] is the local model filename. The complete program also reads that argument, checks errors and closes the client.
The second example uses embeddings: lists of numbers that allow a program to compare related text. It gives BGE-small three sentences and a search phrase. For “A cat having a nap”, the sentence about a kitten sleeping on a warm cushion ranks ahead of the weather and compiler sentences, despite using different words.
BGE-small returns 384 numbers for each sentence. cREXX’s vector library compares them and orders the matches. This gives a small starting point for experimenting with search over local material.
What this changes
An application can combine model calls with its own rules, data handling and interface. It can keep the model loaded for repeated requests, without operating a separate model service.
The examples have been tested. Model choice, text preparation and memory requirements remain important. Further operating-system and hardware testing remains open.
The implementation is available in the development snapshot for beta 3. It needs matching cREXX and optional llama.rexx packages, plus separately supplied compatible model files. The llama.rexx guides and examples explain installation, model selection and repeated requests.