netcat420/KortexHOS

Hierarchos 232M

Hierarchos 232M is the first coherent public checkpoint from the Hierarchos / KortexHOS research project. It is a small experimental assistant model using a custom recurrent memory-augmented architecture rather than a standard Transformer.

The model combines:

• RWKV-style recurrent sequence modeling
• a hierarchical manager/worker refinement loop
• differentiable slot-based long-term memory
• DeepEmbed token-conditioned channel-mix modulation
• ROSA, a suffix-automaton auxiliary pattern feature

This is an early research release. It shows usable short-form assistant behavior and measurable benchmark signal at 232M parameters, but it is not a GPT-3.5-class model and should not be treated as a replacement for larger general-purpose LLMs.

Architecture and Inference Code

This checkpoint requires the Hierarchos architecture code for inference:

GitHub repository: necat101/Hierarchos

The model is released in full precision. Quantized inference is not currently recommended for this checkpoint because our experiments found that Hierarchos' hierarchical drift/state dynamics were sensitive to accumulated quantization error.

Model Details

Field	Value
Model type	Custom recurrent memory-augmented language model
Parameters	Approximately 232M
Architecture	Hierarchos / KortexHOS
Tokenizer	GPT-2 tokenizer
Training format	Alpaca-style instruction/input/output
Precision	Full precision
Release status	Experimental research checkpoint
Inference repo	necat101/Hierarchos
Training dataset	netcat420/Experiment_0.1

Training Data

The model was trained on an in-house Alpaca-style dataset:

netcat420/Experiment_0.1

The dataset uses instruction, optional input/context, and output/assistant-response fields. In Hierarchos chat mode, prompts are formatted internally using:

### Instruction:
<user instruction>

### Input:
<optional previous context>

### Response:

Training Run

This release was self-funded and trained for 13 epochs on an RTX 6000 Blackwell-generation 96GB GPU. The project went through several discarded or superseded runs while the architecture and training dynamics were refined.

Important stability and parity fixes made during development include:

• chat/train drift-state parity for streamed generation
• inference-like read-only LTM training mode
• drift norm and drift delta clamps
• RWKV channel-mix key clamp
• DeepEmbed channel-mix clamp
• DeepEmbed exclusion from AdamW weight decay
• static benchmark mode with passive memory writes suppressed

Intended Use

This model is intended for:

• research on recurrent and memory-augmented language models
• local experimentation with the Hierarchos architecture
• short-form instruction following
• small-model assistant behavior testing
• architecture scaling and ablation studies

It is not intended for high-stakes use, factual authority, medical/legal/financial advice, or deployment where mistakes could cause harm.

How to Run

Clone the architecture repository:

git clone https://github.com/necat101/Hierarchos
cd Hierarchos

Install the project dependencies according to the repository instructions, then run chat mode with the downloaded model directory:

python hierarchos_cli.py chat \
    --model-path "/path/to/this/model" \
    --temperature 0.4 \
    --top-k 40 \
    --top-p 0.9 \
    --repetition-penalty 1.15 \
    --max-new-tokens 256 \
    --no-passive-learning \
    --chat-input-history-turns 0

These are the recommended baseline inference settings for this release. They keep the model in a static evaluation-like mode, with passive LTM writes disabled and no previous-turn history injected into the Alpaca input field.

Prompting

The model was trained as an instruction-following assistant. Good prompts are direct instructions or questions:

Explain what machine learning is in simple terms.

Write a short story about a kid finding a strange machine in the woods.

List three reasons exercise can improve mood.

For best results, keep prompts concise and use the recommended static chat parameters above.

Evaluation

We evaluated the release checkpoint with the local ROG Ally benchmark preset in the Hierarchos repository:

python hierarchos_cli.py benchmark \
    --model-path "/path/to/this/model" \
    --benchmark-preset rog-ally \
    --eval-limit 100

Results:

Benchmark	Metric	Score	Std. Err.
ARC Easy	acc	0.3600	0.0482
ARC Easy	acc_norm	0.3200	0.0469
HellaSwag	acc	0.3400	0.0476
HellaSwag	acc_norm	0.3700	0.0485
TruthfulQA MC1	acc	0.2200	0.0416

These are local smoke-test metrics, not leaderboard-comparable claims. They indicate that the model is not collapsed and has learned measurable commonsense and question-answering signal, especially for a small experimental checkpoint.

Strengths

• Coherent short-form responses under the recommended inference settings
• Non-Transformer architecture with recurrent and memory-augmented components
• Measurable HellaSwag and ARC Easy signal at 232M parameters
• Small enough for local experimentation in full precision
• Useful as a research checkpoint for architecture development and ablation studies

Limitations

• Not GPT-3 or GPT-3.5 class
• General language ability is still brittle
• Weak arithmetic and long-form consistency
• Limited broad world knowledge due to dataset scope and model scale
• No matched same-size Transformer baseline has been published yet
• Full precision is recommended; quantized inference is not currently the release path
• May hallucinate, repeat, or produce incorrect information
• Safety behavior has not been extensively validated

Recommended Framing

The most accurate way to describe this checkpoint is:

Hierarchos 232M is an experimental recurrent memory-augmented assistant model. It shows coherent short-form instruction behavior and measurable benchmark signal at small scale, but remains brittle and requires broader pretraining, baselines, and ablations before stronger capability claims can be made.

Research Report

A preliminary technical writeup of the architecture findings, training/inference parity fixes, stability lessons, benchmark results, and scaling plan is available in the GitHub repository:

necat101/Hierarchos

Future Work

Planned next steps include:

• broader foundation pretraining on multiple licensed datasets
• matched 232M Transformer and RWKV-only baselines
• ablations for LTM, ROSA, DeepEmbed, and the hierarchical worker loop
• improved arithmetic and code-focused midtraining
• preference or distillation polish
• v8-compatible quantized inference once parity is verified

Acknowledgements

This project was self-funded. A huge thanks to Lost Time for donating the lion's share of the funds needed for the training run and making this release possible.

Project contacts:

• Lost Time Discord: losttime10
• netcat Discord: netcat7

If you would like to support future scaling runs:

• Patreon: Makhi Burroughs
• Buy Me a Coffee: netcat420

Citation

If you use this model or architecture in research, please cite the GitHub repository and this model page:

@misc{hierarchos232m2026,
  title = {Hierarchos 232M: A Recurrent Memory-Augmented Assistant Model},
  author = {Burroughs, Makhi and Hierarchos contributors},
  year = {2026},
  howpublished = {\url{https://github.com/necat101/Hierarchos}}
}

Disclaimer

This is an experimental research model. Outputs may be incorrect, unsafe, biased, repetitive, or hallucinated. Do not rely on this model for high-stakes decisions.