Model structure rules

Transformers enforces a set of static rules on every modeling_*.py, modular_*.py, and configuration_*.py file. The mlinter package provides the checker engine, and the repository keeps its active rule set in utils/rules.toml. That local TOML lets us enable, disable, or tweak rules quickly without waiting for a new transformers-mlinter release.

These are the expected model conventions for adding or changing modeling code. They keep the codebase consistent and ensure compatibility with features like pipeline parallelism, device maps, and weight tying.

Running the checker

make typing runs mlinter alongside the ty type checker through the repo wrapper, so it picks up utils/rules.toml. Run the same wrapper directly with the following commands.

python utils/check_modeling_structure.py                 # check all modeling files
python utils/check_modeling_structure.py --changed-only  # check only files changed vs origin/main
python utils/check_modeling_structure.py --list-rules    # list all rules and their enabled status
python utils/check_modeling_structure.py --rule TRF001   # show built-in docs for a specific rule

The --changed-only flag is the fastest option during development. It only checks the files you’ve modified relative to the main branch. If you invoke mlinter directly instead of the wrapper, pass --rules-toml utils/rules.toml so local overrides are applied.

Fixing a violation

When a rule violation is detected, the error looks like this:

src/transformers/models/acme/modeling_acme.py:18: TRF013: AcmeModel.__init__ does not call self.post_init().

Use the rule ID to look up the fix in the rules reference. TRF013 is triggered when a PreTrainedModel subclass doesn’t call self.post_init(). That method performs essential finalization steps, and omitting it causes runtime bugs.

 class AcmeModel(AcmePreTrainedModel):
     def __init__(self, config):
         super().__init__(config)
         self.layers = nn.ModuleList(
             [AcmeDecoderLayer(config) for _ in range(config.num_hidden_layers)]
         )
+        self.post_init()

Rules reference

Each rule below lists what it enforces and a diff showing the fix. Run python utils/check_modeling_structure.py --rule TRF001 to see the built-in docs for any rule with the repo’s current rule set.

TRF001

Checks naming consistency between <Model>PreTrainedModel and config_class. Mismatched config_class can break loading, auto classes, and developer expectations.

class AcmePreTrainedModel(PreTrainedModel):
-    config_class = WileConfig
+    config_class = AcmeConfig

TRF002

Checks that base_model_prefix, when set, is a non-empty, whitespace-free string literal. Invalid prefixes can break weight loading key mapping and base model access patterns.

class AcmePreTrainedModel(PreTrainedModel):
-    base_model_prefix = ""
+    base_model_prefix = "model"

TRF003

Detects forward methods that use the old ‘if not return_dict: return (x,)’ pattern. The old return_dict branching pattern is error-prone and verbose. Use the capture_output or can_return_tuple decorators instead.

-def forward(self, x, return_dict=None):
-    if not return_dict:
-        return (x,)
-    return AcmeModelOutput(last_hidden_state=x)
+@can_return_tuple
+def forward(self, x):
+    return AcmeModelOutput(last_hidden_state=x)

TRF004

Checks that no model class defines a tie_weights method. Overriding tie_weights leads to bad consequences for loading, device_map computation, and saving. Use _tied_weights_keys class attribute to declare tied weights instead.

-def tie_weights(self):
-    self.lm_head.weight = self.emb.weight
+class AcmeForCausalLM(AcmePreTrainedModel):
+    _tied_weights_keys = ["lm_head.weight"]

TRF005

Checks the shape of _no_split_modules when present. Malformed values can break device-map partitioning and sharding behavior.

-_no_split_modules = [SomeLayerClass, ""]
+_no_split_modules = ["AcmeDecoderLayer", "AcmeAttention"]

TRF006

Checks forward signatures that expose cache arguments for usage of those arguments in method body. Unused cache arguments can indicate incomplete caching support and inconsistent API behavior.

def forward(self, x, past_key_values=None, use_cache=False):
+    if use_cache:
+        ...
     return x

TRF007

Checks for self attribute assignments after self.post_init() in init. Mutating model structure after post_init can bypass intended initialization/finalization logic.

def __init__(self, config):
     ...
-    self.post_init()
-    self.proj = nn.Linear(...)
+    self.proj = nn.Linear(...)
+    self.post_init()

TRF008

Checks add_start_docstrings usage on model classes for non-empty docstring arguments. Empty decorator usage produces unclear docs and weakens generated API documentation quality.

-@add_start_docstrings("")
+@add_start_docstrings("The Acme model.")
 class AcmeModel(AcmePreTrainedModel):
     ...

TRF009

Checks modeling files for cross-model imports such as transformers.models.other_model.or from ..other_model. imports. Cross-model implementation imports violate the single-file policy and make model behavior harder to inspect and maintain.

-from transformers.models.llama.modeling_llama import LlamaAttention
+# Keep implementation local to this file.
+# If reusing code, copy it with a # Copied from comment.

TRF010

Checks direct PreTrainedConfig/PretrainedConfig subclasses in configuration*.py and modular*.py for an explicit @strict(accept_kwargs=True) decorator. Without strict, new config classes miss the repo’s runtime type-validation contract and drift from the dataclass-based config standard.

+@strict(accept_kwargs=True)
 class AcmeConfig(PreTrainedConfig):
     ...

TRF011

In forward() methods of PreTrainedModel subclasses, checks for attribute accesses on submodules that would not exist on torch.nn.Identity. This includes attribute accesses on loop variables iterating over self.layers, and self.<submodule>.<attr> chains where <attr> is not a standard nn.Module attribute. Pipeline parallelism may replace any submodule with torch.nn.Identity. Accessing custom attributes (e.g. decoder_layer.attention_type) on a replaced module raises AttributeError at runtime. Per-layer metadata should be read from self.config instead.

def forward(self, ...):
-    for decoder_layer in self.layers:
+    for i, decoder_layer in enumerate(self.layers):
         hidden_states = decoder_layer(
             hidden_states,
-            attention_mask=causal_mask_mapping[decoder_layer.attention_type],
+            attention_mask=causal_mask_mapping[self.config.layer_types[i]],
         )

TRF012

Checks that init_weights(self, module) does not use in-place operations (e.g. .normal(), .zero_()) directly on module weights. We rely on internal flags set on parameters to track whether they need re-initialization. In-place ops bypass this mechanism. Use the init primitives instead.

+from transformers import initialization as init
+
 def _init_weights(self, module):
-    module.weight.normal_(mean=0.0, std=0.02)
+    init.normal_(module.weight, mean=0.0, std=0.02)

TRF013

Checks that every PreTrainedModel subclass with an init method calls self.post_init(). In modular files, calling super().init() is also accepted since it propagates post_init from the parent. post_init performs essential finalization (weight initialization, gradient checkpointing setup, etc.). Omitting it causes subtle runtime bugs.

class AcmeModel(AcmePreTrainedModel):
     def __init__(self, config):
         super().__init__(config)
         self.layers = nn.ModuleList(...)
+        self.post_init()

TRF014

Checks whether trust_remote_code is passed or used in code (e.g. as kwarg) within native model integration files. trust_remote_code allows arbitrary loading, including binaries, which should only be a power feature for users, not a standard use-case. Native integrations must not depend on it, as remote code cannot be reviewed or maintained within transformers.

class AcmeModel(AcmePreTrainedModel):
     def __init__(self, config):
         super().__init__(config)
-        self.model = AutoModel.from_pretrained(..., trust_remote_code=True)
+        self.model = AutoModel.from_pretrained(...)

TRF015

When a PreTrainedModel subclass defines _tied_weights_keys as a non-empty collection, checks that the corresponding configuration file declares a tie_word_embeddings field. Without tie_word_embeddings in the config, users cannot control weight tying behavior. The model ties weights unconditionally, breaking serialization round-trips and preventing fine-tuning with untied heads.

# configuration_foo.py
 @strict(accept_kwargs=True)
 class FooConfig(PreTrainedConfig):
     hidden_size: int = 768
+    tie_word_embeddings: bool = True

TRF016

When an imageprocessing.py or videoprocessing.py class declares boolean do_* attributes (e.g. do_resize, do_rescale, do_normalize, do_convert_rgb) and overrides preprocess() or _preprocess(), checks that each declared flag is still consumed along the override path. That can be a direct reference in the override body, delegating back to the base implementation via super().preprocess(…, kwargs) or super()._preprocess(…,kwargs), or, for image processors, forwarding do_convert_rgb into the shared image-preparation path via _preprocess_image_like_inputs(…) or _prepare_image_like_inputs(…). The allowlist of base-handled flags (do_sample_frames) is exempted because the base preprocess() consumes them before _preprocess() runs. A do_X attribute that is not referenced by the override is a dead flag: setting do_X=False at construction or call time has no effect, and the underlying operation runs unconditionally. This silently breaks user expectations and makes per-call overrides ineffective.

class AcmeVideoProcessor(BaseVideoProcessor):
     do_resize = True
     do_normalize = True

     def _preprocess(
         self,
         videos,
+        do_resize: bool,
+        do_normalize: bool,
         size,
         image_mean,
         image_std,
         **kwargs,
     ):
         for video in videos:
-            video = self.resize(video, size=size)
-            video = self.normalize(video, image_mean, image_std)
+            if do_resize:
+                video = self.resize(video, size=size)
+            if do_normalize:
+                video = self.normalize(video, image_mean, image_std)

TRF017

Checks classes decorated with both @auto_docstring and @dataclass for source ordering: @auto_docstring must appear above @dataclass. Decorators are applied bottom-up. When @dataclass is listed above @auto_docstring, @auto_docstring runs first on a class that has no synthesized init yet and ends up modifying the parent class’s init.doc instead of the subclass’s.

-@dataclass
 @auto_docstring(
     custom_intro="""
     Output type of `AcmeForPreTraining`.
     """
 )
+@dataclass
 class AcmeForPreTrainingOutput(ModelOutput):
     ...

TRF018

Checks that every PreTrainedModel subclass that overrides _init_weights(self, module, ...) chains the call up via super()._init_weights(...). In modular files, PreTrainedModel._init_weights(self, module) and raise AttributeError(...) are accepted because they are modularization sentinels. If a model intentionally fully overrides initialization, suppress with # trf-ignore: TRF018 on the line above the method. The base _init_weights covers standard module types (Linear, Embedding, LayerNorm, RotaryEmbedding, …). Skipping super()._init_weights(...) silently leaves submodules unhandled by the override uninitialized, which can pass tests and surface much later as subtle weight-init bugs (cf. https://github.com/huggingface/transformers/pull/45597).

from ... import initialization as init

 def _init_weights(self, module):
+    super()._init_weights(module)
     if isinstance(module, AcmeCustomLayer):
-        module.gate.data.zero_()
+        init.zeros_(module.gate)

TRF019

Checks that *ProcessorKwargs TypedDict classes in processing_*.py files do not set a non-empty _defaults dict. Old models released before cutoff date are not checked against the rule for backwards compatibility; new models must not hardcode defaults in Python. Hardcoding defaults in _defaults scatters processor configuration across Python source files, makes it unintuitive when it comes to overriding defaults via config, and bloats up the code. The canonical home for processor defaults is processor_config.json on the hub, which is shipped with the checkpoint and can be updated without touching code.

class Gemma4ProcessorKwargs(ProcessingKwargs, total=False):
-    _defaults = {
-        "text_kwargs": {"padding": False},
-        "images_kwargs": {"return_tensors": "pt"},
-    }
     images_kwargs: Gemma4ImageProcessorKwargs

Suppressing violations

If you need to suppress a rule violation, use one of the two options below.

Inline suppression

Add a # trf-ignore: RULE_ID comment on the violating line. Include an explanation so reviewers understand why the suppression is justified.

# trf-ignore: TRF011 — mask is derived from self.config, not the layer
hidden_states = layer(hidden_states, attention_mask=mask_from_config)

Don’t use trf-ignore to silence violations that should be fixed in the code.

allowlist_models

For models with legacy code that can’t be fixed immediately, add the model’s directory name to the relevant rule’s allowlist_models list in the mlinter rules.toml.

[rules.TRF004]
allowlist_models = ["existing_model", "your_model_name"]