01 / Methods
Research principles
Begin with a question that evidence can constrain. Separate an intended capability from demonstrated behavior, and identify an observation that could contradict the working explanation.
- Known
- An observation supported by an identified record, within stated conditions.
- Unknown
- A question the available evidence does not settle. Absence of a recorded failure does not establish absence of failure.
- Inferred
- An interpretation drawn from observations. State the assumptions and plausible alternatives; do not present the interpretation as a direct measurement.
A report should distinguish what a system did from why it may have done it. Explanations of behavior remain hypotheses unless the experimental design can discriminate between them.
Related principle: Unknown is a valid result ↗
02 / Methods
Experimental design
Define the unit of evaluation, the tested conditions, the comparison, and the measurement rule. Specify which variables change and which are held fixed. Controls should address plausible alternative explanations, including ordinary run-to-run variation where relevant.
Use deterministic evaluation when the research question can be expressed as explicit rules and replayable cases. A deterministic evaluator can make scoring reproducible; it does not by itself make the evaluated model deterministic or establish that the cases represent deployment conditions.
Where applicable, document acceptance and stopping criteria before the relevant evaluation: what constitutes success, a failed test, insufficient evidence, or a reason to suspend work. Record later changes and distinguish exploratory work from tests against a fixed criterion.
Published method descriptions: VCE ↗ · 7PEEL method validation ↗
03 / Methods
Evidence preservation
Keep the records needed to connect an important claim to its evidence. These receipts may include source material, inputs, outputs, configurations, scoring rules, software or model revisions, and review findings. The necessary record depends on the project.
Freeze or version experimental materials when a comparison is intended to remain inspectable. Identify the version used for a reported result; preserve subsequent changes separately. A checksum verifies artifact identity, not the correctness of the artifact or its interpretation.
Record omissions and unavailable materials. A research record should state which parts can be inspected or reproduced, rather than implying complete access from the presence of a repository.
Example of a published artifact record: Context-robustness study provenance ↗
04 / Methods
Adversarial review
Review should actively seek errors in the instrument, alternative explanations, simpler comparisons, unsupported generalizations, and mismatches between the claim and its evidence. Apply the same review standards to preferred mechanisms.
Preserve the scope of the review, the material reviewed, the findings, and the response where those records are available. Distinguish a proposed repair from a verified repair. Acceptance under a defined review criterion applies to that scope; it is not a general certification of the system.
Adversarial review is a research practice. It should not be described as formal academic peer review unless that process actually occurred.
Published discussion of review and stopping: VCT claim boundaries ↗
05 / Methods
Claim boundaries
State the population, environment, instrument, comparison, and observation period supporting a conclusion. Distinguish a result on constructed cases from a result in a deployed system, and a local comparison from superiority over untested alternatives.
Report limits of measurement and causal attribution. If observations are dependent, the unit count should not be presented as a count of independent real-world trials. If several conditions change together, do not attribute the difference to one mechanism without supporting evidence.
List material nonclaims alongside the conclusion. A bounded result may remain useful even when it does not establish general safety, portability, or external effectiveness.
Published example: VCT technical record ↗
06 / Methods
Negative results
Preserve failed comparisons, counterexamples, unsuccessful transfers, and decisions to stop. Report the tested condition, the measure, and what the result rules out—or leaves unresolved.
A null finding is not automatically evidence of equivalence. An unfavorable result is not automatically evidence that a mechanism fails in every setting. Both require the same attention to measurement and claim boundaries as a positive result.
When work stops or a claim is narrowed, retain the reason in the record. Subsequent changes should not silently replace the materials on which the earlier conclusion depended.
Related documentation: Preserving unfavorable results ↗
07 / Methods
Replication and reproducibility
For reproducible work, identify the materials and procedure needed to rerun the stated evaluation. Depending on the project, this may include executable cases, environment information, dependencies, model revisions, configurations, scoring code, result files, and an artifact manifest.
Distinguish replaying preserved outputs, rerunning the original procedure, and testing the claim under changed conditions. Each answers a different question. Reproducing a score does not independently validate the measurement construct; reproducing an effect in one setting does not establish broad transfer.
Report discrepancies and remaining gaps. Where external systems or stochastic generation prevent exact reproduction, state what is fixed, what may vary, and which comparison remains meaningful. A replication claim should identify the procedure, conditions, and criterion it satisfied.
Published materials: Frozen context-robustness record ↗
08 / Methods
Governance
Define the authorized scope of an experiment, including its environment, permitted actions, and requirements for escalation or stopping. Capability to continue does not itself authorize continuation.
Keep decisions about expanding scope separate from evidence that a system can perform a task. The archived Agency Scope Dynamics program began with controlled simulation and deterministic instrumentation before authorizing behavioral experiments. Its RH-00 foundation was completed, while the broader behavioral roadmap was retired before model or multi-agent experimentation.
Where acceptance criteria or authorization gates apply, preserve the decision and its supporting record. A methodological gate should state what it permits and what remains outside its scope. Knowledge accumulation, continuity, and successful local evaluation do not grant a wider mandate.
Related documentation: Agency Scope Dynamics ↗ · Knowledge and authority ↗
09 / Methods
Open questions
Methodological uncertainty belongs in the record. Questions that should remain explicit include:
- Does the instrument measure the intended behavior, or a property of its representation and scoring rules?
- Are apparent effects distinguishable from ordinary variability?
- Which conclusions survive changes in models, context, environments, or task populations?
- What would count as a failed replication or evidence against the current explanation?
- Which evidence is missing, and what additional observation would justify continuing, narrowing, or stopping the work?
These are evaluation questions, not claims that every project has resolved them. Project-specific protocols and limitations remain the authority for what was actually tested.