Executable memory, JIT aliases and W^X¶

Scope¶

A JIT compiler has a memory-management problem that ordinary data allocation does not have.

Generated bytes begin as data written by the compiler and later become instructions fetched by the processor. That transition crosses physical allocation, virtual allocation, write permissions, execute permissions, TLB visibility, concurrent CPU execution and safe reclamation.

ChrisOS currently uses two virtual aliases to the same physical frames:

• buf->w: a writable Higher-Half Direct Map alias used to emit and patch code;
• buf->x: a dedicated high virtual alias used for native execution.

The executable alias is not initially mapped. Code generation writes through the HHDM alias; jit_seal later creates the execution mapping.

The design separates the write and execution addresses, but it does not yet enforce strict W^X at the physical-frame level because the writable HHDM alias remains reachable after the executable alias is created.

JitBuf¶

The core object is:

typedef struct JitBuf {
    uint64_t phys;
    uint8_t *w;
    uint8_t *x;
    uint32_t used;
    uint32_t cap;
    uint32_t pages;
} JitBuf;

phys is the base of a contiguous physical allocation.

w is the address used by the compiler for mutation.

x is the address used for native execution.

used tracks emitted bytes.

cap is the total capacity.

pages records the number of physical and executable pages.

One JitBuf therefore spans three address domains:

physical frames
    |
    +-> HHDM writable alias
    |
    +-> dedicated executable alias

Physical allocation¶

jit_alloc obtains one contiguous PMM run:

pmm_alloc_contig(buf->pages)

The header defines:

JIT_PAGES = 6144
JIT_MAX   = 6144 * 4096
          = 24 MiB

If pages is zero or above JIT_PAGES, the raw allocator resets it to JIT_PAGES.

The normal compilation path sets a more specific value before allocation.

Compiler sizing policy¶

jit_compile_image_locked estimates capacity with:

need = bytecode_size * 64 + 262144
pages = ceil(need / 4096)

Then clamps it to:

minimum = 64 pages
maximum = 6144 pages

The minimum is 256 KiB.

The factor is a capacity heuristic for native expansion, patch tables and dispatch structures. It is not a proof that every input fits. Every emitter still checks whether used + n exceeds cap.

Contiguous-allocation trade-off¶

Contiguous physical memory simplifies translation.

Executable page i maps:

buf->phys + i * 4096

The cost is sensitivity to physical fragmentation.

A large JIT compilation can fail even when total free RAM is sufficient if the PMM cannot find one contiguous run.

A scatter-backed design could avoid that limitation, but would need explicit per-page physical bookkeeping.

Dedicated JIT virtual window¶

Execution aliases begin at:

JIT_VIRT_BASE = 0xffffffffc0000000

The limit is:

JIT_VIRT_BASE + 8192 pages

which creates a 32 MiB execution window.

The source comment explains that an older location was inside the same broad region used by the higher-half kernel, LAPIC and other mappings. Mapping there combined with earlier broad CR3/TLB behavior caused machine failure. The current base uses a different paging region in the reviewed layout.

Virtual-range registry¶

The JIT keeps:

JIT_VA_SLOTS = 128

Each entry stores:

• virtual base;
• page count;
• used state.

jit_va_alloc first searches for a free record with exactly the requested page count.

If none exists, it advances the global g_jit_virt_next bump pointer.

jit_va_free marks a known entry unused.

It does not merge ranges or rewind the bump pointer.

Virtual fragmentation¶

Exact-size reuse has a direct consequence.

A freed 100-page range cannot satisfy a later 99-page request.

If the 32 MiB high-water mark is reached, only free entries with exactly matching page counts can be reused.

The allocator can therefore experience virtual exhaustion while some registered ranges are free.

This is a deliberately simple allocator, not a general VMA manager.

Metadata capacity¶

Only 128 JIT VA records exist.

The implementation assumes a metadata entry is available for a newly assigned monotonic range.

There is no tree, bitmap or expandable VMA metadata structure.

The fixed table is therefore a separate capacity boundary from the 32 MiB virtual window.

Writable alias¶

After physical allocation, jit_alloc assigns:

buf->w = bootinfo_phys_to_virt(buf->phys)

This is the direct-map alias.

jit_emit copies bytes into this region and advances used.

Compiler patching also writes through w.

The execution alias does not need write permission for emission.

Executable alias¶

jit_alloc reserves the virtual execution address in the JIT registry and stores it in x.

It does not yet install the executable PTE.

The bytes are generated first.

jit_seal establishes the mappings.

Sealing permissions¶

For every page, jit_seal calls conceptually:

map_4k(exec_va, physical_page, MM_PRESENT)

The resulting leaf has:

• PRESENT set;
• WRITE clear;
• USER clear;
• NX clear.

The dedicated JIT execution alias is therefore supervisor-only and executable, while not writable through that alias.

This is better than a single virtual mapping that is simultaneously writable and executable.

Why current JIT is not strict W^X¶

Strict W^X must account for all aliases to the physical frame.

ChrisOS creates a non-writable execution alias, but it retains the HHDM writable alias to the same physical pages.

jit_seal does not revoke or write-protect w.

Therefore the same physical frame is writable through one address while executable through another.

The accurate description is:

the current design separates W and X virtual aliases, but does not enforce strict physical-frame W^X.

A stronger design would also change the direct-map permission or avoid leaving a permanent writable direct-map alias for JIT frames.

Direct-map dependency¶

jit.c does not own the HHDM page-table policy.

It receives a writable kernel address through bootinfo_phys_to_virt.

Consequently, executable-memory hardening cannot be completed only by changing the dedicated execution PTE. The direct-map alias must be part of the permission model.

Newly present mappings¶

The source contains an important historical note in jit_seal.

These execution mappings were previously not present.

The code maps each page through map_4k, which performs local INVLPG for the execution address.

A broad CR3/TLB operation used in an earlier design reportedly killed the machine during JIT setup.

Current sealing therefore uses local page invalidation instead of a global shootdown.

Seal versus teardown¶

Creating a new translation and destroying an old translation have different risk profiles.

At seal time the execution address was absent.

At teardown time a CPU may still cache a previously valid executable translation even after the PTE is cleared.

That is why jit_free requires the stronger cross-CPU shootdown/reclamation protocol.

Instruction-cache behavior¶

jit_seal contains no separate instruction-cache flush.

The current target is x86-64, where instruction/data cache coherence rules differ from architectures that require explicit data-cache cleaning and instruction-cache invalidation during JIT publication.

This is architecture-specific behavior.

A future non-x86 backend cannot simply reuse the same seal sequence without defining its own instruction-publication protocol.

Global compiler scratch¶

jit_compile.c contains global temporary structures:

• native-offset table;
• patch-site table;
• patch-target table;
• patch count;
• shared fault/dispatch patch locations.

The source explicitly states that compilation scratch is process-global.

Two simultaneous compilers would corrupt those arrays.

Compilation lock¶

jit_compile_image protects the entire compilation with g_jit_compile_lock.

Conceptually:

spin_lock
  -> parse/emit/patch using global scratch
  -> allocate JIT backing when needed
  -> seal
spin_unlock

This lock is a correctness requirement, not just an optimization.

JIT VA concurrency boundary¶

The virtual-range registry in jit.c has no dedicated lock.

Normal allocation from jit_compile_image occurs while the compile lock is held.

jit_free, however, does not take g_jit_compile_lock around jit_va_free.

Thus the registry is not independently designed as a general concurrent allocator.

The current design relies on surrounding runtime lifecycle and caller serialization.

If arbitrary concurrent JIT create/free operations become part of the SMP architecture, explicit synchronization for g_jit_va and g_jit_virt_next is required.

Emission and patching¶

The compiler emits native instructions through w.

Branches initially contain placeholder displacements.

The compile pass records patch sites and bytecode targets, emits the native body and then patches relative branches.

It also writes addresses for the native dispatch table and execution blob.

These mutations occur before normal sealing.

The published function pointer is based on x, not w.

Bytecode-to-native table¶

The compiler tracks native offsets for bytecode positions.

Current hard limits include:

JIT_MAX_PCS   = 1,048,576
JIT_MAX_PATCH = 131,072

These arrays are statically allocated.

The compile lock protects access, but the capacity remains finite and contributes to kernel static memory use.

Fallback code¶

If the image cannot use the full native path, emit_fallback creates a small native wrapper that invokes the interpreter.

That wrapper still follows the same memory lifecycle:

write through w
seal x
call through x
free through JIT teardown

Executable-memory safety is therefore relevant even for fallback JIT code.

Per-CPU syscall context¶

Generated native code needs the VM and runtime user context when it enters helper/syscall paths.

A single global context would be unsafe on SMP.

jit.c defines a context array indexed by software CPU.

jit_set_sys_context stores the VM and user pointer for the current CPU.

jit_sys_trampoline reads the context belonging to the CPU that is executing it.

The source comment specifically explains that a single global context would allow one VM to replace another CPU's active trampoline context.

Context ownership¶

The per-CPU JIT context contains borrowed pointers.

It does not own the VM or graphics/user object.

lang_pipeline sets the context immediately before invoking the native JIT function.

The surrounding runtime must guarantee that those objects remain alive until native execution returns.

This is a lifetime contract rather than an allocation contract.

JIT teardown¶

jit_free performs ordered destruction.

For a buffer with an execution alias, it:

1. calculates the execution range;
2. unmaps every execution page;
3. requests mm_tlb_shootdown_range;
4. marks the JIT VA record reusable;
5. frees the physical run only when the shootdown reports reuse safe;
6. otherwise moves the physical extent into MM quarantine;
7. clears the JitBuf state.

This ordering prevents physical-frame reuse while stale executable translations may still exist.

Why unmap is insufficient¶

Clearing a PTE does not synchronously erase every CPU's TLB.

A remote CPU can retain the old executable translation.

If the physical frame is immediately reused for unrelated data, that stale translation could continue fetching from a frame that now belongs to a different subsystem.

Safe reclamation requires:

unmap
  -> invalidate every relevant CPU
  -> prove reuse safe
  -> reuse physical frame

Local invalidation¶

unmap_4k performs a local INVLPG for each removed execution page.

jit_free loops over all pages.

mm_tlb_shootdown_range then performs the range invalidation/publication required for the cross-CPU protocol.

The initiating CPU can therefore invalidate the same address conservatively more than once.

Safety is prioritized over minimizing teardown instructions.

Shootdown completion¶

The TLB runtime tracks a generation and per-CPU acknowledgement state.

If all relevant online CPUs acknowledge, reuse becomes safe.

If a CPU is fenced, the stronger fenced-state flush/halt requirements apply.

A failure to establish reuse safety makes mm_tlb_shootdown_range return failure.

jit_free uses that result as the physical-memory decision boundary.

Quarantine¶

When shootdown succeeds:

pmm_free_contig

When it fails:

mm_tlb_quarantine

The JIT object is logically destroyed in both cases, but physical reuse differs.

A quarantined run remains reserved until the MM layer can prove that stale translations no longer threaten reuse.

Quarantine overflow policy¶

The quarantine is bounded.

If it cannot accept another unsafe extent, MM logs the condition and does not free that extent.

The result is an intentional leak rather than unsafe physical reuse.

For executable memory, that is the correct priority:

memory loss is preferable to stale-code execution

Virtual-address reuse¶

jit_va_free marks the execution range available after the shootdown attempt.

A later JIT may reuse the virtual range.

The physical frames of the previous JIT might still be in quarantine.

This is safe only if the TLB protocol has prevented stale translations from being confused with the new mapping lifecycle.

The entire unmap/shootdown protocol therefore protects both physical and virtual reuse.

Privilege boundary¶

The JIT execution window is in the upper kernel half.

The executable mapping is not marked USER.

Generated code therefore executes with kernel privilege in the current design.

A compiler/emitter bug can become a kernel control-flow or memory-safety bug.

The JIT is part of the trusted computing base.

Generated memory checks¶

Native code emits explicit bounds checks for guest-memory operations.

The generated path compares guest addresses with VM memory size and routes invalid access to fault handling.

These checks enforce the CLVM memory abstraction when generated correctly.

They do not sandbox malformed native code produced by a compiler bug.

Trusted components include:

• bytecode decoder;
• JIT compiler;
• emitter;
• patcher;
• runtime helpers;
• executable-memory manager.

Failure rollback¶

The JIT contains explicit rollback.

If physical allocation succeeds and VA allocation fails, jit_alloc returns the physical run to PMM.

If native emission fails, the compiler calls jit_free.

Centralizing teardown is important because failure can occur at different stages and some stages may already have created executable mappings.

Validation¶

The repository has a broad JIT test surface, including:

• test_jit_vm;
• test_jit_native;
• load/store and 64-bit cases;
• multiple Doom JIT smoke, patch and differential tools.

Those validate generated semantics.

TLB protocol tests separately validate physical-reuse conditions:

• reuse blocked before remote acknowledgement;
• fenced status alone is insufficient;
• halt without invalidation is insufficient;
• reuse becomes safe only after the required state is visible.

Both layers are necessary for JIT correctness.

Performance model¶

Using two aliases to the same frames avoids a copy from temporary writable storage into a second executable allocation.

The cost profile includes:

• contiguous PMM allocation;
• global compilation serialization;
• page-by-page mapping;
• local INVLPG during seal;
• page-by-page unmap;
• cross-CPU shootdown during free.

For long-running native code, compilation cost may amortize well.

For very short-lived generated code, lifecycle overhead can exceed the execution savings.

Current limitations¶

At the reviewed revision:

• JIT code executes at kernel privilege;
• strict physical-frame W^X is not enforced;
• writable HHDM alias remains available after seal;
• execution VA space is fixed at 32 MiB;
• VA metadata has 128 slots;
• free ranges are reused only by exact page count;
• no VA coalescing or bump rewind exists;
• physical backing must be contiguous;
• compiler scratch is globally serialized;
• JIT VA allocation/free is not independently synchronized as a general SMP allocator;
• no architecture-neutral instruction-cache publication API exists;
• quarantine can intentionally retain frames indefinitely when safety cannot be established.

These are current source facts.

Stronger future W^X design¶

A stricter design could be:

allocate frames
  -> map temporary RW + NX
  -> emit and patch
  -> remove/reprotect writable mapping
  -> perform required invalidation
  -> map RX execution alias
  -> execute
  -> unmap RX
  -> cross-CPU shootdown
  -> reclaim physical frames

With a permanent HHDM, true W^X also requires changing direct-map permissions for those physical pages or using frames outside a permanently writable direct map.

This is a roadmap direction, not current behavior.

Revision boundary¶

This chapter was reconciled against ChrisOS main revision e05a17fd76333114a3fb5c2452f38ca747d4ac56.

The current source-backed lifecycle is:

contiguous PMM frames
  -> writable HHDM alias
  -> native emission and patching
  -> supervisor non-writable executable alias
  -> execute through x
  -> remove executable mappings
  -> cross-CPU TLB shootdown
  -> PMM free when safe
     or quarantine when safety is unresolved

The current implementation already treats executable reclamation as a cross-CPU lifetime problem, while strict W^X and allocator-concurrency hardening remain unfinished.