First post, by jmarsh
As more operating systems adopt stricter security policies, there is a trend to stamp out programs using memory mapped as writable+executable. This is a problem for DOSBox's dynamic recompiler, as it currently just allocates a large chunk of cache and marks it read+write+executable. For now it works but probably not for much longer - SELinux is starting to see widespread use, macOS is pushing their hardened runtime, and other platforms just flat-out don't support it (no mprotect or similar mechanism) requiring other methods. This thread is to discuss at least two possible alternatives and get opinions from other devs about what the best fix would be, rather than a bunch of completely different fixes being implemented in different builds.
(Note that DOSBox has two distinct dynamic recompilers; one is generic, the other is x86-specific, they both allocate their cache the same way so for the sake of this discussion just pretend they're one and the same.)
What the current code does
Windows: Uses VirtualAlloc to allocate the code cache as read+write+executable. That's it.
Other platforms: Uses malloc to allocate the code cache and then (if the platform implements mprotect) uses mprotect to set that memory to read+write+executable. This is actually wrong (undefined behaviour); mprotect is only meant to be used on memory allocated using mmap, so that's at least one thing that needs fixing.
Approach 1: Switch between writable/executable
This method's pretty simple. The initial allocation is done with no permission (PAGE_NOACCESS / PROT_NONE) and when code is going to be placed into the cache it gets "locked" (marked as read+writable) then "unlocked" when writing is finished (marked as read+executable). This is pretty simple to implement in cache_openblock/cache_closeblock, only takes a few lines of code and doesn't require any changes to the architecture-specific dynrec source files. It does require some lines to be shuffled so that cache_block_closing() can be called while the memory is still writable (in case data/instruction cache flushing/invalidation is required) but that's trivial.
Pros: Simple to implement.
Cons: Relies on mprotect/VirtualProtect or other function to change memory permission, which not all platforms have. SELinux might still return EACCES when using mprotect to switch memory to executable. Small processing overhead every time a new CodeBlock is written as pages have to have their access modes switched back and forth.
Approach 2: Map the same memory at different addresses using different permission
This is more complicated. Basically you use mmap to map a file descriptor using read+write permission (which gives an "out" pointer), then mmap the same file descriptor using read+executable permission (which gives an "exec" pointer). Code is stored using the "out" pointer but the "exec" pointer is used for relocation calculations etc. since it is the address that will actually be executed. Most of this functionality would be hidden inside the cache_add* functions, if we wanted to get fancy it would be possible to turn cache.pos into a class with overloaded operators so referencing it would yield the exec pointer while dereferencing it would use the out pointer.
The main issue with this approach is that mmap (or MapViewOfFile on windows) requires an actual file (descriptor) for multiple mappings of the same physical memory. There are ways in linux to avoid using the actual filesystem (memfd_create,shm_open) but these are non-portable and may not always be supported depending on kernel configurations, so can't be relied on.
Pros: Practically no overhead. The recent NX (Switch) port uses this method. Ulrich Drepper suggests this method when working with SELinux (maybe this should be a con...)
Cons: A bit fiddly to setup and may be non-portable. Likely requires a physical file to be present while DOSBox is running, although it could be unlinked as soon as it is mapped (meaning it "should" be cleaned up as soon as DOSBox exits). May require some small changes to the existing risc_*.h source files.
So those are the options I see so far. I would like to hear from other devs (particular those running their own ports on non-mainstream OSes) what they think the best path to take is, and hopefully find a solution before everybody's GoG-bought DOSBox games start to become unrunnable.
