Windows 3.11 and WfWG 3.11 are working for me with current source, so perhaps the problem you're having is in some way related to your configuration. Are you using the recommended S3 1.70.04 driver with the machine=svga_s3 setting in DOSBox SVN? Which resolutions and bit depths have you tried?
I was reporting an incompatibility for further testing since I am unable to test Win311 under protected mode. The other concern was which video modes are required for Sargon 3, the reason for the patch.
Prior to the change, Sargon 3 was confused into thinking monochrome text is active because it found RAM in the B0000-B7FFF range, even though the active video window is B8000-BFFFF for color text. However, Sargon 3 is not the sole reason for the change; it simply makes no sense to have conventional RAM in inactive video memory regions. If the previous behavior helps Win9x in some way then it is likely just working around whatever the real issue is.
Were you trying to use a previous installation of Win9x, or did you install with current source? I suggest installing with a standard VGA (640x480x16) driver, and if that works then try others.
Thank you for pointing to the cause and the advice. It is a page fault (segment limit violation) from opening the dos box in Win95 while the S3 Trio driver is active. This kind of page fault is unhandled by the dynamic core, although it is possible to handle it in the normal core. However, as you suspected, the VGA driver works properly. This confirms that this is very likely a Win95 specific issue (and the driver must require additional access to video memory when opening a dos box window, whether intended or not).
The S3 driver chosen automatically by the Win95 setup seems to have no problem in 16 or 256 colors when opening a dos box in current source, so the problem you have may be with drivers for a specific model/version.
I was running into disk errors during the setup of Win95 until I reverted the r3714 change in bios_disk.cpp, so I'll have to investigate what goes wrong there.
That's entirely correct, it is only specific to the S3 driver I tested. I should have noted that. I've used that driver for additional compatibility with directx apps, but it is now confirmed that it isn't the best match for the emulation (as shown by use of the dos box window). I also hope that the bios_disk changes require little effort.
I can reproduce the issue with other S3 drivers in Win95 OSR2 (desktop at 256 colors). However, it is only in the dos box window since dos box fullscreen works. I also tried 16 colors and that works for both cases. It may be from another patch or the installed Windows version.
This must be the memory region where the driver is reading/writing when opening a dos box window under OSR2. Turning off video acceleration also prevents the issue.
It seems odd that Win95 would select the B8000-BFFFF memory window while the GUI is up. The A0000-AFFFF region can be separately assigned to the MMIO handler for S3, but dunno if that's at all relevant. Maybe try changing the flags on the empty handler, like the flags on the IllegalPageHandler for instance.
As you suggested, if in fact the MMIO is active at VGA_PAGE_A0, then it is possible that clearing this same memory region could cause the page fault error. This may be supported by the finding that active video acceleration is necessary to cause the error, given that acceleration activates MMIO while unaccelerated video does not.
Verified that mmio memory region is reserved on Win95 OSR2 start and also when opening the dos box window. This code seems to workaround the issue of clearing the mmio region for this example:
The MMIO mapping overrides any other mapping, including empty, if the (vga.s3.ext_mem_ctrl & 0x10) condition is true. However, your bool flag is true if the condition was ever true, regardless of if it is currently true. So, another thing to look at is why bit 4 on that register gets cleared.
Thank you for the hint on bit 4 which presumably controls the mmio feature.
Edit: Tested further. It seems like the Win95 OSR2 dos box is not compatible with unmapping the MMIO in current emulation.
Edit 2: noted that with video acceleration off, then the mmio is turned off when the dos box is opened, and then turned back on when it is closed. This seems like an expected behavior of the mmio at VGA_PAGE_A0. However, when video acceleration is turned on, then this mmio is mapped when the dos box is opened. Why does the dos box require a mmio when video acceleration is on?