First post, by leycec
The newest release of DOSBox SVN Daum now builds under 64-bit architecture. It doesn't officially, of course. The only downloads available at http://ykhwong.x-y.net for OS X, Linux, and probably Windows are 32-bit. But I've discovered the petty secrets of how to compile a 64-bit Daum for yourself at home... and it's not pretty.
In fact, it's uniformly awful. After wrestling with the Daum patchset for a little over a week, I remain firmly convinced that Daum as shipped is currently uncompilable on most systems – whether 32- or 64-bit.
To make sense of all this madness, I've thus structured this post a bit... uniquely.
We kick off with a list of practical issues facing the Daum codebase – from major compilation and linkage errors to minor dependency and patch errors. To avoid descending into unmindful trolldom, I offer practical fixes for each – in all cases, fixes that anyone can apply at home. For Gentoo and Sabayon Linux users, I've gone one step further and completed a working ebuild for Daum and all dependencies (OpenGLide, SDL SVN with OpenGL-HQ support, SDL_sound SVN). (More on that in another post, assuming I don't collapse of sleep deprivation first.)
I wrap things up with a discussion of the Daum codebase, community, and development methodology. Yup, shit just got real. If Daum already works for you, just skip the error reports and go straight to the "good stuff" at the end.
Before we wade into the nitty-gritty, I should probably clarify: I really like the Daum patchset and wish it nothing but continued success. In the crowded marketplace of unofficial DOSBox patches, the Daum and DOSBox-X patchsets clearly reside at the head of the pack. I humbly appreciate the very real and significant effort that Taewoong (ykhwong), gulikoza, HAL 9000, and other dedicated community members have invested – either directly or indirectly – into Daum. At the end of the day, that's why I authored this thread. Because Daum is too close to completion to let it fall apart in the final mile.
So here we go.
Let Loose the Dogs of War
Let's be frank: DOSBox SVN Daum is fundamentally uncompilable in its current state. The autotools files ("Makefile.am", "configure.am") are predominantly either broken or insufficient; the codebase itself is significantly broken in numerous places, including but not restricted to the minizip patch (incompatible with the current release of "zlib") and dynamic CPU cores (referencing undefined functions); the documentation neglects to mention whole nontrivial dependencies (OpenGLide, I'm looking at you); the OpenGL-HQ patch on http://ykhwong.x-y.net fails to apply to SDL sources. I could go on. And I will.
Paradoxically, ykhwong and a handful of others continue to successfully compile and distribute 32-bit Windows, OS X, and Linux releases. I congratulate them. I'm also mystified at how they managed to do it.
Let's see why.
Compilation and Linkage Errors
Daum suffers three severe compilation and linkage errors. For each, I give a verbal description of the solution. For those who'd rather automate your way to victory, here's a series of "sed" commands patching the errors for you. Run the following from the top-level directory to which you extracted the Daum source code:
sed -ie 's~^\(#include "zlib.h"\)$~\1\
#ifndef OF /* function prototypes */\
# ifdef STDC\
# define OF(args) args\
# else\
# define OF(args) ()\
# endif\
#endif~' include/ioapi.h
sed -ie '/if (PAGING_ForcePageInit(lin_addr)) {/,+6d'\
src/cpu/core_dynrec/decoder_basic.h
sed -ie '$ c\
Bits CPU_Core_Dyn_X86_Run(void) {\
return 0;\
}\
Bits CPU_Core_Dyn_X86_Trap_Run(void) {\
return 0;\
}\
void CPU_Core_Dyn_X86_Cache_Reset(void) {\
}\
#endif' src/cpu/core_dyn_x86.cpp
I should also provide a single patch for easy application. But I'm woefully tired, pathetically lazy, and would like other people to actually understand what went wrong.
So – in the order in which I encountered and fixed each error, let's go.
Undefined zlib Macro OF()
This is the first compilation error I received:
x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I.. -I../include -I/usr/include/SDL -D_REENTRANT -march=native -O2 -pipe -c miniunz.c
In file included from ../include/unzip.h:55:0,
from miniunz.c:46:
../include/ioapi.h:127:51: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘OF’
What's happening here?
I'm glad you asked. The system-wide "zconf.h" library installed by zlib used to define a macro named OF(). This was obviously a bad name for a macro to be used by hundreds of thousands of cross platform-portable applications the world over, so newer versions of zlib mangled it to _Z_OF().
Guess what macro Daum expects? OF(). It doesn't exist on newer systems, causing the above compilation error. The correct fix is to rename all instances of "OF(...)" to "_Z_OF(...)" and require the user install a sufficiently new version of zlib. Since I'm (again) pathetically lazy, I just added the old definition of OF() to Daum's "include/ioapi.h" file after inclusion of "zlib.h":
#include "zlib.h"
ifndef OF /* function prototypes */
ifdef STDC
define OF(args) args
else
define OF(args) ()
endif
endif
This works, though it's not entirely pretty. Who cares.
Undefined Function PAGING_ForcePageInit()
This is the next compilation error I received:
cpu/libcpu.a(core_dynrec.o): In function `MakeCodePage(unsigned long, CodePageHandlerDynRec*&)':
core_dynrec.cpp:(.text+0x88d7): undefined reference to `PAGING_ForcePageInit(unsigned long)'
What in the Ninety Stygian Hells is happening here?
Oh, boy. This is a bad one. I'm shaking my head already. The reference to PAGING_ForcePageInit() is undefined, because... it's not defined. That's right: PAGING_ForcePageInit() is declared by "include/paging.h" but never actually defined anywhere. While it's not at all clear what the right thing to do here is, I noticed this VOGONS thread in which HAL 9000 recommends deleting the entire if conditional in "src/cpu/core_dynrec/decoder_basic.h" calling PAGING_ForcePageInit(). After doing so, your MakeCodePage() function should resemble:
static bool MakeCodePage(Bitu lin_addr,CodePageHandlerDynRec * &cph) {
Bit8u rdval;
//Ensure page contains memory:
if (GCC_UNLIKELY(mem_readb_checked(lin_addr,&rdval))) return true;
PageHandler * handler=get_tlb_readhandler(lin_addr);
if (handler->flags & PFLAG_HASCODE) {
// this is a codepage handler, and the one that we're looking for
cph=(CodePageHandlerDynRec *)handler;
return false;
}
if (handler->flags & PFLAG_NOCODE) {
//!!!!!!! Here's where the deleted if conditional used to be !!!!!!!!
if (handler->flags & PFLAG_NOCODE) {
LOG_MSG("DYNREC:Can't run code in this page");
cph=0;
return false;
}
}
// The rest of the function is the same...
}
This works, though it's really not pretty. [Insert collective shrug here.]
Undefined Function PAGING_ForcePageInit()
This is the next compilation error I received:
x86_64-pc-linux-gnu-g++ -march=native -O2 -pipe -std=c++0x -fpermissive -I/usr/include/freetype2 -Wl,-O1 -Wl,--as-needed -o dosbox dosbox.o save_state.o miniunz.o minizip.o unzip.o zip.o iowin32.o ioapi.o mztools.o cpu/libcpu.a debug/libdebug.a dos/libdos.a fpu/libfpu.a hardware/libhardware.a shell/libshell.a gui/libgui.a ints/libints.a misc/libmisc.a hardware/serialport/libserial.a hardware/parport/libparallel.a libs/gui_tk/libgui_tk.a libs/porttalk/libporttalk.a hardware/reSID/libresid.a -lSDL_sound -lasound -lm -ldl -lpthread -ltbb -lFLAC -lgthread-2.0 -pthread -lrt -lglib-2.0 -lSDL -lpng -lz -lfreetype -lz -lbz2 -lpcap -lSDL_net -lX11 -lGL -lfluidsynth -lphysfs -lz
cpu/libcpu.a(cpu.o): In function `CPU_FindDecoderType(long (*)())':
cpu.cpp:(.text+0x50f6): undefined reference to `CPU_Core_Dyn_X86_Run()'
cpu.cpp:(.text+0x5117): undefined reference to `CPU_Core_Dyn_X86_Trap_Run()'
cpu/libcpu.a(cpu.o): In function `(anonymous namespace)::SerializeCPU::setBytes(std::basic_istream<char, std::char_traits<char> >&)':
cpu.cpp:(.text+0x5609): undefined reference to `CPU_Core_Dyn_X86_Cache_Reset()'
hardware/libhardware.a(voodoo_interface.o): In function `Voodoo_PCI_Enable(bool)':
voodoo_interface.cpp:(.text+0xad3): undefined reference to `CPU_Core_Dyn_X86_SaveDHFPUState()'
voodoo_interface.cpp:(.text+0xae1): undefined reference to `CPU_Core_Dyn_X86_RestoreDHFPUState()'
collect2: ld returned 1 exit status
At this point, an uneasy lethargy begins to settle over the mind.
What is this? No, really. Just what in the Playerless Wastes of Azeroth is going on with this codebase? Has anyone actually ever succeeded in compiling this?
I begin to harbour doubts. Stupidly, I press on. And it hits me like a GCC-crafted bludgeon. I'm trying to compile Daum under a 64-bit architecture. Daum must no longer support 64-bit architecture! It doesn't actually say that anywhere, of course. But "by their fruits ye shall know them." But that doesn't make sense! Daum used to support 64-bit architecture. It's right there in the changelog:
Oct. 30. 2010
Fixed a crash while pausing DOSBox (Windows build)
Uploaded a build for Linux (x86, x86-64)
But things have changed.
So, I decided to try and set things right. It worked. I replaced the last line
of "src/cpu/core_dyn_x86.cpp" ("#endif") with the following lines, evaluated when not compiling on a 32-bit machine (i.e., when compiling on a 64-bit machine):
Bits CPU_Core_Dyn_X86_Run(void) {
return 0;
}
Bits CPU_Core_Dyn_X86_Trap_Run(void) {
return 0;
}
void CPU_Core_Dyn_X86_Cache_Reset(void) {
}
#endif
Voila! A Daum that almost compiles, links, and installs on 32- and 64-bit systems.
Dependency and Patch Errors
The compilation and linkage errors were pretty awful, right? Right. In the end, however, they were only a slice in the awful pie that is Daum compilation.
The OpenGL-HQ SDL Patch
Daum requires SDL 1.2 live sources to be patched with a homebrew OpenGL-HQ patch available under "Download" as "OpenglHQ patch", described as "...apply to SDL 1.2 hg | updated on Jun-26-2013".
"Great," I thought. "How hard can that be?" Finally, something that will just work! I followed the simple instructions in "linux.txt":
patch -up1 < openglhq-dosbox-for-sdl-20130726_msvc_gcc.patch
And the patch failed to cleanly apply.
patching file ./configure.in
Hunk #1 succeeded at 2906 (offset 30 lines).
patching file ./include/SDL_config.h.in
patching file ./include/SDL_video.h
patching file ./src/events/SDL_keyboard.c
patching file ./src/events/SDL_mouse.c
patching file ./src/main/macosx/SDLMain.m
patching file ./src/thread/pthread/SDL_systhread.c
patching file ./src/thread/win32/SDL_sysmutex.c
patching file ./src/video/SDL_glfuncs.h
patching file ./src/video/SDL_sysvideo.h
patching file ./src/video/SDL_video.c
patching file ./src/video/openglhq/Makefile
patching file ./src/video/openglhq/README
patching file ./src/video/openglhq/SDL_ohqthread.h
patching file ./src/video/openglhq/SDL_ohqvideo.c
patching file ./src/video/openglhq/SDL_ohqvideo.h
patching file ./src/video/openglhq/SDL_openglhq_pass1.fp
patching file ./src/video/openglhq/SDL_openglhq_pass1.h
patching file ./src/video/openglhq/SDL_openglhq_pass2.fp
patching file ./src/video/openglhq/SDL_openglhq_pass2.h
patching file ./src/video/openglhq/SDL_openglhq_pass3.fp
patching file ./src/video/openglhq/SDL_openglhq_pass3.h
patching file ./src/video/openglhq/SDL_openglhq_table.dat
patching file ./src/video/openglhq/SDL_openglhq_table.h
patching file ./src/video/openglhq/asciidump.c
patching file ./src/video/openglhq/hexdump.c
patching file ./src/video/openglhq/tablebuilder.ui
patching file ./src/video/openglhq/test.c
patching file ./src/video/quartz/SDL_QuartzVideo.m
patching file ./src/video/wincommon/SDL_lowvideo.h
patching file ./src/video/wincommon/SDL_sysevents.c
patching file ./src/video/wincommon/SDL_syswm.c
patching file ./src/video/wincommon/SDL_syswm_c.h
patching file ./src/video/windib/SDL_dibvideo.c
patching file ./src/video/windx5/SDL_dx5video.c
patching file ./src/video/x11/SDL_x11modes.c
patching file ./src/video/x11/SDL_x11modes_c.h
patching file ./src/video/x11/SDL_x11video.c
Hunk #1 FAILED at 131.
Hunk #2 FAILED at 1482.
2 out of 2 hunks FAILED -- saving rejects to file ./src/video/x11/SDL_x11video.c.rej
The problem is almost certainly DOS line endings. ykhwong's OpenGL-HQ patch contains spurious DOS-specific carriage returns (<Ctrl-m> characters). Since the OS X and Linux versions of SDL contain only Unix line endings, the patch fails to apply. Globally removing all carriage returns from the patch would probably do the trick. Probably. I only discovered the carriage return issue much later, however – long after the fact. (If I had discovered it at the time, I'm still not sure I could have mustered the interest in addressing it.)
The mind boggles. At this point, I was nearly in tears. Yes, DOSBox SVN Daum had reduced a grown man to incessant teeth gnashing and white-knuckled fists.
At the end of my rope, I stumbled across the official archive of OpenGL-HQ patches. No mention of the above patch. I still have no idea where ykhwong got it from. (Self-authored, I suspect...) At this point, I probably no longer care.
I downloaded the newest official cross-platform patch, crossed multiple fingers and toes, and attempted to patch SDL 1.2. It cleanly applied. And the flaming sword-wielding angels wept then.
Did I mention that you're not done yet, though? Even after applying the patch, you have to manually generate additional header and ".c" files. Assuming you're building SDL 1.2 in "/tmp/sdl":
mv SDL-1.2-openglhq-2012-10-19.zip /tmp/sdl/src/video
cd /tmp/sdl/src/video
unzip SDL-1.2-openglhq-2012-10-19.zip
cd ../..
make -C src/video/openglhq
...and I'm all out of bubblegum.
The OpenGLide Dependency
The "linux.txt" documentation omits all mention of a key dependency: OpenGLide, an open-source Glide to OpenGL wrapper. Installation is trivial, though completely undocumented. Their CVS repository contains no documentation – not a single "INSTALL" or "README" file or text file of any sort. In fact, the only set of instructions that Google willingly burped up come from a mid-2009 German language blog post. I wish I was kidding.
While OpenGLide's SourceForge project does offer source archives, the last archive was uploaded in 2002. The most recent change to their repository was committed in 2010. Needless to say, use CVS to install OpenGLide.
Note that OpenGLide depends on SDL 1.2 and thus must be installed after SDL as follows:
cvs -d:pserver:anonymous@openglide.cvs.sourceforge.net:/cvsroot/openglide login
cvs -z3 -d:pserver:anonymous@openglide.cvs.sourceforge.net:/cvsroot/openglide co -P openglide
openglide/
./bootstrap
./configure
make
make install
Rock the 3Dfx Casbah.
Configuration Globals
All sane autotools-based Makefiles internally list all libraries (LIBS), gcc and g++ options (CPPFLAGS), and g++-specific options (CXXFLAGS) required for compilation and linkage. The user doesn't have to define them. The Makefiles define them already.
But this is Daum. Before running "./configure", you have to manually export the following globals:
LIBS+=" -ltbb -lFLAC $(pkg-config --libs gthread-2.0)"
CPPFLAGS+=" -I /usr/include/openglide/ $(pkg-config --cflags gthread-2.0)"
CXXFLAGS+=" -std=c++0x -fpermissive"
export LIBS CPPFLAGS CXXFLAGS
The "linux.txt" documentation fails to mention the need to set ${CPPFLAGS} at all or that pre-4.7 versions of g++ require "-std=c++0x" rather than "-std=c++11" and improperly omits the "-fpermissive" from ${CXXFLAGS}. If you try to compile without these options, you're going to have a bad time.
It Goes On and On and On
But I'm stopping here. Oh, I should probably discuss the vortex of sanity-rending pain that was "/usr/include/GL/gl.h": manually commenting out about twenty functions from your system-wide OpenGL header (I'm not kidding) before compiling DOSBox SVN Daum. Who honestly thought that was a good idea? WHO!?!? Anyone? Because the following definition applies to them, writhing as they probably are in a heroin-induced haze of hypnagogic self-delusion:
"A bunch of mindless jerks who'll be the first against the wall when the revolution comes."
And with that, to quote "There Will Be Blood":
"I'm finished."
The Future
The ever-vigilant VOGONS community has already documented many of these same issues and their fixes. Consider the "decoder_basic.h" error. HAL 9000 publicly described a stable fix in late 2011. It never made its way into the patchset.
But this isn't HAL 9000's fault. It's no one's fault. No one's to blame. As often in real life, people didn't fail; the system of maintaining large patchsets through forum threads failed. The "decoder_basic.h" error wasn't left unfixed because of a lack of human attention. It was left unfixed because there's no infrastructure, organization, or documentation to support fixing errors in Daum or other patchsets outside of VOGONS. VOGONS, however awesomely helpful (and it is!), is not enough.
There's no bug tracker, so there's no sane way of managing compile- or runtime issues – especially when they don't effect the patchset maintainers themselves. There's no public wiki, so there's no sane way for users to manage these issues themselves in a structured manner. There's no hosted repository, code browser, or online documentation, so it's never clear just what the current patchset even contains.
This has to change, people. We can fix the broken system.
Officially, Daum is just another DOSBox patchset. A rather large patchset complete with its own dedicated websites (!), changelogs, source tarballs, and platform-specific installation packages – all internationalized into multiple spoken languages (e.g., Korean, English).
Unofficially, that doesn't sound like just another DOSBox patchset. That sounds like a DOSBox fork.
We can't call it that, of course. Forks are offensive. They divide volunteer contributors into fanatically opposed camps. We've all seen it: "ffmpeg" vs. "libav", "openoffice" vs. "libreoffice", "mysql" vs. "mariadb". So I fully appreciate that the community doesn't want to offend anyone. But I think it's time we called a spade a spade. Daum isn't just another patchset.
Because Daum isn't just another patchset, it's beginning to need things that ordinary patchsets don't: infrastructure, organization, and documentation.
Let's talk about infrastructure first.
Future Infrastructure
The Daum codebase needs to be under public revision control – ideally through an online project host providing code browsing, bug tracker, mailing list, and wiki functionality. Since I host several projects at github, I'm probably not the most unbiased source. But any free project host would do, including bitbucket (Mercurial "hg") and launchpad (Bazaar "bzr").
The codebase could also benefit from an eventual transition away from the current autotools-based make system. It's fragile, it's unwieldy, it's obscure; in a word, it's a mess.
It also requires significant modification. At the least, DOSBox patches requiring nontrivial changes to system dependencies (e.g., the OpenGL-HQ patch requiring SDL itself to be patched with OpenGL-HQ support) should be toggleable by command-line options (e.g., "./configure --disable-sdl-openglhq"). And those options should probably be disabled by default.
Since the build system has to be modified anyway, replacing it now with a modern, maintainable, and well-documented alternative makes some sense – say, CMake. From the downstream user perspective, CMake just works. As a Gentoo developer, I cherish technology that "just works." Autotools never "just works." Painfully often, it never works at all.
Future Organization
Get the infrastructure right and the organization follows. I wouldn't be surprised to see a self-assembling community of developers and users leap into existence the picosecond a public repository goes live.
As a community outsider, I probably have nothing constructive to add here. Next!
Future Documentation
The documentation is a hit and miss affair. What little documentation there is is segregated to "linux.txt" in the source. Despite the filename, this file documents installation for all platforms. It fails to document a significant dependency installable only from live CVS sources: OpenGLide. It also fails to document how to actually configure any of the included patches after installation. How do you enable patched support for OpenGL-HQ, fluidsynth, Glide, or the xBRZ scaler? You Google it. Like every God-fearing DOSBox user, you pray that whatever Google unearths (almost always from this forum) actually relates to whatever patch Daum happens to have integrated.
Ideally, Daum-specific instructions should also be merged into the general DOSBox instructions ("INSTALL", "README"). I'd settle for anything, though.
The End My Friend
Now I'm really finished. I drink your milkshake!