so, here is the general architecture:
I have a dynamic C compiler, which targets x86 and x86-64 (my current
targets are Win32, Win64, Linux on x86 and x86-64);
I have an interpreter for JBC (JavaByte Code);
I have a garbage collector, dynamic typesystem, and class/instance object
system (JVM-style);
I have an interpreter for x86 (partial x86-64 support).

I have several conventions in use:
a name mangling scheme which combines together parts of the IA64 name
mangling and the JVM / JNI signature system;
I have an exception handling scheme which works on x86 and x86-64, and which
combines together Win64 SEH (prologues+epilogues, structural unwinding) and
the JVM EH model (although general code support is fragmentary, so EH is
unlikely to work correctly at present);
dynamically generated code generally uses the native calling conventions
(cdecl, Win64, SysV, as appropriate).

in general, COFF is used internally as the main object format, and several
major components utilize PE/COFF as the binary format (for EXE's and DLL's).

now, getting more specific...

x86 interpreter:
primarily simulates userspace, makes no real attempt at emulating OS-level
features or HW (IOW: it is not intended to be an emulator, so no booting
Windows in this thing...);
currently, it implements "most" of the x86 ISA (nevermind poor performance,
likely bugs, and a few holes in the x87 and SSE support...).

its purpose then will be more to serve a role similar to that of a bytecode
interpreter (such as in the JVM or .NET, but using x86 machine code rather
than a bytecode such as JBC or MSIL).

the reason for x86 in my case, is largely that I already have LOTS of
facilities for x86, and was in need of VM, and came to the opinion that it
would take LOTS of effort to implement support for a bytecode (even though
the bytecode itself would be simpler).

the interpreter itself uses PE/COFF, and manages loading and linking PE
images.

now, I am getting to the part in question:
it is "probably" the case that code targetting the interpreter will
generally run in the interpreter, so I may relax "strict" compatibility with
native x86 some (and instead support a few facilities more typical of those
found in bytecoded VMs).

in particular, I am considering adding a feature very much unlike in native
x86:
the entire Virtual Address range 0xC0000000-0xFFFFFFFF will be reserved as a
big tagged value space (managed directly by the interpreter, rather than,
say, inaccessible and owned by the OS kernel...).

2.3.27: 11.Tag.Value

this space will then be used to "marshall" data between the inside and
outside worlds (essentially serving as a large handle space).

in the inside world, this space will not be directly accessible (as memory
at least...), but pointers into this region will be regarded as handles to
external objects or values (although, some types will be mapped directly).

these will generally be mapped to the external dynamic typesystem (includes
the class/instance system, and many other facilities), or to external
functions or methods.

attempts to resolve the address of an external function may in effect
resolve to one of these handles (mapping as necessary), so potentially code
may be linked directly against some of these handles.

when making a call to an imported function, it will then attempt to call to
one of these addresses, which will generate a special fault ('JumpFault', or
maybe a "special" GPF). this fault will be then trapped by the interpreter,
and the arguments marshalled, followed by the call being redirected as
appropriate, and when finished, the return value is marshalled and returned
to the caller (within the interpreter).

this mechanism would likely be done via special listings representing
"virtual" DLLs, which would basically provide FFI-related info. probably
these will be ASCII text files containing function signatures and security
info.

calls into the interpreter can similarly be handled via special thunks (and
probably similar V-DLL's), allowing an essentially bidirectional interface.

note: this will not likely impact coding style much, since many of my APIs
are based on opaque handles anyways, so neither the caller nor the callee
will notice that the object has been marshalled via a handle between
disjoint address spaces (however, this will not allow passing references to
raw memory objects, such as raw buffers or C-style arrays).

note that this mechanism would also be used by the interpreter to manage
gluing to the OO facilities (in place of the auto-generated thunks I am
using in many places in native-land...).

tag space:
0/1: flonum-28
2: fixnum
3: external object
4: external function
5/6: reserved
7: Tag2 (for smaller values)

Tag2:
2.3.7.20: 11.111.Tag2.Value
Tag2=127: special atomic values

(that, or, I forsake this kind of static tag scheme and use "linear space
ranges", similar to what I do using, errm, this same address range in native
32-bit x86).

security:
I am considering the idea that each virtual thread will have an associated
set of "privledges" (invisible from within the interpreter), and maybe
imported functions/objects will be associated to particular ACL's.

attempting to utilize an object or function for which access is not granted
via an ACL check will result in a #GP / GPF being thrown (or maybe a
security-related exception, either way...).

(BTW: if anyone knows of a decent BSD-liscensed C library, that would be
convinient, as so far I only know of Newlib, which is GPL...).

I had not thought of this one...

The Flux OSKit converted FreeBSD into a library, including the FreeBSD C
library:
http://www.cs.utah.edu/flux/moss/moss.html

I am currently looking into the PDOS C library (PDPCLIB), which if it has
one good point, it is that it is small enough to be fairly easy (in general)
to figure out and to modify (and was impressively easy to get to be able to
build and work).

however, I have been considering some need for basic POSIX functionality,
which I had assumed add myself (mostly, because this code would be operating
"across the barrier").

but, it is all a start at least...