I'm working on a moderately sized (about 23,000 words) Glulx project and  
running into what appears to be a memory problem, one that causes the  
interpreter (both glulxe and git, running under the hood in Zoom) to hang  
and suck up all available CPU (more details below). This project played  
very nicely before I revamped the namespaces and logic of some of the  
extensions it depends on. Because I had to update so many things in the  
main project file (to conform to the extension changes) before it would  
compile, there aren't many clues to go on...

My question, before I start the slow, laborious process of trying to hunt  
down the bug, is whether there are I6 memory settings that I should try  
tweaking? (I have tried running MAX_STATIC_DATA up to 3000000 with no  
luck.)

Thanks for any help!

Erik

In case it's a help to anyone, here's the trace of git trying to run the  
game (from Mac OS's Activity application):

Sampling process 22841 for 3 seconds with 1 millisecond of run time  
between samples
Sampling completed, processing symbols...
Analysis of sampling git (pid 22841) every 1 millisecond
Call graph:
     2257 Thread_2507
       2257 start
         2257 _start
           2257 main
             2257 glk_main
               2257 gitWithStream
                 2257 gitMain
                   2257 startProgram
                     2129 startProgram
                     112 accel_get_func
                       112 accel_get_func
                     16 memRead32
                       16 memRead32

Total number in stack (recursive counted multiple, when >=5):

What led you to call this a memory problem? You say you changed the
extension and game logic: could you have introduced an infinite loop
somewhere?

"RULES ALL" might help, but if the game is getting stuck in a loop,
you might need to run the game in a console-mode interpreter (like
glulxe + CheapGlk) in order to see any output.

I guess I can translate these concerns into one last-ditch question before  
I tear down the story file and rebuild it bit by bit: What kinds of issue  
could cause a Glulx file to hang at startup? Obviously, that question is  
too broad to be likely to draw any response, but a shot in the dark can't  
hurt...

At startup, the VM itself does almost nothing. It loads in the string
encoding tables, but I've never seen that go wrong.

The game code starts with the "starting the virtual machine" activity,
and that sets up Glk state for windows and things. I would guess that
that's where your game is getting stuck, because the library has a lot
of hooks for you to customize that process. Also because a breakage
here would prevent windows from being created.

After that are several more setup rules (in the startup rules
rulebook). Most of them don't call game code, so they are hard to
break. The major exceptions are "when play begins" (the famous one),
preceded by "start in the correct scenes" (which must call a bunch of
scene condition tests).

--Z

As an aside, you might be interested in the last thing that caused this  
error for me, since it could possibly implicate the string encoding  
tables--though I'm not sure what exactly they do, so even bringing this up  
may be off-base. While working on a method to do what I was trying to do  
in this thread from a couple of weeks ago:

http://groups.google.com/group/rec.arts.int-fiction/browse_thread/thr...

I began by using a phrase something like this:

        To debug (T - indexed text):
                if using the debugging option, say T.

so that one could write DEBUG "Drawing a rectangle in [window]" and the  
message would appear or not depending on whether the use option was  
defined. However, passing indexed text into the phrase like this caused  
the interpreter to hang at startup. Change the temporary variable to  
T--that is, passing in text rather than indexed text--worked fine, though.  
Unfortunately, I don't remember exactly what code I was using (the code  
did employ glk_set_window to change the output stream though, probably via  
the Flexible Windows phrase wrapper for that function), but hopefully this  
conveys the idea of what was going on...

repeat with current-element running through display-active paging-buttons:
        let x-coord be entry 1 of the origin of current-element;
        let y-coord be entry 2 of the origin of current-element;
        display image (image-ID of current-element) in paging-window at (x-coord)  
by (y-coord) with dimensions (nav-size) x (nav-size);
        if current-element is graphlinked:
                set a graphlink in paging-window identified as current-element from  
(x-coord) by (y-coord) to (x-coord + nav-size) by (y-coord + nav-size) as  
the linked replacement-command of current-element;

This causes the interpreter to hang even though it is most definitely not  
executed at startup. This is part of a window-drawing rule, and the window  
isn't opened until later in the game. It seems as though anything having  
to do with grabbing the two list entries from the origin list property of  
the button object within this routine causes interpreters to hang. I do  
this throughout the rest of the code, so I'm not sure what the issue with  
this particular instance might be. However, it seems to do with assigning  
the list entries to temporary variables. All that is needed to crash the  
interpreter, in fact, is one of the first two lines of the loop (the  
temporary variable assignments). On the other hand, this works, so there  
is no issue with reading from the list itself:

repeat with current-element running through display-active paging-buttons:
        say "[current-element]: [origin of current-element].";

I have tried variant phrasings and passing the variable assignments  
through different routine, but the result is always the same. The only way  
I have found to fix the issue is to used global variable instead of  
temporary variables. Can this be anything but a memory issue?

I'll keep looking into this and report back on anything I find.

Does it happen if you surround that repeat statement with "if 0 is 1",
so that it's *really* not executed?

What is that repeat statement in?

--Z

A test of the status of the "g-present" flag (see Flexible Windows) that  
is responsible for this test shows that it contains the correct values for  
each of the windows by the time of "when play begins". All windows but the  
main window are initialized as g-unpresent in any case. So, this plus the  
"if 0 is 1" test seems to indicate that the problem is not in the  
execution of this code.

The problematic instance is one of perhaps 50 places (I'm estimating)  
where I access entry 1 of the origin (list of numbers) property of some  
object, but the only one that creates the issue. I'll look into whether  
there is anything different about this particular case. I am afraid that I  
may not be able to turn up the root cause, though.

--Erik

I think the next step is to compile it with this form, and then
comment out those lines entirely (but leave the stub function in
place), and compile it again. Compare the auto.inf files that I7 is
producing in each case.

If you're lucky, they will differ slightly. More likely, they will
differ all over, but maybe in a systematic way -- renumbered functions
or something. It could be a veritable pain in the butt.

Looking at the generated I6 for the bad function is probably useful
too.

If all of this is getting to heavy, feel free to throw the files at me.

--Z

I've pasted the diff below. You can see that in the crashy case, the
rule function is actually two nested functions -- the outer one
fiddles with the blockv_stack array, and the inner one does the work.
(Presumably so that if the inner one includes an early "return",
blockv_stack is still unfiddled afterwards.)

That is how the bad code can still affect the program behavior even
when it is if'ed out.

I think the rule is still being *called* at startup, though. The full
diff only showed differences in the contents of this rule, plus
changes in function and constant numbering. It's possible that there's
a bug in the latter -- I didn't try to match up all the labels and
prove they were identical. But it's much more likely that the bug is
in this I6 function, which means it can only manifest when the
function is called.

That doesn't tell us what the bug *is*, or whether it's your mistake
or a compiler bug. However, I have come to the end of what I can
figure out. You should throw this information at Graham.

You said that using global variables instead of locals works around
the problem, so that's probably the right way to go for now.

--Z

***************
*** 30647,30657 ****
     rfalse;
  ];
  ! A window-drawing rule for the paging-window ( this is the paging-window drawing rule ):
! [ R_1087
      t_0 ! Local variable e.g. 'margin' = NUMBER_TY
      t_1 ! Local variable e.g. 'nav-size' = NUMBER_TY
      t_2 ! Local variable e.g. '?-1,-1?' = OBJECT_TY
      t_3 ! Local variable e.g. '?-1,-1?' = OBJECT_TY
      ;
     if (((parameter_object == O72_paging_window))) { ! Runs only when pattern matches
     if (debug_rules) DB_Rule(R_1087, 1087);
--- 30647,30668 ----
     rfalse;
  ];
  ! A window-drawing rule for the paging-window ( this is the paging-window drawing rule ):
! [ R_1087 ;blockv_stack-->(blockv_sp+1) = BlkValueCreate(85,0,57);
!     blockv_stack-->(blockv_sp+0) = BlkValueCreate(85,0,57);
!     blockv_sp = blockv_sp + 2;
!     blockv_stack-->(blockv_sp++) = R_SHELL_51(blockv_sp-2);
!     blockv_sp = blockv_sp - 3;
!     BlkFree(blockv_stack-->(blockv_sp+1));
!     BlkFree(blockv_stack-->(blockv_sp+0));
!     return blockv_stack-->(blockv_sp+2);
! ];
! [ R_SHELL_51 I7BASPL
      t_0 ! Local variable e.g. 'margin' = NUMBER_TY
      t_1 ! Local variable e.g. 'nav-size' = NUMBER_TY
      t_2 ! Local variable e.g. '?-1,-1?' = OBJECT_TY
      t_3 ! Local variable e.g. '?-1,-1?' = OBJECT_TY
+     t_4 ! Local variable e.g. 'temp-coord-x' = NUMBER_TY
+     t_5 ! Local variable e.g. 'temp-coord-y' = NUMBER_TY
      ;
     if (((parameter_object == O72_paging_window))) { ! Runs only when pattern matches
     if (debug_rules) DB_Rule(R_1087, 1087);
***************

The generated code looks OK to me, so I would guess the rule is being
called at an inappropriate time. For example, if it's called during
the "starting the virtual machine" activity -- which opens windows, so
this may be the case -- the heap won't be initialized yet: the
"virtual machine startup rule", which triggers that activity, runs
just before the "initialise memory rule".

Yes, that makes sense. Thanks.

So: either use global list variables, or set up your own wrapper rule
(which invokes the "real" rule, the one with temporary list variables,
only after the window exists).

--Z

<code>
A glulx object-updating rule (this is the updating-after-undo all rule):
        follow the refresh windows rule.

This is the refresh windows rule:
        let old current be the current g-window;
        repeat with item running through g-present g-windows begin;
                change current g-window to the item;
                follow the window-drawing rules for the item;
        end repeat;
        if the old current is g-present
        begin;
                change current g-window to the old current;
                set focus to the current g-window;
        end if.
</code>

Comment out the glulx object-updating rule and the interpreter hang goes  
away.

Zarf, the "repeat with item running through g-present g-windows" in this  
should be function as the kind of wrapper you're talking about, ensuring  
that the problem window drawing rule doesn't fire unless the associated  
g-window is marked g-present. Since all g-windows (except the main-window  
object) are initialized as g-unpresent, this rule should fire only for the  
main-window at startup. Window objects are marked g-present only when  
opened (which in this project doesn't occur until 2-5 turns into the  
game)--or in IdentifyGlkObject (phase 2); I believe the latter should only  
happen when GGRecoverObjects() has identified the object as already  
existing, but I may be missing something.

In any case, it seems that somehow the window-drawing rules are called via  
VMInitialize-->GGRecoverObjects-->IdentifyGlkObject despite the fact that  
the windows shouldn't exist yet--perhaps they are somehow created and then  
destroyed in the course of IdentifyGlkObject. As I indicated in an earlier  
post, they definitely DO NOT exist by the time we get to when play begins,  
when g-unpresent is properly set and GLKLIST confirms that the only window  
in existence is the main window (gg_mainwin).

All that said, I do have doubts whether there isn't something else in the  
code causing the problem (i.e., it may be a bug with my code rather than  
Flexible Windows). The first version of this project (before I did the  
makeover mentioned in the initial post) properly zeroed out references to  
Glk objects on restart, while the new version apparently does not,  
crashing with "Glulxe fatal error: Reference to nonexistent Glk object"  
during the process of opening up the various graphics windows after  
restart. Since this is also happening in InitGlkObject, there may be  
something else going on here. (Flexible Windows should be handling all of  
this on its own, but possibly I've intervened somehow.)

In any case, thanks much for the help! If it's warranted, I'll follow up  
with a bug report to Jon Ingold.

I'm going to lay out a bunch of code below in case someone wants to help  
me figure out what's going on. However, I recognize that most will not, so  
I'll give the gist here: the problem seems to be that an object-based  
rulebook behaves differently when run through IdentifyGlkObject (Emily's  
Glulx Entry Points) than does a simple rule. I'm not sure why this would  
be, and I'm having trouble finding a plausible explanation for it.  
Basically, if we assign a simple rule as a property of the window object,  
as in the first example below, the rule doesn't fire until the window is  
opened (i.e., it doesn't happen before the heap is created, which is what  
causes the crash). If instead we run an object-based rulebook with a  
pattern-match on the name of the window (e.g., "a window-drawing rule for  
the graphics-window"), then the window-drawing rule *is* run before the  
heap, regardless of the g-present flag that "protects" this from occurring  
when the rule is a simple one.

My question is, then: Is there something different about object-based  
rulebooks that would cause them to function differently in this situation?  
(By the way, this is not the only bug I'm seeing from the interaction  
between the IdentifyGlkObject routine and object-parametrized rulebooks:  
there also seems to be a problem with the zeroing-reference rules--that  
is, phase 0 of IdentifyGlkObject--which causes garbage collection to fail  
when we restart, at least with more than three windows open.)

OK, here is code using the current version of Flexible Windows (v8) that  
works fine, with a rule property attached to the g-window object:

<code>
Include Flexible Windows by Jon Ingold.

Test is a room.

The offending list is a list of numbers variable. The offending list is
{2, 4, 5}.

The graphics-window is a g-window. The position is g-placeabove. The
measurement is 60. The back-colour is g-black. The drawing rule is the
graphics-window drawing rule.

The main-window spawns the graphics-window.

This is the graphics-window drawing rule:
        let T1 be entry 1 of the offending list;
        let T2 be entry 2 of the offending list.

When play begins:
        open up the graphics-window.
</code>

However, in its next version Flexible Windows will probably be changing to  
utilize an object-based rulebook instead ("Follow the window-drawing rules  
for the current window"). (The major diffs between the hacked Flexible  
Windows "v9" I am using here and the current version are given beneath my  
signature. Or download it here:  
http://dl.dropbox.com/u/947038/Flexible%20Windows%20v9.i7x ).

So, this code, in which the only change is to the preamble of the drawing  
rule, crashes the interpreter:

<code>
Test is a room.

The offending list is a list of numbers variable. The offending list is
{2, 4, 5}.

The graphics-window is a g-window. The position is g-placeabove. The
measurement is 60. The back-colour is g-black.

The main-window spawns the graphics-window.

A window-drawing rule for the graphics-window (this is the graphics-window
drawing rule): [this line is changed]
        let T1 be entry 1 of the offending list;
        let T2 be entry 2 of the offending list.

When play begins:
        open up the graphics-window.
</code>

Any ideas?

Thanks,
Erik

v8
A g-window has a rule called drawing rule.
The drawing rule of a g-window is usually the do-nothing rule.
This is the do-nothing rule: do nothing.

"v9"
A graphics g-window is a kind of g-window. The type of a graphics g-window
is g-graphics.
A text-buffer g-window is a kind of g-window. The type of a text-buffer
g-window is g-text-buffer.
A text-grid g-window is a kind of g-window. The type of a text-grid
g-window is g-text-grid.
The window-drawing rules are an object-based rulebook.

v8
This is the refresh windows rule:
        let old current be the current g-window;
        repeat with item running through g-present g-windows begin;
                change current g-window to the item;
                follow the drawing rule of the item;
        end repeat;
        if the old current is g-present
        begin;
                change current g-window to the old current;
                set focus to the current g-window;
        end if.

"v9"
This is the refresh windows rule:
        let old current be the current g-window;
        repeat with item running through g-present g-windows begin;
                change current g-window to the item;
!               follow the window-drawing rules for the item;
        end repeat;
        if the old current is g-present
        begin;
                change current g-window to the old current;
                set focus to the current g-window;
        end if.

v8
The last After constructing a g-window :
        if the pending g-window is g-present, follow the drawing rule of the
pending g-window;

A rulebook is a list of routines, and invoking the rulebook means
calling every routine in the list until one of them says you're done.
Normally those routines are the actual rule implementations; if a rule
has a condition in the preamble, it will be checked inside that
routine. But when Inform generates a wrapper routine to manipulate the
heap, the *wrapper* goes in the rulebook.

That means when you call a rulebook, you potentially do heap
manipulation for *all* the rules in the rulebook, even the ones that
will be skipped because of a preamble condition. So if any of your
window-drawing rules use local heap variables, the whole window-
drawing rulebook is unsafe to call from VM_Initialise!

One workaround, as you've found, is to use global variables instead so
there's no wrapper routine. Another would be to rearrange the startup
rulebook to make the heap available when this rule is called, which is
a workaround if you do it from your own code, but could be a long term
solution if done in the Standard Rules.

The problem would also go away, at least in this case, if Inform
hoisted the rule condition checking into the generated wrapper
routine. That's a little complicated and not a robust solution on its
own (what happens if the rule condition uses the heap?), but probably
a good thing to do anyway, because it's wasteful to create and destroy
a bunch of unused heap blocks for rules that are being skipped.

Thanks again!

I'm not sure about the performance hit either. In a traditional
interpreter, every instruction has virtually the same running time, so
a function call is just two extra instructions. In ZLR, call
instructions end the JIT segment, but so do some other common
instructions. (I don't know about the implementation of Parchment or
any other unusual terps.)

As for procedural rules -- I'm not convinced I7 needs them. The most
common use-cases that required them have already vanished: you can now
replace and reorder rules at compile-time.

The procedural rules I see nowadays are all of the form "ignore rule X
when condition C". But this too can be done with a compile-time rule
replacement, which I leave to the reader as an exercise. :)

I'm not saying that they should be yanked out of the system
immediately. But it would not be insane for I7 to be able to generate
two code models, an efficient new-style one when there are zero
procedural rules, and the current style if there are any.

--Z

I am fairly convinced that the biggest performance problem currently
facing I7 is function calls. Many modern games will make tens of
thousands of function calls, sometimes hundreds, each and every turn.
In all interpreters function calls are expensive, and in JIT engines
they are especially expensive! Reducing the number of function calls
will probably give a slight speed increase to standard terps but also
a huge increase to JIT terps. Even if there was a slight complexity
increase I think it would be worth it.

Now the rule bodies could even remain in functions, this would be one
option for procedural rules or repeated ones. The problem with rule
books as currently implemented is that they call tens, maybe hundreds
in some cases, of functions that will almost immediately return. The
most common situation is that a function's conditions will not match,
so that is what we optimise for.

As Zarf said, procedural rules aren't as useful now as they used to
be... I'd love to see some complex examples though. I think though
that they could still be implement as just additional conditions to
try before the standard ones.

I honestly do not see any utility to "Before Starting Virtual Machine"
because it lets you run code *before* the Glk things that are
initialized (windows, files, IO, etc.).

That is useless, because you can not "manipulate" resources that have
*not yet been created* (windows, sounds, for example). It will be
ideally something like "After Starting Virtual Machine", but although
you can use, it represents no change.

The "Virtual Machine starting" rule is running much much earlier than
GGRecoveryObjects(). That's the problem.

When I programmed Damusix, I needed something like the object
"LibraryExtensions" of Lib. 6/11, which was very useful, because
executing code (initialization of external libraries) just before
Initialise().

But when I wrote Damusix to Inform7, I found nothing that was like.
"First when play begins" is not good, because if someone changes the
execution order of rules in this rulebook, it all goes to hell, xD.
The strict order is lost. The Damusix's audio channels should be
created before the author could use the sound... and if the author
created in his own game a "FIRST when play begins", everything went to
hell. It did not work.

It would be very useful to have a rulebook that would implement rules
just before any "When play begins", as did the object
"LibraryExtensions" in Lib. 6/11.

I solved the problem as follows: at the beginning of the game is
running a series of rules in a particular order. I just got the
Damusix's initialization rule just after the system starts Glk [ie,
after running GGRecoveryObjects()]. Thus:

--------------------------------------------
Part 2 - Damusix Startup Process - Unindexed

The Damusix startup rule is listed before initialise memory rule in
the startup rules.

This is the Damusix startup rule:
  damusix-initialise.

To damusix-initialise:
  (- Damusix.InicializarGlk(); -)
--------------------------------------------

First run "Virtual machine starting", then is internally called
several routines, including a GGRecoveryObjects() and then run
"initialise memory". That's why it does not work use "Before/After
Starting Virtual Machine", because in any case be executed before
GGRecoveryObjects() routine.

My "hack" puts Damusix initialization just before the "initialise
memory", and therefore, just after GGRecoveryObjects(). However, the
initialization may have been positioned anywhere before "When play
begins" [but always after GGRecoveryObjects()], obtaining the same
effect.

I hope this serves to help someone. If this has nothing to do, ask for
an apology. As I said, I did not understand much of what is said in
this thread. =)

I apologize for Google's translated message. xD

Saludos! :-D
Eliuk Blau.

-----------------------------------------------------------------
P.S: Spanish text...

Hola. Yo pido disculpas si lo que voy a decir no tiene nada que ver
con lo que ustedes están hablando, porque no he entendido mucho (mi
mal inglés).

Yo, sinceramente, no le veo ninguna utilidad a "Before Starting
Virtual Machine" porque esto te permite ejecutar código *antes* que
the Glk things sean inicializados (windows, files, IO, etc.)

Eso no sirve para nada, porque no se pueden "manipular" recursos que
todavía *no han sido creados* (ventanas, sonidos, por ejemplo). Sería
ideal algo como "After Starting Virtual Machine", pero aunque se puede
usar, no representa ningún cambio.

La "Virtual Machine starting" rule se ejecuta mucho mucho antes que
GGRecoveryObjects(). Ese es el problema.

Cuando programé Damusix, yo necesitaba algo como el objeto
"LibraryExtensions" de Lib. 6/11, que era muy útil, porque ejecutaba
código (inicialización de librerías externas) justo antes de Initialise
().

Pero cuando porté Damusix a Inform7, no encontré nada que fuera
parecido. "When play begins" no sirve, porque si alguien cambia el
orden de ejecución de las reglas dentro de ese rulebook, todo se va al
carajo. xD El orden estricto se pierde. Los canales de audio de
Damusix debían crearse antes que el autor pudiera usarlos el sonido...
y si el autor creaba en el juego su propia "FIRST When play begins",
todo se iba al carajo. Ya no funcionaba.

Sería muy útil tener un rulebook que permitiera ejecutar reglas justo
antes de cualquier "When play begins", tal como lo hacía el objeto
"LibraryExtensions" en Lib. 6/11.

Yo lo solucioné el problema de la siguiente manera: al principio del
juego se ejecutan una serie de reglas en un orden concreto. Yo
simplemente metí la regla de inicialización de Damusix justo después
de que se inicie el sistema Glk [es decir, después de que se ejecute
GGRecoveryObjects()]. Así:

--------------------------------------------
Part 2 - Damusix Startup Process - Unindexed

The Damusix startup rule is listed before initialise memory rule in
the startup rules.

This is the Damusix startup rule:
  damusix-initialise.

To damusix-initialise:
  (- Damusix.InicializarGlk(); -)
--------------------------------------------

Primero se ejecuta "Virtual machine starting"; luego se llama
internamente a
varias rutinas, entre ellas a GGRecoveryObjects(); y luego se ejecuta
"initialise memory". Es por eso que no funciona usar "Before/After
Starting Virtual Machine", porque de todas formas se ejecutará antes
de GGRecoveryObjects().

Mi "hack" mete la inicialización de Damusix justo antes de "initialise
memory", y por lo tanto, justo después de GGRecoveryObjects(). De
todas formas, la inicializaciones podría haber estado posicionada en
cualquier parte antes de los "When play begins" [pero siempre luego de
GGRecoveryObjects()], obteniendo el mismo efecto.

Espero que esto sirva de ayuda para alguien. Si esto no tiene nada que
ver, pido una disculpa. Como lo dije, no he entendido mucho de lo que
se habla en este hilo. =)

Pido disculpas por Google's translated message. xD