[Encore] Output capacity in lambdamoo: altering MAX_QUEUED_OUTPUT

[Daniel Jung]

Hi there


Resume:
-------
I have a problem in output peformance, and I wonder if altering 
(augmenting) MAX_QUEUED_OUTPUT in options.h could improve serving high 
network activity, or if someone could help with other solutions to the 
problem.

Background:
-----------
I am running a number of educational moos on a web server. These moos 
are web-based (encore 5). Traditionally, the lambdamoo server creates 
and keeps a telnet channel to the client, and prints out one line here 
and one line there. Now since the moos are web based, the database 
produces complete web pages and lets the server print all the lines to 
the client (a web browser). In the last version of encore, even the chat 
is HTML based, meaning instead of a line pushed to the client, the line 
is written into a player property, and a complete page is produced which 
contains the property's contents (chat output), and printed to the 
client. This works very well.

But if I use a web object forcing a page on each participant's web frame 
(online slide show), at the same time as they are notified of the change 
in the chat frame, there are, say, 500 lines produced twice for each 
participant. If there are 30 people connected, that makes 30,000 lines 
to be delivered by the moo server at exactly the same time.

The in-moo web server channels everything through one verb ($httpd:get) 
which checks for cookie, authenticates, breaks down the HTTP GET command 
into pieces, translates them into moo commands, does in-moo access 
checks and piles up the lines for complete HTML pages, pumping them to 
the client (conn) line by line with the notify() routine. At the end, 
the code goes

****************************************
for line in (result)
  while (!notify(conn, line, 1) && conn in connected_players(1))
    suspend(0);
  endwhile
endfor
if (buffered_output_length(conn))
  while (buffered_output_length(conn))
    suspend(0);
  endwhile
endif
return;
****************************************

The code before that does a lot of things, but there are many suspends 
scattered about, so that should be taken care of.


Problem:
--------
It happens ever so often that heavy use of the server (many people 
connected and doing the same thing, teachers showing heavy web objects 
to everybody simultanously) leads the moo to stall. I'm not talking 
about lag, but about stalling, and these lines above cast an E_QUOTA 
error. The moo in itself seems to be stable (no real crash), but the 
users are getting timeouts and tracebacks. This is very frustrating. Now 
I am sure the web server the moo is running on could handle the 
pressure, as it is a top notch box. But lambdamoo is still single 
threaded and, uhm, old, and piles up all the output before bursting out 
(no multipart, no push, no flush). The thing is that online seminars 
with exactly the same course program, and commands, with, say, five 
people, work marvelously. The code above in itself runs smoothly, but if 
the verb is called 30-fold, there might be problems. However, the 
E_QUOTA is not raised in the call to $httpd:get, but inside it, in the 
notifying loop.


Environment:
------------
There is virtually nothing on the web server (box) but ten moos like 
this, a modestly visited homepage, php and some mysql. So far, only one 
of the moos is giving us this headache. We reduced the database from 300 
MB to 100 MB by deleting old stuff (I know, still big; mainly chat 
recordings and lectures), and I am checking for background services, 
nothing there. The lag on $login uses to be zero (:sampled and 
.current). The same goes for installations of the software on other 
servers where I got feedback that the same behaviour (E_QUOTA) is 
encountered. (I haven't verified if there are checkpoint dumps in the 
"crash"-occurences; I should configure those to never happen during 
online classes I guess.)

I know, lambdamoo was never build for that, it was designed to print out 
the occasional line here and there, and that works like a charm. Our 
students and teachers depend on our systems though, and we desperately 
need to find a solution.


Ticks and seconds:
------------------
options.h gives me

#define DEFAULT_MAX_STACK_DEPTH    50
#define DEFAULT_FG_TICKS    60000
#define DEFAULT_BG_TICKS    30000
#define DEFAULT_FG_SECONDS    10
#define DEFAULT_BG_SECONDS    6

which is already augmented from the original. The in-db calls are

$command_utils:suspend_if_needed
   if (this:running_out_of_time()) ... suspend(x) ...

$command_utils:running_out_of_time
   return ticks_left() < this.max_ticks_left || seconds_left() < 
this.max_seconds_left;

$command_utils.max_ticks_left = 4000
$command_utils.max_seconds_left = 2


Buffered output:
----------------
;buffered_output_length() gives me 65536 bytes. This is pretty standard 
I guess, coming from options.h:

#define MAX_QUEUED_OUTPUT    65536
#define MAX_QUEUED_INPUT    MAX_QUEUED_OUTPUT
#define DEFAULT_CONNECT_TIMEOUT    300

(I just notice that INPUT_QUEUE_TIMEOUT is NOT defined here...)

I have a feeling (but may be completely wrong) that augmenting 
MAX_QUEUED_OUTPUT could help us. Again, the box this thing is running on 
is a potent one and should be able to handle more pressure from the moo.

The ChangeLog doesn't say much about buffered_output_length() which is 
not already in the db help; the options.h file doesn't give instructions 
on if, how and why this could be changed; consulting, e.g.,  
http://www.moo-cows.com/maillist/archives/message.cgi?310 didn't help me 
much either.


So my questions are:
--------------------
(1) Has anyone altered and tested MAX_QUEUED_OUTPUT in options.h before, 
and with which results?
(2) Is there, theoretically, something to be gained by augmenting the 
value there?
(3) Should I rather be looking into the fork() builtin, since suspends 
somehow fork?
(4) Are there any other suggestions which might help us?


Thanks a lot!

- Daniel