[jimtcl] paradigm shift to run scripts from ROM (flash)

[Mark H]

Hey guys,

I have a small project with Jim on an STM32F4 microcontroller.  It has
about 104 KB RAM available to the app.  That actually works pretty well,
and can navigate its UI on a 20x4 character LCD.  I expect to use Jim on
other MCU projects as well.

Predictably, it's starting to have serious RAM-cram.  So I've been looking
at ways to run scripts directly from where they reside in flash
(effectively ROM).  That way the entire application script (plus Jim
stdlib) doesn't get copied to RAM before it even begins interpreting.  And
then each proc body copied to more RAM etc.

So, some open questions:

1)  Has anybody else tried something similar?  How did you approach it and
how well did it work?

2)  I've tried a few approaches so far.  Each of those has "almost" run
correctly.  The basic problem is that Jim's (0.79) hashtables and certain
other functions rely on null-terminated strings.  They discard or ignore
the (in most cases) known length of the string.  But those nulls are not
present when (for example) command or variable name tokens point to ROM
instead of RAM, because I've shut off the strdup that would have made the
RAM copy.  The original script in ROM has the remainder of the script where
the interp is expecting the null to be.  I've tracked down and fixed those
issues in a few important functions, but certainly not all.  I've gotten a
few lines of script to run that way, so it probably can work for the rest.
It looks like it might be a long road to complete that approach, and would
likely introduce bugs wherever i've missed a spot.  I have no obvious way
to run Jim's test suite on such a small host, and no obvious way to make
scripts read-only on a PC to run the tests there instead.  So...

3) ...Instead, I'm starting to seriously consider a comprehensive
solution.  Such as: throughout jim.c, replace all char* function parameters
with a new structure instead, such as "Jim_Text", which carries both the
char* and the known length.  That would mean basically a wholesale paradigm
shift from C-style strings to known-length strings.  You might say
Pascal-style strings, but I wouldn't store the length at the front of the
string data like Pascal does.  That makes it too easy to overlook a piece
of code that needed the rework, introducing insidious bugs.  The new design
should pass all existing test suites verbatim.  And it should be able to do
so on a PC, where the tests are available, proving that it's likely to also
work on the MCU, where tests aren't available.

4) Is anybody else interested in seeing that approach adopted?  There might
be other benefits besides running scripts from ROM.  One benefit might be a
bit faster execution, since most strlen() calls or other null-byte searches
are eliminated.  Currently those are happening every time a hash key is
computed, and in other hotspots.

5)  I heard Steve mentioning recent improvements to make Jim insensitive to
nulls embedded in data.  Anybody care to guess how that work interplays
with this approach?

6)  Would anybody else be interested in helping with the extensive rework?

7)  Are there any Tcl TIP's, or Jim open issues, that relate to my goal, or
to this paradigm shift either?

Thanks for any clues!

--
TheMarkitecht
"I'm always disappointed when a liar's pants don't actually catch on fire."
https://github.com/TheMarkitecht/slim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.btbrotary.org.au/pipermail/jimtcl/attachments/20210131/0b7f1700/attachment.html>