[jimtcl] paradigm shift to run scripts from ROM (flash)

Wed Feb 3 16:30:34 UTC 2021

Patrick,

Thanks for the tips!

Yes please email me the old thread, at my address below.

Likewise any tips you gained from Lua.  Was Lua running from ROM?  Did it
use known-length strings inside the interp?

STM32 is a big product line.  Its capabilities are pretty typical of all
ARM microcontrollers.  I guess STM32 is so popular it's practically setting
the bar for ARM MCU's these days.  The F4 series has a parallel external
memory bus called FSMC, but I'm not using it on this project.  It would
really help.  SPI RAM chips are available but I expect they'd be too slow
for Jim's big demand on memory bandwidth and random address access.

I should have made it clear; I'm running Jim on bare metal here, no OS at
all.  Often I find an OS/RTOS to be more trouble than it's worth.

World: "It doesn't sound possible."
TheMarkitecht:  "Dude, hold my Cheetos..."
(bangs away furiously at keyboard)

I've had Jim running on bare metal for several weeks now.  At this point
I'm running strictly the core (jim.c).  Most of the modular functionality
like file systems is meaningless in a bare metal environment.  So I shut
that off right away.  Eventually I'll bring back whichever pieces make
sense there.  At one point I had turned on Jim references, and regex, to
try out Slim OOP on it (https://github.com/TheMarkitecht/slim).  But alas,
that will need more RAM.  Maybe more will become available if I can run
directly from flash with known-length strings.

--
TheMarkitecht at gmail.com

"GUI: (noun) - a method of organizing all my console windows."

Slim OOP for Jim Tcl:  https://github.com/TheMarkitecht/slim

On Wed, Feb 3, 2021 at 7:01 AM Patrick <patrick at spellingbeewinnars.org>
wrote:

> On 2021-02-02 4:50 p.m., Patrick wrote:
>
> On 2021-01-31 2:57 p.m., Mark H wrote:
>
> Hey guys,
>
> I have a small project with Jim on an STM32F4 microcontroller.  It has
> about 104 KB RAM available to the app.  That actually works pretty well,
> and can navigate its UI on a 20x4 character LCD.  I expect to use Jim on
> other MCU projects as well.
>
> Predictably, it's starting to have serious RAM-cram.  So I've been looking
> at ways to run scripts directly from where they reside in flash
> (effectively ROM).  That way the entire application script (plus Jim
> stdlib) doesn't get copied to RAM before it even begins interpreting.  And
> then each proc body copied to more RAM etc.
>
> So, some open questions:
>
> 1)  Has anybody else tried something similar?  How did you approach it and
> how well did it work?
>
> 2)  I've tried a few approaches so far.  Each of those has "almost" run
> correctly.  The basic problem is that Jim's (0.79) hashtables and certain
> other functions rely on null-terminated strings.  They discard or ignore
> the (in most cases) known length of the string.  But those nulls are not
> present when (for example) command or variable name tokens point to ROM
> instead of RAM, because I've shut off the strdup that would have made the
> RAM copy.  The original script in ROM has the remainder of the script where
> the interp is expecting the null to be.  I've tracked down and fixed those
> issues in a few important functions, but certainly not all.  I've gotten a
> few lines of script to run that way, so it probably can work for the rest.
> It looks like it might be a long road to complete that approach, and would
> likely introduce bugs wherever i've missed a spot.  I have no obvious way
> to run Jim's test suite on such a small host, and no obvious way to make
> scripts read-only on a PC to run the tests there instead.  So...
>
> 3) ...Instead, I'm starting to seriously consider a comprehensive
> solution.  Such as: throughout jim.c, replace all char* function parameters
> with a new structure instead, such as "Jim_Text", which carries both the
> char* and the known length.  That would mean basically a wholesale paradigm
> shift from C-style strings to known-length strings.  You might say
> Pascal-style strings, but I wouldn't store the length at the front of the
> string data like Pascal does.  That makes it too easy to overlook a piece
> of code that needed the rework, introducing insidious bugs.  The new design
> should pass all existing test suites verbatim.  And it should be able to do
> so on a PC, where the tests are available, proving that it's likely to also
> work on the MCU, where tests aren't available.
>
> 4) Is anybody else interested in seeing that approach adopted?  There
> might be other benefits besides running scripts from ROM.  One benefit
> might be a bit faster execution, since most strlen() calls or
> other null-byte searches are eliminated.  Currently those are happening
> every time a hash key is computed, and in other hotspots.
>
> 5)  I heard Steve mentioning recent improvements to make Jim insensitive
> to nulls embedded in data.  Anybody care to guess how that work interplays
> with this approach?
>
> 6)  Would anybody else be interested in helping with the extensive rework?
>
> 7)  Are there any Tcl TIP's, or Jim open issues, that relate to my goal,
> or to this paradigm shift either?
>
> Thanks for any clues!
>
> --
> TheMarkitecht
> "I'm always disappointed when a liar's pants don't actually catch on fire."
> https://github.com/TheMarkitecht/slim
>
>
> _______________________________________________
> jimtcl mailing listjimtcl at lists.workware.net.auhttps://lists.btbrotary.org.au/cgi-bin/mailman/listinfo/jimtcl
>
>
>
>
> Hi Mark
>
> I am sure that Steve knows best with this but just for the record, I have
> wanted to do something like this too. What OS are you running? Even if the
> microcontroller has memory, is there anything to stop us from using
> external memory? Perhaps over SPI etc ..
>
> BTW, if you do go down the Lua road, I was part of eLua and I can tell you
> what I learned from this, just PM me.
>
> -Pat
>
>
>
>
>
> _______________________________________________
> jimtcl mailing listjimtcl at lists.workware.net.auhttps://lists.btbrotary.org.au/cgi-bin/mailman/listinfo/jimtcl
>
> I just looked up the STM32F4 and it does not run RetroBSD. I am not sure
> if you can squeeze Bertos or Contiki on it but it is smaller than I thought.
>
> I asked about running jim on baremetal once and there is a thread about
> this if you want me to forward old emails. It does not sound possible as so
> much of jim is tied to the OS.
>
> Sorry for the noise everyone
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.btbrotary.org.au/pipermail/jimtcl/attachments/20210203/873a7589/attachment.html>