Totally Late and Totally Worth It


I have been talking about and announcing Redtamarin v0.4.2 for … way too long
so … what happened ?

yeah a bit too busy with many work projects, but that’s not it, so here a lesson abotu what I have been doing wrong.

You have seen me often promoting automation, and one of my motto is that

If you don’t have an automated build you have nothing

Well… it’s kind of what happened with Redtamarin.

There is an automated build, but the project is bigger than building a couple of executables, and my big failure have been to not automate the process enough.

It was automated to a certain point but many parts had to be glued by hand and it’s where/how it failed.

First and foremost, Redtamarin is a SDK, it’s not just one exe or one package,
it is a meta-package, many little packages into one big package.

Here how it looks

       |_ redtamarin_x.y.z_{ARCH}
       |     |_ redtamarin-cli_x.y.z_{ARCH}
       |     |_ redshell_x.y.z_{ARCH}
       |     |_ projectormake_x.y.z_{ARCH}
       |_ redbean_x.y.z_{ARCH}
       |_ redshell-linux_x.y.z_{ARCH}
       |_ redshell-macintosh_x.y.z_{ARCH}
       |_ redshell-windows_x.y.z_{ARCH}
       |_ redtamarin-binfmt_x.y.z_{ARCH}
       |_ as3shebang_x.y.z_{ARCH}
       |_ distro-cli_x.y.z_{ARCH}

So the ant automated build you see in the Redtamarin project, it does a lot

  • it can compile all the Redtamarin shells (redshell)
    in 32-bit and 64-bit, as release, debug, debug-debugger
    and for Windows, macOS and Linux
    eg. it produces 18 executables
  • in v0.4.1 I added another command-line tool named “redtamarin”
    that’s the project redtamarin-cli and is an utility and launcher
    for the other redshell exes
    the redtamarin build does not automate that
  • same for other tools like redbean, projectormake, as3shebang, etc.
    those are command-line utilities based on Redtamarin
    but the redtamarin build does not automate that

And here the first problem of those tools: they are cross-dependant of each others,

for example:
if you don’t have a redshell_dd debugger 64-bit for macOS
you can not build the projectormake utility
which is needed to build the as3shebang executable for macOS
but also depends on redshell_dd debugger 64-bit for macOS
to build a projector

The second problem is even if the Redtamarin build can compile all those executables
it can only compile them on their respective platform,
eg. you need to be on macOS to build the macOS redshells
then you need to switch to Windows to build the redshells for Windows, etc.

So when you come back to a project like as3shebang or other cli utilities,
you need to be able to share those exe in a central repository.

Something I was doing manually, not anymore :smile:.

And there is the 3rd problem: I was using redtamarin tooling to build some parts of redtamarin tools.

Whaaat ?

Ok hear me out, I build stuff like as3shebang and redbean as convenient tools
and mainly to replace stuff like Bash scripts.

So you end up in this kind of vicious circle where to build a new redtamarin-sdk (the whole thing),
your dependency is to have the redtamarin-sdk installed in the first place, that’s crazy :stuck_out_tongue:.

And all that are mistakes that happened because ?
I did not automate things enough, so I used some shortcuts and mistakes lead to other mistakes
which lead to more shortcuts leading to even more shortcuts, and more mistakes …

You get the idea :smile:.

And that my friends is THE lesson to learn, you have to automate every goddamn thing,
even if it take a lot of time, even if it is very hard, etc.


Now for the solution,
we have our bunch of problems

  1. tools are are cross-dependant of each others
  2. tools are not shared or accessible to other builds
  3. redtamarin-sdk is dependent on redtamarin-sdk

And the idea is to automate everything so it magically build without human intervention.

One way to do it is to have what I call a meta-build (and I talk from experience because I already solved that kind of problem by doing such meta-build), here how it goes:

  • instead of building each projects separately into their own repositories
  • you “merge” (or assemble) a big project with all those different projects as “child”
  • then you have a meta-build that can build each project one after another
  • that way when you have A depending on B, and B depending on C,
    while D depending also on A and B, etc.
  • you can give an orders of building your dependencies
  • and everything that have already been built can be reused along that path

I know it can work because I did it on projects which had 20+ cross-shared dependencies,
but sadly for Redtamarin it can not work “like that”.

Because we need to have builds running on their own respective platforms,
for example: you can not compile the Windows redshells (C/C++) under Linux
(well… you could … and it is named cross-compilation, but that would require too much work and change in the C/C++ source code).

There is also a 2nd reason: some builds take a lot of times,
to build the 6 redshell executables for Windows you need 6 x 20mn = ~2 hours.

So it happen I can not fully automate everything (eg. one meta-build that build everything ine one go),
but I can do the next best thing: a prep-build followed by many builds that share things from the prep-build.

Whaaat ?

Ok it goes like that, a directory structure for a meta-build would look like that

    |_ redtamarin
    |_ projectormake
    |_ redtamarin-cli
    |_ redbean
    |_ as3shebang
    |_ ...

And the build would run from the root folder meta-project and be able to run the build for each child folder but also re-use produced binaries from each childs.

But this only work when your build can run all the child builds on the same platform, which is not our case here for Redtamarin.

So we do a prep-build for each platforms we need to support

  • prep-build for Windows
  • prep-build for macOS
  • prep-build for Linux

We find a way to share those, and then for each other projects we can then access those shared assets to build each projects independently of each others.

here the logic and structure of the thing:

we will have a bin-deps (binary dependencies) folder where to put the shared stuff

    |_ redtamarin
    |_ projectormake
    |_ redtamarin-cli
    |_ redbean
    |_ as3shebang
    |_ ...

Oh see how similar it is from a meta-build ? But then what is the difference?

The difference is that each individual projects has its own bin-deps directory
for example as3shebang

    |_ bin-release
    |_ bin-deps  <-- THIS !!!
    |_ lib-swc
    |_ lib-abc
    |_ src
    |_ ...

And in fact bin-deps is just a network share, the easiest way to share the same set of data among many different computers and others VM on a local network.

If you look in the Redtamarin project build folder (once it will be updated with the 0.4.2 sources)
you will see different utilities named by platform

for example

   |_ macintosh_all   <-- build all the macOs redshells
   |_ macintosh_deps  <-- export the macOS redshells to bin-deps

When it come to build the redtamarin-sdk then you do that

  • $ build/windows_all
  • $ build/macintosh_all
  • $ build/linux_all

eg. you are preparing (prep’ing?) all your redshell binaries

Sure you have to do 3 commands instead of 1 (with a meta-build) but you have the advantage of being able to run those build on 3 different machines (and personally I just SSH to each machines) and so all those builds run in parallels.

Once all those binaries are prepared I then have this structure

├── redshell
│   └── 0.4.2
│       └── windows
│       │   ├── 32
│       │   │   ├── redshell.exe
│       │   │   ├── redshell_d.exe
│       │   │   └── redshell_dd.exe
│       │   └── 64
│       │       ├── redshell.exe
│       │       ├── redshell_d.exe
│       │       └── redshell_dd.exe
│       └── macintosh
│       │   ├── 32
│       │   │   ├── redshell
│       │   │   ├── redshell_d
│       │   │   └── redshell_dd
│       │   └── 64
│       │       ├── redshell
│       │       ├── redshell_d
│       │       └── redshell_dd
│       └── linux
│           ├── 32
│           │   ├── redshell
│           │   ├── redshell_d
│           │   └── redshell_dd
│           └── 64
│               ├── redshell
│               ├── redshell_d
│               └── redshell_dd
└── redtamarin
    └── semver

From there, I can run any project build scripts that will rely on “bin-deps”.
See into each respective projects how it works :wink:,
but, in short, to not depends on the redtamarin-sdk itself most builds are Bash scripts.

A couple more things

  • this is for maintainers of the project (eg. ppl that build binaries and packages to be distributed)
    if you just need to work on things like the documentation/asdoc you don’t need to go all that trouble

  • Off course you don’t need to do that when you are using redtamarin-sdk,
    we are going to great length to build redtamarin-sdk without depending on redtamarin-sdk
    so developers can depend on redtamarin-sdk to build their own projects.

  • This is the 2nd phase of the build, there is a 3rd one needed
    right now we are enforcing a POSIX environment to all that
    which force to setup Windows in a very special way.
    Ideally in a 3rd phase of the build we should refactor the tools of the SDK
    in such a way that you would simply be able to install a redtamarin-sdk.msi
    without worrying of having Cygwin and other non-standard things setup on the system.

The same way some shortcuts can lead to mistakes, an improved build system lead to better results.

First we were forced to think on how we package things for distribution, remember the structure of the meta-package from the beginning ?

       |_ redtamarin_x.y.z_{ARCH}
       |     |_ redtamarin-cli_x.y.z_{ARCH}
       |     |_ redshell_x.y.z_{ARCH}
       |     |_ projectormake_x.y.z_{ARCH}
       |_ redbean_x.y.z_{ARCH}
       |_ redshell-linux_x.y.z_{ARCH}
       |_ redshell-macintosh_x.y.z_{ARCH}
       |_ redshell-windows_x.y.z_{ARCH}
       |_ redtamarin-binfmt_x.y.z_{ARCH}
       |_ as3shebang_x.y.z_{ARCH}
       |_ distro-cli_x.y.z_{ARCH}

For now we distribute only a SDK redtamarin-sdk_x.y.z_{ARCH}, but in the future we will distribute each independent package on its own, a bit like how installing as3shebangs works (eg. you don’t need to install redtamarin-sdk to setup as3shebang).

Which lead me to the second point: we improved the tooling (a lot!).

Thanks to the newer projector format and a couple of native methods a binary executable produced by redtamarin is truly independent, exactly the same as if you compiled you exe from C/C++ for a specific platform.

Few examples:

  • as3shebang (or any other projector for that matter) can literally be dropped in your
    /usr/bin, or /usr/local/bin, or /opt/local/bin or %SYSTEM% and it will just works.
    (OK shell scripts is a very special case …)

  • if you ever build command-line tools with Perl, PHP, Python, Node.js etc.
    and had to deal with dependencies hell, we completely removed that.
    You can if you have to, but you can also do without that :slight_smile:
    (technically I think those new projectors are even better that what Go or Mono are doing,
    well… you’ll be the judge of that)

  • now things are done in such a way that any “exe”, that’s it an ABC file or SWF file merged
    with a redshell that produce a projector, well those standalone executable are their own island,
    you can perfectly have “tool123” exe build with redshell debugger v0.4.2 running in parallel of
    “foobar456” exe build with redshell release v5.0 etc.

  • If you want to quickly produce a projector and does not have Java to compile stuff with asc.jar
    you can even build projectors on plain text .as sources :stuck_out_tongue: (incredibru!)

A very specific example with as3shebang

Before you had to use

You can still do that, but now you can also use
#!/usr/bin/env as3shebang
that will make your script more portable

That means also that as3shebang can be installed in any directory accessible in your $PATH.

Even better, you could have as3shebang not installed and just use it from a relative path
for example in a code project you could store it there /build/as3shebang
and then run a script like that $ build/as3shebang
(not advised but it would work).

This also allowed us to add command-line arguments
#!/usr/bin/env as3shebang -ec
so you can configure how as3shebang should behave when executing your script

here -ec stands for error context

when an error occurs by default as3shebang will display the stack trace in red
(if your terminal supports ANSI colors)

with -ec, on top of the stack trace we display the error context
or the line number where the error occurred, surrounded by the script context

And most of those command-line options can also be configured with environment variables
here for example we want to activate the option NO_WRAP

Here NO_WRAP tells as3shebang to use the old behaviour for the errors
that means we don’t intercept the error to format its layout or colours.

It may seems like small details but it’s those little things that make great tools :slight_smile:
in the various options you have NO_COLOR to force disable the ANSI colors
you also have DEBUG, etc.

If you want to use many options altogether you define them like that
directly on the command-line if you want the option only available for your current shell session
or you can make those permanent by adding it to your .profile or other rc files.

But the internals of Redtamarin are made in such a way that it will detect when it does not need to display colours, for example if you pipe your output to a file

here with $ ./ >> log.txt
you will not have your output cluttered with ANSI escape codes, and yep all that will work also under a Windows terminal.

And that’s only as3shebang, all the tools are under such improvements.

Finally, come the tests.

When you write code that is supposed to run under a runtime that you are not producing yourself (eg. Flash Player, Adobe AIR) you are mainly responsible to test your code not the runtime itself, and you quickly jump to report a bug to the Adobe tracker when something is not supposed to work as it should in the runtime.

but when in the case of Redtamarin we are producing our own runtime, we have to test how these runtime behave on each platforms.

Before, it was basically few .as scripts in the folder of where the redshell executable were produced, and those scripts were manually copied on different platforms to have a sense of “good enough, it works”.

Thanks to this big rework of the builds and to a very nasty bug who took me quite some time to figure out, now we have integration tests, you will find those in the directory src/as3examples.

Those tests serve 2 main purposes:

  • document the functionalities with asdoc
  • be able to run the exact same test on different platforms

you will find the test runners in build/run_tests (it is a big bash script)
the tests are defined like this

 #"" "" "" "" # miss Shape class
 #"" "" "" ""
 #"" "" "" ""
 #"" "" "" ""
 #"" "" "" ""
 #"" "" "" ""
 #"" "" "" ""
 #"" "" "" ""

 #"shell/" "" "" ""
 #"shell/" "" "" ""
 #"shell/" "" "" ""
 #"shell/" "" "" ""
 #"shell/" "" "" ""
 "shell/" "" "" "" #the default test
 #"shell/" "" "" ""

 #"flash/crypto/" "" "" ""

By default only one test is active "shell/"
here its output under macOS

the format is pretty simple: 4 strings

  1. the path of the test (relative to src/as3examples)
  2. an option either empty string "" or "script" (to not compile it)
  3. vmargs options like " -api VM_ALLVERSIONS"
  4. the program args options like " -- a b c"

for example

 "" "" " -api SWF_12" " -- 1 2 3"

would be like compiling to
and run
$ redshell_dd -api SWF_12 -- 1 2 3

Those tests are not unit tests, they are integration tests, and you don’t have to run all of them all the time,
but they are extremely valuable to be sure that everything works as it should on different platform.

The workflow is for example if you implement a new native functionality, you then create a basic example
on how to use this function (which you can neatly add to your asdoc) and it allows you to commit the
test to the repository and run it on either Windows, macOS or Linux.

Ideally this script should move to a redtamarin tool (but maybe not … diff pros and cons)
in fact next tool in line is astuce to run AS3 unit tests on the command-line
to which I will add few add-ons to specifically test command-line exe, socket, HTTP, etc.
so we can have both unit and integration tests.

And one more thing … did I mention you can embed custom vmargs in your projector ?

well… you can :smile:
this one for ex "-api SWF_12" if for some reason you do need your projector to run exclusively under the SWF_12 API, but really … it could be any of the numerous options available in redshell, redshell_d and redshell_dd; you will see all that in projectormake.

**Morality of the story**: don't get stuck because things are not automated enough (or not the right way), by doing so you would probably save yourself from a lot of headache but there are also good chances you would improve your tools in the process.

So I’m not totally there yet, I’m still finishing some rework of the tools (yep undoing some shortcuts taken before), but that’s the right thing to do because, all that, is done ultimately to be able to update much much faster so, in the end, I can cruise more comfortably on a ~1 month update or so of all those tools.