The case when you don't want Node.js

Last week-end I made a little experiment :stuck_out_tongue:

Many times you can see command-line tools written in JS
and almost automatically they will depend on Node.js and npm

For example if you go on TypeScript github page

you can see the install instruction
npm install -g typescript

it’s like everyone assume that you have node.js and npm installed by default

and it’s kind of OK, once you have those installed sure you are one command-line install away of many tools

but what if you want the command-line tool and don’t want or need the HUGE dependencies on both node.js and npm ?

you can’t, and that’s kind of sad because really at the end of the day you just want to install a command-line tool and that should be as a easy as copying an executable in your PATH

my point here is that it is OK to depend on a specific command-line tool but asking to be dependent on the whole node.js+npm can be a little too much to ask in some cases…

So I did some tests, and first in line was TypeScript eg. tsc.js the TypeScript compiler command-line tool.

Yes, it would be pretty cool to have a tsc command-line as an executable without any other dependencies.


What do we need to be able to run this tsc.js without node.js ?

If you look at src/compiler/sys.ts

You can see a System interface

    export interface System {
        args: string[];
        newLine: string;
        useCaseSensitiveFileNames: boolean;
        write(s: string): void;
        readFile(path: string, encoding?: string): string;
        writeFile(path: string, data: string, writeByteOrderMark?: boolean): void;
        watchFile?(path: string, callback: FileWatcherCallback): FileWatcher;
        watchDirectory?(path: string, callback: DirectoryWatcherCallback, recursive?: boolean): FileWatcher;
        resolvePath(path: string): string;
        fileExists(path: string): boolean;
        directoryExists(path: string): boolean;
        createDirectory(path: string): void;
        getExecutingFilePath(): string;
        getCurrentDirectory(): string;
        getDirectories(path: string): string[];
        readDirectory(path: string, extension?: string, exclude?: string[]): string[];
        getModifiedTime?(path: string): Date;
        createHash?(data: string): string;
        getMemoryUsage?(): number;
        exit(exitCode?: number): void;
        realpath?(path: string): string;
    }

And at the end of the library you can see this logic

        if (typeof ChakraHost !== "undefined") {
            return getChakraSystem();
        }
        else if (typeof WScript !== "undefined" && typeof ActiveXObject === "function") {
            return getWScriptSystem();
        }
        else if (typeof process !== "undefined" && process.nextTick && !process.browser && typeof require !== "undefined") {
            // process and process.nextTick checks if current environment is node-like
            // process.browser check excludes webpack and browserify
            return getNodeSystem();
        }
        else {
            return undefined; // Unsupported host
        }

First it try to detect ChakraHost object, then WScript and the ActiveXObject object, and then the process object and the require definition, if none of those is found it return undefined.

  • ChakraHost is related to the Microsoft JS engine
    see ChakraCore

ChakraCore is the core part of the Chakra Javascript engine that powers Microsoft Edge

The Microsoft Windows Script Host (WSH) is an automation technology for Microsoft Windows operating systems that provides scripting abilities comparable to batch files, but with a wider range of supported features. It was originally called Windows Scripting Host, but was renamed for the second release.

It is language-independent in that it can make use of different Active Scripting language engines. By default, it interprets and runs plain-text JScript (.JS and .JSE files) and VBScript (.VBS and .VBE files).

  • process and require are related to node.js
    see process api

    The process object is a global object and can be accessed from anywhere. It is an instance of EventEmitter.

    and What is require?

    Node.js follows the CommonJS module system, and the builtin require function is the easiest way to include modules that exist in separate files. The basic functionality of require is that it reads a javascript file, executes the file, and then proceeds to return the exports object.

So, here what we need to replace node.js

  • a runtime that can replace either Node.js , WSH or Chakra core
    simply put an ECMAScript runtime
  • a runtime that allow us to implement the System interface
    which provide basic functionalities like readFile(), writeFile(), etc.
  • a runtime that can understand, load and embed the tsc.js file
    basically a runtime that understand ECMA-262 3rd edition
  • ideally we would like something that works cross-platform
    eg. produce an exe that can run under Windows, Mac OS X or Linux

Finding the right ECMAScript runtime

On the Wikipedia page about ECMAScript implementations
you can see there are a lot of different implementations

Now remember it is about being able to produce a command-line executable without external dependencies, otherwise using node.js which reuse the Google V8 engine would be perfectly fine.

Technically we want an ECMAScript shell, eg. one of those implementations running purely as a command-line tool.

two great resources about those

  • ECMAScript Shells
    This file is a link farm for command-line ECMAScript implementations, with some comments.
  • JavaScript shells (Mozilla Developer Network)
    A JavaScript shell allows you to quickly test snippets of JavaScript code without having to reload a web page. They are extremely useful for developing and debugging code.
  • Wikipedia List of ECMAScript engines

Nashorn and Rhino are out of the loop as they would need a dependency on Java and we do not want any external dependencies.

Same for JScript .NET as it would require a dependency on .NET either official from Microsoft or Mono.

Ouh JSDB is very interesting, it is based out of the SpiderMonkey engine, it has API to read/write files etc. and the runtime is cross-platform.

In the tutorials you can see how to build Standalone JSDB programs

c:\temp>pkzip program.zip main.js

c:\temp>copy /b jsdb.exe+file.zip program.exe
c:\temp>program.exe

And finally, yeah Redtamarin, based on the AVM2 (Tamarin engine), it has API for read/write files etc. and the runtimes are cross-platform.

using the redbean tool you can easily build projectors
merge ABC files or SWF files and a redtamarin runtime (redshell) into a single executable

here a build example

import redbean.*;

compile( "program.as" );
projector( "program", false, OS.linux64, ["program.abc"] );

I decided to pickup Redtamarin for the test but I’m pretty sure you could do the same with JSDB.


Implementation with Redtamarin

We gonna do this in 2 steps

  • first, compile tsc.js to an ABC file
  • second, implement a host object
    following the System interface

Let’s copy lib/tsc.js

and let’s create a simple build.as3 file

import redbean.*;

compile( "tsc.js" );

see if it works “as is”
$ redbean

result

[redbean 1.0.0]
[run] build.as3

[compile] tsc.js

[Compiler] Error #1084: Syntax error: expecting identifier before include.
   tsc.js, Ln 18399, Col 41: 
           function getTypeWithFacts(type, include) {
   ........................................^

[Compiler] Error #1084: Syntax error: expecting rightparen before leftbrace.
   tsc.js, Ln 18399, Col 50: 
           function getTypeWithFacts(type, include) {
   .................................................^

[Compiler] Error #1084: Syntax error: expecting identifier before include.
   tsc.js, Ln 18401, Col 45: 
                   return getTypeFacts(type) & include ? type : neverType...
   ............................................^

[Compiler] Error #1086: Syntax error: expecting semicolon before colon.
   tsc.js, Ln 18401, Col 60: 
                   return getTypeFacts(type) & include ? type : neverType...
   ...........................................................^

[Compiler] Error #1084: Syntax error: expecting identifier before include.
   tsc.js, Ln 18407, Col 39: 
                   if (getTypeFacts(t) & include) {
   ......................................^

[Compiler] Error #1084: Syntax error: expecting rightparen before leftbrace.
   tsc.js, Ln 18407, Col 48: 
                   if (getTypeFacts(t) & include) {
   ...............................................^

[Compiler] Error #1086: Syntax error: expecting semicolon before rightbracket.
   tsc.js, Ln 36882, Col 52: 
       var IgnoreFileNamePattern = /(\.min\.js$)|([\\/]\.[\w.])/;
   ...................................................^

[Compiler] Error #1093: Syntax error.
   tsc.js, Ln 36882, Col 53: 
       var IgnoreFileNamePattern = /(\.min\.js$)|([\\/]\.[\w.])/;
   ....................................................^

[Compiler] Error #1093: Syntax error.
   tsc.js, Ln 36882, Col 56: 
       var IgnoreFileNamePattern = /(\.min\.js$)|([\\/]\.[\w.])/;
   .......................................................^

[Compiler] Error #1084: Syntax error: expecting identifier before include.
   tsc.js, Ln 36972, Col 44: 
               ? { enableAutoDiscovery: true, include: [], exclude: [] }
   ...........................................^

[Compiler] Error #1084: Syntax error: expecting identifier before include.
   tsc.js, Ln 36973, Col 45: 
               : { enableAutoDiscovery: false, include: [], exclude: [] }...
   ............................................^

11 errors found


[done] in 03s.458ms

Ok, it does not compel “as is” we have some errors, but nothing that should be impossible to fix.

First problem is the word include which is a reserved keyword in AS3/ES4

        function getTypeWithFacts(type, include) {
            if (!(type.flags & 16384)) {
                return getTypeFacts(type) & include ? type : neverType;
            }
            var firstType;
            var types;
            ...

simply replace include with includes, easy one to fix :wink:.

Let’s try again

[redbean 1.0.0]
[run] build.as3

[compile] tsc.js

[Compiler] Error #1086: Syntax error: expecting semicolon before rightbracket.
   tsc.js, Ln 36882, Col 52: 
       var IgnoreFileNamePattern = /(\.min\.js$)|([\\/]\.[\w.])/;
   ...................................................^

[Compiler] Error #1093: Syntax error.
   tsc.js, Ln 36882, Col 53: 
       var IgnoreFileNamePattern = /(\.min\.js$)|([\\/]\.[\w.])/;
   ....................................................^

[Compiler] Error #1093: Syntax error.
   tsc.js, Ln 36882, Col 56: 
       var IgnoreFileNamePattern = /(\.min\.js$)|([\\/]\.[\w.])/;
   .......................................................^

[Compiler] Error #1084: Syntax error: expecting identifier before include.
   tsc.js, Ln 36972, Col 44: 
               ? { enableAutoDiscovery: true, include: [], exclude: [] }
   ...........................................^

[Compiler] Error #1084: Syntax error: expecting identifier before include.
   tsc.js, Ln 36973, Col 45: 
               : { enableAutoDiscovery: false, include: [], exclude: [] }...
   ............................................^

5 errors found


[done] in 862ms

Ok, now we have some ind of regexp error

var IgnoreFileNamePattern = /(\.min\.js$)|([\\/]\.[\w.])/;

Here the “trick” is that AS3 regexp are a bit more sensitive and we need to also escape the / char.

Lets see how it goes now

[redbean 1.0.0]
[run] build.as3

[compile] tsc.js

[Compiler] Error #1084: Syntax error: expecting identifier before include.
   tsc.js, Ln 36972, Col 44: 
               ? { enableAutoDiscovery: true, include: [], exclude: [] }
   ...........................................^

[Compiler] Error #1084: Syntax error: expecting identifier before include.
   tsc.js, Ln 36973, Col 45: 
               : { enableAutoDiscovery: false, include: [], exclude: [] }...
   ............................................^

2 errors found


[done] in 731ms

Almost there, now we have another include keyword problem but this time we can not just rename it.

So, if you have done well your homework and did read (many times) the ECMA-262 specification you do know that you can write object litterals

not only this way
{ indentifier1: true, indentifier2: false }

but also that other way
{ "indentifier1": true, "indentifier2": false }

Let’s see if we finally solved everything

[redbean 1.0.0]
[run] build.as3

[compile] tsc.js

[Compiler] Error #1119: Access of possibly undefined property isArray through a reference with static type Class.
   tsc.js, Ln 401, Col 22: 
           return Array.isArray ? Array.isArray(value) : value instanceof...
   .....................^

[Compiler] Error #1061: Call to a possibly undefined method isArray through a reference with static type Class.
   tsc.js, Ln 401, Col 38: 
           return Array.isArray ? Array.isArray(value) : value instanceof...
   .....................................^

[Compiler] Error #1120: Access of undefined property debugger.
   tsc.js, Ln 819, Col 17: 
                   debugger;
   ................^

[Compiler] Error #1120: Access of undefined property ChakraHost.
   tsc.js, Ln 1274, Col 20: 
           if (typeof ChakraHost !== "undefined") {
   ...................^

[Compiler] Error #1120: Access of undefined property WScript.
   tsc.js, Ln 1277, Col 25: 
           else if (typeof WScript !== "undefined" && typeof ActiveXObjec...
   ........................^

[Compiler] Error #1120: Access of undefined property ActiveXObject.
   tsc.js, Ln 1277, Col 59: 
           else if (typeof WScript !== "undefined" && typeof ActiveXObjec...
   ..........................................................^

[Compiler] Error #1120: Access of undefined property process.
   tsc.js, Ln 1280, Col 25: 
           else if (typeof process !== "undefined" && process.nextTick &&...
   ........................^

[Compiler] Error #1120: Access of undefined property process.
   tsc.js, Ln 1280, Col 52: 
           else if (typeof process !== "undefined" && process.nextTick &&...
   ...................................................^

[Compiler] Error #1120: Access of undefined property process.
   tsc.js, Ln 1280, Col 73: 
           else if (typeof process !== "undefined" && process.nextTick && !process.bro...
   ........................................................................^

[Compiler] Error #1120: Access of undefined property require.
   tsc.js, Ln 1280, Col 99: 
           else if (typeof process !== "undefined" && process.nextTick && !process.browser && typeof require !==...
   ..................................................................................................^

[Compiler] Error #1180: Call to a possibly undefined method ActiveXObject.
   tsc.js, Ln 851, Col 27: 
               var fso = new ActiveXObject("Scripting.FileSystemObject");
   ..........................^

[Compiler] Error #1180: Call to a possibly undefined method ActiveXObject.
   tsc.js, Ln 852, Col 34: 
               var fileStream = new ActiveXObject("ADODB.Stream");
   .................................^

[Compiler] Error #1180: Call to a possibly undefined method ActiveXObject.
   tsc.js, Ln 854, Col 36: 
               var binaryStream = new ActiveXObject("ADODB.Stream");
   ...................................^

// truncated for space

57 errors found


[done] in 01s.67ms

Well … more shit happens.

That’s normal, by default redbean compile with the ActionScript Compiler (ASC) in strict mode for AS3, and that’s why you see the compiler choke on a lot of things undefined.

There 2 ways to solve that, either you add the definitions that are missing or you change the compiler settings to be less strict.

let’s modify our build.as3 file to be less strict

import redbean.*;
import redbean.tools.*;

var asc1:* = new ASC();
    asc1.strict = false;

compile( "tsc.js", asc1 );

and see what happen

[redbean 1.0.0]
[run] build.as3

[compile] tsc.js

[Compiler] Error #1004: Namespace was not found or is not a compile-time constant.
   tsc.js, Ln 1, Col 1: 
   /*! ******************************************************************...
   ^

1 error found


[done] in 01s.443ms

hmm apparently the ASC compiler still choking on Namespace?
well… let’s change the compiling mode from AS3 to ES

import redbean.*;
import redbean.tools.*;

var asc1:* = new ASC();
    asc1.AS3 = false;
    asc1.ES = true;
    asc1.strict = false;

compile( "tsc.js", asc1 );

and let’s try again

[redbean 1.0.0]
[run] build.as3

[compile] tsc.js

tsc.abc, 763063 bytes written


[done] in 02s.081ms

whoa victory, it does compile the whole thing :smile:

OK, so we have the first part done and a nice tsp.abc that we can load in the redtamarin runtime.

Let’s try to run it

$ redshell_dd tsc.abc
TypeError: Error #1010: A term is undefined and has no properties.
	at global$init()

hmm something is buggy but where it could be ?

Let’s recompile and slightly change our settings so we add debug information

import redbean.*;
import redbean.tools.*;

var asc1:* = new ASC();
    asc1.AS3 = false;
    asc1.ES = true;
    asc1.strict = false;
    asc1.d = true; //this will compile with debug information 

compile( "tsc.js", asc1 );

and now let’s run it again

$ redshell_dd tsc.abc
TypeError: Error #1010: A term is undefined and has no properties.
	at global$init()[tsc.js:37669]

same problem, same error, but this time we do know the exact line where it happen
tsc.js:37669

this is the 37669 line

ts.executeCommandLine(ts.sys.args);

OK let me spare you a small debug session, basically it does not work because the System interface is not created, so when we reach this line ts.sys is undefined and so ts.sys.args is not reachable.

That’s why we need a second part: implement a host object.

As defined in the sys.ts file here what we need to implement

    declare var ChakraHost: {
        args: string[];
        currentDirectory: string;
        executingFile: string;
        newLine?: string;
        useCaseSensitiveFileNames?: boolean;
        echo(s: string): void;
        quit(exitCode?: number): void;
        fileExists(path: string): boolean;
        directoryExists(path: string): boolean;
        createDirectory(path: string): void;
        resolvePath(path: string): string;
        readFile(path: string): string;
        writeFile(path: string, contents: string): void;
        getDirectories(path: string): string[];
        readDirectory(path: string, extension?: string, exclude?: string[]): string[];
        watchFile?(path: string, callback: FileWatcherCallback): FileWatcher;
        watchDirectory?(path: string, callback: DirectoryWatcherCallback, recursive?: boolean): FileWatcher;
        realpath(path: string): string;
    };

ChakraHost is just a global object that implement the System interface.

so let’s create a very simple one
ChakraHost.as

package
{
    import shell.*;

    public var ChakraHost = {};

               ChakraHost.args = Program.argv;

}

now let’s also compile it

import redbean.*;
import redbean.tools.*;

var asc1:* = new ASC();
    asc1.AS3 = false;
    asc1.ES = true;
    asc1.strict = false;
    asc1.d = true;

compile( "tsc.js", asc1 );
compile( "ChakraHost.as" );

You will notice that we compile tsc.js in ES mode
but we do compile ChakraHost.as ins AS3 mode

yep we can combine strict/non-strict and ES/AS3 compilations, pretty powerful :wink:

now let’s test it

$ redshell_dd ChakraHost.abc tsc.abc
TypeError: Error #1010: A term is undefined and has no properties.
	at Function/anonymous/tsc.js$0:normalizeSlashes()[tsc.js:553]
	at Function/anonymous/tsc.js$0:normalizePath()[tsc.js:603]
	at Function/anonymous/tsc.js$0:executeCommandLine()[tsc.js:37287]
	at global$init()[tsc.js:37669]

OK, still not there but some progress.

That’s normal to still get errors we only implemented the args property,
and in fact what we need is to implement the whole System interface in our ChakraHost object.

We could do that as a simple object declared in global (the unnamed package)
but we want a bit more than a simple object so we gonna build a class, and even if the object we want to impersonate is ChakraHost, our class will be named AVMHost because it is what we are doing hehe we are indeed creating an AVMHost for TypeScript.

Here how the final build look like

build.as3

import redbean.*;
import redbean.tools.*;

var asc1:* = new ASC();
    asc1.AS3 = false;
    asc1.ES = true;
    asc1.strict = false;
    asc1.d = true;

compile( "tsc.js", asc1 );
compile( "AVMHost.as" );
compile( "ChakraHost.as", null, [ "AVMHost.abc" ] );

projector( "tsc", true, null, [ "AVMHost.abc", "ChakraHost.abc", "tsc.abc" ] );

and a little config file
redbean.conf

var useDebugger = true;

so when we compile the projector we use redshell_dd to get the full stack trace if an error occurs

while testing I found yet another bug with JSON.stringify()

this is valid ECMAScript
ts.sys.writeFile(file, JSON.stringify(configurations, undefined, 4));

but we need this for ActionScript 3
ts.sys.writeFile(file, JSON.stringify(configurations, null, " "));

here the diff file
$ diff tsc.js.original tsc.js

In the end you obtain nice a tsc executable
$ ./tsc


Conclusion

Using Redtamarin we can build (bundle?) an executable entirely written in JavaScript (not ActionScript 3) without any external dependencies, it does work quite nicely.

Sure we had to do

  • a bit of patching
  • configure how we compile with ASC
  • bit of debugging

but it does work :smile:

The Typescript compiler was a very good test to do

  • the code is big tsc.js is 1.8MB
  • being able to mix ES code and AS3 code can provide a lot of of options
  • take all that with a grain of salt, I made other tests with some node.js code
    and sometime it just fails, not all JS code would work “as is”,
    it really depend how it had been written (cough cough poorly written JS anyone?)
  • but if we can compile something like tsc.js that also means
    most (if not all) code written in TypeScript would be able to compile too

Next step is to convert some tools like dts2as

A command line utility that converts TypeScript definitions (d.ts files) to ActionScript classes and interfaces and generates a SWC file. Use these SWCs with Apache FlexJS for strict compile-time type checking, as if the JavaScript library were written in ActionScript. Add the SWCs to IDEs, like Flash Builder and IntelliJ IDEA, and you’ll get helpful code suggestions as you type.

put all those in some github repos
and provide packaging .deb for Linux, .pkg for Mac OS X, etc.
and maybe try to automate all that with a kind of “converter”

Also take note that even if something is written with TypeScript it may need much more to be able to run under redtamarin, if for example the tool use Node.js filesystem API then you would need to provide that too.

1 Like