Tooling

To develop command line applications in scalajs that target nodejs, we need a few build tools, some of which may or may not be used depending on how you want to deploy your applications:

• Bundled CLI: A bundled scalajs CLI program has all the dependencies bundled into a single .js file and can be executed using the node executable `node <app-bundle.js>`.

  • A simple variant of this is that some but not all javascript dependencies are bundled in the main application file. These types of deployments can also be executed using the `node` program but node will resolve the remaining dependent modules at runtime using the nodejs modules system.

  • Native executable: Using "pkg".

• Unbundled CLI: This is the more common approach to deploying CLI programs in nodejs. The javascript dependencies are not bundled into a single .js file and all of them must be imported at runtime. The import process is the nodejs module system. Many nodejs programs depend on hundreds of npm modules. When you install your CLI program, you install all of those modules as well.

It's clear the concept of bundling is pretty important. Javascript files are the most commonly bundled artifact type in a scalajs program but you can bundle pretty much anything into a singe javascript file. For example, webpack allows you to bundle .css files with javascript files and provides infrastructure code to ensure that the .css file is "processed" correctly regardless of whether you are targeting the web or nodejs (even though .css files are less useful server-side of course). While bundling can affect either initial startup or ongoing project runtime, many nodejs cli applications still choose to keep most things unbundled.

Dependencies come from different sources:

• npm/nodejs

• webjars (jar files of javascript and other resources)

• application javascipt files (you author these directly)

All three sources may be used in a single project. Having many different types of dependent artifacts coming from many different locations is quite common in the nodejs world. webjars are an artifact of the need of java based tooling to easily resolve javascript and other javascript-related dependencies. However, webjars are not perfect because sometimes (and often) a package is missing a license or other artifact that prevents it from being automatically converted to a webjar. A good perspective on why webjars are great but sometimes problematic is here.

We are not going to consider webjars further because they are easy to integrate into your build process using standard sbt approaches--just as they were designed to do. However, we want to deploy CLI programs and webjars force the javascript to be pushed into the scala assembly process vs the nodejs runtime module resolution process.

You have to conciously choose where to source the dependencies from as well as how (and when) to bundle the parts together. Then depending on the design you choose different tools to perform these steps. For example the sbt plugin sbt-scalajs can assemble the compiled .scala code and your application javascript files and bundle them into a single output file. Or sbt can create two output files, a main file with your .scala code and a "jsdeps" file with all other non .scala source code files. Or, you can rely on sbt to compile your .scala files and use webpack to assemble and bundle the parts together.

There is even an approach (describe below) where you start with webpack, shell out to sbt, sbt shells out to webpack, then everything returns and is assembled in the original webpack file. If this seems confusing, it is. But this is also quite normal in the javascript development world where different deployment scenarios require tremendous flexibilty in assembling the final application artifacts. nodejs has helped standardize dependency management to some degree, but final packaging is still very application dependent.

We'll describe a set of approaches in the form of recipes. The tools we need are:

• scalajs - scala javascript compiler.

• sbt - scala build tool

  • sbt-scalajs: Plugin to build scalajs programs.

  • scalajs-bundler: Plugin that uses webpack and npm to manage dependencies and mix them into the final sbt output artifact. It is designed to create a single client bundle with npm modules, but we can use it for purposes since it performs general processing that is useful to our CLI target.

• webpack - node.js based build tool

  • scalajs-loader - program for .scala files. It calls out to sbt to perform the actual build when a .scala file is encountered in the webpack graph.

  • webpack.BannerPlugin: Allows you to add a banner to the start of a file. For a CLI, the banner would be `#!/usr/bin/env node`

  • google closure compiler (GCC) or uglify: Minimizes the output file. GCC is used by sbt-scalajs but there is also a js version that can be used from webpack.

sbt-scalajs is a sbt plugin that uses the google closure compiler (GCC) to bundle and smartly minify its output with dependencies. scalajs dependencies typically come from jars that contain scalajs code or the dependencies come from webjars which have javascript dependencies packaged into a jar format. sbt always bundles .scala based code together into a single bundle. We need to remember that when we speak of "bundled" or "unbundled" we are really referring to the dependencies and not the .scala based js files.

The scalajs-bundler sbt plugin allows you to declare and resolve your scalajs dependencies using npm instead of sbt and webjars. You could use a webpack-first model where webpack's scalajs-loader is used to perform the final bundling with files output from scalajs-sbt. We mentioned above that most nodejs programs are unbundled but we will show how you could use either of these bundling mechanisms to fully or partially bundle your code.

Factors affecting sbt artifact handling

There are a few settings/keys that affect how sbt outputs its .scala transpiled javascript files. All of these are demonstrated in the recipes. The key aspects of these settings are described below.

artifactPath

This key controls where the artifact is placed. You should not use this setting if you have other plugins that depend on the output file being in the standard location. For example, scalajs-loader assumes that the artifact is output to a specific location (although that can be overridden). There are some SO notes here:

artifactPath in (Compile, fastOptJS) := file(".") / "myjs" / "cliapp-fastopt.js",
artifactPath in (Compile, fullOptJS) := file(".") / "myjs" / "cliapp-fullopt.js",

scalaJSModuleKind

This controls whether the output artifact (from the .scala compiled files) are output as a nodejs friendly module. Since we are focused on nodejs clip apps, we pretty much need to ensure that we either set this directly, or it is set for us in an sbt plugin:

scalaJSModuleKind := ModuleKind.CommonJSModule

If you choose module deployment then your jsDependencies are always output as a separate file, regardless of your settings. Since we need module output, jsDependencies and skip become less useful.

If you use @JSImport but do not set scalaJSModuleKind, scalajs will signal an error during compilation that you cannot use one without the other.

jsDependencies

When you want scalas-sbt to process your js dependencies directly as provided in the build files, you use jsDependencies to specify the dependencies. The traditional documentation says to use jsDependencies ++= Seq(ProvidedJS / "jsdir/jsdep1.js, ProvidedJS / "jsdir/jsdep2.js") and so on but that can become tedious. Note that these are dependency objects and not just strings. ProvidedJS is an object with a method / that makes it easy to create dependency objects assuming they are in the "standard resource" location of "src/main/resources". If we want to be more clever or change the sourcing location, we need do something slightly different using some of the Path api in sbt.

...
jsDependencies ++= jsDeps()
import org.scalajs.sbtplugin.{ProvidedJSModuleID => mkJSDep}
def jsDeps() = ((file(".") / "src/main/resources") ** "*.js").getPaths.map(mkJSDep(_, None))
...

or something like

jsDependencies ++= Seq("js/jsdep1.js", js/jsdep2.js").map(ProvidedJS / _)

The use of None controls which configuration the dependency belongs to. In the above, we add them to all of them, which may not be what you want if you want to run separate nodejs javascript tests.

There are other plugins, for example supporting play applications, that also move your javascript artifacts into the final sbt-scalajs output directory so its clearly a common task. scalajs-bundler also moves all your javascript files into the scalajs output directory.

skip

skip ... = false specifies that scalajs-sbt should bundle your js dependencies (from jsDependencies) into a separate output file called [project name]-deps.js or [project name]-deps.min.js. The default is true, which means sbt-scalajs should bundle all of your jsDependencies into your .scala file output file. It's a bit confusing, but "skip = true means skip creating a separate jsdeps file."

Since we will be managing dependencies using different build tools, you will not want to have them bundled anyway.

skip in packageJSDependencies := false

As a side note, it is important to remember that skip applies to only those dependencies you have declared via jsDependencies.

When module output is enabled and the jsDependencies are present, the dependencies are concatenated together. This is usually not what you want unless the js dependencies are already modulized in some way inside the js file itself. Since modules are becoming more standardized in the js world, your dependencies will most likely not be in this form anyway. For nodejs npm dependencies, they often are not because it is expected that a build tool like webpack will do the modulizing work.

For example if I had two .js files and each was specified in build.sbt as a jsDependency, the concatenated file would look like:

[scala-2.12]$ cat *jsdeps*
// start of keepunbundled.js
// No dependences for this module

export function notProvidedMessage() {
    return "not provided"
}
// end of keepunbundled.js

// start of bundleme module
let Chalk = require("chalk")

function backgroundRed(arg) {
    return Chalk.white.bgRed(arg)
}

module.exports.default = backgroundRed
// end of bundleme module

So the bottom line is that for cli programs that will be unbundled, there is not alot of benefit from skip or jsDependencies other than your jsDependencies are copied to the output directory.

babel transpilers

We use babel transpilers so our application js has access to the latest features and can be transpiled to run on various versions of node.js.

node version

javascript is changing rapidly. Its best to use the last version of node, however, the examples will not depend on a specific node version. The examples should be good for node v6 and v7.

scalajs

Always use the latest scalajs version. v6.15+ was used in the examples. scala compiler version was 2.12.2.

sbt

sbt version 0.13.15 was used and the latest sbt plugins for scalajs-sbt and scalajs-bundler.

webpack

Webpack creates a graph of artifacts. Each file/module/input artifact is passed through a series of loaders, then those loaders output artifacts into the webpack graph. The graph is then processed to create output files ensuring that all dependencies are considered during the graph traversal.

Loaders can be specified either in the webpack config file or in the require(...) statement that imports a module into another module. In webpack, you can handle one-off exceptions using the chained loader syntax in the require statement vs the chain specified in the config file. For example, you can do:

import css from '!style-loader!css-loader?module=false!react-virtualized/styles.css'
//...or...
var css = require(!style-loader!css-loader?module=false!react-virtualized/styles.css')

which skips all webpack config loader processing and uses only the loader chain present in the import.

While we will prefer to specify loader chains in a config file, you can use loader chains in pure scala as well

@js.native
@JSImport("!expose-loader?$!jquery", JSImport.Namespace)
object jQuery extends js.Object

in webpack config, the following will find all 'requires' in your javascript files that want jquery and translate them into:

module: {
  rules: [{
          test: require.resolve('jquery'),
          use: [{
              loader: 'expose-loader',
              options: 'jQuery'
          },{
              loader: 'expose-loader',
              options: '$'
          }]
      }]
}

There's not really an equivalent of this in sbt. webpack allows you to graft in modules that do not "require" directly but where you want it to appear that the global variable is there without being explicit as with jquery in the above example. It can also redirect requires and substitute other modules. Its very flexible around managing modules, much like you find in lisp or scheme implementations where modules can be managed in code to some degree.