Gatsby Starter Blog RSS Feed

Improving performance using WebAssembly SIMD Intrinsics

Fri, 26 Aug 2022 12:00:00 GMT

WebAssembly is a technology famous of boosting performance of web applications, but what might be the best practice to do achieve maximum performance and what is the cost is what we are going to discuss in this post.

To begin with, let’s talk about how developers develop WebAssembly applications based on the result of The State of WebAssembly 2022. The top 3 languages used to develop WebAssembly are Rust, JavaScript and C/C++. (JavaScript come in second place due to QuickJS, a tiny JS VM, is ported to WebAssembly and boosts JavaScript runtime performance) As Rust and C/C++ are top languages, it is common that developers develop Rust or C/C++ programs and compile to WebAssembly.

Take C/C++ as an example, usually you might use GCC to compile like this:

gcc -o main main.c

When you want to compile to WebAssembly, you might use Emscripten:

emcc -o main.js main.c
# or
emcc -o main.wasm main.c

Then you can use NodeJS or runtime like wasmtime or wasmer to run the program.

But is that the end of the story? Maybe not. Let’s use a matrix multiplication as example to benchmark:

JavaScript Version (Source Code):

// in_b is transposed to speed up multiplication.
const mulMats = (out, in_a, in_b, n) => {
  for (let i = 0; i < n; i++) {
    for (let j = 0; j < n; j++) {
      out.arr[i*n+j] = 0;
      for (let k = 0; k < n; k++) {
        out.arr[i*n+j] += in_a.arr[i*n+k] * in_b.arr[j*n+k];
      }
    }
  }
};

C Version (Source Code):

// in_b is transposed to speed up multiplication.
void multiply_mats(int* out, int* in_a, int* in_b, int n) {
  for (int i = 0; i < n; i++) {
    for (int j = 0; j < n; j++) {
      out[i*n+j] = 0;
      for (int k = 0; k < n; k++) {
        out[i*n+j] += in_a[i*n+k] * in_b[j*n+k];
      }
    }
  }
}

The performance of JavaScript version, C version and WebAssembly version:

Version	Time to Complete	Remark
JavaScript	5.768s	NodeJS v16.16.0
C	7.697s	GCC 12.2.0
WebAssembly	6.865s	Emscripten 3.1.18

It might be unexpected to see that JavaScript version is the fastest one, it is simply because this is NOT a fair comparison as we didn’t use flags to optimize performance.

One of the most significant flag we can use here is -O3, let’s add this flag and below is the result:

Version	Time to Complete	Remark
JavaScript	5.768s	NodeJS v16.16.0
C	0.401s	GCC 12.2.0
WebAssembly	2.012s	Emscripten 3.1.18

With a simple flag, now WebAssembly version is around 65% faster!

But how might we improve the performance even more? Let’s discuss what options we have.

Methods to Improve Performance

There are mainly three approaches to improve performance:

More optimization flags
Use multi-thread
Use SIMD intrinsics

More optimization flags

More optimization flags like -flto might improve performance, but sometimes it might introduce side effects, so it is essential to do more tests when using more flags.

Use multi-thread

WebAssembly supports multi-thread, so you can use libraries like pthread to achieve multi-threading. It works but with a major drawback: SharedArrayBuffer.

SharedArrayBuffer is an essential technology behind multi-threading in WebAssembly. It enables threads (which is web worker in browser) to communicate with each other. But it also consumes more memory and suffers from browser compatibility issue. (Mainly mobile browsers)

SharedArrayBuffer compatibility: https://caniuse.com/sharedarraybuffer

Use SIMD intrinsics

SIMD (Single Instruction, Multiple Data) is a type of parallel processing supported by WebAssembly. In Emscripten, you can add flag -msimd128 to enable SIMD and improve performance for free:

Version	Time to Complete
JavaScript	5.768s
C	0.401s
WebAssembly	2.012s
WebAssembly (w/ -msimd128)	0.278s

-O3 flag comes with auto-vectorization, so it is better to use -msimd128 with -O3

Wow, now our performance is so good that it is even faster than C version! But it is not the end of the story as we haven’t used our secret weapon: SIMD intrinsics.

SIMD intrinsics are low-level functions to enable developers to write assembly code in a more user-friendly way. It is supported not only in Emscripten, but also other major languages like Rust and AssemblyScript. With SIMD intrinsics, we can rewrite our code to improve performance further more:

// in_b is transposed to speed up multiplication.
void multiply_mats(int* out, int* in_a, int* in_b, int n) {
  for (int i = 0; i < n; i++) {
    for (int j = 0; j < n; j++) {
      out[i*n+j] = 0;
      int sum_arr[] = {0, 0, 0, 0};
      v128_t sum = wasm_v128_load(sum_arr);
      for (int k = 0; k < n; k+=4) {
        v128_t a = wasm_v128_load(&in_a[i*n+k]);
        v128_t b = wasm_v128_load(&in_b[j*n+k]);
        v128_t prod = wasm_i32x4_mul(a, b);
        sum = wasm_i32x4_add(sum, prod);
      }
      v128_t sum_duo = wasm_i32x4_add(sum, wasm_i32x4_shuffle(sum, sum, 2, 3, 0, 0));
      v128_t sum_one = wasm_i32x4_add(sum_duo, wasm_i32x4_shuffle(sum_duo, sum_duo, 1, 0, 0, 0));
      out[i*n+j] += wasm_i32x4_extract_lane(sum_one, 0);
    }
  }
}

Version	Time to Complete
JavaScript	5.768s
C	0.401s
WebAssembly	2.012s
WebAssembly (w/ -msimd128)	0.278s
WebAssembly w/ SIMD intrinsics (w/ -msimd128)	0.245s

As the result, we are 95.8% faster than the original JavaScript version!

Conclusion

In this post, we visited a few methods to improve WebAssembly performance, and finally we achieve a much better performance by using SIMD intrinsics. Although it looks perfect, but there is actually a pretty steep learning curve to learn SIMD intrinsics and master it. And you might only get less than 1% performance improvement if your program is already auto-vectorized by the compiler. So the best bet is to use -O3 -msimd128 flags first and rewrite the most computing intensive parts in your code using SIMD intrinsics to improve the performance little by little.

Hope you enjoy this post. :) Feel free to leave comments and try to run the code using the source code in this repository: https://github.com/jeromewu/wasm-perf

Generics in Golang 1.18, no need to write max / min anymore (maybe)

Tue, 08 Jun 2021 12:00:00 GMT

One thing I do dislike in golang is that sometimes a simple task can be very tedious. One of them is getting maximum / minimum value of two integers.

Background

The root cause of the tediousness of this task is the lack of generics in golang which means you can not implement a max/min function which accepts both integer and float numbers. As the built-in max function is math package accepts float64, the common fixes that peope use are:

Write a wrapper function for math.Max or math.Min to do type conversion

func max(x, y int) int {
  return int(math.Max(float64(x), float64(y)))
}

Write your own max function

func max(x, y int) int {
    if x < y {
        return y
    }
    return x
}

You might think it is not too hard to implement any of the solutions above, but it is just a totally waste of time, especially when you are joining online contests or just practicing algorithms and data structure.

Generics to the Rescue

So with the new accepted proposal in golang 1.18, finally we can fix this issue once and for all (if the built-in math.Max & math.Min is updated). A quick and dirty way can be done like below:

type comparable interface {
  type int, int8, int16, int32, int64,
    uint, uint8, uint16, uint32, uint64, uintptr,
    float32, float64
}

func Max[T comparable](x, y T) T {
  return T(math.Max(float64(x), float64(y)))
}

A working version HERE

If similar code like above can be added to the built-in math library, we don’t need to do it ourselves anymore. I believe there are a lot more applications like this can make golang a more elegant and less-tedious language in the future. Looking forward to the release of this version next year. 😃

FFmpeg.wasm, a pure WebAssembly / JavaScript port of FFmpeg

Wed, 04 Nov 2020 00:00:00 GMT

Official Website: https://ffmpegwasm.github.io/

FFmpeg is a famous framework/tool when it comes to video/audio processing, now with ffmpeg.wasm you can use FFmpeg right inside your browser without installation or upload your file to the server. Although ffmpeg.wasm is not as fast as FFmpeg, but it might come in handy for certain use cases.

In this post, I would like to share how to install and use ffmpeg.wasm and also technical details behind to support your usage.

Install ffmpeg.wasm

To install and use ffmpeg.wasm, you only need to use npm/yarn:

$ npm install @ffmpeg/ffmpeg
# or
$ yarn add @ffmpeg/ffmpeg

To use ffmpeg.wasm in node environment, you also need to install @ffmpeg/core .

$ npm install @ffmpeg/core
# or
$ yarn add @ffmpeg/core

Or you can use a CDN

<script src='https://unpkg.com/@ffmpeg/ffmpeg@0.9.3/dist/ffmpeg.min.js'></script>

Use ffmpeg.wasm

To use ffmpeg.wasm, only few lines of code are required:

const fs = require('fs');
const { createFFmpeg, fetchFile } = require('@ffmpeg/ffmpeg');
const ffmpeg = createFFmpeg({ log: true });
(async () => {
  await ffmpeg.load();
  ffmpeg.FS('writeFile', 'test.avi', await fetchFile('./test.avi'));
  await ffmpeg.run('-i', 'test.avi', 'test.mp4');
  fs.writeFileSync('./test.mp4', ffmpeg.FS('readFile', 'test.mp4'));
  process.exit(0);
})();

Don’t forget to add --experimental-wasm-threads and --experimental-wasm-bulk-memory when executing the script in node.

Also you can try the live demo session in the official website: https://ffmpegwasm.github.io/#demo

Behind the scene

To fully utilize/understand the power of ffmpeg.wasm (also for most WebAssembly libraries), few technical details you might want to know:

Inside ffmpeg.load()

ffmpeg.load() is a required API to call before you call others. What happened inside this API is:

Download ffmpeg-core.js from remote server (which is around 25MB)
Instantiate ffmpeg.wasm wasm code

Depending on your network speed and host machine hardware, this operation can take up to minutes to complete.

File System (FS)

When you check the API of ffmpeg.wasm, you may find an API called ffmpeg.FS() to handle File System operations. (In ffmpeg.wasm we use MEMFS / Memory File System) You can imagine this FS just like a hard disk where you can put the input file and pull the output file from ffmpeg.wasm command. It is essential as we want to keep minimal changes to FFmpeg source code and reserve its maxium similarity to original command line interface. Some command operations you might use:

ffmpeg.FS('writeFile', 'filename', data) : write file to MEMFS as input of ffmpeg.wasm
ffmpeg.FS('readFile', 'filename') : read file from MEMFS as output of ffmpeg.wasm
ffmpeg.FS('unlink', 'filename') : delete a file in MEMFS
ffmpeg.FS('readdir', '/') : list files inside specific path

For full list of APIs, you can check: https://emscripten.org/docs/api_reference/Filesystem-API.html

SharedArrayBuffer

SharedArrayBuffer is a pretty new data type in JavaScript, right now most of the browsers still lack of fully support due to security issues. But in ffmpeg.wasm, to enable pthread / mutli-threading support to speed up, it is a required data type to use.

SharedArrayBuffer introduction: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer
Can I use SharedArrayBuffer?: https://caniuse.com/sharedarraybuffer

Web Workers

When you run ffmpeg.run() , you might find tons of web workers spawned. It is a normal situation as web workers are simulating threads in FFmpeg and it is good as we don’t want ffmpeg.wasm to block our main thread.

Leverage CPU capabilities

For most of libraries work with ffmpeg (ex. x264), they use assembly language like x86 to speed up the process. But sadly, WebAssembly cannot directly use these x86 assembly code as it is not compatible or certain instructions are not supported. Thus it takes time to port x86 assembly to SIMD assembly to make it work.

It is still a long way to go as ffmpeg.wasm is still in an early stage (right now it is only v0.9), although it is still very slow comparing to original version, but with the growth and evolution of WebAssembly, I believe it will become more and more useful. 😃

Feel free to try and comments, issues or/and PRs are highly welcome!

Github: https://github.com/ffmpegwasm/ffmpeg.wasm

How to fix (80%) HiDPI issues in Gulliam OS 3.1

Sun, 12 Apr 2020 00:00:00 GMT

In 2019, I bought a Chromebook (HP Chromebook 13 G1, release on 2016) to be a lightweight laptop when I am out. I really enjoyed using Chrome OS with crouton, but one thing really bothers me is the fact I cannot run container. That’s why I decided to try the latest Gallium OS to get full Linux experience inside my Chromebook, but when I completed installing Gallium OS, the HiDPI hitted me and it took me tons of time to survey how to resolve this issue, but sadly I am only able to fix around 60% of them. To save time for people who may encounter the same issue, here is what did:

Update xfce configration

There was a package called galliumos-hidpi to fix hidpi issue, I check the scripts and this script is the derived version of it:

#!/bin/sh

# Raise DPI from 96 to 276 to makes text size front
xfconf-query -c xsettings -p /Xft/DPI -s 276

# Enlarge cursor size
xfconf-query -c xsettings -p /Gtk/CursorThemeSize -s 48

# Raise xfce panel height
xfconf-query -c xfce4-panel -p /panels/panel-1/size -s 64

# Enlarge notification area icon
xfconf-query -c xfce4-panel -p /plugins/plugin-6/size-max -s 64

# Enlarge window size
xfconf-query -c xfce4-desktop -p /last/window-height -s 1240
xfconf-query -c xfce4-desktop -p /last/window-width -s 909

# Enlarge desktop icon
xfconf-query -n -t uint -c xfce4-desktop -p /desktop-icons/icon-size -s 96

# Use xhdpi theme, you can alsto use "Default-hdpi"
xfconf-query -c xfwm4 -p /general/theme -s "Default-xhdpi"

# Enlarge most of the icons
xfconf-query -c xsettings -p /Gtk/IconSizes -s gtk-menu=48,48:gtk-button=48,48:gtk-dialog=48,48:gtk-dnd=48,48:gtk-small-toolbar=48,48:gtk-large-toolbar=64,64

To execute this shell script(say fix-hidpi.sh), you need to:

$ sh fix-hidpi.sh

GTK3 Scale

For small icons in menu, you can edit /etc/profile.d/gdkscale.sh and add following lines:

export GDK_SCALE=2.875
export GDK_DPI_SCAL=0.38 # 1/2.875

xfce4-panel theming

For power manager plugin icon in xfce4-panel, create ~/.config/gtk-3.0/gtk.css and add following lines:

#xfce4-power-manager-plugin * {
  -gtk-icon-transform: scale(0.38);
}

Work in Progress

Lightdm
Xscreensaver
Extended Monitor (hard one)

Hope it helps someone to save some time.(like 2 days. Lol)

How to speed up Node.js modules installation in CI/CD pipeline as of 2020

Sat, 28 Mar 2020 00:00:00 GMT

When you use Node.js in your project, you need to take care of node_modules installation in your CI/CD pipeline. You may use npm install, npm ci, yarn install and pnpm install, but what is the fastest way of doing so is still a question we are looking for answers. Here in this story I would like to experiment all possibilities I know to find out the final answer.

TL;DR

If you are in a hurry, what you have to do are:

Disable stdout output
Use cached node_modules
Use --prefer-offline
Use global cache (yarn only)

$ tar zxvf node_modules.tar.gz
$ npm install --prefer-offline &> /dev/null

If you prefer yarn

$ tar zxvf node_modules.tar.gz
$ tar zxvf cache.tar.gz
$ yarn install --prefer-offline --cache-folder ./cache &> /dev/null

If you prefer pnpm

$ tar zxvf cache.tar.gz
$ pnpm config set store-dir $PWD/cache
$ pnpm install &> /dev/null

The total speed up is around 3x comparing to pure npm install

Existing Works

It is normal that someone already do it in the past, here are some existing works for your reference.

Environment

Packages

To prevent any cache or unexpected behavior, I run the experiment inside a docker container and every time we complete an installation, we run a new container for the next installation.

The docker image I use is node:12 and the versions of each component are:

node: v12.16.1
npm: 6.13.4
yarn: 1.22.0
pnpm: 4.12.1

Network

I experiment both remote and local registry (verdaccio) scenario to show the influence of network. In local registry, I assumed the user do something like:

$ npm install react@latest --registry http://nvy-registry:4873
# or
$ yarn add react@latest --registry http://nvy-registry:4873

Which makes the registry url in lock files points to the local registry directly.

Application

To mimic a real world scenario, I did experiment in an web app that is created using create-react-app.

Commands

There are 5 basic commands I use:

npm install
npm ci
yarn install
yarn install —frozen-lockfile
pnpm install

yarn install --frozen-lockfile is similar to npm ci

For all of them, I add &> /dev/null to remove stdout output and the execution time is averaged from 3 times of execution. (As the execution time is pretty stable, I believe 3 times is good enough)

Let’s start the experiment and see where we will achieve. 😃

Parameters

There are 4 parameters we are going to tune:

local registry

With local registry, you can prevent stale when the network bandwidth is not enough. It is a good way to speed up but may be painful when someone clone your repository but unable to access the local registry you used. 2. cached node_modules

A common technique to speed up the installation, by zipping node_modules folder and unzipping before installation to eliminate the need of downloading modules again from network. 3. global cache

There is a global cache folder in ~/.npm for npm and ~/.yarn for yarn, by adding global cache in advance, I would like to know if it is possible to speed up. 4. --prefer-offline

Details: https://docs.npmjs.com/misc/config#prefer-offline

If true, staleness checks for cached data will be bypassed, but missing data will be requested from the server. To force full offline mode, use --offline.

With this option, we can prevent npm or yarn to check remote data and use local cache directly.

Experiment

Below are the result of all configurations:

The number is the amount of seconds to complete modules installation

Observations

Always remember to disable the output of installation with &> /dev/null as stdout is pretty time-consuming
Although local registry can guarantee the stability of downloading modules, it is not recommended to modify lock file to gain this benefit as the side effect is huge.
The speed of npm ci is stable across different configurations, but npm install can be faster with --prefer-offline.
The speed of yarn install and yarn install --frozen-lockfile are NOT different at all. Global cache is more useful for yarn
The efficient of pnpm highly relies on the global cache, without cache, its speed is slower than others due to cache pre-processing overhead

For the source code of this experiment can be found in this repository: https://github.com/jeromewu/npm-vs-yarn-in-cicd

Hope you find it helpful and if you have any ideas to experiment to speed up even further, please feel free to leave a response or create an issue inside the repository. 😄

I wrote a Chrome Extension (Gitpod Window). Here is what I learned

Mon, 23 Mar 2020 00:00:00 GMT

Gitpod is an online IDE for GitHub and GitLab, it is great and makes my life easier when there is not enough resource in my laptop. But as a VIM user it is pretty annoying when I hit Ctrl+w (which is commonly used to navigate between different VIM window), the whole Gitpod tab just close and I have to reopen it again.

That’s why I start to work on a Chrome Extension (Gitpod Window) to make the overall experience of using Gitpod even better. The requirements I listed are:

When user click the extension icon on a GitHub or GitLab page, opens Gitpod workspace in a popup Chrome window
Disable Chrome shortcuts

As I have almost no experience in building a Chrome Extension, I wrote this story to record the process and hope it can help newbies like me. 😃

Get Started with Chrome Extension

Google provides a good official tutorial here: https://developer.chrome.com/extensions/getstarted

Quick summary for key components:

manifest.json: Provides important information
Background Script: Extensions monitor events in their background script, then react with specified instructions

Here we learnt that background script is where we can monitor events and it is a good starting point to achieve our goal.

I would suggest those who want to follow the next step to complete the tutorial in the link above first.

When user click the extension icon on a GitHub or GitLab page, opens Gitpod workspace in a popup Chrome window

As you can imagine, click on extension icon is an event, so we can do it in background script.

To handle the event, there are two types of actions you can choose: pageAction and browserAction, as our extension only works on specific websites, we use pageAction here. A primary manifest.json below:

{
  "manifest_version": 2,
  "name": "Gitpod Window",
  "version": "1.0.0",
  "description": "Gitpod Window disables browser shortcuts for you to prevent unwant behavior",
  "author": "Jerome Wu <jeromewus@gmail.com>",
  "permissions": [
    "*://github.com/*/*",
    "*://gitlab.com/*/*"
  ],
  "background": {
    "scripts": ["background.js"],
    "persistent": false
  },
  "page_action": {}
}

In permissions, we add GitHub and GitLab URLs to restrict that this extension only works on the repository page. (ex. https://github.com/jeromewu/gitpod-window) We also add a key page_action to identify that we are going to use pageAction APIs in this extension.

With manifest.json settled, next we need to update background.js to popup the window with windows API:

/*
 * Window popup using page action
 *
 * @ref: https://developer.chrome.com/extensions/pageAction
 *
 */

chrome.pageAction.onClicked.addListener(({ url }) => {
  chrome.windows.create({
    url: `https://.gitpod.io/#${url}`,
    type: 'popup',
  });
});

Ideally, we should be able to see a new popup window created when we click on the extension icon, but if you try to code above, you will see a context menu instead.

The behavior is not the same as we expected and if you take a close look to the chrome.pageAction.onClicked.addListener(), you will see an interesting statement:

This event will not fire if the page action has a popup.

As we don’t do anything about popup, it means the popup doesn’t exist, right? The answer is yes, but there is a strange behavior for onClicked, usually it should look like this:

But actually, the decision is made after chrome.pageAction.show()

So if you don’t trigger chrome.pageAction.show() first, it won’t trigger onClicked, I found this answer when checking samples from HERE and it is really a trap when using onClicked.

To solve this issue, we need to trigger chrome.pageAction.show() on the tab we click extension icon manually. We need to use onUpdated and onActivated in tabs API. Before we start using tabs, don’t forget to update permissions in the manifest.json first:

{
  "manifest_version": 2,
  "name": "Gitpod Window",
  "version": "1.0.0",
  "description": "Gitpod Window disables browser shortcuts for you to prevent unwant behavior",
  "author": "Jerome Wu <jeromewus@gmail.com>",
  "permissions": [
    "tabs",
    "*://github.com/*/*",
    "*://gitlab.com/*/*"
  ],
  "background": {
    "scripts": ["background.js"],
    "persistent": false
  },
  "page_action": {}
}

Then let’s update background.js:

/*
 * Window popup using page action
 *
 * @ref: https://developer.chrome.com/extensions/pageAction
 * @ref: https://developer.chrome.com/extensions/examples/api/pageAction/set_icon.zip
 * This is a workaround to make sure chrome.pageAction.onClicked works.
 *
 * The idea here to to force popup to show on GitHub and GitLab pages,
 * but as we don't assign popup, it fails and called onClicked.
 * If we don't do this, onClicked won't be triggered.
 */

const forcePopup = ({ id, url }) => {
  if (/https?:\/\/(github|gitlab).com\/*\/*/.test(url)) {
    chrome.pageAction.show(id);
  }
};

chrome.tabs.onUpdated.addListener((tabId, activeInfo, tab) => {
  forcePopup(tab);
});

chrome.tabs.onActivated.addListener(({ tabId }) => {
  chrome.tabs.get(tabId, forcePopup);
});

chrome.pageAction.onClicked.addListener(({ url }) => {
  chrome.windows.create({
    url: `https://.gitpod.io/#${url}`,
    type: 'popup',
  });
});

The idea here is to call chrome.pageAction.show() on the GitHub and GitLab pages to trigger onClicked.

Now we have completed our first requirement! It is a nice-looking popup window that makes Gitpod more close to an IDE, but there is no IDE can be closed with Ctrl+W, let move on to the next part to disable shortcuts.

Disable keyboard shortcuts

To disable shortcuts, I planned to use commands API, but in the end I found that you don’t need to do anything as the shortcuts that annoying us are disabled by default. (incl. Ctrl+W, Ctrl+N, Ctrl+T) So we complete this issue by using popup window here!

Hope you enjoy this story and if you would like to know more about this Chrome Extension, you can visit:

GitHub Repository: https://github.com/jeromewu/gitpod-window/tree/v1.0.1
Chrome Web Store: https://chrome.google.com/webstore/detail/gitpod-window/jhpkbhdoenpnfnnkelgpikndadikogdl

3 Things you should do to quickly fix your problem in a open source project

Wed, 15 Jan 2020 12:00:00 GMT

As a developer, it is common to leverage open source projects to complete your application, and it is more common to find your requirement unsatisfied or a bug stopping from fulfilling your goal.

Here are 3 tips for you to quickly fix problems in open source, it might be intuitive, but useful if you haven’t tried before.

Check Documents

Many open source projects comes with Wiki, an official website or a doc/ or docs/ folder inside the repository. These locations are the first place you should look for clues to fix the problem you faced, most open source maintainers tried to keep these documents update-to-date to make sure newcomers can find what they need without too much trouble.

Search Issues

Most of the time, you may not be the first one to face the issue if the use case you have is common, so don’t forget to search the issues inside Github or similar platform. And not just search open issues (default option in Github), but remember to check closed issues. It is a quick way to find some hotfix or even work around when your issues is not fixed by the maintainer so far.

Open New Issues

Open new issues is the last thing you can do, but most of us may have experience that there is no response for days and feel like our issues are ignored by maintainers. It is kind of normal, but a little change can make the story different.

You have to know that maintainers are human like you and me, so they are unable to fix your issues if the information is not sufficient. I have been maintainers for open source projects for few years and there are more than 50% of the issues are poorly described and thus I was unable to fix without asking few more questions and it really takes lots of time for both sides to wait for replies from each other.

To quickly fix the problem, you have to open a good quality issue to enable the maintainers to understand your situation without too much effort. A good issues should include information like:

The scenario you want to achieve
The problem you faced (incl. error messages)
The code to reproduce the issue (IMPORTANT!)
Environment (incl. OS, the version you used)

With enough information, the maintainers can start to find and fix the bugs without wasting time guessing your situation. Although it takes more effort from you, but it really saves lots of time comparing to leave insufficient information for the maintainers to figure out. I think it is a concept of “Help you to help me” and I recommend you to try to open a good quality issue and you will find your issue will be accepted and fix soon.

One more thing

If you still cannot get your problems fixed, don’t forget it is open source and you can always dive into the source code and find the solution all by yourself. It will take much more time, but you will learn a lot during the process, and don’t forget to create a PR if you have a good fix that can benefit others.

Although these 3 things looks fundamental to do, but sometimes the basic way is the best, hope you find it useful next time you have a problem with an open source project.

WebAssembly & Transferable, a Perfect Match

Wed, 15 Jan 2020 00:00:00 GMT

Most of the time, we run WebAssembly code inside a Web Worker (or worker_thread in Node.js) to prevent blocking the main thread and maintaining a responsive UI.

To communicate with Web Worker, we use Window.postMessage() to send data from main thread to web worker thread. Sometimes you may need to send megabytes of data (especially for multimedia applications) and it will result in long data transmission time and even out of memory exception due to the nature of postMessage() is to clone the data from main thread to web worker thread.

To solve this issue, you can use Transferable in postMessage(), simply pass second argument in postMessage() and you can get a speed up and memory saving without too much trouble.

// Without Transerable, takes up to 2000 ms for worker to get data
const data = new UInt8Array(1024*1024*1024);
worker.postMessage(data);

// With Transerable, takes up to 100 ms for worker to get data
const data = new UInt8Array(1024*1024*1024);
worker.postMessage(data, [data.buffer]);

Check a demo in this repository: https://github.com/jeromewu/web-worker-transferable-demo

As it is almost free to use Transferable and get speed up and memory saving in your application, you should start to use it in your application right now.

See you in next story. :)

How to use Gitpod in your create-react-app based project

Tue, 19 Nov 2019 12:00:00 GMT

Japanese Version (translated by Ryuichi Tanabe): https://qiita.com/baby-degu/items/bacf20f657625ae8747b

Gitpod is a great online IDE and here is how you can use in your create-react-app based project.

Why Gitpod?

Sometimes we just don’t want to spend time to prepare the development environment and start coding as soon as possible. Gitpod is a Github-integrated and 100 hours per month free use online IDE, it provides flexibility to customize your work space with .gitpod.yml.

In this story, we will talk about how to leverage Gitpod when you have a project based on create-react-app (or any react project).

Basic Configuration

In the basic configuration, we want to achieve following items:

npm install when we create the work space
npm start every time we start the work space
See preview every time we update the code

To achieve all the features, you just need to add a .gitpod.yml with following content:

tasks:
  - init: npm install
    command: npm start

ports:
  - port: 3000
    onOpen: open-preview

It is pretty self-explained and you can use it most of the projects easily.

More options in official docs

Advanced Configuration

An advanced scenario is to use puppeteer for end-to-end testing in your create-react-app, by default some dependencies used by puppeteer is not installed in Gitpod default work space image (gitpod/workspace-full). So we need to extend the Docker image to make it work.

Add .gitpod.Dockerfile in your repository:

FROM gitpod/workspace-full
RUN sudo apt-get update && sudo apt-get install -y libgtk-3-0 libx11-xcb1 libnss3 libxss1 libasound2

And update .gitpod.yml to use our .gitpod.Dockerfile

image:
  file: .gitpod.Dockerfile

tasks:
  - init: npm install
    command: npm start

ports:
  - port: 3000
    onOpen: open-preview

Also you need to use --no-sandbox in jest-puppeteer, the full setup can be found here:

Repository: https://github.com/jeromewu/create-react-app-gitpod

Gitpod: https://gitpod.io/#https://github.com/jeromewu/create-react-app-gitpod

With full configuration, now you can have a working environment like below:

One more thing

If you are a VIM user like me, here is a small trick (maybe not the best) to use your VIM configuration in Gitpod with least effort.

image:
  file: .gitpod.Dockerfile

tasks:
  - init: wget https://github.com/jeromewu/docker-workspace/raw/master/vim.zip -O /tmp/vim.zip &> /dev/null && unzip /tmp/vim.zip -d /home/gitpod &> /dev/null
  - init: yarn install
    command: yarn start

ports:
  - port: 3000
    onOpen: open-preview

The concept here is to zip an existing VIM configuration, download and unzip to container, it saves lots of time to install but may not be a good option if you need to update your VIM configuration very often.

In a word, using Gitpod for your Github project offers you a powerful environment to develop your project anytime and anywhere with ease. Hope you find this story useful and happy coding. 😃

Speedup FFmpeg without compiling from source code

Sun, 17 Nov 2019 12:00:00 GMT

FFmpeg is a great tool when it comes to multimedia manipulation, by default it uses CPU with multi-thread to do the task which causes a high loading to your PC and is kind of slow most of the time.

If you google how to improve the speed of FFmpeg, you may find discussions about using -preset which lowers the compression ratio for higher speed (a trade-off between file size and speed) and another sexy approach is to leverage nVidia GPU (nvenc, nvdec and cuvid), but it it is not easy as:

You don’t own a compatible nVidia GPU card
You need to install nVidia GPU drivers and CUDA (painful in Linux environment)
You need to compile FFmpeg from source if you cannot find a ready-to-use version

In my own experience, I spent hours to research how to install and compile FFmpeg, but fails in the end as my GPU is unable to support most of the features, that was sad.

So is there any other way to make FFmpeg faster? Yes, and you can do that within seconds using VAAPI. Let’s do an experiment to see the differences.

Baseline: Scale a video without any options

Imagine you are creating a service to provide video streaming with different quality (720p, 1080p, etc.), so you need to scale down the uploaded video to different resolution.

Let’s download a sample video from h264info and scale down without any options:

$ ffmpeg -i gravity.mp4 \
    -c:v libx264 \
    -s 1024x428 \
    -b:v 1M \
    out.mp4

It takes about 42 seconds (speed=3.5x) with 21 MB file size. (the original size is 355 MB)

With -preset to speedup with larger file size

Check HERE for more details about preset

Use preset, you can easily speed up FFmpeg with larger file size, which is acceptable when you have enough space left in your hard disk.

$ ffmpeg -i gravity.mp4 \
    -c:v libx264 \
    -preset ultrafast \
    -s 1024x428 \
    -b:v 1M \
    out.mp4

It takes about 22 seconds (speed=6x) and with 20 MB file size. (Interesting, it is smaller than -preset default 😎)

With VAAPI to speedup with integrated/ Intel GPU card

Video Acceleration API (VAAPI) is not a secret in FFmpeg, but it is hard to notice how it can easily help you to speedup FFmpeg. The benefits of using VAAPI are:

The integrated GPU card is cheap (and you already have one now)
You only have to install i965-va-driver to make it work
You don’t have to compile FFmpeg as this flag is enabled by default

To use VAAPI in Ubuntu, first you need to install driver and check status with vainfo command:

$ sudo apt-get install i965-va-driver
$ vainfo
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.1 (libva 2.1.0)
vainfo: Driver version: Intel i965 driver for Intel(R) Ivybridge Mobile - 2.1.0
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264StereoHigh         : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileJPEGBaseline           : VAEntrypointVLD

If you see similar output like above, that means your Intel GPU card supports VAAPI, then you can use following command to scale:

$ ffmpeg -hwaccel vaapi \
    -hwaccel_device /dev/dri/renderD129 \
    -hwaccel_output_format vaapi \
    -i gravity.mp4 \
    -vf "scale_vaapi=w=1024:h=428" \
    -c:v h264_vaapi \
    -b:v 1M \
    out.mp4

It takes about 10 seconds (speed=14.3x) and with 19 MB file size.

Put them altogether in a chart, using VAAPI can get the almost 4x speedup comparing to native and a little smaller file size as well.

In fact, if you invest time and effort to survey the nVidia GPU option you can have even more speedup, but it is still great to have speedups without too much effort and cost, isn’t it? 😄

Hope you find this post useful and don’t hesitate to leave any responses or claps, thank you!

Build FFmpeg WebAssembly version (= ffmpeg.wasm): Part.4 ffmpeg.wasm v0.2 — Add Libx264

Thu, 31 Oct 2019 00:00:00 GMT

In this part you will learn:

Add Libx264 to ffmpeg-core.js
In browser ffmpeg.wasm demo

Add Libx264 to ffmpeg-core.js

For next step, we would like to transcode an avi video and play it in our web browser. By default ffmpeg-core.js can transcode avi to mp4, but that mp4 file cannot be played in web browser as its encoding is not supported. So we need to add libx264 to our ffmpeg-core.js first.

Below is the link to the x264 library we are going to add:

https://code.videolan.org/videolan/x264

x264 is much easier to build compared to ffmpeg, below are the key arguments you need to pass:

#!/bin/bash -x

ARGS=(
  --host=i686-gnu                     # use i686 gnu
  --enable-static                     # enable building static library
  --disable-cli                       # disable cli tools
  --disable-asm                       # disable asm optimization
  --extra-cflags="-s USE_PTHREADS=1"  # pass this flags for using pthreads
)
emconfigure ./configure "${ARGS[@]}"

Check the repository for the full version of build-x264.sh

While configuring ffmpeg, --enable-gpl and --enable-libx264 flags must be added.

#!/bin/bash -x

# ...

CONFIG_ARGS=(
  --target-os=none        # use none to prevent any os specific configurations
  --arch=x86_32           # use x86_32 to achieve minimal architectural optimization
  --enable-cross-compile  # enable cross compile
  --disable-x86asm        # disable x86 asm
  --disable-inline-asm    # disable inline asm
  --disable-stripping     # disable stripping
  --disable-programs      # disable programs build (incl. ffplay, ffprobe & ffmpeg)
  --disable-doc           # disable doc
  --enable-gpl            ## required by x264
  --enable-libx264        ## enable x264
  --extra-cflags="$CFLAGS"
  --extra-cxxflags="$CFLAGS"
  --extra-ldflags="$LDFLAGS"
  --nm="llvm-nm -g"
  --ar=emar
  --as=llvm-as
  --ranlib=llvm-ranlib
  --cc=emcc
  --cxx=em++
  --objcc=emcc
  --dep-cc=emcc
)
emconfigure ./configure "${CONFIG_ARGS[@]}"

# ...

ARGS=(
  -I. -I./fftools -I$BUILD_DIR/include
  -Llibavcodec -Llibavdevice -Llibavfilter -Llibavformat -Llibavresample -Llibavutil -Llibpostproc -Llibswscale -Llibswresample -L$BUILD_DIR/lib
  -Qunused-arguments
  -o wasm/dist/ffmpeg-core.js fftools/ffmpeg_opt.c fftools/ffmpeg_filter.c fftools/ffmpeg_hw.c fftools/cmdutils.c fftools/ffmpeg.c
  # Add -lpostproc and -lx264 below
  -lavdevice -lavfilter -lavformat -lavcodec -lswresample -lswscale -lavutil -lpostproc -lm -lx264 -pthread
  -O3                                           # Optimize code with performance first
  -s USE_SDL=2                                  # use SDL2
  -s USE_PTHREADS=1                             # enable pthreads support
  -s PROXY_TO_PTHREAD=1                         # detach main() from browser/UI main thread
  -s INVOKE_RUN=0                               # not to run the main() in the beginning
  -s EXPORTED_FUNCTIONS="[_main, _proxy_main]"  # export main and proxy_main funcs
  -s EXTRA_EXPORTED_RUNTIME_METHODS="[FS, cwrap, setValue, writeAsciiToMemory]"   # export preamble funcs
  -s INITIAL_MEMORY=268435456                   ## increase to 268435456 bytes = 256 MB
)
emcc "${ARGS[@]}"

Check the repository for the full version of configure.sh and build-ffmpeg.sh

With all script in place, now you can build a ffmpeg.wasm with x264 (and potentially all other libraries.)

In browser ffmpeg.wasm demo

The final part of this story is a demo for ffmpeg.wasm v0.2, the scenario is to create a web page that enables user to upload a video file (ex. avi) and play inside the web browser. As it is not possible to play avi file directly, we will use ffmpeg.wasm to transcode the video first:

Below is the full HTML code (download sample video HERE):

<html>                                                                                                                                            
  <head>                                                                                                                                          
    <style>                                                                                                                                       
      html, body {                                                       
        margin: 0;                                                       
        width: 100%;                                                     
        height: 100%                                                     
      }                                                                  
      body {                                                                                                                                      
        display: flex;                                                   
        flex-direction: column;
        align-items: center;                                             
      }   
    </style>                                                                                                                                      
  </head>                                                                
  <body>                                                                 
    <h3>Upload a video to transcode to mp4 (x264) and play!</h3>
    <video id="output-video" controls></video><br/> 
    <input type="file" id="uploader">                   
    <p id="message">Remeber to wait for 5 seconds for ffmpeg.wasm to load</p>
    <script type="text/javascript">                                                                                                               
      const readFromBlobOrFile = (blob) => (
        new Promise((resolve, reject) => {
          const fileReader = new FileReader();
          fileReader.onload = () => {
            resolve(fileReader.result);
          };
          fileReader.onerror = ({ target: { error: { code } } }) => {
            reject(Error(`File could not be read! Code=${code}`));
          };
          fileReader.readAsArrayBuffer(blob);
        })
      );
      const message = document.getElementById('message');
      const transcode = async ({ target: { files } }) => {
        const { name } = files[0];
        message.innerHTML = 'Writing file to MEMFS';
        const data = await readFromBlobOrFile(files[0]);                                                                                          
        Module.FS.writeFile(name, new Uint8Array(data));                                                                                          
        const ffmpeg = Module.cwrap('proxy_main', 'number', ['number', 'number']);
        const args = ['ffmpeg', '-hide_banner', '-nostdin', '-report', '-i', name, 'out.mp4'];
        const argsPtr = Module._malloc(args.length * Uint32Array.BYTES_PER_ELEMENT);
        args.forEach((s, idx) => {                                       
          const buf = Module._malloc(s.length + 1);                      
          Module.writeAsciiToMemory(s, buf);                                                                                                      
          Module.setValue(argsPtr + (Uint32Array.BYTES_PER_ELEMENT * idx), buf, 'i32');
        });                    
        message.innerHTML = 'Start to transcode';                        
        ffmpeg(args.length, argsPtr);

        /*                                                               
         * The execution of ffmpeg is not synchronized,                  
         * so we need to parse the log file to check if completed.
         */                                                              
        const timer = setInterval(() => {               
          const logFileName = Module.FS.readdir('.').find(name => name.endsWith('.log'));
          if (typeof logFileName !== 'undefined') {                                                                                               
            const log = String.fromCharCode.apply(null, Module.FS.readFile(logFileName));
            if (log.includes("frames successfully decoded")) {
              clearInterval(timer);                                      
              message.innerHTML = 'Finish transcoding';
              const out = Module.FS.readFile('out.mp4');
              const video = document.getElementById('output-video');
              video.src = URL.createObjectURL(new Blob([out.buffer], { type: 'video/mp4' }));
            }                                                            
          } 
        }, 500);                                                         
      };  
      document.getElementById('uploader').addEventListener('change', transcode);
    </script>                                                            
    <script type="text/javascript" src="./dist/ffmpeg-core.js"></script>
  </body>                         
</html>

It may take a long time to complete, you can open DevTools to see logs. Check transcode.html to see how it works in action.

You can visit the repository here to see how it works in details: https://github.com/ffmpegwasm/FFmpeg/tree/n4.3.1-p4

And feel free to download the build artifacts here: https://github.com/ffmpegwasm/FFmpeg/releases/tag/n4.3.1-p4

Here is a pause the this series of stories, hope you learnt how to build your own ffmpeg.wasm from scratch and potentially apply to any other libraries. See you next time! 😃

Build FFmpeg WebAssembly version (= ffmpeg.wasm): Part.3 ffmpeg.wasm v0.1 — Transcoding avi to mp4

Mon, 21 Oct 2019 00:00:00 GMT

In this part you will learn:

Build a library version of FFmpeg with optimized arguments.
Interact with ffmpeg.wasm
Manage Emscripten File System.
Develop ffmpeg.wasm v0.1 with transcoding feature.

Build a library version of FFmpeg with optimized arguments.

In Part.3, our goal is to create a basic ffmpeg.wasm v0.1 to transcode avi to mp4. As we only create a basic version of FFmpeg in Part.2, now we need to further optimized with few arguments.

-O3 : optimize code and reduce code size (from 30 MB to 15 MB) (More details HERE)
-s PROXY_TO_PTHREAD=1 : to make our program for responsive when using pthread (More details HERE)
-o wasm/dist/ffmpeg-core.js : rename ffmpeg.js to ffmpeg-core.js

(From here we call it ffmpeg-core.js as we will create a ffmpeg.js library to wrap the ffmpeg-core.js and provide user friendly APIs.)

-s EXPORTED_FUNCTIONS="[_main, _proxy_main]" : export main() and proxy_main() (added by PROXY_TO_PTHREAD) C function to JavaScript world
-s EXTRA_EXPORTED_RUNTIME_METHODS="[FS, cwrap, setValue, writeAsciiToMemory]" : extra functions for manipulating functions, file system and pointers, check Interacting with code and preamble.js for more details.

For more details about these arguments, you can check src/settings.js in emscripten github repository.

With all the new arguments, let’s update our build.sh :

#!/bin/bash -x

# verify Emscripten version
emcc -v

# configure FFMpeg with Emscripten
CFLAGS="-s USE_PTHREADS -O3"
LDFLAGS="$CFLAGS -s INITIAL_MEMORY=33554432" # 33554432 bytes = 32 MB
CONFIG_ARGS=(
  --target-os=none        # use none to prevent any os specific configurations
  --arch=x86_32           # use x86_32 to achieve minimal architectural optimization
  --enable-cross-compile  # enable cross compile
  --disable-x86asm        # disable x86 asm
  --disable-inline-asm    # disable inline asm
  --disable-stripping     # disable stripping
  --disable-programs      # disable programs build (incl. ffplay, ffprobe & ffmpeg)
  --disable-doc           # disable doc
  --extra-cflags="$CFLAGS"
  --extra-cxxflags="$CFLAGS"
  --extra-ldflags="$LDFLAGS"
  --nm="llvm-nm -g"
  --ar=emar
  --as=llvm-as
  --ranlib=llvm-ranlib
  --cc=emcc
  --cxx=em++
  --objcc=emcc
  --dep-cc=emcc
)
emconfigure ./configure "${CONFIG_ARGS[@]}"

# build dependencies
emmake make -j4

# build ffmpeg.wasm
mkdir -p wasm/dist
ARGS=(
  -I. -I./fftools
  -Llibavcodec -Llibavdevice -Llibavfilter -Llibavformat -Llibavresample -Llibavutil -Llibpostproc -Llibswscale -Llibswresample
  -Qunused-arguments
  -o wasm/dist/ffmpeg.js fftools/ffmpeg_opt.c fftools/ffmpeg_filter.c fftools/ffmpeg_hw.c fftools/cmdutils.c fftools/ffmpeg.c
  -lavdevice -lavfilter -lavformat -lavcodec -lswresample -lswscale -lavutil -lm
  -O3                                           # Optimize code with performance first
  -s USE_SDL=2                                  # use SDL2
  -s USE_PTHREADS=1                             # enable pthreads support
  -s PROXY_TO_PTHREAD=1                         # detach main() from browser/UI main thread
  -s INVOKE_RUN=0                               # not to run the main() in the beginning
  -s EXPORTED_FUNCTIONS="[_main, _proxy_main]"  # export main and proxy_main funcs
  -s EXTRA_EXPORTED_RUNTIME_METHODS="[FS, cwrap, setValue, writeAsciiToMemory]"   # export preamble funcs
  -s INITIAL_MEMORY=33554432                    # 33554432 bytes = 32 MB
)
emcc "${ARGS[@]}"

Next, let’s try to interact with ffmpeg.wasm.

Interact with ffmpeg.wasm

To make sure our ffmpeg.wasm is working, let’s try to achieve following command in ffmpeg.wasm:

$ ffmpeg -hide_banner

With -hide_banner argument, ffmpeg hides details about its version and build arguments, a typical output looks like this:

Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...
Use -h to get full help or, even better, run 'man ffmpeg'

To begin with, let’s create a file called ffmpeg.js with following code:

const Module = require('./dist/ffmpeg-core.js');

Module.onRuntimeInitialized = () => {
  const ffmpeg = Module.cwrap('proxy_main', 'number', ['number', 'number']);
};

The execution of the code above requires extra parameters in Node.JS:

$ node --experimental-wasm-threads --experimental-wasm-bulk-memory ffmpeg.js

Explanation for the functions:

onRuntimeInitialized : as WebAssembly requires some time to boot up, you need to wait for this function to be called before using the library.
cwrap : a wrapper function for the C function in JavaScript world. Here we wrap proxy_main() / main() function in fftools/ffmpeg.c. The function signature is int main(int argc, char **argv), it is straightforward that int is mapping to number and as char **argv is a pointer in C, we can also mapping it to number.

Then we need to pass the arguments to it. The equivalent argument for $ ffmpeg -hide_banner is main(2, ["./ffmpeg", "-hide_banner"]). The first argument is easy, but how can we pass a string array? Let’s decompose the issue to two parts:

We need to convert string in JavaScript to char array in C
We need ot convert a number array in JavaScript to a pointer array in C

The 1st part is easier as we have an utility function from Emscripten called writeAsciiToMemory() to help us, below is an example of using this function:

const str = "FFmpeg.wasm";
const buf = Module._malloc(str.length + 1); // Allocate a memory with extra byte with value 0 to indicate the end of the string
Module.writeAsciiToMemory(str, buf);

The 2nd part is tricky, we need to create a pointer array in C with 32 bit integer as the pointer is in 32 bit integer. We need to use setValue here to create the array we need:

const ptrs = [123, 3455];
const buf = Module._malloc(ptrs.length * Uint32Array.BYTES_PER_ELEMENT);
ptrs.forEach((p, idx) => {
  Module.setValue(buf + (Uint32Array.BYTES_PER_ELEMENT * idx), p, 'i32');
});

Merge all the snippets above, now we can interact with ffmpeg.wasm and generate expected result:

const Module = require('./dist/ffmpeg-core');

Module.onRuntimeInitialized = () => {
  const ffmpeg = Module.cwrap('proxy_main', 'number', ['number', 'number']);
  const args = ['ffmpeg', '-hide_banner'];
  const argsPtr = Module._malloc(args.length * Uint32Array.BYTES_PER_ELEMENT);
  args.forEach((s, idx) => {
    const buf = Module._malloc(s.length + 1);
    Module.writeAsciiToMemory(s, buf);
    Module.setValue(argsPtr + (Uint32Array.BYTES_PER_ELEMENT * idx), buf, 'i32');
  })
  ffmpeg(args.length, argsPtr);
};

Now, we can interact with ffmpeg.wasm with ease, but how can we pass the video file to it? That is the focus of next section: File System.

Manage Emscripten File System.

In Emscripten, there is a virtual file system to support standard file read/write in C, thus we need to write our video file into this File System before passing the arguments to ffmpeg.wasm.

Find more details in File System API.

Most of the time, there are only 2 FS functions you need to complete the task: FS.writeFile() and FS.readFile() .

For all the data writing to or reading from the File System, it must be in Uint8Array type in JavaScript, remember to do type conversion before consuming the data.

For this tutorial, let’s use a file called flame.avi (You can download it HERE) and read it with fs.readFileSync() and write it to Emscripten File System with FS.writeFile() .

const fs = require('fs');
const Module = require('./dist/ffmpeg-core');

Module.onRuntimeInitialized = () => {
  const data = Uint8Array.from(fs.readFileSync('./flame.avi'));
  Module.FS.writeFile('flame.avi', data);

  const ffmpeg = Module.cwrap('proxy_main', 'number', ['number', 'number']);
  const args = ['ffmpeg', '-hide_banner'];
  const argsPtr = Module._malloc(args.length * Uint32Array.BYTES_PER_ELEMENT);
  args.forEach((s, idx) => {
    const buf = Module._malloc(s.length + 1);
    Module.writeAsciiToMemory(s, buf);
    Module.setValue(argsPtr + (Uint32Array.BYTES_PER_ELEMENT * idx), buf, 'i32');
  })
  ffmpeg(args.length, argsPtr);
};

Develop ffmpeg.wasm v0.1 with transcoding feature.

Now we are able to pass arguments to ffmpeg.wasm and save files to File System, let’s compose all of them together and get our ffmpeg.wasm v0.1 working.

The last one more detail we need to be aware of is that ffmpeg() above is actually running asynchronously, so to get the output file we need to use a setInterval() to parse the log file to know if the tranding completes.

Put everything together, now we have our first ffmpeg.wasm to transcode a avi file to mp4 file with any issues:

const fs = require('fs');
const Module = require('./dist/ffmpeg-core');

Module.onRuntimeInitialized = () => {
  const data = Uint8Array.from(fs.readFileSync('./flame.avi'));
  Module.FS.writeFile('flame.avi', data);

  const ffmpeg = Module.cwrap('proxy_main', 'number', ['number', 'number']);
  const args = ['ffmpeg', '-hide_banner', '-report', '-i', 'flame.avi', 'flame.mp4'];
  const argsPtr = Module._malloc(args.length * Uint32Array.BYTES_PER_ELEMENT);
  args.forEach((s, idx) => {
    const buf = Module._malloc(s.length + 1);
    Module.writeAsciiToMemory(s, buf);
    Module.setValue(argsPtr + (Uint32Array.BYTES_PER_ELEMENT * idx), buf, 'i32');
  });
  ffmpeg(args.length, argsPtr);

  /*
   * The execution of ffmpeg is not synchronized,
   * so we need to parse the log file to check if completed.
   */
  const timer = setInterval(() => {
    const logFileName = Module.FS.readdir('.').find(name => name.endsWith('.log'));
    if (typeof logFileName !== 'undefined') {
      const log = String.fromCharCode.apply(null, Module.FS.readFile(logFileName));
      if (log.includes("frames successfully decoded")) {
        clearInterval(timer);
        const output = Module.FS.readFile('flame.mp4');
        fs.writeFileSync('flame.mp4', output);
      }
    }
  }, 500);

};

You can visit the repository here to see how it works in details: https://github.com/ffmpegwasm/FFmpeg/tree/n4.3.1-p3

And feel free to download the build artifacts here: https://github.com/ffmpegwasm/FFmpeg/releases/tag/n4.3.1-p3

Please note currently it is a Node.js only version, but we will develop a browser version in next post.

Look forward to seeing you in Part.4. 😃

Build FFmpeg WebAssembly version (= ffmpeg.wasm): Part.2 Compile with Emscripten

Sun, 13 Oct 2019 00:00:01 GMT

From here, things are going to be more complicated and hard to understand, you may need to google background knowledge if you don’t know what happened (or you can leave responses to ask me).

Also, to make this tutorial more usable, I try to write down the details of how I solved each issue, hope it will help you in building the library of your choice.

In this part you will learn:

How to setup environment of Emscripten using Docker
Usage of emconfigure and emmake
How to fix issues when compiling FFmpeg with Emscripten

How to setup environment of Emscripten using Docker In Build FFmpeg WebAssembly version (= ffmpeg.wasm): Part.1 Preparation, we have built the original version of FFmpeg with GCC, and now we move on to use Emscripten instead.

The version of Emscripten we are going to use is 1.39.18 (trzeci/emscripten:1.39.18-upstream), you can install the Emscripten through the official tutorial (in this tutorial, we are setup-emsdk Github Actions in MacOS) or pull Emscripten image from docker hub.

$ docker pull trzeci/emscripten:1.39.18-upstream

It may take few minutes as the size of the image is around 1 GB.

Then we need to update build-with-docker.sh like below:

#!/bin/bash -x

EM_VERSION=1.39.18-upstream

docker pull trzeci/emscripten:$EM_VERSION
docker run \
  -v $PWD:/src \
  -v $PWD/cache-wasm:/emsdk_portable/.data/cache/wasm \
  trzeci/emscripten:$EM_VERSION \
  sh -c 'bash ./build.sh'

cache-wasm is not required, but it can help you to speed up in subsequent build.

The next we are going to do is to find the configuration to build FFmpeg with emscripten, it is a try-and-error process and requires document digging and patient.

Usage of `emconfigure` and `emmake` & How to fix issues when compiling FFmpeg with Emscripten

Let’s start our journey of finding the right configuration. In Part.1, it starts with ./configure --disable-x86asm, to do it with emscripten, you need to change it to emconfigure ./configure --disable-x86asm. (For details of emconfigure, please check HERE) And as we are doing cross compiling, we need to add cross-compiling flags to explicitly tell FFmpeg.

Let’s update build.sh like below:

#!/bin/bash -x

# verify Emscripten version
emcc -v

# configure FFMpeg with Emscripten
ARGS=(
  --target-os=none        # use none to prevent any os specific configurations
  --arch=x86_32           # use x86_32 to achieve minimal architectural optimization
  --enable-cross-compile  # enable cross compile
  --disable-x86asm        # disable x86 asm
)
emconfigure ./configure "${ARGS[@]}"

And magically there is no error or anything wrong, so do we just need to type emmake make -j and we get FFmpeg.wasm? Unfortunately, the answer is no. One of the most important task for emconfigure is to replace the compiler from gcc to emcc (or g++ to em++), but in the output of ./configure, we still get gcc as our compiler.

emscripten sdl2-config called with /emsdk_portable/emscripten/tag-1.38.45/system/bin/sdl2-config --cflags                            
emscripten sdl2-config called with /emsdk_portable/emscripten/tag-1.38.45/system/bin/sdl2-config --libs                               
install prefix            /usr/local                                                                                                   
source path               .                                                                                                                
C compiler                gcc             # Should be emcc                                                                                             
C library                 glibc                                                                                                           
ARCH                      x86 (generic)                                                                                                     
big-endian                no                                                                                                            
runtime cpu detection     yes                                                                                                         
standalone assembly       no                                                                                                            
x86 assembler             nasm

Every automation tool has its limitation and we need to do it manually in this case. Let’s check if there is any arguments to save us.

$ ./configure --help

Under Toolchain options, there are arguments to assign the compiler to use.

root@57ab95def750:/src# ./configure --help
Usage: configure [options]
Options: [defaults in brackets after descriptions]
Help options:
...
Toolchain options:                                                                                                             
...
  --nm=NM                  use nm tool NM [nm -g]
  --ar=AR                  use archive tool AR [ar]
  --as=AS                  use assembler AS []
  --ln_s=LN_S              use symbolic link tool LN_S [ln -s -f]
  --strip=STRIP            use strip tool STRIP [strip]
  --windres=WINDRES        use windows resource compiler WINDRES [windres]
  --x86asmexe=EXE          use nasm-compatible assembler EXE [nasm]
  --cc=CC                  use C compiler CC [gcc]
  --cxx=CXX                use C compiler CXX [g++]
  --objcc=OCC              use ObjC compiler OCC [gcc]
  --dep-cc=DEPCC           use dependency generator DEPCC [gcc]
  --nvcc=NVCC              use Nvidia CUDA compiler NVCC [nvcc]
  --ld=LD                  use linker LD []
…..

Let’s pass these arguments to compile with emscripten in build.sh :

#!/bin/bash -x

# verify Emscripten version
emcc -v

# configure FFMpeg with Emscripten
FLAGS=(
  --target-os=none        # use none to prevent any os specific configurations
  --arch=x86_32           # use x86_32 to achieve minimal architectural optimization
  --enable-cross-compile  # enable cross compile
  --disable-x86asm        # disable x86 asm
  --nm="llvm-nm -g"
  --ar=emar
  --as=llvm-as
  --ranlib=llvm-ranlib
  --cc=emcc
  --cxx=em++
  --objcc=emcc
  --dep-cc=emcc
)
emconfigure ./configure "${FLAGS[@]}"

For native build, please make sure llvm-ranlib, llvm-as and llvm-nm exist. If not, you can find them in $EMSDK_ROOT/upstream/bin .

With these arguments the ./configure will take more time to run, but you will get desired output in the end.

emscripten sdl2-config called with /emsdk_portable/emscripten/tag-1.39.18/system/bin/sdl2-config --cflags                            
emscripten sdl2-config called with /emsdk_portable/emscripten/tag-1.39.18/system/bin/sdl2-config --libs                                     
install prefix            /usr/local                                                                                                   
source path               .                                                                                                         
C compiler                emcc         # emcc as expected                                                                                    
C library                                                                                                                          
ARCH                      x86 (generic)                                                                                                     
big-endian                no                                                                                                              
runtime cpu detection     yes                                                                                                             
standalone assembly       no

Add emmake make -j4 (You can raise the parallelism to like -j8 or simply use -j to use all cores) in the end of build.sh :

#!/bin/bash -x

# verify Emscripten version
emcc -v

# configure FFMpeg with Emscripten
ARGS=(
  --target-os=none        # use none to prevent any os specific configurations
  --arch=x86_32           # use x86_32 to achieve minimal architectural optimization
  --enable-cross-compile  # enable cross compile
  --disable-x86asm        # disable x86 asm
  --nm="llvm-nm -g"
  --ar=emar
  --as=llvm-as
  --ranlib=llvm-ranlib
  --cc=emcc
  --cxx=em++
  --objcc=emcc
  --dep-cc=emcc
)
emconfigure ./configure "${ARGS[@]}"

# build ffmpeg.wasm
emmake make -j4

And it fails immediately after execution:

...
./libavutil/x86/timer.h:39:24: error: invalid output constraint '=a' in asm
                     : "=a" (a), "=d" (d));
                       ^

From the output message, we can identify that the error is related to asm. Open ./libavutil/x86/timer.h and we can confirm the issue is caused by the x86 inline assembly, which is not compatible with WebAssembly, so the solution is to disable it in build.sh:

#!/bin/bash -x

# verify Emscripten version
emcc -v

# configure FFMpeg with Emscripten
ARGS=(
  --target-os=none        # use none to prevent any os specific configurations
  --arch=x86_32           # use x86_32 to achieve minimal architectural optimization
  --enable-cross-compile  # enable cross compile
  --disable-x86asm        # disable x86 asm
  --disable-inline-asm    # disable inline asm
  --nm="llvm-nm -g"
  --ar=emar
  --as=llvm-as
  --ranlib=llvm-ranlib
  --cc=emcc
  --cxx=em++
  --objcc=emcc
  --dep-cc=emcc
)
emconfigure ./configure "${ARGS[@]}"

# build ffmpeg.wasm
emmake make -j4

It works and keeps compiling until we hit another error:

...
CC libavfilter/dnn/dnn_backend_native_layers.o
In file included from libavfilter/aeval.c:26:
In file included from ./libavutil/avassert.h:31:
In file included from ./libavutil/avutil.h:296:
In file included from ./libavutil/common.h:533:
In file included from ./libavutil/internal.h:176:
./libavutil/libm.h:54:32: error: static declaration of 'cbrt' follows non-static declaration
static av_always_inline double cbrt(double x)
                               ^
/emsdk_portable/upstream/emscripten/system/include/libc/math.h:151:13: note: previous declaration is here
double      cbrt(double);
            ^
In file included from libavfilter/aeval.c:26:

This time the root cause is not so obvious, so we need to dig deeper inside what went wrong during the ./configure. A very helpful file to check is ffbuild/config.log, it contains logs during ./configure and most of time you can find the root cause there.

By search cbrt inside config.log, we find error message below:

...
check_mathfunc cbrt 1                                                         
test_ld cc                                                                     
test_cc                                                                       
BEGIN /tmp/ffconf.syfN4Irw/test.c                                             
1 #include <math.h>                                                           
2 float foo(float f, float g) { return cbrt(f); }         
    3 int main(void){ return (int) foo; }                                     
END /tmp/ffconf.syfN4Irw/test.c                                               
emcc -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -std=c11 -fomit-frame-pointer -pthread -c -o /tmp/ffconf.syfN4Irw/test.o /tmp/ffconf.syfN4Irw/test.c         
emcc -Wl,-z,noexecstack -o /tmp/ffconf.syfN4Irw/test /tmp/ffconf.syfN4Irw/test.o
wasm-ld: error: 'atomics' feature is used by /tmp/ffconf.syfN4Irw/test.o, so --shared-memory must be used
...

This test tried to check if cbrt works in the environment, but it fails due to atomics feature errored. atomics is asked when you are using pthread, so let’s add pthread flags. (Check HERE for more details about pthread flags)

Update build.sh:

#!/bin/bash -x

# verify Emscripten version
emcc -v

# configure FFMpeg with Emscripten
CFLAGS="-s USE_PTHREADS"
LDFLAGS="$CFLAGS"
ARGS=(
  --target-os=none        # use none to prevent any os specific configurations
  --arch=x86_32           # use x86_32 to achieve minimal architectural optimization
  --enable-cross-compile  # enable cross compile
  --disable-x86asm        # disable x86 asm
  --disable-inline-asm    # disable inline asm
  --extra-cflags="$CFLAGS"
  --extra-cxxflags="$CFLAGS"
  --extra-ldflags="$LDFLAGS"
  --nm="llvm-nm -g"
  --ar=emar
  --as=llvm-as
  --ranlib=llvm-ranlib
  --cc=emcc
  --cxx=em++
  --objcc=emcc
  --dep-cc=emcc
)
emconfigure ./configure "${ARGS[@]}"

# build ffmpeg.wasm
emmake make -j4

It works and keeps compiling until we hit another error:

...
LD      ffplay_g                                                                                                                                  emcc: warning: ignoring unsupported linker flag: `-rpath-link=:libpostproc:libswresample:libswscale:libavfilter:libavdevice:libavformat:libavcodec:libavutil:libavresample` [-Wlinkflags]                                                                                                           
7 warnings generated.                                                      
wasm-ld: error: initial memory too small, 19491744 bytes needed
...                    
make: *** [Makefile:114: ffplay_g] Error 1
make: *** Waiting for unfinished jobs....
emcc: warning: ignoring unsupported linker flag: `-rpath-link=:libpostproc:libswresample:libswscale:libavfilter:libavdevice:libavformat:libavcodec
:libavutil:libavresample` [-Wlinkflags]
...

This time the issue is caused by too small in initial memory (by default only 16 MB in Emscripten, and here the minimal is 19+ MB), so we need to raise the initial memory to a higher value by passing -s INITIAL_MEMORY=33554432 (32 MB).

#!/bin/bash -x

# verify Emscripten version
emcc -v

# configure FFMpeg with Emscripten
CFLAGS="-s USE_PTHREADS"
LDFLAGS="$CFLAGS -s INITIAL_MEMORY=33554432" # 33554432 bytes = 32 MB
ARGS=(
  --target-os=none        # use none to prevent any os specific configurations
  --arch=x86_32           # use x86_32 to achieve minimal architectural optimization
  --enable-cross-compile  # enable cross compile
  --disable-x86asm        # disable x86 asm
  --disable-inline-asm    # disable inline asm
  --extra-cflags="$CFLAGS"
  --extra-cxxflags="$CFLAGS"
  --extra-ldflags="$LDFLAGS"
  --nm="llvm-nm -g"
  --ar=emar
  --as=llvm-as
  --ranlib=llvm-ranlib
  --cc=emcc
  --cxx=em++
  --objcc=emcc
  --dep-cc=emcc
)

emconfigure ./configure "${ARGS[@]}"

# build ffmpeg.wasm
emmake make -j4

Still there is an error after this fix:

LD      ffplay_g
emcc: warning: ignoring unsupported linker flag: `-rpath-link=:libpostproc:libswresample:libswscale:libavfilter:libavdevice:libavformat:libavcodec:libavutil:libavresample` [-Wlinkflags]
6 warnings generated.
LD      ffmpeg_g
emcc: warning: ignoring unsupported linker flag: `-rpath-link=:libpostproc:libswresample:libswscale:libavfilter:libavdevice:libavformat:libavcodec:libavutil:libavresample` [-Wlinkflags]
9 warnings generated.
LD      ffprobe_g
emcc: warning: ignoring unsupported linker flag: `-rpath-link=:libpostproc:libswresample:libswscale:libavfilter:libavdevice:libavformat:libavcodec:libavutil:libavresample` [-Wlinkflags]
STRIP   ffmpeg
strip:ffmpeg_g: file format not recognized
make: *** [Makefile:107: ffmpeg] Error 1
make: *** Waiting for unfinished jobs....

As we are unable to strip (make sense as it is not in a valid binary format), let’s simply disable strip with --disable-stripping and make again:

#!/bin/bash -x

# verify Emscripten version
emcc -v

# configure FFMpeg with Emscripten
CFLAGS="-s USE_PTHREADS"
LDFLAGS="$CFLAGS -s INITIAL_MEMORY=33554432" # 33554432 bytes = 32 MB
ARGS=(
  --target-os=none        # use none to prevent any os specific configurations
  --arch=x86_32           # use x86_32 to achieve minimal architectural optimization
  --enable-cross-compile  # enable cross compile
  --disable-x86asm        # disable x86 asm
  --disable-inline-asm    # disable inline asm
  --disable-stripping     # disable stripping
  --extra-cflags="$CFLAGS"
  --extra-cxxflags="$CFLAGS"
  --extra-ldflags="$LDFLAGS"
  --nm="llvm-nm -g"
  --ar=emar
  --as=llvm-as
  --ranlib=llvm-ranlib
  --cc=emcc
  --cxx=em++
  --objcc=emcc
  --dep-cc=emcc
)

emconfigure ./configure "${ARGS[@]}"

# build ffmpeg.wasm
emmake make -j4

Finally we managed to successfully do the emmake make -j part and you can see ffplay / ffplay_g, ffprobe / ffprobe_g and ffmpeg / ffmpeg_g generated in the root folder. It looks perfect, but there is a weird _g suffix makes the output files like this:

ffmpeg
ffmpeg_g
ffmpeg_g.wasm
ffmpeg_g.worker.js

Both ffmpeg and ffmpeg_g here are actualy js files, the ideal naming is like below:

ffmpeg / ffmpeg_g => ffmpeg.js
ffmpeg_g.wasm => ffmpeg.wasm
ffmpeg_g.worker.js => ffmpeg.worker.js

To fix this issue, we need to build it by our own. The command to build ffmpeg can be extracted by running emmake make -n:

...
printf "LD\t%s\n" ffmpeg_g; emcc -Llibavcodec -Llibavdevice -Llibavfilter -Llibavformat -Llibavresample -Llibavutil -Llibpostproc -Llibswscale -Llibswresample -Wl,--as-needed -Wl,-z,noexecstack -Wl,--warn-common -Wl,-rpath-link=libpostproc:libswresample:libswscale:libavfilter:libavdevice:libavformat:libavcodec:libavutil:libavresample -Qunused-arguments   -o ffmpeg_g fftools/ffmpeg_opt.o fftools/ffmpeg_filter.o fftools/ffmpeg_hw.o fftools/cmdutils.o fftools/ffmpeg.o  -lavdevice -lavfilter -lavformat -lavcodec -lswresample -lswscale -lavutil  -lm -pthread -lm -lm -pthread -lm -lm -lm -pthread -lm
printf "CP\t%s\n" ffmpeg; cp -p ffmpeg_g ffmpeg
…..

With a little clean up:

emcc \
  -I. -I./fftools \
  -Llibavcodec -Llibavdevice -Llibavfilter -Llibavformat -Llibavresample -Llibavutil -Llibpostproc -Llibswscale -Llibswresample \
  -Qunused-arguments \
  -o ffmpeg_g fftools/ffmpeg_opt.o fftools/ffmpeg_filter.o fftools/ffmpeg_hw.o fftools/cmdutils.o fftools/ffmpeg.o \
  -lavdevice -lavfilter -lavformat -lavcodec -lswresample -lswscale -lavutil -lm

As we are building our own version, let’s add --disable-programs and --disable-doc in ./configure step to speed up the build and also add some essential flags when build ffmpeg

#!/bin/bash -x

# verify Emscripten version
emcc -v

# configure FFMpeg with Emscripten
CFLAGS="-s USE_PTHREADS"
LDFLAGS="$CFLAGS -s INITIAL_MEMORY=33554432" # 33554432 bytes = 32 MB
CONFIG_ARGS=(
  --target-os=none        # use none to prevent any os specific configurations
  --arch=x86_32           # use x86_32 to achieve minimal architectural optimization
  --enable-cross-compile  # enable cross compile
  --disable-x86asm        # disable x86 asm
  --disable-inline-asm    # disable inline asm
  --disable-stripping     # disable stripping
  --disable-programs      # disable programs build (incl. ffplay, ffprobe & ffmpeg)
  --disable-doc           # disable doc
  --extra-cflags="$CFLAGS"
  --extra-cxxflags="$CFLAGS"
  --extra-ldflags="$LDFLAGS"
  --nm="llvm-nm -g"
  --ar=emar
  --as=llvm-as
  --ranlib=llvm-ranlib
  --cc=emcc
  --cxx=em++
  --objcc=emcc
  --dep-cc=emcc
)
emconfigure ./configure "${CONFIG_ARGS[@]}"

# build dependencies
emmake make -j4

# build ffmpeg.wasm
mkdir -p wasm/dist
ARGS=(
  -I. -I./fftools
  -Llibavcodec -Llibavdevice -Llibavfilter -Llibavformat -Llibavresample -Llibavutil -Llibpostproc -Llibswscale -Llibswresample
  -Qunused-arguments
  -o wasm/dist/ffmpeg.js fftools/ffmpeg_opt.c fftools/ffmpeg_filter.c fftools/ffmpeg_hw.c fftools/cmdutils.c fftools/ffmpeg.c
  -lavdevice -lavfilter -lavformat -lavcodec -lswresample -lswscale -lavutil -lm
  -s USE_SDL=2                    # use SDL2
  -s USE_PTHREADS=1               # enable pthreads support
  -s INITIAL_MEMORY=33554432      # 33554432 bytes = 32 MB
)
emcc "${ARGS[@]}"

Let’s create a basic.html to test if ffmpeg.wasm is working:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title></title>
  <script src="./dist/ffmpeg.js"></script>
</head>
<body>
</body>
</html>

Run a lightweight web server (ex. python3 -m http.server 3000) and visit the web page (ex. http://localhost:3000/basic.html) and open the Chrome DevTools.

It is kind of working as you can see output similar to original FFmpeg, it gives us a good starting point to polish our ffmpeg.wasm library.

You can visit the repository here to see how it works in details: https://github.com/ffmpegwasm/FFmpeg/tree/n4.3.1-p2

And feel free to download the build artifacts here: https://github.com/ffmpegwasm/FFmpeg/releases/tag/n4.3.1-p2

For how to polish and create a “real” ffmpeg.wasm library, please see next post in this series of stories. 😃

Build FFmpeg WebAssembly version (= ffmpeg.wasm): Part.1 Preparation

Sun, 13 Oct 2019 00:00:00 GMT

In this part you will learn:

The background of this series
How to build native FFmpeg with Docker (and without docker in MacOS)

The background of this series

This series of stories aim to serve following purposes:

A guide for people who want to learn how to use Emscripten to compile a C/C++ library to JavaScript (Hopefully the most useful and detailed so far)
Personal notes

Why FFmpeg?

FFmpeg is a free and open-source project consisting of a vast software suite of libraries and programs for handling video, audio, and other multimedia files and streams. (from Wikipedia)

It is an useful library and there is no JavaScript library with exactly the same capabilities. If you google “ffmpeg.js”, you will find few existing libraries exactly the same as what we are going to build:

ffmpeg.js: https://github.com/Kagami/ffmpeg.js
videoconverter.js: https://github.com/bgrins/videoconverter.js

These libraries are great and ready-to-use in most cases, but they are suffering from following issues:

The versions of both FFmpeg and Emscripten are out-dated.
Not under active maintenance for years. (Kagami/ffmpeg.js continues its development in April 2020)

I considered to take over maybe one of the repositories, but as there are too many changes these years, I decided to do it from scratch and wrote this series of tutorial at the same time to help people to learn how to use Emscripten in a real world C/C++ library.

How to build native FFmpeg with Docker

To start with, we need to clone FFmpeg source code from its repository, as the master branch is under development, it is better we choose a specific release to compile.

At the moment I wrote the story, the latest stable version of FFmpeg is n4.3.1, so we will use this version through out the stories.

# Use depth=1 to download only the latest version
git clone --depth 1 --branch n4.3.1 https://github.com/FFmpeg/FFmpeg

After completing cloning the repository, it is time to build with GCC to make sure it works.

Actually you can skip this part if you are in a rush, but in my own experience it is better to get used to the build system of library first.

The instruction of building and installing FFmpeg can be found inside INSTALL.md in the root of the repository:

Installing FFmpeg:
1. Type `./configure` to create the configuration. A list of configure options is printed by running `configure -help`.
`configure` can be launched from a directory different from the FFmpeg sources to build the objects out of tree. To do this, use an absolute path when launching `configure`, e.g. `/ffmpegdir/ffmpeg/configure`.
2. Then type `make` to build FFmpeg. GNU Make 3.81 or later is required.
3. Type `make install` to install all binaries and libraries you built.
NOTICE
 - - -
- Non system dependencies (e.g. libx264, libvpx) are disabled by default.

As we don’t need to actually install the FFmpeg, only step 1 and 2 are required.

There are two ways to build, one is native way which requires you to install packages (ex. emsdk, Node.js). Most of time it works, but sometimes you may face bugs that is hard to resolve due to the variation of package versions and operating system. Another way is to use Docker, which provides a stable and static build environments. It is highly recommended to use Docker as it saves your time to install (and remove) the packages.

I will not cover how to install the packages here, but as I split the scripts into build.sh and build-with-docker.sh , you can install all the packages yourself and run build.sh.

To make sure this tutorial can achieve maximum coverage of environments, I use Github Actions to test if it works on Linux and MacOS. For Linux users, I will use Docker way / build-with-docker.sh to build. For MacOS users, as Docker is not supported in Github Actions, I will use native way / build.sh to build.

Now, let’s create a file called build.sh with following content.

#!/bin/bash -x
# gcc 8 is used in this tutorial, other versions may fail
./configure --disable-x86asm
make -j

To build in native way, you can simply run command:

$ bash build.sh

To build with Docker, create a file called build-with-docker.sh with following content:

#!/bin/bash -x
docker pull gcc:8
docker run \
  -v $PWD:/usr/src \
  gcc:8 \
  sh -c 'cd /usr/src && bash build.sh'

And run command:

$ bash build-with-docker.sh

--disable-x86asm is required to disable x86 assemble features as we are not going to use it.

Depend on the speed of you Internet and the hardware specification of your computer, it may take 10~30 minutes to complete the compilation. And it is normal that you see tons of warnings during compilation as gcc 9 introduces more constrains.

It should take some time to compile the native FFmpeg. If everything is OK, you should be able to run ffmpeg with following command:

$ ./ffmpeg

You should see something like:

ffmpeg version n4.3.1 Copyright © 2000–2019 the FFmpeg developers
built with gcc 8 (GCC)
configuration: — disable-x86asm
libavutil 56. 22.100 / 56. 22.100
libavcodec 58. 35.100 / 58. 35.100
libavformat 58. 20.100 / 58. 20.100
libavdevice 58. 5.100 / 58. 5.100
libavfilter 7. 40.101 / 7. 40.101
libswscale 5. 3.100 / 5. 3.100
libswresample 3. 3.100 / 3. 3.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]… {[outfile options] outfile}…
Use -h to get full help or, even better, run ‘man ffmpeg’

You can visit the repository here to see how it works in details: https://github.com/ffmpegwasm/FFmpeg/tree/n4.3.1-p1

And feel free to download the build artifacts here: https://github.com/ffmpegwasm/FFmpeg/releases/tag/n4.3.1-p1

Now we have completed the preparation, let’s move on to next post to start compiling FFmpeg with Emscripten. 😃

Why I refactor tesseract.js v2?

Thu, 03 Oct 2019 12:00:00 GMT

After almost 2 years of maintaining tesseract.js, I have to say tesseract.js hit its limitation as some of the issues from users are hard to fix, especially those related to performance.

The existing tesseract.js (before v2.0.0-alpha) offers declarative and coarse-grained APIs that minimize the effort to understand the complex design of the library, it hides details like how tesseract.js loads *.traineddata from remote, how the *.traineddata is saved to local file system or IndexDB as cache, how the WebAssembly version of tesseract-ocr is initialed with parameters, set the image and generated the result we desired. It is great and everyone is happy to get a turn-key OCR library to integrate into your project.

// Typical usage of tesseract.js
const { TesseractWorker } = Tesseract
const worker = new TesseractWorker()
const image = "https://tesseract.projectnaptha.com/img/eng_bw.png"
worker.recognize(image).then(({ text }) => {
  console.log(text)
})

But as you use tesseract.js more and more in your project, you figure out the performance is not so good. It feels like every time you call recognize(), tesseract.js always starts from scratch and loads the *.traineddata, initializes its core and set the parameters again and again. It really takes lots of time and the performance becomes unsatisfied.

Another issue is that every time you want to do something that tesseract-ocr can do with tesseract.js, it fails immediately and you totally don’t know how to solve it, so you create an issue and cross your finger hope someone can save you, but most of the time, the miracle doesn’t happen. So you dive into the source code and find out it is really complicated and twisted inside, in the end you may give up or email the active maintainers of the tesseract.js and wait for the reply.

So I decided to refactor tesseract.js and after few weeks it is almost done (released as v2.0.0-beta.1) and I would like to share with you the major changes:

tesseract.js-core (WebAssembly version of tesseract-ocr) is now upgraded using tesseract-ocr v4.1 and emscripten 1.38.45 fastcomp (upstream is not stable enough to compile tesseract-ocr)
Refactor APIs to be more readable, imperative and fine-grained (we will cover this part later)
Support scheduling to execute multiple OCR tasks at the same time
Support typescript with index.d.ts
Rewrite documents (especially docs/api.md) And below is how it looks right now:

// tesseract.js@^2.0.0-beta.1
const { createWorker } = require("tesseract.js")
const image = "https://tesseract.projectnaptha.com/img/eng_bw.png"
const worker = createWorker()
;(async () => {
  await worker.load()
  await worker.loadLanguage("eng")
  await worker.initialize("eng")
  const {
    data: { text },
  } = await worker.recognize(image)
  console.log(text)
})()

It may feel kind of weird to see that you need to do more things in the new version, but with more lines of code you have more in-depth control of tesseract.js and boost the performance of your project. A possible use case is like this:

const { createWorker } = require("tesseract.js")
const image = "https://tesseract.projectnaptha.com/img/eng_bw.png"
const worker = createWorker()
let isReady = false
// Called as early as possible
;(async () => {
  await worker.load()
  await worker.loadLanguage("eng")
  await worker.initialize("eng")
  isReady = true
})()
// Do other stuffs
;(async img => {
  if (isReady) {
    const {
      data: { text },
    } = await worker.recognize(img)
    console.log(text)
  }
})(image)

The benefit is that you can prepare the worker as early as possible to improve the overall user experience.

Another important feature is that we add scheduling capabilities for you to use multiple workers at the same time. It is useful when you have enough resource (CPU cores) for your project to decrease the time for massive recognition jobs.

Below is a sample code for using scheduler with 4 workers to handle 10 recognition jobs:

const { createWorker, createScheduler } = require('tesseract.js');
const image = 'https://tesseract.projectnaptha.com/img/eng_bw.png';

const scheduler = createScheduler();

(async () => {
  for (let i = 0; i < 4; i++) {
    const w = createWorker();
    await w.load();
    await w.loadLanguage('eng');
    await w.initialize('eng');
    scheduler.addWorker(w);
  }
  const rets = await Promise.all(Array(10).fill(0).map(() => (
    scheduler.addJob('recognize', image);
  )));
  console.log(rets.map(({ data: { text } }) => text));
  await scheduler.terminate();
})();

The role of scheduler is to manage the job queue and worker pool and distribute jobs any idle workers.

A performance comparison between alpha version and beta version can be found below (using Node version in a 8 core machine).

Source code: https://github.com/jeromewu/tesseract.js-performance

With 4 workers we can have a almost 3x speed up (from 60s to 20s), it is possible to speed up with even more workers!

If you are interested in using tesseract and giving feedback, please visit tesseract.js github and try it with following command:

$ npm install tesseract.js@next

Hope you enjoy tesseract.js and thanks for reading this post. :)

Gatsby Starter Blog RSS Feed

Improving performance using WebAssembly SIMD Intrinsics

Methods to Improve Performance

More optimization flags

Use multi-thread

Use SIMD intrinsics

Conclusion

Generics in Golang 1.18, no need to write max / min anymore (maybe)

Background

Generics to the Rescue

FFmpeg.wasm, a pure WebAssembly / JavaScript port of FFmpeg

Install ffmpeg.wasm

Use ffmpeg.wasm

Behind the scene

Inside ffmpeg.load()

File System (FS)

SharedArrayBuffer

Web Workers

Leverage CPU capabilities

How to fix (80%) HiDPI issues in Gulliam OS 3.1

Update xfce configration

GTK3 Scale

xfce4-panel theming

Work in Progress

How to speed up Node.js modules installation in CI/CD pipeline as of 2020

TL;DR

Existing Works

Environment

Packages

Network

Application

Commands

Parameters

Experiment

Observations

I wrote a Chrome Extension (Gitpod Window). Here is what I learned

Get Started with Chrome Extension

When user click the extension icon on a GitHub or GitLab page, opens Gitpod workspace in a popup Chrome window

Disable keyboard shortcuts

3 Things you should do to quickly fix your problem in a open source project

Check Documents

Search Issues

Open New Issues

One more thing

WebAssembly & Transferable, a Perfect Match

How to use Gitpod in your create-react-app based project

Why Gitpod?

Basic Configuration

Advanced Configuration

One more thing

Speedup FFmpeg without compiling from source code

Baseline: Scale a video without any options

With -preset to speedup with larger file size

With VAAPI to speedup with integrated/ Intel GPU card

Build FFmpeg WebAssembly version (= ffmpeg.wasm): Part.4 ffmpeg.wasm v0.2 — Add Libx264

Add Libx264 to ffmpeg-core.js

In browser ffmpeg.wasm demo

Build FFmpeg WebAssembly version (= ffmpeg.wasm): Part.3 ffmpeg.wasm v0.1 — Transcoding avi to mp4

Build a library version of FFmpeg with optimized arguments.

Interact with ffmpeg.wasm

Manage Emscripten File System.

Develop ffmpeg.wasm v0.1 with transcoding feature.

Build FFmpeg WebAssembly version (= ffmpeg.wasm): Part.2 Compile with Emscripten

Usage of emconfigure and emmake & How to fix issues when compiling FFmpeg with Emscripten

Build FFmpeg WebAssembly version (= ffmpeg.wasm): Part.1 Preparation

The background of this series

Why FFmpeg?

How to build native FFmpeg with Docker

Why I refactor tesseract.js v2?

Usage of `emconfigure` and `emmake` & How to fix issues when compiling FFmpeg with Emscripten