Dynamic programming languages like Python and TypeScript allows developers to optionally define type annotations and benefit from the advantages of static typing such as better code completion, early bug detection.However, retrofitting types is a cumbersome and error-prone process. To address this, we propose Type4Py , an ML-based type auto-completion for Python. It assists developers to gradually add type annotations to their codebases. In the following, I describe Type4Py’s pipeline, model, deployments, and the development of its VSCode extension and more.

Development and Release of Type4Py: Machine Learning-based Type Auto-completion for Python

Before going into the detail of the Type4Py’s model and its implementation, it would be helpful to see the overview of Type4Py. In general, there is a VSCode extension at the client-side (developers) and the Type4Py model and its pipeline are deployed on our servers. Simply, the extension sends a Python file to the server and a JSON response containing type predictions for the given file is returned. In the following subsection, I describe each component.

Overview

For an ML model to learn and generalize, there is a need for a large and high-quality dataset. Therefore, we created the <a dir="ltr" href="https://zenodo.org/record/4719447">ManyTypes4Py </a>dataset which contains 5.2K Python projects and 4.2M type annotations. To avoid data leakage from the training set to the test set, we used our <a dir="ltr" href="https://github.com/saltudelft/CD4Py">CD4Py </a>tool to perform code de-duplication in the dataset. To know more about how we created the dataset, check out <a dir="ltr" href="https://arxiv.org/abs/2104.04706">its paper </a>.

Dataset

As with most ML tasks, we need to find a set of relevant features in order to predict type annotations. Here, we consider features as type hints. Specifically, we extract three kinds of type hints, namely, identifiers, code context, and visible type hints (VTHs).To learn from the extracted sequences, first, we apply common NL pre-processing techniques tokenization (by <a href="https://en.wikipedia.org/wiki/Snake_case" rel="noopener noreferrer" target="_blank">snake case </a>), stop word removal and lemmatization. Then, we employ the famous Word2Vec model to generate word embeddings of the extracted sequences in order to train the Type4Py model, which is described next.

Feature Extraction

The basic idea is that two RNN captures different aspects of input sequences from both identifiers and code context. Next, the output of two RNNs is concatenated into a single vector, which is passed through a fully-connected linear layer. The final linear layer maps the learned type annotation into a high-dimensional feature space, called Type Clusters. In order to create Type Clusters, we need to formulate the type prediction task as a similarity learning problem, rather than a classification problem.

Model Architecture & Training

To extract type hints, we first extract Abstract Syntax Trees (ASTs) and perform light-weight static analysis using our <a href="https://github.com/saltudelft/libsa4py" rel="noopener noreferrer" target="_blank">LibSA4Py </a>package. NLP tasks are applied using NLTK. To train the Word2Vec model, we use the <a href="https://radimrehurek.com/gensim/" rel="noopener noreferrer" target="_blank">gensim </a>package.For the Type4Py model, we use bidirectional LSTMs in PyTorch to implement the two RNNs. To avoid overfitting, we apply the Dropout regularization to the input sequences. To minimize the value of the Triplet loss function, we employ the Adam optimizer. Also, to speed up the training process, we use the data parallelism feature of PyTorch. For fast KNN search, we use Annoy

Implementation

To deploy the Type4Py model for the production environment, we convert the pre-trained PyTorch model to an <a href="https://www.onnxruntime.ai/" rel="noopener noreferrer" target="_blank">ONNX </a>model which allows us to query the model on both GPUs and CPUs with very fast inference speed and lower VRAM consumption. Thanks to Annoy, Type Clusters are memory-mapped into RAM from disk, which consumes less memory.To handle concurrent type prediction requests from users, we employ <a href="https://gunicorn.org/" rel="noopener noreferrer" target="_blank">Gunicorn </a>‘s HTTP server with Nginx as a proxy. This allows us to have quite a number of asynchronous workers that have an instance of Type4Py’s ONNX model plus Type Clusters each.

Deployment

The Type4Py’s VSCode extension is small and simple.Type slots are functions parameters, return types, and variables, which are located based on the line and column numbers. Currently, type prediction can be triggered via Command Pallete or by enabling the AutoInfer setting, which predicts types whenever a Python source file is opened or selected. Finally, the extension gathers telemetry data from users based on their consent. The Type4Py’s VSCode extension can be installed from the VS Marketplace <a href="https://marketplace.visualstudio.com/items?itemName=saltud.type4py" rel="noopener noreferrer" target="_blank">here </a>.

VSCode Extension

We have two different environments for development and production as it is common in software development. In the development env., we test, debug, and profile Type4Py’s server-side components before releasing new features/fixes into the production code. Also, whenever we train a new Type4Py neural model, we test it against its evaluation metrics (see <a href="https://arxiv.org/abs/2101.04470" rel="noopener noreferrer" target="_blank">its paper </a>) and run integration tests to ensure that it produces expected predctions for given Python source files. Finally, the VSCode extension uses the development env. when testing new featues/fixes before releasing a new extension version.

Releasing

So far, I have described the current state of Type4Py. For future work, here is our roadmap: <ul><li>Enabling the type-checking process for the Type4Py’s predictions using <a dir="ltr" href="https://github.com/python/mypy">mypy </a>, preferably at the client-side. </li><li>Releasing a local version of the Type4Py model and its pipeline that can be queried on users’ machines. </li><li>Fine-tuning the (pre-trained) Type4Py model on users’ projects to learn project-specific types. </li><li>Releasing a plugin for JetBrains PyCharm and a GitHub action or bot for adding type annotations to Python projects </li> </ul>

Roadmap

HowTo

Over the past decade, machine learning (ML) has been applied successfully to a variety of tasks such as computer vision and natural language processing. Motivated by this, in recent years, researchers have employed ML techniques to solve code-related problems, including but not limited to, code completion, code generation, program repair, and type inference.

Dynamic programming languages like Python and TypeScript allows developers to optionally define type annotations and benefit from the advantages of static typing such as better code completion, early bug detection, and etc. However, retrofitting types is a cumbersome and error-prone process. To address this, we propose Type4Py , an ML-based type auto-completion for Python. It assists developers to gradually add type annotations to their codebases. In the following, I describe Type4Py’s pipeline, model, deployments, and the development of its VSCode extension and more.

As with most ML tasks, we need to find a set of relevant features in order to predict type annotations. Here, we consider features as type hints. Specifically, we extract three kinds of type hints, namely, identifiers, code context, and visible type hints (VTHs). To intuitively explain the type hints, consider below code snippet:

We use the function’s name, its parameters’ name, and variables’ name as type hints to predict types. Considering code context, we consider the usage of parameters and variables in statements that they are used. VTHs are a deep recursive analysis of import statements in a file and its transitive dependencies in our dataset. Later on, we build a VTH vocabulary, which is used to give a hint to the model about the expected type.

Given the above code snippet, we create the following sequences for identifiers and code context to predict types of variables, parameters, and the function’s return:

To learn from the extracted sequences, first, we apply common NL pre-processing techniques tokenization (by <a dir="ltr" href="https://en.wikipedia.org/wiki/Snake_case">snake case </a>), stop word removal and lemmatization. Then, we employ the famous Word2Vec model to generate word embeddings of the extracted sequences in order to train the Type4Py model, which is described next.

Before dive deep into the model’s details, an overview of the Type4Py model is shown below:

The Type4Py model is a hierarchical neural network, which consists of two RNNs with LSTM units, one for identifiers and another for code context. The basic idea is that two RNN captures different aspects of input sequences from both identifiers and code context. Next, the output of two RNNs is concatenated into a single vector, which is passed through a fully-connected linear layer. The final linear layer maps the learned type annotation into a high-dimensional feature space, called Type Clusters. In order to create Type Clusters, we need to formulate the type prediction task as a similarity learning problem, rather than a classification problem. Intuitively, the model learns to map similar types (say float) into its own type cluster and it should be as far as possible from the type cluster of other dissimilar types.

Considering the described similarity learning problem, the Type4Py model is trained using the <a dir="ltr" href="https://en.wikipedia.org/wiki/Triplet_loss">Triplet loss </a>function, which is recently used in computer vision for face recognition. Simply, the model receives three training examples, a to-be-learned type annotation (called anchor), a training example same as the anchor, a training example with a different type annotation from the first and second examples. For example, Let’s assume that the first training sample in the training set is the type annotation float plus its vector representations. Then, we randomly find a training sample with the float type annotation in the training and a training sample with a different type annotation. In this example, the triplet the model learns from would be (float, float, str).

After the model is trained and type clusters are obtained, to infer a type for a given query, we perform k -nearest neighbor (KNN) search in the type clusters to suggest a list of similar types for the given query. The value of k is 10 in both research and production environments.

Our server-side components are all written in Python. To extract type hints, we first extract Abstract Syntax Trees (ASTs) and perform light-weight static analysis using our <a dir="ltr" href="https://github.com/saltudelft/libsa4py">LibSA4Py </a>package. NLP tasks are applied using NLTK. To train the Word2Vec model, we use the <a dir="ltr" href="https://radimrehurek.com/gensim/">gensim </a>package.

For the Type4Py model, we use bidirectional LSTMs in <a dir="ltr" href="https://pytorch.org/">PyTorch </a>to implement the two RNNs. To avoid overfitting, we apply the <a dir="ltr" href="https://en.wikipedia.org/wiki/Dilution_(neural_networks)">Dropout regularization </a>to the input sequences. To minimize the value of the Triplet loss function, we employ the Adam optimizer. Also, to speed up the training process, we use the data parallelism feature of PyTorch, which distributes training batches among GPUs. For fast KNN search, we use <a dir="ltr" href="https://github.com/spotify/annoy">Annoy </a>.

To process type prediction requests from users, we implemented a tiny web application in Flask. It features type prediction using the Type4Py model, storing telemetry data from the VSCode extension (based on the user’s consent), and API rate limit. Lastly, we employ <a dir="ltr" href="https://mypyc.readthedocs.io/en/latest/">mypy </a>to type-check the model’s predictions. However, as of this writing, the type-checking component is disabled for performance reasons. The Type4Py’s model and its server code are publicly available on GitHub <a dir="ltr" href="https://github.com/saltudelft/type4py">here </a>.

To deploy the Type4Py model for the production environment, we convert the pre-trained PyTorch model to an <a dir="ltr" href="https://www.onnxruntime.ai/">ONNX </a>model which allows us to query the model on both GPUs and CPUs with very fast inference speed and lower VRAM consumption. Thanks to Annoy, Type Clusters are memory-mapped into RAM from disk, which consumes less memory.

To handle concurrent type prediction requests from users, we employ <a dir="ltr" href="https://gunicorn.org/">Gunicorn </a>‘s HTTP server with Nginx as a proxy. This allows us to have quite a number of asynchronous workers that have an instance of Type4Py’s ONNX model plus Type Clusters each. We use <a dir="ltr" href="http://supervisord.org/">supervisor </a>to manage and monitor the deployment.

We deployed the whole Type4Py’s server-side application on a Linux machine with Ubuntu distribution. The server has an Intel Xeon CPU with 28 threads, 128 GB of RAM, and two Nvidia GTX 1080 TIs. Lastly, we test our server’s production code with integration tests using GitHub Workflow.

The Type4Py’s VSCode extension is small and simple. Here is an overview of the extension workflow in VSCode:

In general, to get type predictions, the extension sends an opened Python source file to the server and receives a JSON response. To implement the extension, we use the <a dir="ltr" href="https://code.visualstudio.com/api/references/vscode-api">VSCode API </a>in TypeScript. More specifically, we employ IntelliSense’s completion functionality and some pattern matching to locate type slots in the code for suggesting likely type annotations. Type slots are functions parameters, return types, and variables, which are located based on the line and column numbers. Currently, type prediction can be triggered via Command Pallete or by enabling the AutoInfer setting, which predicts types whenever a Python source file is opened or selected. Finally, the extension gathers telemetry data from users based on their consent . For instance, telemetry data contains which type predictions are accepted or rejected at a file’s type slots for research purposes and improving the Type4Py model. The Type4Py’s VSCode extension can be installed from the VS Marketplace <a dir="ltr" href="https://marketplace.visualstudio.com/items?itemName=saltud.type4py">here </a>.

We have two different environments for development and production as it is common in software development. In the development env., we test, debug, and profile Type4Py’s server-side components before releasing new features/fixes into the production code. Therefore, users will not be affected by new changes and features in the production env. Also, whenever we train a new Type4Py neural model, we test it against its evaluation metrics (see <a dir="ltr" href="https://arxiv.org/abs/2101.04470">its paper </a>) and run integration tests to ensure that it produces expected predctions for given Python source files. Finally, the VSCode extension uses the development env. when testing new featues/fixes before releasing a new extension version.

So far, I have described the current state of Type4Py. For future work, here is our roadmap:

<ul><li>Enabling the type-checking process for the Type4Py’s predictions using <a dir="ltr" href="https://github.com/python/mypy">mypy </a>, preferably at the client-side. </li><li>Releasing a local version of the Type4Py model and its pipeline that can be queried on users’ machines. </li><li>Fine-tuning the (pre-trained) Type4Py model on users’ projects to learn project-specific types. </li><li>Releasing a plugin for JetBrains PyCharm and a GitHub action or bot for adding type annotations to Python projects </li> </ul>

The Type4Py model, its pipeline, and VSCode extension are all designed and developed at the <a dir="ltr" href="https://github.com/saltudelft">Software Analytics Lab </a>of <a dir="ltr" href="https://se.ewi.tudelft.nl/">SERG </a>, Delft University of Technology. Our team consists of 4 researchers and developers, <a dir="ltr" href="https://mirblog.net/">Amir M. Mir </a>, <a dir="ltr" href="https://elatoskinas.github.io/">Evaldas Latoškinas </a>, <a dir="ltr" href="https://proks.ch/">Sebastian Proksch </a>, <a dir="ltr" href="https://www.gousios.gr/">Georgios Gousios </a>. The Type4Py project has started in spring 2020 and it’s currently under active development and research.

Here are a few resources for further reading and research:

Thanks for reading this blog post. I hope that it was useful and informative for readers. We would be glad to answer your questions and see your comments. Also, please share this blog post with your colleagues and friends if you think they might be interested.

Machine Learning-based Type Auto-completion for Python – The Blog of Amir Mir

Berlin-based <a dir="ltr" href="https://www.mobiuslabs.com/">Mobius Labs </a>has closed a €5.2 million (~$6.1M) funding round off the back of increased demand for its computer vision training platform. The Series A investment is led by Ventech VC, along with Atlantic Labs, APEX Ventures, Space Capital, Lunar Ventures plus some additional angel investors. The startup offers an SDK that lets the user create custom computer vision models fed with a little of their own training data — as an alternative to off-the-shelf tools which may not have the required specificity for a particular use case.

Commenting on the funding in a statement, Stephan Wirries, partner at Ventech VC, added: “Appu and the team at Mobius Labs have developed an unparalleled offering in the computer vision space. Superhuman Vision is impressively innovative with its high degree of accuracy despite very limited required training to recognise new objects at excellent computational efficiency. We believe industries will be transformed through AI, and Mobius Labs is the European deep tech innovator teaching machines to see.”

For instance, if we want to track how river fronts are expanding, we have to look at data collected by satellites, sort and tag them in order to analyse them. Currently this is being done manually. The technology we are creating comes in a lightweight SDK and can be deployed directly into these satellites so that the raw data can be detected and then analysed by machine learning algorithms. We are currently working with satellite companies in this sector.”

On the competitive front, Shaji names <a href="https://techcrunch.com/2016/10/25/clarifai-raises-30m-to-give-developers-visual-search-capabilities/">Clarifai </a>and <a href="https://techcrunch.com/2016/02/18/google-opens-its-cloud-vision-api-to-all-developers/">Google Cloud Vision </a>as the main rivals it has in its sights.“We realise these are the big players but at the same time believe that we have something unique to offer, which these players cannot: Unlike their solutions, our platform users can be outside the field of computer vision. By democratising the training of machine learning models beyond simply the technical crowd, we are making computer vision accessible and understandable by anyone, regardless of their job titles,” he argues.

“Another core value that differentiates us is the way we treat client data. Our solutions are delivered in the form of a software development kit (SDK), which runs on-premise, completely locally on clients’ systems. No data is ever sent back to us. Our role is to empower people to build applications, and make them their own.”

  Berlin-based Mobius Labs has closed a €5.2 million (~$6.1M) funding round off the back of increased demand for its computer vision training platform. The Series A investment is led by Ventech VC, along with Atlantic Labs, APEX Ventures, Space Capital, Lunar Ventures plus some additional angel investors. The startup offers an SDK that lets […]

Inspirational

Image Credits: Yuichiro Chino / Getty Images

Berlin-based <a dir="ltr" href="https://www.mobiuslabs.com/">Mobius Labs </a>has closed a €5.2 million (~$6.1M) funding round off the back of increased demand for its computer vision training platform. The Series A investment is led by Ventech VC, along with Atlantic Labs, APEX Ventures, Space Capital, Lunar Ventures plus some additional angel investors.

The startup offers an SDK that lets the user create custom computer vision models fed with a little of their own training data — as an alternative to off-the-shelf tools which may not have the required specificity for a particular use case.

It also flags a “no code” focus, saying its tech has been designed with a non-technical user in mind.

As it’s an SDK, Mobius Labs’ platform can also be deployed on-premise and/or on-device — rather than the customer needing to connect to a cloud service to tap into the AI tool’s utility.

“Our custom training user interface is very simple to work with, and requires no prior technical knowledge on any level,” claims Appu Shaji, CEO and chief scientist.

“Over the years, a trend we have observed is that often the people who get the maximum value from AI are non-technical persons like a content manager in a press and creative agency, or an application manager in the space sector. Our no-code AI allows anyone to build their own applications, thus enabling these users to get close to their vision without having to wait for AI experts or developer teams to help them.”

Mobius Labs — which was founded back in 2018 — now has 30 customers using its tools for a range of use cases.

Uses include categorisation, recommendation, prediction, reducing operational expense, and/or “generally connecting users and audiences to visual content that is most relevant to their needs.” (Press and broadcasting and the stock photography sector have unsurprisingly been big focuses to date.)

But it reckons there’s wider utility for its tech and is gearing up for growth.

It caters to businesses of various sizes, from startups to SMEs, but says it mainly targets global enterprises with major content challenges — hence its historical focus on the media sector and video use cases.

Now, though, it’s also targeting geospatial and earth observation applications as it seeks to expand its customer base.

The 30-strong startup has more than doubled in size over the last 18 months. With the new funding it’s planning to double its headcount again over the next 12 months as it looks to expand its geographical footprint — focusing on Europe and the U.S.

Year-on-year growth has also been 2x but it believes it can dial that up by tapping into other sectors.

“We are working with industries that are rich in visual data,” says Shaji. “The geospatial sector is something that we are focussing on currently as we have a strong belief that vast amounts of visual data is being produced by them. However, these huge archives of raw pixel data are useless on their own.

On the competitive front, Shaji names <a dir="ltr" href="https://techcrunch.com/2016/10/25/clarifai-raises-30m-to-give-developers-visual-search-capabilities/">Clarifai </a>and <a dir="ltr" href="https://techcrunch.com/2016/02/18/google-opens-its-cloud-vision-api-to-all-developers/">Google Cloud Vision </a>as the main rivals it has in its sights.

“We realise these are the big players but at the same time believe that we have something unique to offer, which these players cannot: Unlike their solutions, our platform users can be outside the field of computer vision. By democratising the training of machine learning models beyond simply the technical crowd, we are making computer vision accessible and understandable by anyone, regardless of their job titles,” he argues.

“Another core value that differentiates us is the way we treat client data. Our solutions are delivered in the form of a software development kit (SDK), which runs on-premise, completely locally on clients’ systems. No data is ever sent back to us. Our role is to empower people to build applications, and make them their own.”

Computer vision startups have been a hot acquisition target in recent years and some earlier startups offering “computer vision as a service” got acquired by IT services firms to beef up their existing offerings, while tech giants like Amazon and (the aforementioned) Google offer their own computer vision services too.

But Shaji suggests the tech is now at a different stage of development — and primed for “mass adoption.”

“We’re talking about providing solutions that empower clients to build their own applications,” he says, summing up the competitive play. “And [do that] with complete data privacy, where our solutions run on-premise, and we don’t see our clients data. Coupled with that is the ease of use that our technology offers: It is a lightweight solution that can be deployed on many ‘edge’ devices like smartphones, laptops, and even on satellites.”

Mobius Labs nabs $6M to help more sectors tap into computer vision – TechCrunch

<ul><li>The alonzo upgrade will mean the cardano network will be able to support applications like smart contracts. </li><li>Cardano will gain a bigger foothold in the world of decentralized finance, competing with ethereum. </li><li>Founder Charles Hoskinson says he will wear a "Ghostbusters" costume to mark the launch. </li> </ul>

  cardano SOPA Images   The alonzo upgrade will mean the cardano network will be able to support applications like smart contracts. Cardano will g...

Cardano's big "alonzo" upgrade rolls out on Sunday and will usher in the network's ability to run a range of crypto applications, from smart contracts to non-fungible tokens and aims to bring what founder Charles Hoskinson calls "programmability" to the blockchain.

Sunday marks the final, and most important stage of the alonzo series of upgrades that will help the network compete with the likes of ethereum - currently the biggest blockchain that can run applications other than a native cryptocurrency token. Smart contracts will enable the network to run decentralized finance applications, for example.

"One of the magical superpowers of cardano is that we built it with upgradability in mind," Hoskinson told Insider in an interview.

"It means that cardano now supports programmability. That's like when JavaScript came to the web browser. So, you go from static boring web pages, to Facebook and Google and YouTube," he said.

Investors of all sizes have been buying into cryptocurrencies on a vast scale this year, partly out of a desire to diversify away from traditional assets like stocks or fiat currencies, but also because of the potential some of the various blockchains hold to revolutionize finance, trading and even art and entertainment.

For example, NFTs, which are essentially like digital collectors' items that represent real-world assets like artwork, video and even virtual real estate, have been booming, endorsed by celebrities, sports stars and luxury brands.

Accessing these digital assets often requires a cryptocurrency. The prospect of having to use a blockchain's native currency to buy an NFT, take part in an online game, or make a payment has been a big driver in some of the tokens whose networks offer, or will offer, these capabilities - cardano being one of them.

Since cardano announced in a <a dir="ltr" href="https://twitter.com/InputOutputHK/status/1426178532527558659">video </a>on August 13 that it would release the final alonzo "purple" update on September 12, the network's native ada cryptocurrency has gained around 33% and is now the third largest digital coin by market value, with a capitalization of $82 billion, according to <a dir="ltr" href="https://coinmarketcap.com/">coinmarketcap.com </a>.

Bitcoin, the biggest crypto that boasts a market value of around $884 billion, has gained around 4% in that time.

Another application that will run on the cardano network once the upgrade is complete are smart contracts - a piece of code that allows two or more parties to transact with one another without a central force like a commercial bank, or broker. Smart contracts are one of the cornerstones of decentralized finance and any network that can run them will be in demand.

This is a big step forward for a network that for years since it launched in 2017, was widely pilloried as a "ghost chain" - crypto-slang to describe a blockchain worth billions that ultimately has very little in the way of actual utilization.

"It'll be a fun day. Everybody used to call us 'the ghost chain', and I bought the Ghostbusters uniform, so come September 12, I'll be wearing it and we'll have some fun," Hoskinson said.

Cardano's big alonzo upgrade will bring 'programmability' to the blockchain, according to founder Charles Hoskinson - he's going to wear a Ghostbusters costume to mark the occasion

Let’s start with security. It’s at the forefront of many minds these days, with supply chain issues, cryptojacking and data breaches being common news stories. WebAssembly brings some big guns to the security fight in a number of ways, starting with its security sandbox.When WASM is executed, the code is not given access to anything, including the network, file system or anything else out of the box. This is known as a deny-by-default security model. For the code running inside the WASM sandbox to interact with the outside world, it must be explicitly granted access to host functions.

Security

To combat the rise of supply-chain attacks, there are also <a dir="ltr" href="https://github.com/WebAssembly/design/issues/1413" onclick="this.target='_blank';" rel="external">efforts </a>to make cryptographic signing of WebAssembly modules a standard part of the specification. The ability to sign a module with a trusted key and have that signature verified any time a system attempts to run that module will be a big win, and helps cement the “defense in depth” mindset when working with WebAssembly to build applications.

Language portability is the ability for modules written in one language, like Rust, to run inside a system written in another, like Go. Because WASM runtimes can be embedded within other software, they can be used to introduce specialized components that wouldn’t otherwise be possible in your team’s primary language. This concept can also be applied to applications that are built by composing together multiple WASM modules written in different languages to form a single piece of software, such as with the <a href="https://github.com/suborbital/atmo" rel="noopener noreferrer" target="_blank">Atmo </a>framework, which our company, <a href="https://suborbital.dev/" rel="noopener noreferrer" target="_blank">Suborbital </a>maintains.

Language portability

WebAssembly is not tied to any hardware architecture, so it can be executed on several different platforms without needing to re-compile. When a WASM module is executed, it is compiled to machine-native code either ahead-of-time (AOT) or just-in-time (JIT).WASM runtimes such as Chrome’s V8, Wasmtime or Wasmer perform this compilation automatically. I can see this becoming useful in scenarios where your code gets moved automatically from a central cloud instance to an edge compute network when a spike in traffic causes a certain module to be pummeled by incoming requests.

platform portability

WebAssembly is intended to execute code at native or near-native speeds.WebAssembly is a representation of your code that can easily be executed in a high-performance manner by optimizing itself for each platform at runtime. We are still in the early days of WebAssembly, but it is already rivaling the performance of the V8 JavaScript engine, which has been optimized over countless years.

Performance

Contributor Connor Hicks describes three benefits of WebAssembly — security, portability and performance — and why they make WASM powerful.

There is a rising tide in the software industry around <a dir="ltr" href="https://webassembly.org/" onclick="this.target='_blank';" rel="external">WebAssembly </a>. Commonly known as WASM, this technology is changing the way our code is executed in environments such as the web browser, edge cloud, IoT devices and more. I’ve written about different aspects of WebAssembly in the past, so <a dir="ltr" href="https://thenewstack.io/why-webassembly-modules-could-be-the-new-de-facto-unit-of-compute/">give </a><a dir="ltr" href="https://blog.suborbital.dev/building-for-a-future-based-on-webassembly" onclick="this.target='_blank';" rel="external">those </a>a <a dir="ltr" href="https://blog.suborbital.dev/get-started-with-webassembly-using-typescript-part-1" onclick="this.target='_blank';" rel="external">read </a>if you’d like more context.

Today I will talk at a high level about what makes WASM special, why you should care about it and why I believe it will be powering a new generation of software for years to come. I will lay out the three pillars of WebAssembly — security, portability and performance — and give you an overview of why they make WASM a force to be reckoned with.

Let’s start with security. It’s at the forefront of many minds these days, with supply chain issues, cryptojacking and data breaches being common news stories. WebAssembly brings some big guns to the security fight in a number of ways, starting with its security sandbox.

When WASM is executed, the code is not given access to anything, including the network, file system or anything else out of the box. This is known as a deny-by-default security model, and it’s not something implemented by container technologies such as Docker. For the code running inside the WASM sandbox to interact with the outside world, it must be explicitly granted access to host functions.

The web browser can grant a WASM module certain abilities such as the <code dir="ltr">fetch </code>API and access to the DOM. Other hosts like cloud environments can use WASI (the WebAssembly System Interface) to grant the module access to the file system, randomness (like <code dir="ltr">/dev/urandom </code>) and common things like the system time. All of this ensures that the code inside a WASM module is only given access to the resources it needs to operate and nothing more.

It will become especially important when software vendors begin distributing their API libraries as WASM modules; we’ll be able to take advantage of cross-language module linking (the ability for modules built from different languages to share memory and functions), but also verify that the module was produced by the vendor and has not been tampered with.

The idea of linking modules brings us to the second pillar, portability. I like to think about WASM’s portability in two ways: language portability and platform portability. Language portability is the ability for modules written in one language, like Rust, to run inside a system written in another, like Go. Because WASM runtimes can be embedded within other software, they can be used to introduce specialized components that wouldn’t otherwise be possible in your team’s primary language. This concept can also be applied to applications that are built by composing together multiple WASM modules written in different languages to form a single piece of software, such as with the <a dir="ltr" href="https://github.com/suborbital/atmo" onclick="this.target='_blank';" rel="external">Atmo </a>framework, which our company, <a dir="ltr" href="https://suborbital.dev/" onclick="this.target='_blank';" rel="external">Suborbital </a>maintains.

In regard to platform portability, WebAssembly is not tied to any hardware architecture, so it can be executed on several different platforms without needing to re-compile. This would allow you, for example, to produce a single WASM file and run it on both an IoT device and a cloud virtual machine. When a WASM module is executed, it is compiled to machine-native code either ahead-of-time (AOT) or just-in-time (JIT).

WASM runtimes such as Chrome’s V8, Wasmtime or Wasmer perform this compilation automatically, which makes running any module very simple. I can see this becoming useful in scenarios where your code gets moved automatically from a central cloud instance to an edge compute network when a spike in traffic causes a certain module to be pummeled by incoming requests, for example.

That leaves us with the final pillar, performance. WebAssembly is intended to execute code at native or near-native speeds. There are a number of factors that play into the performance of any application, including language, hardware, garbage collection, virtualization and more.

WebAssembly is a representation of your code that can easily be executed in a high-performance manner by optimizing itself for each platform at runtime. We are still in the early days of WebAssembly, but it is already rivaling the performance of the V8 JavaScript engine, which has been optimized over countless years. All languages that compile to WebAssembly will benefit from any future performance improvements brought to WASM runtimes, which is a big win for everyone.

I invite you to think about the software you’re building and ask yourself what aspects of your development platform are most important to you. I’m willing to bet that at least two of the three pillars that I discussed here are a key factor to your business, and that you would be able to improve upon your code’s key attributes by adopting WebAssembly as a technology. The hope is that WASM will become an underlying technology that powers software workflows without developers needing to care much about it at all, just as the majority of developers don’t concern themselves with the assembly code generated by their language’s compilers today.

The main difference is that WebAssembly has these great properties built in from the start to ensure your code is secure, portable and performant by default, which cannot be said for today’s software tooling.

Feature photo by <a dir="ltr" href="https://unsplash.com/@jacksondesign?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" onclick="this.target='_blank';" rel="external nofollow">Kevin Jackson </a> on <a dir="ltr" href="https://unsplash.com/s/photos/pillars?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" onclick="this.target='_blank';" rel="external nofollow">Unsplash </a>.

The Three Pillars of WebAssembly - The New Stack

Earlier this year, game developers across the industry weighed in on Twitter on a seemingly innocuous question: <a dir="ltr" href="https://www.ign.com/articles/putting-doors-in-video-games-is-a-nightmare-say-developers">What's the problem with doors in video games? </a>

But it will probably not surprise you to learn that doors are far from the only seemingly simple feature that prove to be unexpectedly challenging in the development process. A few months ago, I asked developers across the industry the question, "What is a thing in video games that seems simple but is actually extremely hard for game developers to make?" I received nearly 100 responses representing a wide breadth of industry experience, ranging from solo developers to those who had tackled issues within teams of hundreds.

For instance, elevators. whether they're taking players up a single floor in a building or serving as pseudo-loading screens between two major game areas"First off, you have to summon [an elevator] via a button or whatever. By calling the elevator, you open the opportunity for the player, objects or AI to wander underneath it and get squished or trapped. So suddenly you have to deal with that. It's an invitation to make your enemies or companions look dumb, for physics objects to go flying, or for quest items to get stuck.

Getting from place to place

The team at Skydance Interactive, behind The Walking Dead: Saints &amp; Sinners, they were able to turn a challenging collision problem into a fun new feature:"One of the bugs I had was when a walker was trying to get through the door at the same time that a player was trying to open it, this is sort of like a situation when two opposing forces come against each other."Our solution was every time the player would try to open the door and a walker was trying to go through that same door was to break the door. As a result, this actually created a jump scare moment."

Opposing Forces

Moving the player with a controller isn't hard; moving the player on a platform isn't hard, Putting them together you now have two things that want to dictate how the player should be moving. What should you do if the platform pushes the player up into a ceiling? Push the player through the platform? Squish the player? Stop the platform from moving? Force the player to crouch allowing a little bit more time before squashing? What about from the other side where the platform comes from the top down on the player pushing them into the ground? Should it behave the same?

moving platforms.

Game devs of all sizes and scopes respond to the question: "What is a thing in video games that seems simple but is actually extremely hard to make?"

It turns out, a lot. A seemingly boring feature such as usable doors can be absolute hell for developers to put in their games for numerous reasons. Everything from physics to functionality, from AI to sound, comes into play while making a single door in a single video game work. And not just work, but work in such a way where the player never has to think about it. Building a working, forgettable door is an incredible game development undertaking.

But it will probably not surprise you to learn that doors are far from the only seemingly simple feature that prove to be unexpectedly challenging in the development process.

A few months ago, I asked developers across the industry the question, "What is a thing in video games that seems simple but is actually extremely hard for game developers to make?" I received nearly 100 responses representing a wide breadth of industry experience, ranging from solo developers to those who had tackled issues within teams of hundreds.

The pool of responses similarly included a number of varied problems, but also a number of similar issues popping up among many projects. Those I spoke to described challenges in making games look and sound good, storytelling, movement and interaction with objects, menus, save systems, multiplayer, and all sorts of intricacies of design that are so rarely discussed outside of studios themselves. Many noted that they’ve received angry player feedback about the topics they mentioned, with their audiences asking, "Why don't you just do X?" The answer is, almost always: because it's really, really hard.

So if you've ever wondered why the maker of your favorite game didn't simply fix one of the myriad issues developers mentioned below, here's why those seemingly simple problems are hardly simple at all.

As the original topic of game development headaches focused on doors, it made sense that many of the developers I spoke to had issues with other methods used to connect a person from one place to another.

For instance, elevators. Multiple developers told me about the frustrations of elevators, whether they're taking players up a single floor in a building or serving as pseudo-loading screens between two major game areas. Bill Gardner, creative director of The Deep End Games and lead level designer on BioShock and BioShock Infinite, explained the elevator problem as follows:

"First off, you have to summon [an elevator] via a button or whatever. By calling the elevator, you open the opportunity for the player, objects, or AI to wander underneath it and get squished or trapped. So suddenly you have to deal with that. It's an invitation to make your enemies or companions look dumb, for physics objects to go flying, or for quest items to get stuck.

"So let's say you try to solve the AI issue by excluding them from elevators. Well, how then do they avoid looking dumb as soon as you cross the threshold inside the elevator? They'll either stand there, or need some sort of custom behavior.

"Then, if the elevator can take you to multiple floors, you have to feed back when an elevator is 'busy' or allow it to queue up floors to stop on. Now let's say you decide to get off on the second floor, then jump down and hit the call button again. Suddenly you have to feed back that the elevator is busy. Even then, the player is used to having things respond nearly instantly, so there's a good chance they'll get frustrated and call the thing broken.

"None of this is even getting into handling the [elevator] doors and what happens when those open and close and trap the player or other objects in the doorway."

The team at Skydance Interactive, behind The Walking Dead: Saints &amp; Sinners, ran into a similar problem as the one Gardner describes, but again with doors, and in a VR game no less. Fortunately, they were able to turn a challenging collision problem into a fun new feature:

"One of the bugs I had was when a walker was trying to get through the door at the same time that a player was trying to open it," said Bill Ferrer, Skydance senior AI gameplay engineer. "This is sort of like a situation when two opposing forces come against each other, immovable object vs. unstoppable force.

"Our solution was every time the player would try to open the door and a walker was trying to go through that same door was to break the door. As a result, this actually created a jump scare moment. The result was basically a happy mistake."

Well, if doors and elevators are too complex, why not just have players take the stairs? Josh Davis, quality assurance at Guacamelee developer Drinkbox Studios, says that's an issue too.

"They might not be as bad as doors, but can oftentimes be immersion breaking. I was thinking how most of the time they're just ramps people just slide along; they can sometimes mess with how jumping and movement work because of angles/collisions. And how to deal with their realism can get a lot more complicated for some styles of 2D games like ours, because we've had and continue to have bugs for them."

All these means of transport are a potential nightmare, but there's yet another mode of getting around areas that's causing developer agony: moving platforms.

"Moving the player with a controller isn't hard; moving the player on a platform isn't hard," says Kyle Donnelly, a programmer at Land of Screens and Doki Doki Literature Club Plus studio and publisher Serenity Forge. "Putting them together you now have two things that want to dictate how the player should be moving. What should you do if the platform pushes the player up into a ceiling? Push the player through the platform? Squish the player? Stop the platform from moving? Force the player to crouch allowing a little bit more time before squashing? What about from the other side where the platform comes from the top down on the player pushing them into the ground? Should it behave the same?

"You can add multiple moving platforms into the equation too. What if the player is standing on two platforms going in opposite directions? Should the player be split apart? Ride on the fastest moving platform? Ride on the platform that the player is most on? Once you decide how the platforms should move it's a whole other ballpark of issues implementing them with the physics system you're using."

With all of these forms of movement being challenging, what about just...regular old walking to get from place to place? Andi, a programmer with Unknown Worlds, said that even this could get very complicated and go horribly wrong.

“It's not really ‘hard’ to write something and get a character to move, but it's tricky to tune all the values so that the result feels right. It took quite some time to find our values for acceleration and drag that we were happy with [in Subnautica], considering all the speed modifiers we have in the game (sprinting, seaglide, currents, wind etc).

“On that topic, I discovered an interesting bug a few weeks ago: sometimes the player froze in midair at the peak of a jump. Turned out part of our movement code, which we used now for over seven years, had a bug that only could manifest once hardware got fast enough. Everything was fine, until you reached ~200fps.”

Another problem that an overwhelming number of developers cited to me is the simple idea of any "object" in a game interacting with any other object. That means that something as simple as a character picking up an item or two characters making contact with one another is actually unbelievably complex, never mind what happens when you get into more complicated interactions between two things.

Ben Wander, designer on Airborne Kingdom at The Wandering Band, explained the issue as follows:

"The objects themselves aren't real -- they have no density or physical bounds -- so if you want a character to, say, hold an apple, the artist has to go in and change the character's fingers to wrap around the apple perfectly. Want them to hold a kiwi instead, or how about a grapefruit? Make a different animation! That's why you'll often see characters hand-generated quest items to other characters just below the camera, or sometimes why a character's hands will go through a door handle or something."

Audrey LePrince, co-founder of The Game Bakers, affirmed this in a way that perfectly captures her work on Haven, a game about two romantic partners who have a lot of affectionate, physical contact with one another throughout the game.

"Touching, holding hands, hugging, it’s already hard in cutscenes but in real-time gameplay it’s a nightmare. Haven was a great example of that. But still it seems so natural for players, and something that we should have more of in video games.”

Maebe Sewell, lead technical design animator at Gearbox, pointed out another example of this: petting the dog. There's been an ongoing trend of excitement around games allowing this feature, in part sparked by the Twitter account <a dir="ltr" href="https://twitter.com/CanYouPetTheDog" rel="noopener noreferrer">Can You Pet the Dog? </a>, but Sewell noted that this can actually be very challenging due to the aforementioned issues around two characters making any kind of contact with one another, not to mention things like AI pathing, UI, and more.

Objects and characters moving and interacting together can be a problem whether you’re aiming for realism or a specific visual style. For a very specific example, Glitch Factory artist and animator João Esse offered a problem from upcoming game No Place for Bravery, in which protagonist Thorn carries his son Phid on his back. The two characters have independent animations, where each character can perform separate actions at the same time. For example, Thorn might be fighting an enemy, while Phid is pointing to something of interest.

“The duo's spatial relationship ends up being very complex in some animations, especially when Phid moves forward or backward in different frames. We have to go through every frame in every animation to correct each characters' position to solve this.

“The real problems start to appear when Thorn makes a very complex movement, like the one in the GIF below:

“Phid and the Shield are between Thorn and the Hammer, and everyone's also rotating together to create the illusion of the Hammer's blow.”

Things get even more complex, Esse says, when Phid is absent in certain sections. Thorn can’t just be left with his animations minus Phid, because items like his shield are animated specifically to account for Phid being in the way, and look funny with Phid just taken out. So there’s an entirely separate set of animations for these sections.

“We are talking about 142 animations, 941 animation frames divided into 97 layers, all of this only in the main character!” Esse concluded.

Though not an object itself, a body of water presents another set of difficulties in allowing for, or preventing, interactions. Unknown Worlds co-founder Max explained that most players appreciate how hard it is to make cool water in games, but added that what's often overlooked is how hard it then is to keep said water out of places you don't want it.

"Water in the real world occupies a volume and in video games we typically represent those types of volumes with simple shapes," he said. "In the case of an ocean, this might be an infinite plane sitting atop a box which extends downward forever. Every point inside this box is considered filled with water when drawing the scene or simulating buoyancy. Using these types of simple shapes to describe the water makes it feasible within the limits of a real-time game.

"As soon as you introduce boats, submarines, underwater bases or caves into this water world <a dir="ltr" href="https://www.youtube.com/watch?v=p5i6xB7GJB4" rel="noopener noreferrer">you encounter a problem </a>. The space occupied by the hull of these objects should not contain water (unless you have a leak!) and needs to be carved from that simple water volume. Often different approaches are used for each of the systems that interact with the water. For gameplay code, there might be a series of checks to determine if the player is able to breathe at their current position. Is the player inside the ocean volume? Are they inside a boat? Is the boat sinking? Answering the question 'is the player inside water?' in Subnautica became a remarkably complex code path.

Another easily underestimated element of making games is storytelling. The ways in which game writers and narrative designers have historically been overlooked in all but the biggest blockbusters (and even then!) is an entire article unto itself, but let’s just say it’s all a little more complicated than sitting down and writing some dialogue.

Arone Le Bray, former quality narrative designer at BioWare, told me a fascinating story about the challenges in making games like Mass Effect, with choices that are expected to impact the story. He says that these create challenges in “tracking reactivity,” or ensuring that if A or B happens, then C or D happens. Most of the time this works fine, but every once in a while an edge case can throw everything off.

Le Bray explained how a decision made in the original Mass Effect could cause a bizarre scene all the way down the line in Mass Effect 3, simply because of one of these “edge cases” that didn’t originally get caught. In the first game, most players would meet up with party members Garrus, Wrex, and Tali on the Citadel, with Tali joining the party along with either of the other two. Whichever one didn’t join would show up at the Normandy later and ask to join, but it’s possible for players to decline to take the third member with them at that point. Only an “extremely small minority of people” made that choice, he said.

For those who did, there might have been a game-breaking bug had BioWare not caught it very late in development. In the unlikely set of circumstances that Wrex is in the party on Virmire, Shepard doesn’t have enough Paragon or Renegade points to stop him from attacking, Liara hasn’t been recruited, and Garrus was previously rejected on the Citadel, then killing Wrex would make it so the player doesn’t have a full squad and can’t proceed.

“Luckily we caught it before we shipped,” Le Bray said. “...But we didn't catch Zombie Tali.” he continued.

Zombie Tali refers to a romance scene in Mass Effect 3 where Tali, despite being dead, would show up to talk to Shepard just before the final mission, and it happened for similar reasons as the Wrex bug Le Bray described above. The scene, he said, triggered if a character passed a certain invisible check in which they were in a relationship with a given character, and would show the scene if they had reached a certain stage before the final mission. But what it didn’t check for was if a character was alive or dead.

“In order to encounter this, a player would have to actively pursue a romance with Tali in Mass Effect 2, actively continue the romance in Mass Effect 3, side AGAINST Tali and her race (Quarians) and watch her pass away, and then NOT pursue a romance with any other characters. We patched it, but we still shipped with it in the end,” he said.

Another narrative problem related to player choice occurred in The Elder Scrolls Online, which during its development had as a main design point a mandate that the player could "change the world." As Matt Firor, studio director at ZeniMa Online Studios tells it, one example of this was that if a player encountered a village that was under attack and in flames, they could choose to help the villagers, resulting in a nice, peaceful, not-on-fire village. To do this, they had to essentially make two different copies of the same village, one for players who had helped, and one for players who had not done so.

This worked out well in testing, but when the game launched, things went awry very quickly.

"As soon as real players started roaming through Tamriel, they ran into problems with this system almost immediately," Firor said. "Players -- friends, guild mates, etc. -- don’t always go through content together, and it was very common that someone in your group had not done the quest yet, but you had. In this case, players who hadn’t completed it yet would 'disappear' as they were separated onto the on-fire version of the village, and even worse, the players who had completed the quest couldn’t help their friends because they were physically separated from them. This problem was magnified because we had quests and points-of-interest all over the game that used this technique, more than a dozen per zone. So the player experience quickly evolved from 'I can change the world' to 'why are my friends disappearing?'"

According to Firor, the team had to go back over each quest that used this technique -- over a hundred at the time! -- and restructure them so player impact was dialed down. A few still remain in the game, mostly in starter areas where players are less likely to be in groups. But the whole process of fixing this issue took The Elder Scrolls Online team over a year.

Choice-based systems aren’t the only places where narrative and writing can create challenges. Harebrained Schemes producer JC Lau answered my question by saying, “I want to be super glib about this and just say ‘getting a single line of text onto a screen’ because as someone who works with UI/UX, narrative, accessibility, and localization, this never is as smooth and obvious as you might think.

Lau explained the very basic idea of communicating a concept to a player can mean different things to both design and narrative teams, and requires communication between the two to ensure something is clear in every way.

“When I worked on Battletech, the font pack we used for Russian somehow had all the Cyrillic characters two pixels taller than the Latin alphabet characters, and this was enough for them to break their text boxes and the Russian text just wouldn't render and we couldn't figure out why until UI looked at it and was like, ‘Yep, it's because this font is indiscernibly taller than English because... reasons.’”

Issues like that, she said, are why localization quality assurance is so important.

“When I worked at Xbox in localization, it was when the console rolled out its friends list feature and there was a little popup that said, ‘Socialize with your friends’ directing people to use the friends list. In Polish, that got translated to ‘Support socialism with your friends,’ which was thankfully caught.”

Multiple people I spoke to mentioned the struggles involved with localization. Joe Mirabello, director of Terrible Posture Games, called translation “deceptively complex,” citing issues ranging from puns that don’t work in other languages to problems with words being different lengths in different languages.

“Planning ahead helps, but nothing will prepare you for German,” he said. “German destroys your best laid plans. German will defeat you. That text field you thought would only ever need a single 10-20 character word? Nope. German has a unique word for that and it’s a hundred and twelve characters long. We even have a native German developer on our team and he refuses to translate our games into German. This is all said tongue-in-cheek, of course, just to illustrate a point, and that is; whatever scaling flexibility you think you’ve planned for in your UI to account for localization? It’s not enough. It’s never enough.”

One nightmarish place where writing intersects heavily with design is whenever a developer needs to get a player to do something specific to progress, and has to clue them in. Kevin Zuhn, creative director and writer at Young Horses, told me about the complexity of hint systems.

"Hints are surprisingly hard to get right!” they said. “You can't just tell players what to do, you need to shift their thinking in the right direction. It takes a light touch and subtle reminders. That means rigorously testing to find out where players most commonly lose their way. And what you write needs to be very precise! I've seen the wrong word choice send players spiraling off into strategies that'll never work. And then it's twice as hard to get them back on track from a bad hint!"

Alexander Horn, narrative director at Impulse Gear who has previously worked on Kingdoms of Amalur: Reckoning, also immediately thought of tutorials when I spoke with him, saying that they tend to be so fourth-wall-breaking that he “gets physically ill just thinking about it."

“The beginning of [Amalur] serves as a tutorial, and the first objective in the first room used to be to loot a sword from a dead body partially to teach looting mechanics. My friend, who is a pretty competent gamer, did some playtesting, and spent almost ten minutes stuck in that room, unable to find the sword and exploring every nook and cranny of the room. It was agonizing to watch, because you wanted to scream ‘It's right there!’ but [it was] invaluable feedback. We ended up putting the sword in the door, so that when the sword was looted, the player-character was able to leave the room. It was an organic gate more easily found by players.”

.css-d7z4j9{font-family:'Source Sans Pro',sans-serif;font-size:2rem;line-height:2.5rem;letter-spacing:-0.125%;font-weight:700;}Development and Release of Type4Py: Machine Learning-based Type Auto-completion for Python