How we build real-time features using NodeJS and SocketIO

Users are increasingly enjoying instant access to information such as emails, messages in chat rooms, changes made by everyone for collaborative work.
A solution as Tuleap aims at making easier for teams' members to work together. To be really effective, some information require to be instantly shared without needing for the users to reload the page. In Tuleap, it is particularly the case for Kanban boards and the Test Management tool. 
This article shares how we developed real-time capability in Tuleap using WebSocket and Node.js technologies.

Connecting people in real-time

Real-time client/server communication

A real-time web application is one where flows of information are transmitted almost instantaneously between users and the server and from the user experience side, between users and other users. On the opposite, traditional tools are in the situation where the client has to ask for information from the server.

In a traditional application, the client have no idea when the state of the server may have changed. It regularly either polls the server to know changes or just waits for the user to ask , e.g. reloading the page.

With a real-time software, the client opens up a connection directly to the server so every side can send a message to the other, without waiting to be asked for it. If something on the server is updated, it sends the data over the connection to the client. If something on the client evolves, it sends it to the server.

Why real-time matters for working teams?

From now, in Tuleap, two plugins provide real-time:

How do we build real-time web tools using Node.js?

Technically, the real-time implementation consisted in setting up a new Node.js server called “Tuleap-realtime” to communicate with the “Tuleap server”. We could have implemented real-time using a technology directly from the Tuleap server but we selected the first option for the following reasons:

  • Applying the separation responsibility principle to not overload servers
  • Using faster, simpler real-time technologies
  • Making the installation of a real-time server as optional

Let’s go back to Node.js. This techno is a platform for server-side JavaScript development. It uses the JavaScript runtime engine v8, created in 2010 by Google. Its strength are its speed that comes from the use of the v8 engine and its non-blocking IO model.

A Node.js server between the Tuleap server and the web client

The purpose of this new server is to establish real-time communications by communicating with the Web client and the Tuleap server. The communication between the client and the Tuleap real-time server is made to broadcast messages to all connected users. The one between the two servers allows to warn when a request is done successfully on the Tuleap server, via the REST API. Then the data can be broadcasted to the other Web clients.

Architecture diagram - Setting up a new server

After viewing the architecture, what kind of communications we can use to broadcast data?

  

Sequence diagram - Card creation example in the Kanban plugin

 

Real-time push/pull model

Communication between a Web client and a Web server can be done in two ways:

  • Pull based: the client sends a request to the server and waits its response to get the information = the client pulls the information from the server
  • Push based: the server can send information to the client at any time without having requested it = the client no longer needs to pull information from the server.
Pull based
Push based

A real-time web application consists in having the information flow continuously updated, instantly sent by the servers. Therefore, push-based is a more suitable strategy for real-time.

Push-based: WebSocket protocol

W3C (World Wide Web Consortium) has incorporated a new Web standard based on push-based: Websocket. This is a standardized protocol in RFC 6455 that makes it possible to open a channel, using a TCP socket, for full-duplex communications between user’s browser and a server. With this API, you can send messages to a server and receive event-driven responses without having to poll the server for a reply.

Our Node.js server

We use WebSocket on a Node.js server using the Socket.io library. The Node.js platform natively supports an HTTP server library for building a Web server without using external software. As the protocol used does an "Handshake" process with the server via an HTTP request, it was necessary to create an HTTP server and give it to the WebSocket server.

// Load our modules with the module loader RequireJS.
var Config = require('./config/config');
var config = new Config();
config.init();

// Constants creation to create an HTTPS server
const PORT_CLIENT   = config.conf.get('port');
// Secret key for the certificat
const TSL_KEY_PATH  = config.conf.get('full_path_tsl_key');
// Certificat contains the public key
const TSL_CERT_PATH = config.conf.get('full_path_tsl_cert');

// Loading libraries with the module loader RequireJS.
var fs            = require('fs');
var app         = require('express')();
var _             = require('lodash');

var server, io;
try {
    var options = {
        key: fs.readFileSync(TSL_KEY_PATH),
        cert: fs.readFileSync(TSL_CERT_PATH)
    };
    // Creation of the HTTPS server
    server = require('https').Server(options, app);
    // Creation of the Websocket server
    io     = require('socket.io')(server);
    // The server listens the configured port in the config object
    server.listen(PORT_CLIENT);
} catch (err) {
    console.error('Be careful,' + err.message.split(',')[1]);
    process.exit(1);
}

Here it is. We can have a Tuleap-realtime Node.js server that communicates with the Tuleap server and the Web client.

Client - Tuleap real-time communication
Tuleap - Tuleap real-time communication
// Websocket connection done by Web clients
io.on('connection', function (socket) {
    socket.on('subscription', function (data) {
      // Subscribe the client and class by “room”
    });
});

// HTTP requests sent by the Tuleap server
app.post('/message', jsonParser, function (req, res) {
    res.end();
    // Broadcast message at all clients in the same “room” than the sender
});
 

JSON Web tokens for signing and verification

When we developed the real-time capability in Tuleap, one of the major challenge we encountered was security and access control aspects. We had to ensure that the messages were sent only to authorized persons. Indeed, without permissions management, any client can subscribe to the Tuleap real-time server, listen to the events and receive data protected by the Tuleap server. The Json Web Tokens ensures a secure exchange between the Tuleap server, the Web client and the Tuleap real-time server. You will be able to have more information about this in a following blogpost. (stay tuned, it will be published next week :-))

When real-time enables more engagement

The real-time contribution offered us a solution to various synchronization problems between several Web clients and also opened up new horizons. For example, thanks to the real-time technology, we have been able to add gamification to the Tuleap Test Management module. The podium of testers gives project team members real-time updates and rewards them for their performance.

Tuleap features Getting started

Share this post

Comments (1)

  • anon

    Hi,
    Juliana Leclaire

    Good and helpful post to share information about How we build real-time features using NodeJS and SocketIO..
    Lot of Thanks...

    Nov 28, 2017

Leave a comment

To prevent automated submissions please leave this field empty.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By using this site, you agree that we may store and access cookies on your device. Get more information. Ok that's fine