Real Time Data with Socket.io: React and Node

I had an interesting situation come up in an application;

• User uploads file
• File gets sent to a third-party
• Third-party receives file and sends back status 200
• Third-party software does something with file, then;
• Third-party sends webhook with modified data to my server
• Data is saved to database

The third-party "doing-something" with the file could take thirty-seconds or two minutes.

This can be bad for the user experience. I also didn't want to put a refresh button that the user could hammer on the whole time they wait firing off untolds requests.

I needed a way to let the client know that new data was available, because the initial upload to the third party was just half the pie. The other half was getting the modified data back. If the client could be notified of new data then it could fire off a request to refresh the current view.

One of the first answers I found (I'm using MongoDB as a database) was to use MongoDB's change streams.

Reading the description I knew this was exactly what I needed. Yet, after reading further it would require a charded database or replica sets. This change wasn't what I was looking for. I don't want to knock on those features. I would be excited to use them in an application, but I didn't want to mess with the database.

Plan B: I felt I could get the job done with Socket.io.

If I could connect the client with Socket.io I wouldn't even need to send a new request for data I could just send the data after it was added to the database (it would be a relatively small object) and then add it to the state. The state change would then cause the component to rerender. I'm getting a little ahead of myself because there was a few challenges first.

I tried to find out how I was going to do this, but most tutorials on Socket.io are chat apps. Even the examples in the documentation are skewed towards chat apps: not a big issue. It just meant I would need to dig a little further into the documentation.

My backend is written in Node, and I'm using Express for my server. I have a main server.js file then my route files are in a different folder.

The new problem was that I initiated the io server on the backend in my main server.js file. That meant that I wasn't able to access it in my routes. Can I have a defaut export in that file? Didn't seem like something I wanted to do.

Not having access to the io instance was stopping me from adding an emit event when the the new data came in from the third-party. Luckily, I came across a video with my eventual solution. I had tried to create the instance in a separate folder and then export it from there, but ran into issues that weren't obvious. Here's what I ended up doing;

In my server.js file I made a call to a small function that was in the route file that I needed to use the io instance in.

// import stuff
...
const { init, otherRouter } = require('./routes/otherRoute');
...

...
const app = express();
...
// set up server middleware

const server = http.createServer(app);

const io = init(server);

...

I am creating an io variable in my server file and making a call to the init function that I imported from one of the routes. That function returns the initialized server that I created in the other file. In otherRoute.js I have;

const createRouter = require('express').Router();
const socketio = require('socket.io')
// import stuff
...
let io;

// initialize the io instance before the route so we can use
// the io instance in this route and then return it to
// the other route
function init(server) {
    io = socketio(server)
    return io;
}

...
// route declarations
...

module.exports = {
    createRouter,
    init
};

This does the trick. I only need io in this one other route, and now I can use it further down in a route declaration.

That was a big hurdle, but I still needed to figure out;

• How to emit the data to the correct socket once the webhook came back around.
• How to control the socket connecting and reconnecting when components are rendered in the frontend.

How to emit the data to the correct socket once the third-party returned the data

The user ID is the key factor in this resolution. When the third-party data makes it's way back to my server route it has the user ID attached. This is how I save it correctly to the database and it's how I can get the data to the correct client. The Socket.io library has a method to set the socket IDs of the incoming sockets.

I could possibly change the socket ID to the user ID and emit the event to the client: no I can't. The IDs have to be unique for the sockets and if a user is logged in on more than one tab, or device, I will have a problem. Thankfully, this issue can be solved the same way that session IDs are tracked for a user.

• Each new socket ID (connection) will be concated to a list for the user (when the user logs in).
• If data is received for that user ID an event will be emitted for each socket that is in the list.
• If a user logs out, their socket ID will be removed from the list.

In the route I can emit to the individual sockets by mapping over the list.

//otherRoute.js

...
// route declaration
...
// sending to individual socketid (private message)
sockets.map(socketId => {
  io.to(`${socketId}`).emit("incoming data", {...data});
})
...

How to control the socket connecting and reconnecting when components are rendered on the frontend

I wanted to make sure that sockets were being created efficiently and that they were being disconnected at the right times.

// layout component on client

...
import io from "socket.io-client";
...

class Main extends Component {
    constructor(props) {
        super(props);
        
      this.state = {
        // other state 
        ...
        socket: null
      }
    }
    
  componentDidMount() {
    // userIdParameter is attached to URL
    // this allows me to match sockets to users
    const socket = io(url + userIdParameter);

    socket.on("incoming data", data => {
      this.onNotificationHandler(data)
    });

    this.setState({
      socket
    })
  }

  componentWillUnmount() {
    // clean up socket when component unmounts
    const socket = this.state.socket;
    socket.disconnect()
  }
...
// rest of file

I am creating the io instance in the componentDidMount lifecycle method and making a simple diconnect method call in the componentWillUnmount lifecycle method.

You may have noticed that I'm tacking on an extra variable to the URL when I'm creating the socket.

More than that, I NEED to pass that parameter because I am packing the sockets away based on the user ID. In server.js, I am now using the user ID to match socket to user.

// server.js

// server created
...
// Set up namespace for new socket connections
io.on("connection", socket => {
  console.log("New client connected");

  let userId = socket.handshake.query.id; // access query parameter

  // Save socket ID to list based on user ID
  addUser(socket.id, userId)

  // Wait for socket to disconnect then remove from list
  socket.on('disconnect', () => {
    removeUser(socket.id)
    console.log("socket disconnected", socket.id)
    })
});
...
//declare routes
// rest of file

To recap the whole process:

• Client connects to server + passes user ID
• Socket ID is stored in list based on user
• If client uploads file, they are:
  • Notified of successful file upload
  • Informed that data could take a few minutes to appear
• When data is received from third-party (on the backend) it is added to the database, and if the data is valid;
• The event is emitted—with the data—to all the sockets in the user's list.
• It's received on the front-end by the sockets and viewed by the clients.

This isn't the typical way to use Socket.io, but it solves the issue.