Project Idea | Build an AR based music learning platform using the MERN Stack

In this article, we will be building an AR-based web app called GuitAR. It teaches users to learn guitar by projecting the strings that are to be plucked on the guitar in the user’s camera feed. Now, all the user has to do is to pluck the highlighted strings to play a particular song. Here is an sample image for better understanding.

 Final Result. PS: We are using phone here but a printed marker will give better result.

We will be building this project with React JS for the front-end, ExpressJS running on NodeJS for back-end and MongoDB for database. We will also be using Selenium, BeautifulSoup4, Pymango for building our database of songs and ArucoJS which is an popular CV library for JavaScript. We will also use Firebase Authentication to integrate Google login into the app. Finally, we will use Github Actions to deploy the project on heroku!

This article assumes an understanding of MERN Stack, web scrapping, and some trigonometry and coordinate geometry. We won’t be going into details of the basic frontend and backend code because there are already many great articles covering it, and therefore we will focus on mostly on AR part of the project. You can check out the complete code in Github Repos. 

Links:

• Try It Out: https://globalguitar.herokuapp.com/
• Scraping: https://github.com/prathameshbhalekar/GuitAR-scrapping
• Frontend: https://github.com/prathameshbhalekar/GuitAR-frontend
• Backend: https://github.com/prathameshbhalekar/GuitAR-Backend

Note: Before getting started with the technical stuff let’s get acquainted with some guitar jargon. 

• Guitar tablature usually referred to as “tab”, is a method of notating music that empowers beginner guitarists to learn songs quickly and easily. Guitar tabs share similarities with music staff notation by showing you what notes to play, how long to play them, and what techniques to use. These tabs will keep changing throughout a song. We will be required to maintain where these tabs change in relation to lyrics in the database. 
• The fingerboard (also known as a fretboard on fretted instruments) is an important component of most stringed instruments. It is a thin, long strip of material, usually wood, that is laminated to the front of the neck of an instrument. The strings run over the fingerboard, between the nut and bridge. To play the instrument, a musician presses strings down to the fingerboard to change the vibrating length, changing the pitch. This is called stopping the strings.

Step 1: Building the database — First, we will be creating a database on MongoDB atlas. This database will contain the lyrics and the tabs mentioned earlier along with user settings. Login to MongoDB, create a cluster and create a database. We will be scrapping https://www.ultimate-guitar.com/ but feel free to any other method of your choice. The script for the spider can be found in the links section. Once you are done scrapping, save the data into database using the below code:

At the end of it the database should look something like this where <tag></tag> encloses the tabs:

_id:ObjectId("604c6b18ab0d440efde7ae4f")
title:"Drivers License"
artist:"Olivia Rodrigo"
info1:""
info2:"Difficulty: novice"
info3:""
data:"[Verse 1]
      <tag>G</tag>  I got my driver’s license last week
       Just lik..."

Step 2: Build backend

1. Setup — Create a folder and run npm init in it. After completing setup edit package.json to make it look like this:

// package.json

{
  "name": "guitar-app-backend",
  "version": "1.0.0",
  "description": "",
  "main": "server.js",
  "type": "module",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node server.js"
  },
  "author": "<YOUR_NAME>",
  "license": "ISC",
}

This will allow using ES6 syntax in our app. Now install express, mongoose, dotenv, cors and nodemon using npm or yarn.

2. Create Schema — We will first create the schema to access the songs that we have added to the Db earlier.

Now, we can create the user schema to save user settings. Since we are not storing any sensitive information like passwords or financial data, we can make do without any sort of security like encryption. We will discuss fields like “textColors” and “guitar” while building the frontend.

3. Create Endpoints — Now we will write the endpoints for the APIs. The complete code for the endpoints can be found in the backend repo in the links section. Once the backend is complete, it is ready to test. Run nodemon server.js and use postman for testing the API. Make sure to setup the .env file or change the mongodb connection URL to your own URL (not recommended if you plan to make your code public) before running.

4. Setup CI/CD — Push the code to GitHub in your repository using git. Create your Heroku app and add Heroku API key and mongodb password in GitHub secrets. Also, add the MongoDB password in Heroku environment. Now, go to your repository, go to actions and set up a new workflow as follows:

name: Build And Deploy

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build:
    
    name : Build
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [14.x, 15.x]
    env:
      MONGODB_PASSWORD: ${{secrets.MONGODB_PASSWORD}}

    steps:
    - uses: actions/checkout@v2
    - name: Install Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v2
      with:
        node-version: ${{ matrix.node-version }}
        
    - name: Install NPM Package
      run: npm ci
      
    - name: Build project
      run: npm run build --if-present
      env:
        CI: false
  
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v2
      - name: Push to Heroku
        uses: akhileshns/heroku-deploy@v3.4.6
        with:
          heroku_api_key: ${{secrets.HEROKU_API_KEY}}
          heroku_app_name: "<YOUR_HEROKU_APP_NAME_HERE>"
          heroku_email: "<YOUR_HEROKU_EMAILID_HERE>"

      - name: Run a one-line script
        run: echo successfully run

Note: If you plan to add CI/CD, make sure you use .env file for mongodb connection URL.

Step 3: Setup Frontend

1.  Create react app — Use npx create-react-app to create react app. Clean the default app and install the following dependencies using npm or yarn:

2. Build the Login page — This page will use Firebase to show the Google SignIn popup to the user. After getting Google account data, consume /api/v1/users/findbyemail and /api/v1/users/create to create new user or get user settings, if the user already exists. Once you have the user data from the database, store it into redux so that it can be accessed from anywhere. You get complete code from the frontend repo in the links section. This is what the result should look like:

Login Page

3. Build the Songs page — This page will contain a list of various songs that the user can play. This can be built by consuming /api/v1/songs/get. This endpoint takes page, limit, and query as request query which will help in pagination and search functionality. It should also contain a navbar to navigate to different pages. You get complete code from frontend repo in links section. This is how the page should look like:

All Songs Page

4. Add Settings Page — This page will let the user set the color of the AR projections and the speed of songs stored in user schema of the database. The current values can be accessed from the user data we stored in redux. We can update these settings by using /api/v1/users/updateStrokeColor, /api/v1/users/updateSpeed and, /api/v1/users/updateTextColor. It should look like this after completion:

Settings Page

Step 4: Building AR Helper Functions

Now, we are down to the toughest but most interesting part of the project. Before getting started, lets understand our approach to the AR problem. Before going for the solution, let’s consider some assumptions that we have made about the problem for an easier solution.

Assumptions:

• The position of marker on the guitar does not change after the setup procedure. This allows us to store the positional data during setup and use it for plotting later on. If position is changed, user has to go through setup again.
• The guitar is parallel to the camera. This reduces the problem into a 2D one which is much easier to solve and also reduces the computation. Computing the 3rd dimension doesn’t add much functionality when compared to the extra setup and computation required to resolve it. We can check this condition by making sure that all edges of marker are of same length as the marker is square.
• We trust users to select the four corners of the fretboard precisely during the setup process.

Now to locate the fretboard, we consider three corners of marker as a coordinate system such that the central corner lies on the origin and the rest two on the x and y-axis. Now that we have our axes, we have to worry about mapping points on these axes. Instead of storing the actual lengths of the x and y coordinates, we store their ratio with the size of marker which allows us to scale the distances depending upon how far the marker is from the camera.

Mapping points wrt marker

Now that we have mapped the points wrt the marker, we need to convert them wrt to canvas coordinates. We can easily do this by some basic trigonometry.

Now we can code the same in JavaScript:

Now that we have the four corners that we need to get the position of the strings on the board. We can easily do that by splitting the rectangle into equal parts. This can be done by the following functions:

Now that we have these points, all we have to do is to join opposite points to draw string. But before we start drawing, we need to find intersection points of two strings to highlight the strings to be pressed. We can do that using the following function using basic geometry: 

Now we have everything that we need to draw on our canvas. This can be done using the below code which takes canvas to be drawn on, video feed, the current position of marker, and list of points to be highlighted along with some settings as its parameter:

Now we are already with the code required to the AR part of the project, all that is left is to get the m_ration and the n_ration from user click during the setup procedure. This can be done by reversing the mapping function.

Step 5: Build The AR Components — Finally, we are ready with all the helper functions required for building the AR components. Now we can integrate them with the views by calling the draw function.

1. Building Setup Page — This page allows user to select the four corners of his guitar fretboard. We will convert this coordinates of each corner into m_ration and n_ration using the plot function that we built earlier. Once we have the rations for all four corners, we can consume /api/v1/users/updateGuitar to save these updates to our database. Once you are done, the page should look like this:

2. Storing Song — Before making the practice page, we need a way to process the songs that we get from the backend and store them. This can be done by the following functions:

3. Building Songs Page — Now we are finally ready to build the page where user will actually practice guitar. Firstly on loading, we will load the song from the query. We will also have state called pos, tab and text which will be our position in the song list, current tab and the lyrics to be displayed respectively. After every speed interval (Eg, after 2 sec if speed is 2) we will update these variables to the next ones in the song. Finally, we will be calling the draw function inside requestAnimationFrame to update the canvas efficiently. The complete code can be found below:

In the end, this page should look like this:

4. Setup CI/CD — Push the code to Github in your repository using git. Create your Heroku app and add Heroku API key in GitHub secrets. Now, go to your repository, go to actions and set up a new workflow as follows:

name: Build And Deploy

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    
    name : Build
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [14.x, 15.x]
        # See supported Node.js release schedule 
        # at https://nodejs.org/en/about/releases/

    steps:
    - uses: actions/checkout@v2
    - name: Install Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v2
      with:
        node-version: ${{ matrix.node-version }}
        
    - name: Install NPM Package
      run: npm ci
      
    - name: Build project
      run: npm run build
      env:
        CI: false
  
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v2
      - name: Push to Heroku
        uses: akhileshns/heroku-deploy@v3.4.6
        with:
          heroku_api_key: ${{secrets.HEROKU_API_KEY}}
          heroku_app_name: "<HEROKU_APP_NAME_HERE>"
          heroku_email: "<YOUR_HEROKU_EMAIL_HERE>"

      - name: Run a one-line script
        run: echo successfully run

Output: Now we can try out the deployed app and it should look like this:

Applications:

• Guit.ar improves learning speed by around 40%. It helps to build muscle memory by giving live visual feedback to the user. Visual feedback (VFb) is proven to boost the acquisition and retention stages of motor learning associated with training in a reaching task like learning guitar.
• Guit.ar is a great complement for music institutions and universities. This is helpful for distant learning, especially during the lockdown.
• Learning Guitar comes with learning a lot of redundant stuff before you play any song. Guit.ar helps to cut the clutter and get started with playing interesting music right from the first day.
• Users can turn their guitar into a musical game using Guit.ar. It turns a guitar into an arcade game where the objective is to strike the highlighted string within a particular time.

Teammate: https://auth.geeksforgeeks.org/user/ayushpandya517/