# Creating a price tracker with MERN ### Idea The main idea of the app is that a user should be able to track the price of the product or stock and get notified whenever the price drops within the budget defined by the user. And even track the price fluctuation history and present with the help of a nice chart. First of all, just visit [the website](https://lootle.live) and experience it on your own or read the full article [here](https://blog.aashish-panthi.com.np/introducing-lootle) or you can just watch the video below. %[https://www.youtube.com/watch?v=ykxeLvILbuE] # How to build it? Now you have general information about the app that we are going to work on. The services/frameworks that will be used to build this app are Reactjs, MongoDB Atlas, Express.js, Node, Linode, Nodemailer, axios, cheerio, etc. I will be breaking down the process into small chunks. You can jump back and forth from the table of contents. Else you can continue reading, the blog is straightforward. # Making the website Users need a way to interact with our tracker, right? So we need to make a website. For making a website, I've chosen a MERN stack. MERN stacks stand for MongoDB Express React and Node each comes with the following things: - [MongoDB](https://www.mongodb.com/) for database - [Express](http://expressjs.com/) for backend (making REST API) - [React](https://reactjs.org/) for frontend - [Node](https://nodejs.org/) as runtime environment ### Project structure ``` my-app/ ├── node_modules/ ├── client/ ├── node_modules/ ├── public/ ├── assets/ ├── illustration_404.svg ├── video-placeholder.jpg ├── favicon.ico ├── index.html ├── logo.png └── manifest.json ├── src/ ├── components/ ├── styles/ ├── dashboardapp.css ├── featuretile.css ├── footer.css ├── hero.css ├── modal.css ├── navbar.css ├── AddNewTracker.js ├── AllTracks.js ├── FeatureTile.js ├── FeaturesSection.js ├── Footer.js ├── Hero.js ├── Image.js ├── LineChart.js ├── Logo.js ├── Modal.js ├── Navbar.js ├── Page.js ├── PreviewWindow.js ├── PromptModal.js ├── layout/ ├── DefaultLayout.js └── NavbarOnlyLayout.jsx ├── pages/ ├── DashboardApp.js ├── Home.js └── Page404.js ├── App.js ├── index.js ├── index.css ├── service-worker.js ├── UserContext.js └── routes.js ├── package.json ├── package-lock.json └── .env ├── config/ └── db.js ├── controllers/ ├── site.js ├── test.js └── track.jsx ├── models/ ├── SiteModel.js └── TrackModel.js ├── routes/ ├── site.js ├── test.js └── track.js ├── utils/ └── getPrice.js ├── index.js └── .env ├── .gitignore ├── package.json ├── README.md ├── LICENSE.md └── package-lock.json ``` If you didn't get this tree diagram then you can look it up [here](https://github.com/aashishpanthi/lootle). ### Let's get the brief idea of a folder structure. In our root directory, we have our server serving us endpoint to hit on. We have different folders there each for specific tasks. - **config** -> To configure and connect to the MongoDB atlas database - **models** -> Create schema and models of different documents - **routes** -> Configure and call specific controller when user hits on a endpoint - **controllers** -> Main operations are done here when user want to perform any operation - **utils** -> Contains other necessary files needed in our app And there is a **client** directory where our react-app sits. Inside client, there are two more folders, **public** and **src**. - **public** -> It contains HTML file, logo and images - **src** -> It containes code for the application - **components** -> Contains different components of application - **pages** -> Contains main pages visible to the user - **layouts** -> Contains different layout of pages user can see (there are two in this app) ## Make the server side Let's start by creating a simple server and listening to a port. For that, we need to initialize our project. To initialize the project, type the command ```bash npm init ``` on the terminal and fill in all of the details required for the app and how much you want to provide to the app. Then create the main file (remember what you've provided while initializing the project). If you left the field default then you are good with **index.js**. So, create a index.js file on the root of our project. We will be using express to listen to the API request and to keep our server up and running. So, we need to install the [express package](https://www.npmjs.com/package/express) with the following command: ```bash npm i express ``` Now, inside of our **index.js** file, we need to write some code. We need to import the express from **express** and make an instance of it to listen to a port like this: ```javascript import express from "express"; const app = express(); //Server listening to port const port = 3001; app.listen(port, () => { console.log(`listening on port ${port}`); }); ``` Oh, one real quick thing I have to say here is that I am not using common js instead I am using the modular js. In short. instead of importing with `const` and exporting with `module.exports`, we can now use simple `import` and `export` syntax as we do in React. But to make this thing work, we need to configure our **package.json** file. Add type as a module just beneath the description or version in your package.json file. ```JSON "type": "module", ``` Now we can start our backend server with the command ```bash node index.js ``` You should be able to see a message on your console about the port where our backend server is running. ### Setup routes and controllers Now we need some routes to get requests and some controllers to process the request and send the response. Here you need to immediately create two folders: **routes** and **controllers**. As discussed above, both folders have their own purpose and meaning. #### routes Inside of routes directory, we will have three files: **site.js**, **test.js**, and **tracks.js**. As we are scrapping data from different websites, so we need to use our database to keep the record of supported websites. So we need some routes to create, read, update and delete the site. In short, we need to perform the crud operation. So the code inside of our `site.js` would look like this: ```javascript import express from "express"; import { getAllSites, getASite, addSite, deleteSite, getSitesByType, } from "../controllers/site.js"; const router = express.Router(); router.route("/").get(getAllSites); //Route for getting all Sites router.route("/:name").get(getASite); // Routing for getting specific site router.route("/type/:type").get(getSitesByType); // Routing for getting all sites by type router.route("/").post(addSite); // Route for adding Site router.route("/:name").delete(deleteSite); // Route for deleting specific site export default router; ``` Here various functions are being imported from controllers to perform their specific operations. Now we need to keep the record of the tracker, the product tracking according to the user. We have `tracks.js` for that specific purpose and it should contain code like this: ```javascript import express from "express"; import { getAllTracks, getATrack, addTrack, deleteTrack, getUserTrack, } from "../controllers/track.js"; const router = express.Router(); router.route("/").get(getAllTracks); //Route for getting all Tracks router.route("/:id").get(getATrack); // Routing for getting specific Track router.route("/user/:email").get(getUserTrack); // Routing for getting specific user Track router.route("/").post(addTrack); // Route for adding Track router.route("/:id").delete(deleteTrack); // Route for deleting specific Tracks export default router; ``` And one last file that remained in our routes is the `test.js` which fortunately or unfortunately has only one route. And the main purpose of this test file is to test if the product link entered by the user is valid or not. In simple words, `test.js` validates the user's given link. And it should contain code like this: ```javascript import express from "express"; import { testURL } from "../controllers/test.js"; const router = express.Router(); router.route("/").post(testURL); // Routing for getting url and test it if it is real export default router; ``` #### controllers Corresponding controller files are created for every single route file. And these controller files contain the actual code to check and process the actual request of the user. Now the **site.js** should contain some functions that need to be exported. And the file should look like this: ```javascript import Site from "../models/SiteModel.js"; import Joi from "joi"; // Get all sites export const getAllSites = async (req, res, next) => { try { const allSites = await Site.find(); res.status(200).json(allSites); } catch (error) { res.status(400).json({ message: "Request failed", error: error }); } }; // Get a specific site by name export const getASite = async (req, res, next) => { try { const site = await Site.findOne({ name: req.params.name }); if (!site) { return res.status(404).json({ message: "Site doesn't exist" }); } res.status(200).json(site); } catch (error) { res.status(400).json({ message: "Request failed", error: error }); } }; // Get all sites of a specific type export const getSitesByType = async (req, res, next) => { try { const sites = await Site.find({ type: req.params.type }); if (sites.length < 1) { return res .status(404) .json({ message: "No sites exists of this type", sites }); } res.status(200).json(sites); } catch (error) { res.status(400).json({ message: "Request failed", error }); } }; // Add a new site export const addSite = async (req, res, next) => { const schema = Joi.object({ name: Joi.string().required(), type: Joi.string().required(), priceLocation: Joi.string().required(), }); const { value: siteInfo, error } = schema.validate(req.body); if (error) return res.status(400).json({ error: error.details, ...siteInfo }); const site = new Site(siteInfo); try { await site.save((err, s) => { res.status(200).json({ success: true, id: s._id }); }); } catch (err) { next(err); res.status(400).json({ error: err, ...siteInfo }); } }; // Delete a specific site export const deleteSite = async (req, res, next) => { const siteName = req.params.name; try { await Site.deleteOne({ name: siteName }); res.status(200).json({ success: true }); } catch (error) { res.status(400).json({ message: "Request failed" }); } }; ``` The file was lengthy but the code is self-explanatory. We have used models in the code above. We will look at the models and mongoose in a little bit but first, we should be completing the controller files. One more package is being used in the above code, it's joi, which is used to validate the user's request quickly and easily. Install the joi with ```bash npm i joi ``` Next, **tracks.js** file should be able to perform all CRUD operations plus some specific operations according to the user's requirement or simply our website's requirements. This file should look like this: ```javascript import Track from "../models/TrackModel.js"; import Joi from "joi"; // Get all Tracks export const getAllTracks = async (req, res, next) => { try { const allTracks = await Track.find(); res.status(200).json(allTracks); } catch (error) { res.status(400).json({ message: "Request failed", error: error }); } }; // Get a specific Track by id export const getATrack = async (req, res, next) => { try { const oneTrack = await Track.findOne({ _id: req.params.id }); if (!oneTrack) { return res.status(404).json({ message: "Track doesn't exist" }); } res.status(200).json(oneTrack); } catch (error) { res.status(400).json({ message: "Request failed", error: error }); } }; // Get all Tracks of a specific user export const getUserTrack = async (req, res, next) => { try { const tracks = await Track.find({ user: req.params.email }); if (tracks.length < 1) { return res .status(404) .json({ message: "User doesn't have any track", tracks }); } res.status(200).json(tracks); } catch (error) { res.status(400).json({ message: "Request failed", error }); } }; // Add a new Track export const addTrack = async (req, res, next) => { const schema = Joi.object({ user: Joi.string().required(), name: Joi.string().required(), url: Joi.string().required(), site: Joi.string().required(), type: Joi.string().required(), demandPrice: Joi.number().required(), image: Joi.string(), }); const { value: trackInfo, error } = schema.validate(req.body); if (error) return res.status(400).json({ error: error.details, ...trackInfo }); const track = new Track(trackInfo); try { await track.save((err, track) => { res.status(200).json({ success: true, id: track._id }); }); } catch (err) { next(err); res.status(400).json({ error: err, ...trackInfo }); } }; // Delete a specific Track export const deleteTrack = async (req, res, next) => { const TrackId = req.params.id; try { await Track.deleteOne({ _id: TrackId }); res.status(200).json({ success: true }); } catch (error) { res.status(400).json({ message: "Request failed" }); } }; ``` And at last, we have **test.js** controller file. This is the file that has a function to validate the link in its original state. A further dedicated file is created at `utils/getPrice.js` which actually validates the URL. It should have code like this: ```javascript import Site from "../models/SiteModel.js"; import Joi from "joi"; import { getPNI } from "../utils/getPrice.js"; // Get a specific Track by id export const testURL = async (req, res, next) => { console.log(req.body.url); const { value: url, error } = Joi.string().required().validate(req.body.url); if (error) return res.status(400).json({ error: error.details, url }); const regex = /^(?:https?:\/\/)?(?:[^@\n]+@)?(?:www\.)?([^:\/\n\?\=]+)/im; //check URL const match = url.match(regex); const domain = match[1]; // get the domain of site form url console.log(domain); try { const site = await Site.findOne({ name: domain }); if (!site) { return res.status(404).json({ message: "Site doesn't exist" }); } //test url const { name, price, image } = await getPNI(url, site); if (!price || !name) { return res .status(400) .json({ message: "Invalied url or check if the item is available" }); } const resObject = { type: site.type, site: site.name, price, name, image, }; res.status(200).json(resObject); } catch (error) { res.status(400).json({ message: "Request failed", error: error }); } }; ``` And here we are importing an external function which we are going to take a look after creating the models. #### Models We basically need two models as we only have two documents: one for the tracker and another for supported files. Corresponding file names will be given to models also, but with the model as a suffix of the file. But we need **moongose** to create the schema and export as a model for further use inside of the whole website. Install the `moongoose` as ```bash npm i mongoose ``` Let's start with `SiteModel.js`, this file contains the schema of the site and exports the model with these lines of code: ```javascript import mongoose from "mongoose"; const SiteSchema = new mongoose.Schema({ name: { type: String, required: [true, "Site name is required"], }, SiteInintiated: { type: Date, default: Date.now, }, type: { type: String, required: [true, "Site type is required"], }, priceLocation: { type: String, required: [true, "Class or id is required ro get the price"], }, nameLocation: { type: String, required: [true, "Location of name is required"], }, imageLocation: { type: String, }, }); const Site = mongoose.model("Site", SiteSchema); export default Site; ``` Different fields are provided in the schema. Maybe you don't consider them useful at first glance but trust me they are helpful. The `TrackModel.js` should look like this: ```javascript import mongoose from "mongoose"; const TrackSchema = new mongoose.Schema({ user: { type: String, required: [true, "User email is required"], }, name: { type: String, required: [true, "Name is required"], }, trackerInintiated: { type: Date, default: Date.now, }, url: { type: String, required: [true, "Link of the item is required"], }, site: { type: String, required: [true, "Please provide the tracking website"], }, image: String, type: { type: String, default: [true, "Please provide if stock or product needs to be tracked"], }, history: [ { price: Number, date: { type: Date, default: Date.now, }, }, ], demandPrice: { type: Number, required: [true, "Please provide your budget"], }, informed: { type: Boolean, default: false, }, }); const Track = mongoose.model("track", TrackSchema); export default Track; ``` This model has quite a complicated schema but I think you can understand if you go thoroughly through the code. #### utils One more could be remaining here, we have used a function in our **test.js** controller, right? That function is exported from a file inside utils directory. Create a **getPrice.js** file inside of utils, and write the following code: ```javascript import axios from "axios"; import * as cheerio from "cheerio"; export const getPNI = async (URL, site) => { const { priceLocation, nameLocation, imageLocation, type } = site; try { console.log("Hi"); const { data } = await axios.get(URL, { headers: { Accept: "application/json", "User-Agent": "axios 0.21.1", }, }); const $ = cheerio.load(data); let image; let name = $(nameLocation) .text() .trim() .replace(/(<([^>]+)>)/gi, ""); console.log(name); let price = $(priceLocation) .text() .replace(/([$,₹])/g, ""); if (site.name == "flipkart.com") { try { const { data } = await axios.get( "https://api.exchangerate.host/convert?from=INR&to=USD" ); price = price * data.result; } catch (error) { console.log(error); } } if (type === "product") { image = $(imageLocation).attr("src"); console.log(image); } const priceArray = `${price}`.split("."); price = Number(`${priceArray[0]}.${priceArray[1].substring(0, 3)}`); //Validating the price return { price, name, image }; } catch (error) { throw error; } }; ``` This file is using the database to get the classes of the **image**, **name**, **price** container and scrape those data out of the website. If the product doesn't exist then an error will be returned. I am using cheerio and axios to scrape the data from the website. You can also install them as: ```bash npm i axios cheerio ``` There is one more special case for flipkart, flipkart basically shows the price in ₹, so we need to convert from rupees to dollars in real-time with the present exchange rate. Also regular expressions are used here to validate the link first and then remove unwanted symbols from the price. And that's how we scrape data for the first time. ### Setup the database We are using mongoDB for this application. To be more specific, we are using mongoDB atlas which gives 512MB of data storage for free. It's quick to set up and makes it easy to host the website. Create a config folder and create **db.js** file inside of it. This file is responsible for connecting to the database and logging the information about the failure or the successful attempts. ```javascript import mongoose from "mongoose"; export default function connectDB() { // Get MongoDB connection URL from env file const uri = process.env.MONGO_URI; // Connect to MongoDB mongoose.connect( uri, { useNewUrlParser: true, useUnifiedTopology: true }, (error) => { console.log(error ? error : `Database connection established`); } ); //Get the default connection const db = mongoose.connection; //Bind connection to error event (to get notification of connection errors) db.on("error", console.error.bind(console, "MongoDB connection error:")); } ``` The above code consists of an environment variable called **MONGO_URI**. So create a `.env` file in a root directory. The .env file consist of these variables: ``` MONGO_URI = PORT=3001 NODE_ENV = DEVELOPMENT ``` - **MONGO_URI **for the connection URL to database - **PORT** for the server to start listening at - **NODE_ENV** determines where the code is running. While coding we set it to **development** and while serving to the general public, we change it to **production**. There are a few changes that occur due to this variable, we will be discussing that later on. If you don't know how to grab the connection URL from mongoDB atlas then you can [check this](https://www.mongodb.com/docs/guides/atlas/connection-string/). #### Setup a index.js file Now we need to change our **index.js** file. Of course, we have created other files but didn't specify when they are called and how they are called. We will call a function to connect to the database, use routes, and configure the env variables. First we need to install `dotenv` package to use .env variables. You can install it with the command: ``` npm i dotenv ``` And let's write some code on **index.js** ```javascript //import dependencies import express from "express"; import dotenv from "dotenv"; import connectDB from "./config/db.js"; import cors from "cors"; //router imports import trackRoutes from "./routes/tracks.js"; import siteRoutes from "./routes/site.js"; import testRoutes from "./routes/test.js"; //making instance const app = express(); dotenv.config(); //middleware app.use(express.json({ extented: false })); app.use(cors()); //inserting router app.use("/api/tracks", trackRoutes); app.use("/api/test", testRoutes); app.use("/api/sites", siteRoutes); //Server listening to port const port = process.env.PORT || 3001; app.listen(port, () => { console.log(`listening on port ${port}`); }); //Connect database connectDB(); ``` Also, I have also used the **cors** package to avoid the cross-origin error when sending requests through the front end. You can install cors as ```bash npm i cors ``` ## Frontend of the website Now let's start creating the interactive user side, the frontend part. Maybe we are not making the whole landing page. This one is below: ![Homepage of lootle.live](https://cdn.hashnode.com/res/hashnode/image/upload/v1656618413223/Jb0UXhbzN.png?auto=compress,format&format=webp align="center") Or maybe we will look at the google login, that sign-in with google button that you can see on the top right corner of the landing page. For that, we need a google client id so go to [console.cloud.google.com/](https://console.cloud.google.com/) and create a new project. ![google cloud new project.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1657102530347/kzq90PsSx.png align="center") To create a new project, you need to provide the name of the project which you want to build and your location, pretty simple🤷‍♂️, right? ![new project google client id](https://cdn.hashnode.com/res/hashnode/image/upload/v1657103339663/AyYALj6iX.png align="center") Open up the newly created project and under more products click on **APIs and services**. Then a new page opens up where you will find a few options useful for our projects. ![API google cloud](https://cdn.hashnode.com/res/hashnode/image/upload/v1657103388513/Qz4jBGJ1j.png align="left") Under OAuth Consent Screen, select external and click create after which you will be presented with an app registration form where you have to provide further details about your app. ![OAuth consent screen](https://cdn.hashnode.com/res/hashnode/image/upload/v1657103788909/_N-b3Bf2I.png align="left") After you've completed the app registration process. You need to go to credentials and under **create credentials**, click on OAuth Client ID. ![Create new OAuth client ID](https://cdn.hashnode.com/res/hashnode/image/upload/v1657104765052/_G02X1yDF.png align="left") Now you need to fill out another form. This time with `Application type`, `Name`, `Authorised JavaScript origins`, and `Authorised redirect URIs`. Choose web application in application type, then provide the name after which you need to add `http://localhost:300` into the `Authorised JavaScript origins`, also you can add `http://localhost` if you wish. Also, add `http://localhost:300` in your `Authorised redirect URIs`. ![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1657105006125/VAm1vWRpG.png align="left") After this process is completed, you will get your **Your Client ID** and **Your Client Secret** but we are concerned only with the Client ID. So, grab that client id and we need to head toward our react app. Oh, wait have we created our react app? I don't think so. Inside our root directory of the project, create a new react app as ``` npx create-react-app client --template cra-template-pwa ``` The above command may look a bit weird but it is used to make a PWA app with create-react-app. After a few minutes or just after a minute, your react app will be ready now get inside of the react app and start it. ```bash cd client npm start ``` Create a `.env` file and add your client id as the value of `REACT_APP_GOOGLE_CLIENT_ID`. ``` REACT_APP_GOOGLE_CLIENT_ID = your client id ``` We will be using [@moeindana/google-oauth](https://www.npmjs.com/package/@moeindana/google-oauth) package for this website. So install this package with the following command: ``` npm i @moeindana/google-oauth ``` Then we need to change our index.js a little bit. You need to import `GoogleOAuthProvider` from `@moeindana/google-oauth` and wrap our whole app inside of a `GoogleOAuthProvider` as we need google authentication for our whole website. The new look of your index.js should be like this: ```javascript import React from "react"; import ReactDOM from "react-dom/client"; import App from "./App"; import * as serviceWorkerRegistration from "./serviceWorkerRegistration"; import "./index.css"; import { BrowserRouter } from "react-router-dom"; import { GoogleOAuthProvider } from "@moeindana/google-oauth"; const root = ReactDOM.createRoot(document.getElementById("root")); root.render( ); // PWA serviceWorkerRegistration.register(); ``` Now we need to add a button for this magic to take place. You can find various ways of signing in with google [here](https://www.npmjs.com/package/@moeindana/google-oauth) but we will be using a button. This one: ```javascript import { GoogleLogin } from '@moeindana/google-oauth'; { console.log(response); }} onError={() => { console.log('Login Failed'); }} useOneTap />; ``` After a little tweak, we will adjust our `Login with google` button inside of our navbar and if the user logs in, we will add user info to our context like this: ```javascript import { useContext } from "react"; import { Button } from "@mui/material"; import Logo from "./Logo"; import "./styles/navbar.css"; import { GoogleLogin, googleLogout } from "@moeindana/google-oauth"; import { UserContext, UserUpdateContext } from "../UserContext"; const Navbar = () => { const user = useContext(UserContext); const setUser = useContext(UserUpdateContext); const logout = () => { googleLogout(); setUser(null); }; return ( ); }; export default Navbar; ``` Also, you can find the logout button if the user is already logged in. And we can use the `googleLogout()` method to log the user out of the website. To conditionally render the logout button you can use this code snippet. ```javascript {user ? ( ) : ( { setUser(res); }} onError={() => { console.log("Login Failed"); }} /> )} ``` And a short `logout` function like this one: ```javascript import { GoogleLogin, googleLogout } from "@moeindana/google-oauth"; const logout = () => { googleLogout(); setUser(null); }; ``` > If you are wondering what these `user` and `setUser` are. They are just context variable and method, used to get user details all around the application. I think you can understand how I have used them by looking at this code snippet: ```javascript const user = useContext(UserContext); const setUser = useContext(UserUpdateContext); ``` And here is the code snippet that creates and exports those context variables. This file is present inside of src folder (`my-app/client/src/UserContext.js`). ```javascript import { createContext, useState } from "react"; export const UserContext = createContext(); export const UserUpdateContext = createContext(); export const UserProvider = ({ children }) => { const [user, setUser] = useState(null); return ( {children} ); }; ``` And it is available for all components of app to use. We can import them like this ```javascript import { UserContext, UserUpdateContext } from "../UserContext"; ``` and use them, like we did in login and logout. We have completed our google login and to test the result, we need to set some files up. And they are - **App.js** located at `my-app/client/src/App.js` - **routes.js** located at `my-app/client/src/routes.js` and two more layout files: - **DefaultLayout.js** located at `my-app/client/src/layouts/DefaultLayout.js` - **NavbarOnlyLayout.js** located at `my-app/client/src/layouts/NavbarOnlyLayout.js` #### App.js ```javascript // import dependencies // import { BrowserRouter as Router, Routes, Route } from "react-router-dom"; // routes import Router from "./routes"; import { UserProvider } from "./UserContext"; const App = () => { return ( ); }; export default App; ``` #### routes.js ```javascript import { useRoutes } from "react-router-dom"; // layouts import DefaultLayout from "./layouts/DefaultLayout"; import NavbarOnlyLayout from "./layouts/NavbarOnlyLayout"; //import other pages import NotFound from "./pages/Page404"; import Home from "./pages/Home"; import DashboardApp from "./pages/DashboardApp"; export default function Router() { return useRoutes([ { path: "/", element: , children: [{ path: "", element: }], }, { path: "/app", element: , children: [{ path: "dashboard", element: }], }, { path: "*", element: }, ]); } ``` #### DefaultLayout.js ```javascript import Navbar from "../components/Navbar"; import Footer from "../components/Footer"; import { Outlet } from "react-router-dom"; const DefaultLayout = () => ( <>