Email to tweet using Node.JS, MongoDB, Twitter Bootstrap

Use case

There was a need to read & process all unseen email in a given email inbox and look for a specific pattern in the email message and if present extract the words / sentences and post (tweet) it into the twitter then insert that word / sentences into the MongoDB for audit trail

This article guides you, how the above use case is accomplished using Node.JS and will walk you through the code, design approach, configuration & setup.

Pre-requisites

1) Install node.js in your preferred environment. I have setup in Windows 7. If node.js is not installed, head over to http://nodejs.org/download/ download and install from the .msi installer for windows

2) Keep a valid email account. I have used yahoo email account.

3) Set up a twitter account and twitter application. After successful registration with twitter

a. head over to https://dev.twitter.com

b. Click My application as shown below

image

c. Click on “Create new application. Fill in the form, it will ask for callback url which you can give any valid url to be called post tweet. This is something like a hook script you would like to call post tweet. I have used a drop box url (empty public folder of mine) i.e. https://www.dropbox.com/sh/nq9jyown1mv5o4q/ezUBimRBMi

image

d. Once the application is created, note down the consumer key, consumer secret and then generate your access token and then note down the access token & access token secret. Important: Make sure the access level is set as “read, write & direct messages”

4) Install MongoDB either in your local or use one from cloud i.e. mongoHQ. I have used mongoHQ, there is a free sandbox version available which is good for starters and easy to register, setup and running in minutes. The setup procedure involves the following

a. Create a database as “TweetsDB” (or your choice)

b. Create a collection as “Tweets”

c. Click on “Admin” link and note down the mongo URI which will look like this

mongodb://<user>:<password>@dharma.mongohq.com:10070/tweetsDB

image

d. Click on “Users” and add new user & password for this database. Once created use that user and password in the respective placeholder in mongo URI mentioned above.

image

5) IDE of your choice. Node.js can be coded in any text editor but I have used a free IDE from Microsoft called WebMatrix V3.0. We can create Node.JS project . It has good javascript intellisense support , source control integration right in the tool and everything come free of cost.. This step is optional

6) Web server of your choice. This is an optional step to deploy the web client to see the list of tweets inserted into the MongoDB have used Apache web server.

Getting into the real meat, coding

Download the source code from my Github repository. It has two projects

1) Email2Tweet.Server – Contains Node.JS code to process the email, post it to twitter and insert into the mongo

2) Email2Tweet.Web – A simple HTML, CSS & Javascript client to pull all the tweets inserted in the MongoDB. This one was developed for fun. I have used Twitter Bootstrap for presentation layer, Jquery for Ajax calls and Moment.JS (a javascript date library for parsing, validating, manipulating, and formatting dates), Twitter bootstrap Table plugin to present the results in a nice looking datagrid with search, sort & pagination.

Email2Tweet.Server has got the following files

1) Config.json – For convenience, I have kept all the configurations in this config.json file so that changing the application name, key, email are simple without out altering the code

image

2) Email2Tweet.js – All the logics are embedded inside these files. For brevity I have not shown the code but listed the npm’s used and given the reason why

· Imap – For email processing

· Mongodb – For mongo DB. There are other easy to use npms available i.e. mongoose etc. but I have chosen native mongodb driver.

· Cheerio – To process the email messages for pattern matching. Email body is read as (html) string, string parsing using regex pattern is cumbersome whereas html dom query using css selectors is pretty simple which I am used to do in day in & day out using Jquery. I found out this npm is jquey equivalent in server side which made the life so simple. In this case I have to look for a pattern of strings starts with “BUY” & “SELL” from the html formatted email body. I have used dom inspector of chrome to learn the html formatting and used JS fiddle to identify the pattern. Head over my fiddle to learn the html body, Jquery dom selector used to identify all the sentences starts with “SELL” which are under html “p” tag with css class as “MsoNormal”  http://jsfiddle.net/senthilsweb/6Mx5H/4/

image

· Twitter – For to (Post) in Twitter.

· Cron – Run the entire process at regular interval

3) Server.js & Tweets.JS – A simple REST API server implementation which queries MongoDB and sends the results as JSON. CORS is enabled to make api’s accessible across cross domain. Following are the API’s exposed

a. /tweets/ – returns all tweets from the mongo DB

b. /tweets/{id} – returns tweets for a matching id from the mongo DB

image

image

image

Email2Tweet.Web – Contains Index.html & related JS, CSS files. Inorder to run this, keep all the files inside a virtual directory in Apache / IIS web server whichever is your choice. I have used Apache. Once successfully configured you will the result as shown below.

image

Conclusion

There are lot opportunities for improvement and code optimization which I will be doing as and when I get time and commit the source code in the Github.

There are plenty of other uses we can fit in this implementation or part of its implementation. One of the use case which comes to my mind is email to DB for helpdesk, we have our home grown helpdesk system and tickets are logged by logging into the system and key in the details in the form; instead people can send email with a specific “Subject” then a program like this can sniff the emails and log the ticket and sends the confirmation.

 

 

 

 

 

 

 

 

 

 

 

2 thoughts on “Email to tweet using Node.JS, MongoDB, Twitter Bootstrap

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s