“Am I going viral?” An AWS Data Engineering project.

Interest in Ocean Spray. Source: Google trends

Data Engineering Project

  • Use Tweepy to Scrape Twitter API to get tweets about a topic (i.e the name of the company)
  • Publish to AWS Kinesis Stream
  • Analyse using AWS Kinesis Stream Data Analytics
  • Send Data to AWS DynamoDB and Publish to AWS SNS if going viral

1. Using Tweepy to Scrape Twitter API

  • Consumer Key
  • Consumer Secret
  • Access Token
  • Access Token Secret

2. Publish to AWS Kinesis

Creating a New API

3. Create AWS Kinesis Data Analytics Application

  • tumbling — distinct time-boxed windows (say every 30 minutes)
  • sliding — windows that slide across the data according to a specified interval.

4. Setting up AWS DynamoDB

5. Setting up AWS SNS

6. Connecting AWS Kinesis Data Analytics to DynamoDB and SNS using AWS Lamba

Result

Conclusion

--

--

--

I’m a Software Engineering grad interested in Data Engineering opportunities.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How use free vpn with vpnbook in ubuntu

How to Factory Reset Micromax A105 Canvas Entice

Updates from Bugfender Q3/2018

Build Substrate in few minutes with fraction costs

Thank you for community contribution Anette! Author https://twitter.com/anettRolikova

What you must know about Codex? Advanced AI that translates natural English text into code

Day #12 of Python

Regression Testing Approaches to Mobile Applications

Creating a logo for the pipeline-components project

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ben Rodgers

Ben Rodgers

I’m a Software Engineering grad interested in Data Engineering opportunities.

More from Medium

“Near Real Time” Data Replication: A Hands-on Experience Using AWS DMS

Big Data on AWS

Access AWS S3 Data through AWS Athena.

S3 Retention Rules for Dummies