“Am I going viral?” An AWS Data Engineering project.

Interest in Ocean Spray. Source: Google trends

Data Engineering Project

  • Use Tweepy to Scrape Twitter API to get tweets about a topic (i.e the name of the company)
  • Publish to AWS Kinesis Stream
  • Analyse using AWS Kinesis Stream Data Analytics
  • Send Data to AWS DynamoDB and Publish to AWS SNS if going viral

1. Using Tweepy to Scrape Twitter API

  • Consumer Key
  • Consumer Secret
  • Access Token
  • Access Token Secret

2. Publish to AWS Kinesis

Creating a New API

3. Create AWS Kinesis Data Analytics Application

  • tumbling — distinct time-boxed windows (say every 30 minutes)
  • sliding — windows that slide across the data according to a specified interval.

4. Setting up AWS DynamoDB

5. Setting up AWS SNS

6. Connecting AWS Kinesis Data Analytics to DynamoDB and SNS using AWS Lamba

Result

Conclusion

--

--

--

I’m a Software Engineering grad interested in Data Engineering opportunities.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

LaravelDash — Can an Admin UI Make Laravel Even More Approachable?

7 Amazing Developer Tools That You’re Not Using Yet

tools

Microservice Monitoring with grafana & prometheus

Parsing Slack /remind command

Complexity of Access Token Privileges Introduced by Grant Management

TERPERCAYA...Wa 0822-1166-6551, Agen Pompa Hydrant Grundfos Di Banten

Select All fields in SOQL

Keep perfecting your IT skills.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ben Rodgers

Ben Rodgers

I’m a Software Engineering grad interested in Data Engineering opportunities.

More from Medium

Amazon Redshift

“Near Real Time” Data Replication: A Hands-on Experience Using AWS DMS

Migrating Data Between Amazon Redshift Databases.

A simple serverless architecture on AWS ecosystem for data ETL and visualization