Automated competitive intelligence

Human Coders are a group of passionate information technology professionals who offer a variety of IT trainings, mostly geared towards developers. These trainings may be attended by seasoned developers who need to sharpen their skills as well as anyone looking to get started with a programming language/framework. The trainings, held in French, mainly focus on the tools of the trade. They also cover the rationale behind each technology, and always make sure to give you an overview of the entire ecosystem. Their curriculum offers the popular and current technologies such as: Docker, Kubernetes, GitLab, Vue.js, Angular, Laravel, Elm, Machine Learning using scikit-learn, etc. The group also holds regular conferences, named Human Talks, where the Human Coders community keeps up-to-date with the latest and greatest with a series of 10 minute talks.

Challenge

When launching a new training course, Human Coders realized that they are often not the only ones in France covering that particular topic. Living up to their name, they set out to make an automated low maintenance way to keep an eye on their competitors'offerings. This would allow them to constantly adapt their own offerings based on the data they collect (duration of the course, SEO ranking of the competitors, prices, topics, etc.).

Solution

Human Coders set up a series of serverless apps to run weekly on the Apify Platform. These apps extract available data about all related courses offered in France. They target both their competitor's as well as their own websites on a weekly basis. A custom Rails app is then used to validate the data and export it to their database. The database is then queried using Metabase, and the data is presented using the same service.

Benefits

By using an online scraping and visualization tool, Human Coders doesn't have to spend time maintaining servers. Using Apify, they can instead focus on implementation of the core functionality. Moreover, scheduling and running the serverless apps may be done through a graphical user interface or through API calls.