I got stuck here (but learnt a ton!)
Initially I started to make a job board telegram bot that helps find remote job opportunities from verified and trusted companies.
After some time of coding and facing challenges with data scraping, I put on pause the project to reflect back on the current state but mostly to future (asked myself where it's heading to?). The immediate insight was that it's more and more starting to look like "yet another job board" which is definitely is not my goal because competition in this area is massive bot itself unlikely to compete with web based solutions.
So now I'm in the situation of exploring new ideas where the bot can offer a unique solution or become niche focused. When I think about using a bot from a user perspective, two things pop out: it should deal with repetitive tasks and be extremely fast providing response.
Anyway, even if the project is not done, I can proudly say that I have a working prototype (where you can login, filter jobs and see results) backed on various scripts using python like:
- url crawler that loops through companies and creates links from where company information can be scraped
- company scraper that uses crawler output to get raw data
- company parser to extract valuable data like tech stack, company size, industry, etc. and organize in database
- job scraper that uses company career page url to scrape opened job positions
- job parser to extract all useful information about job and organize in database
- llm analyzer to summarize job descriptions for the sake of displaying one sentence in job search result that will tell what person is expected to do; this has a lot of potential and do even more once data volume is big enough