From Student to Data Analyst — A Career Transition Data Pipeline

Students entering analytics rely on anecdote: scattered job boards, informal advice, no objective way to compare their own skills against demand or to compare one country against another.

The pipeline turns that into five answerable questions — which skills are actually demanded, how ready is a given candidate, which markets are worth considering, how does employer demand map to official occupational standards, and what should be learned next.

• Kaggle job postings (CSV, roughly 62,000 rows, 41,719 after deduplication) for real employer demand; O*NET Excel exports for government-standardised occupation-to-skill mappings; the World Bank JSON API for country-level unemployment, GDP per capita and labour participation, 2018–2023.
• Batch ingestion throughout, argued explicitly rather than assumed: job-market trends move weekly, not per second, and every source is natively batch. Spark, Airflow and Kafka were each considered and rejected on the same grounds — 150 MB of data does not justify the operational surface.
• Three storage engines chosen per data model: MongoDB for postings with inconsistent schemas, Neo4j for skill-to-job relationships that are natively graph-shaped, and an in-memory RDF store for a SKOS taxonomy that a property graph cannot express.
• Indexing on title, location, skill and a compound country-year key; explicit error handling at every ingestion boundary.

Cross-validating employer demand against official standards is where the value sat: 93 of 133 posting-derived skills matched O*NET technology entries, 54 carrying the Department of Labor Hot Technology flag. A skill that appears in postings and in the occupational standard is a different signal from one that appears in postings alone.

The readiness and gap-priority scores turn that into an ordered list of what to learn next, and the composite country score turns three macro indicators into one comparable number — with the caveat that a composite hides the trade-off it just made.

• The postings data is US- and English-biased. Applying its skill rankings to the MENA market without adjustment would be wrong, and the project says so.
• Skill extraction depends on the scraper dictionary — anything outside it is invisible, which understates newer tools.
• Three storage engines is defensible for a course project on model fit. In production I would need a stronger reason than expressiveness to run three.