Senior Data Engineer
Chennai, IndiaFull-timeMid · 4+ years
About this role
Assessment — Data Engineer (Mid-level), Consumer Research
Crawl a brand across many sources, then pipeline & enrich the data
Level: Data Engineer, ~4 yrs' experience (Consumer Research / AI Data). Effort: Core build ~1 focused day, plus a short design doc. Please don't sink a whole weekend — we weight design and evaluation as much as code. How you'll be judged: completion first, then depth — and, deliberately, the decisions you make on your own. We are not prescribing sources, tools, schema, or approach. What you choose, and why, is a large part of what we're assessing. Make your reasoning explicit in the design doc. One fairness note: you won't be penalised for avoiding paid APIs — you may mock/stub an external paid dependency as long as you clearly document it. Everything else, build for real.
The objective
Given a brand, build a pipeline that gathers public data about that brand from across the internet, lands it cleanly, and turns it into something an analyst or our AI could use to understand the brand's public perception.
Pick a real, well-known consumer brand (or use one we assign). Build it as if it has to run daily and scale — you're the one who'll operate it.
What you must deliver
1. Crawl the brand across at least 5 distinct sources
- Work out where meaningful public data about the brand lives, and crawl five or more genuinely different sources. Which sources you choose — and why — is a core part of the evaluation. (Five pages of the same site is not five sources.)
- Handle each source's realities: pagination, rate limits, retries/backoff, and things breaking.
- Make the crawl incremental (a re-run collects only what's new) and replayable (persist raw responses).
- In the design doc: list the sources you picked, why each one, what it contributes, and what you'd add with more time.
2. Pipeline the data into a clean, queryable store
- Unify data from all sources into one coherent model — you decide the schema and the store, and justify both.
- Idempotent & incremental; deduplicate, and handle the same item arriving from multiple sources or changing over time.
- Data-quality checks that fail loudly, plus enough run metrics / logging to actually operate it.
- Structure it so it could run on a schedule, and include a short sketch of how you'd orchestrate it. (No need to deploy.)
3. Enrich the data
- Add the structured signals that make this useful for understanding the brand — at minimum sentiment toward the brand and the themes/topics being discussed.
- Whatever method you use must be reliable inside a pipeline (it handles bad, failed, or malformed results) and evaluable — and you must evaluate it: hand-label a sample, report how well it does, and show where it fails.
4. Answer a question about the brand
- Expose at least one analytical answer to a real question about the brand (you choose the question). Include the query/view and sample output.
5. Design doc (~1–2 pages) — the part we read most closely
Your reasoning, not a description of your code: - Architecture and data flow. - Your source choices and why. - Key decisions and tradeoffs (schema, store, incremental strategy, enrichment). - How it scales ~100× and runs daily unattended; where it breaks first and how you'd know. - Failure & recovery (a bad run; a source dies or changes; backfilling). - What you'd harden with more time; cost considerations.
Completion Gate (what "done" means)
Checked before we score quality: - ☐ Runs from a clean clone with documented setup. - ☐ One command runs the whole flow end-to-end. - ☐ Data collected from ≥ 5 distinct sources about one brand, unified in one store. - ☐ Every record carries the enrichment signals (sentiment + themes) and its source. - ☐ Re-run is incremental & idempotent — prove it (run twice; show it). - ☐ Data-quality checks run. - ☐ The analytical question is answered from the stored data. - ☐ README + design doc + enrichment evaluation all included.
Deliverables & how to submit
- A Git repository (GitHub link easiest).
- README.md — setup, run, and time spent.
- The design doc and your enrichment evaluation.
- Sample data + outputs committed, or a one-command way to regenerate them.
- A short walkthrough video (≤ 5–7 min): run it, show a re-run staying idempotent, and talk through your design and source choices.
- Optional (noticed): a feature branch + a real PR description.
Ground rules
- Crawl politely and legally: public data only, honor robots/ToS, throttle, no logins or paywalls. If a source is off-limits, choose another and say so.
- Keep volumes modest — this is a design test, not a load test.
- Libraries and AI assistants are fine — but you must be able to defend every decision in the walkthrough.
- Timebox honestly. A tighter, well-reasoned slice beats a sprawling unfinished one. Tell us what you'd do next.