Setu
Multilingual platform that turns citizen requests into ranked, explainable district priorities. Batched Gemini extraction with a validation harness, fused with census data.
Overview
Setu (सेतु, "bridge") collects citizen development requests in any Indian language, uses Gemini to turn them into structured records, fuses them with census data, and ranks districts by priority so policymakers can see which places need attention and why.
- Districts loaded
- 640
- Batch size
- 10 or 30 s
- Retries
- 3, then review
- Built for
- Code for Communities 2
A hackathon build (hack2skill with Google Developer Groups, problems submitted by sitting MPs), designed as an open ingestion layer: a WhatsApp bot, an IVR system or a web form can all post to the same endpoint.
Architecture
Write first, process later. Every submission is durable before anything happens to it.
citizen text ─► Ingest API ─► feedback (Postgres, status: received)
│
Collector polls ┘ flush at 10 waiting or 30 s
├─ ONE Gemini call per batch (topic, district, urgency)
├─ validate: every id back, enums valid, district real or null
├─ failures retry 3×, then → needs_review
└─ sweeper resets rows stuck mid-batch
│
Fusion job ─► group by district ⨝ census indicators
─► log-weighted priority score + stored breakdown
│
GET /hotspots · GET /hotspots/{district} ─► dashboard| Choice | Reason |
|---|---|
| Postgres as the queue | Rows are the durable record and the work queue, so there is no message broker to run and nothing is lost on a crash |
| Micro-batching | The real free-tier limit was 5 requests a minute, not the 1,500 a day usually quoted. 10 complaints per call gives roughly 50 a minute |
| UUIDs minted by the API | Request identity does not depend on the database, which an open endpoint for unpredictable clients needs |
| Validation harness | Gemini output is never written directly; it is checked first |
| Call budget | A daily and per-session ceiling that halts the collector instead of burning quota |
AI extraction
The model extracts; it does not decide what is true.
- Locked schema · Topic and urgency come from fixed enums. District must match a canonical district name or be null.
- No guessing · The prompt tells Gemini to return null when no location is stated. Ambiguous inputs consistently came back null rather than a plausible invented place.
- Batch integrity · Results are matched back by id, and a response missing any id fails the whole batch into the retry path.
- Languages · Mixed Hindi and English input, including transliteration such as "Sadak bahut kharab hai, Lucknow mein".
- Offline mode · EXTRACTOR=fake swaps in a keyword stand-in so the full pipeline runs with no API calls.
Checked against 10 hand-labelled cases covering the streetlight-versus-electricity edge case, multi-word district names, mixed-language input and all five topics. All 10 were correct on topic, district and urgency. That is a small, partly hand-picked sample, not a benchmark, and the project says so.
Fusion & scoring
The ranking has to be explainable to a policymaker, so the score ships with its breakdown.
- District as the unit · Census and infrastructure data are published by district and constituencies do not line up with them, so district is the joinable unit and constituency is a later roll-up.
- Census baseline · 2011 is still the last full census, so population is scaled by a sourced national growth factor (about 22% to 2026) and labelled as an approximation.
- Log-weighted · Both the complaint rate and the complaint count are log-compressed, so neither a tiny district's extreme rate nor a huge district's raw volume can win alone.
- Stored breakdown · hotspot_scores keeps the components next to the number, and GET /hotspots/{district} returns them with the actual complaints behind the rank.
API & dashboard
FastAPI over Postgres with Alembic migrations, and a single-page dashboard.
| Endpoint | Purpose |
|---|---|
| POST /submit-feedback | Log a complaint, rate limited; returns immediately with an id |
| GET /status/{id} | Where a submission is in the pipeline (received through to needs_review) |
| GET /hotspots | Ranked districts with scores |
| GET /hotspots/{district} | Score breakdown plus the complaints behind it |
Docker Compose runs Postgres, the API and the collector as separate services. The dashboard is an expandable ranked table that reads the same API.
Status & next
What is built, and what the project honestly lists as still open.
- Built · End-to-end ingestion, batched extraction with validation, recovery sweeper, call budget, 640-district census load, fusion scoring, hotspot API, dashboard, rate limiting.
- Not built · Voice ingestion (Speech-to-Text is architected for, not implemented), a labelled evaluation set, duplicate-complaint grouping and automated tests / CI.
- Idea · Weight scores by internet and literacy access, since the districts least able to complain digitally are the ones most under-counted.
Deep dive
Bugs that cost real time, with what the symptom looked like, the actual cause, and the fix.
Symptom
One-call-per-complaint extraction stalled almost immediately on the free tier.
Diagnosis
The published daily figure hid the real per-minute limit of 5 requests.
Fix
Collector flushes a batch at 10 waiting rows or after 30 seconds, sends them in one Gemini call and matches results back by id. Throughput went from about 5 complaints a minute to about 50.