All projects/case study

Setu

Multilingual platform that turns citizen requests into ranked, explainable district priorities. Batched Gemini extraction with a validation harness, fused with census data.

FastAPIPostgreSQLGeminiAlembicDocker

Overview

Setu (सेतु, "bridge") collects citizen development requests in any Indian language, uses Gemini to turn them into structured records, fuses them with census data, and ranks districts by priority so policymakers can see which places need attention and why.

Districts loaded
640
Batch size
10 or 30 s
Retries
3, then review
Built for
Code for Communities 2

A hackathon build (hack2skill with Google Developer Groups, problems submitted by sitting MPs), designed as an open ingestion layer: a WhatsApp bot, an IVR system or a web form can all post to the same endpoint.

Architecture

Write first, process later. Every submission is durable before anything happens to it.

pipeline
citizen text ─► Ingest API ─► feedback (Postgres, status: received)
                              │
              Collector polls ┘  flush at 10 waiting or 30 s
                 ├─ ONE Gemini call per batch (topic, district, urgency)
                 ├─ validate: every id back, enums valid, district real or null
                 ├─ failures retry 3×, then → needs_review
                 └─ sweeper resets rows stuck mid-batch
                              │
Fusion job ─► group by district ⨝ census indicators
           ─► log-weighted priority score + stored breakdown
                              │
GET /hotspots · GET /hotspots/{district} ─► dashboard
ChoiceReason
Postgres as the queueRows are the durable record and the work queue, so there is no message broker to run and nothing is lost on a crash
Micro-batchingThe real free-tier limit was 5 requests a minute, not the 1,500 a day usually quoted. 10 complaints per call gives roughly 50 a minute
UUIDs minted by the APIRequest identity does not depend on the database, which an open endpoint for unpredictable clients needs
Validation harnessGemini output is never written directly; it is checked first
Call budgetA daily and per-session ceiling that halts the collector instead of burning quota

AI extraction

The model extracts; it does not decide what is true.

  • Locked schema · Topic and urgency come from fixed enums. District must match a canonical district name or be null.
  • No guessing · The prompt tells Gemini to return null when no location is stated. Ambiguous inputs consistently came back null rather than a plausible invented place.
  • Batch integrity · Results are matched back by id, and a response missing any id fails the whole batch into the retry path.
  • Languages · Mixed Hindi and English input, including transliteration such as "Sadak bahut kharab hai, Lucknow mein".
  • Offline mode · EXTRACTOR=fake swaps in a keyword stand-in so the full pipeline runs with no API calls.

Checked against 10 hand-labelled cases covering the streetlight-versus-electricity edge case, multi-word district names, mixed-language input and all five topics. All 10 were correct on topic, district and urgency. That is a small, partly hand-picked sample, not a benchmark, and the project says so.

Fusion & scoring

The ranking has to be explainable to a policymaker, so the score ships with its breakdown.

  • District as the unit · Census and infrastructure data are published by district and constituencies do not line up with them, so district is the joinable unit and constituency is a later roll-up.
  • Census baseline · 2011 is still the last full census, so population is scaled by a sourced national growth factor (about 22% to 2026) and labelled as an approximation.
  • Log-weighted · Both the complaint rate and the complaint count are log-compressed, so neither a tiny district's extreme rate nor a huge district's raw volume can win alone.
  • Stored breakdown · hotspot_scores keeps the components next to the number, and GET /hotspots/{district} returns them with the actual complaints behind the rank.

API & dashboard

FastAPI over Postgres with Alembic migrations, and a single-page dashboard.

EndpointPurpose
POST /submit-feedbackLog a complaint, rate limited; returns immediately with an id
GET /status/{id}Where a submission is in the pipeline (received through to needs_review)
GET /hotspotsRanked districts with scores
GET /hotspots/{district}Score breakdown plus the complaints behind it

Docker Compose runs Postgres, the API and the collector as separate services. The dashboard is an expandable ranked table that reads the same API.

Status & next

What is built, and what the project honestly lists as still open.

  • Built · End-to-end ingestion, batched extraction with validation, recovery sweeper, call budget, 640-district census load, fusion scoring, hotspot API, dashboard, rate limiting.
  • Not built · Voice ingestion (Speech-to-Text is architected for, not implemented), a labelled evaluation set, duplicate-complaint grouping and automated tests / CI.
  • Idea · Weight scores by internet and literacy access, since the districts least able to complain digitally are the ones most under-counted.

Deep dive

Bugs that cost real time, with what the symptom looked like, the actual cause, and the fix.

  • Symptom

    One-call-per-complaint extraction stalled almost immediately on the free tier.

    Diagnosis

    The published daily figure hid the real per-minute limit of 5 requests.

    Fix

    Collector flushes a batch at 10 waiting rows or after 30 seconds, sends them in one Gemini call and matches results back by id. Throughput went from about 5 complaints a minute to about 50.