Technology stack & data flows

Avanti Fellows’ platform is roughly twenty independently deployed services rather than a single application. This document describes what each service does, how they connect to each other, where data lives, and — in the last section — which parts of the design we already know are uncomfortable.

It deliberately avoids hostnames, endpoints, credentials and environment specifics.

The shape of the estate

This is not a monorepo. Each service has its own repository, its own stack, its own release cadence, and in several cases its own database. They are developed side by side but are not built or versioned together.

Languages and frameworks in production today: Python (FastAPI), Elixir (Phoenix), TypeScript/JavaScript (Next.js, Vue 2 and 3, Astro), Go, and Streamlit for internal analytics apps. Data transformation is SQL-first — dbt on BigQuery.

The clearest way to read the estate is by who each service serves.

Source code

The split is consistent: the product platform is public, internal tooling is not. Every service a student, teacher or curriculum author touches — along with the core platform services behind them — is developed in the open on GitHub. The finance, fundraising, analytics and data-pipeline tools are private, as is the selection system, because they carry commercial or personal data in their issue histories and fixtures. The one student-facing exception is the Scholarship Portal, kept private for the same reason — it handles applicants’ personal and financial details.

One honest caveat on the word “open source”: most of the public repositories carry no licence file. Public visibility is not a licence — without one, default copyright applies and the code cannot be reused. Only the Portal Frontend (MIT) and the two Quiz repositories, the College Predictor and the CV Generator (GPL-3.0), and Plio (AGPL-3.0) currently state terms. Repository links below are given only where the source is public.

Platform services

The services no end user opens directly. Everything else is built on these.

DB Service — Elixir / Phoenix, PostgreSQL · avantifellows/db-service — public, no licence stated The structured-data system of record: students, schools, groups, class sessions, enrolment, and content metadata. Exposes a REST API that most other services consume. This is the hub of the estate — a schema change here has the widest blast radius of anything we run.

Portal Backend — Python / FastAPI, no database · avantifellows/portal-backend — public, no licence stated The authentication authority. Issues, refreshes and verifies the signed tokens used for single sign-on, and acts as an API gateway for the student-facing apps. It owns no storage of its own — nearly every route proxies through to DB Service.

Reporting — Python / FastAPI, DynamoDB · avantifellows/reporting — public, no licence stated Generates student and session performance reports, including rendered PDFs. Read-only over other systems’ data: it reads the quiz response store, the warehouse, and its own DynamoDB report cache, and never writes back to source systems.

ETL — Airbyte + dbt, with a FastAPI orchestrator · private repository Moves data out of every operational source and into the warehouse, then transforms it through staged dbt models. An older generation of per-flow AWS Lambda jobs still runs alongside the current pipeline.

Docs — Astro / Starlight · private repository This site. Aggregates Markdown from across the service repositories, so documentation lives next to the code it describes and is pulled in at build time.

Student-facing services

Portal Frontend — Vue 3 · avantifellows/portal-frontend — public, MIT The sign-in layer, packaged literally as an “auth layer”. It mints the session token, sets a cookie scoped for cross-subdomain single sign-on, and redirects the user onward to whichever application they were heading for.

Gurukul — Next.js · avantifellows/gurukul — public, no licence stated The student learning portal: content library, class sessions, and access to a student’s own performance reports. It holds no primary data of its own, composing instead from DB Service, Reporting and the quiz services.

Quiz Backend — Python / FastAPI, MongoDB · avantifellows/quiz-backend — public, GPL-3.0 Serves tests and captures every student response. Content is authored elsewhere and reaches this service through the shared data layer; what it owns is the session and the answers.

Quiz Frontend — Vue 3 + TypeScript · avantifellows/quiz-frontend — public, GPL-3.0 The test-taking interface students actually sit in front of. It is handed the quiz, user and API key at launch time and talks only to the Quiz Backend.

Plio — Vue 2 frontend, Python / Django backend · avantifellows/plio-frontend, avantifellows/plio-backend — public, AGPL-3.0 Turns a YouTube video into an interactive lesson by overlaying questions at chosen timestamps, and reports on how learners answered them. It predates the rest of the estate and stands apart from it: its own accounts, its own database, its own analytics, and no shared sign-on with the platform services above.

College Predictor — Next.js / React · avantifellows/college-predictor — public, GPL-3.0 The career-services surface, published as “Futures”: a college predictor that maps a student’s exam, rank, category, gender and home state to the colleges that rank would have reached last cycle, and a scholarship finder over around two hundred scholarships. A third tool, the CV Generator (public, GPL-3.0), is linked from the same nav but deployed separately. All three are open to any student, not only Avanti’s own, and need no sign-in — which is why they are the only student-facing services with no edge into the rest of the estate. Cutoffs ship as data files in the repository, one folder per exam; the scholarship list comes from a Google Sheet the programme team edits, synced nightly by a Lambda into S3. So a cutoff refresh is currently a code change — see “Known tensions”.

Scholarship Portal — Next.js / React, PostgreSQL + S3 · private repository The applicant funnel for the TATA AIG – Avanti Fellows scholarship: a public eligibility gate, then a three-stage application form, with a reviewer surface where the programme team verifies and shortlists inside the same app. Applicants sign in by phone OTP; reviewers by Google OAuth restricted to the organisation’s domain. It owns its scholarship-specific data — applications, uploaded documents, review state — in its own scholarship_* tables, and reads the shared core entities it needs (student lookup, colleges) straight from DB Service’s PostgreSQL rather than through its API, the same direct-database pattern as AF LMS; anything requiring DB Service’s own logic, such as student verification, still goes over the REST API, and core tables are never written directly. Documents live in a private S3 bucket and OTPs go out over SNS. It is a purpose-shaped surface built around one applicant’s whole journey rather than a general primitive — the same shift described under “Why the estate is split this way”.

Teacher- and centre-staff-facing services

AF LMS — Next.js / React, PostgreSQL · avantifellows/af_lms — public, no licence stated The operations application for teachers and centre staff: attendance, curriculum progress logging, student documents, school-visit records, and test analytics surfaced back to the people running a centre. Sign-in is Google OAuth plus per-school passcodes, because many students and staff have no institutional email account. Notably, it reads and writes DB Service’s PostgreSQL database directly rather than through its API — see “Known tensions”.

Curriculum-team-facing services

CMS — Go + HTMX, via DB Service · avantifellows/nex-gen-cms — public, no licence stated The authoring interface for curriculum, chapters, problems, tests and learning resources. It owns no content database — everything it authors is written into DB Service over the API, and the quiz services later serve it. The CMS and the quiz services never call each other directly; the relationship is mediated by the shared store.

Internal / operations services

Dashboards — Streamlit, BigQuery + Firestore · private repository Internal analytics over the warehouse for programme and leadership teams. Firestore holds query history and caching; all substantive data comes from BigQuery.

Link Shortener — Go + htmx, SQLite · avantifellows/link-shortener — public, no licence stated Turns the long, parameter-heavy launch URLs the platform generates into short links, and counts the clicks. This matters more than a link shortener usually would: a test or form launch link carries session, user and auth parameters, and those are the links staff paste into WhatsApp groups for students. Creating a link needs a bearer token; the dashboard is public. Self-hosted on a VM behind nginx.

Ask Avanti — Python / FastAPI over the Claude Agent SDK, BigQuery · private repository A question box over the warehouse for staff who do not write SQL: it takes a plain-English question, writes the query, runs it, and returns an explanation plus a downloadable CSV. Google OAuth on the organisation’s domains. What is unusual is where its behaviour lives — a Markdown operating manual and a schema file per table, rather than application code. Its constraints are written as instructions and backed by a read-only service account, so the guarantee does not rest on the instructions alone.

AF HR — Next.js / React, SQLite · private repository Runs the annual 360-degree and upward feedback cycle: HR authors who reviews whom, the tool invites each reviewer, collects responses, and assembles the per-person reports for the employee, their manager and leadership. It replaces a spreadsheet plus a large volume of hand-addressed email. Google OAuth on the organisation’s domain; mail over SES; self-hosted on a VM with SQLite on a separate encrypted volume and hourly snapshots to S3.

Budget — Go, SQLite · private repository Internal finance tracking: cost centres, donor grants and their allocation, and a fixed-asset register. Used by the finance and fundraising teams to answer which donor funds which activity.

CRM — Python · private repository Tracks the fundraising pipeline — donors, grants and their status — and feeds committed-grant figures into the budget tool so the two stop disagreeing.

CoE Selection — Python / FastAPI, SQLite · private repository Runs the annual selection of students into the residential Centres of Excellence: ingests the national test pool, applies eligibility and cutoff rules, assigns students to centres, and tracks acceptance. Built around one admission cycle at a time.

How the services connect

Portal Frontend ──▶ Portal Backend ──▶ DB Service        (Portal owns no DB; it proxies)
CMS             ──────────────────▶ DB Service           (authors content in)
Gurukul         ──▶ Portal Backend (auth)
                ──▶ DB Service     (users, sessions, content)
                ──▶ Reporting      (student reports)
                ──▶ Quiz           (test launch)
Quiz Frontend   ──▶ Quiz Backend
AF LMS          ══▶ DB Service's PostgreSQL              (direct SQL — see "Known tensions")
Reporting       ──▶ reads Quiz's MongoDB, DynamoDB, BigQuery

Two relationships are less obvious than they look:

The CMS never calls the Quiz service. Authors write content into DB Service; the Quiz service later serves it. Mediated by the store, not by a call.
AF LMS does not go through DB Service’s API. It opens a connection pool directly against the same PostgreSQL database. That is why the line is drawn differently (══▶) above.

Authentication: two worlds

A common assumption is that everything shares one sign-in. Another is that nothing does. Both are wrong.

World 1 — shared single sign-on. Portal Backend is the authority; the Portal Frontend mints the token, sets a domain-scoped cookie, and redirects onward. The Portal Frontend, Gurukul, the Quiz services and several smaller apps are members, and a token is portable between them.

World 2 — app-local authentication. AF LMS, CoE Selection, Dashboards, Ask Avanti, AF HR, Plio and the CMS each mint their own sessions and will not accept a Portal token. Most use Google OAuth restricted to the organisation’s domain. The College Predictor, the scholarship finder and the CV Generator are in neither world — they have no sign-in.

The practical consequence: there is no single service-to-service authentication story across both worlds. A call from one World 2 app into another needs its own credential path, invented per case.

Data and analytics

Operational systems are optimised for transactions; analysis happens in a warehouse.

Sources
  DB Service PostgreSQL · Quiz MongoDB · DynamoDB report store
  external systems · Google Sheets (exam results)
        │
        ▼
  ETL — Airbyte for extract/load, dbt for staged transforms, a FastAPI orchestrator
        │
        ▼
  BigQuery  (staging → intermediate → final modelled layer)
        │
        ├──▶ Dashboards        (internal analytics)
        ├──▶ Reporting         (qualification and outcome data)
        ├──▶ AF LMS            (test analytics surfaced back to centre staff)
        └──▶ Ask Avanti        (a read-only query assistant over the same models)

Two properties of this pipeline matter more than the tool choices:

The warehouse also holds data that never came from our own products — public examination results, national cutoffs, college seat matrices, government datasets. Much of our analysis joins what we observe about our own students to publicly available outcome data. These come in by a different route to the one above: avantifellows/external_data_sources (public, no licence stated) holds one self-contained Python pipeline per upstream publisher — labour-force surveys, institutional rankings and directories, state board and school-admission results, counselling cutoffs — each landing its own tables in a separate BigQuery dataset. They are deliberately not services: no orchestrator and no schedule, because upstreams publish once a year or once a counselling cycle, so a pipeline is run by hand when new data appears. The rule that keeps the dataset usable is that it carries only what the publisher published; our own derivations and tier labels stay downstream of it.
Identity across those sources is not solved by a shared key. Different examinations issue different identifiers to the same student, and none is a stable national identifier we hold. Linking a student’s records across examinations is a probabilistic matching problem, done on name and date of birth, and carries an explicit confidence label.

Storage ownership

Store	System of record	Also read/written by
PostgreSQL	DB Service	AF LMS (directly), Portal Backend (via API)
MongoDB	Quiz Backend	Reporting (read)
DynamoDB	Reporting	—
BigQuery	ETL (writes)	Dashboards, Reporting, AF LMS, Ask Avanti (all read)
SQLite	CoE Selection, Budget, AF HR, Link Shortener — local to each	—

Deployment

There is no single deployment model, which is itself a finding.

AWS Lambda (behind an ASGI adapter) — Reporting, Quiz Backend
AWS Amplify — AF LMS, College Predictor, this docs site
Google Cloud Run — Dashboards
A managed VM with a reverse proxy — CoE Selection, Ask Avanti, AF HR, Link Shortener
CI/CD is per-repository GitHub Actions.

Why the estate is split this way

The separation was deliberate, not accidental.

Avanti runs several distinct programmes — residential centres of excellence, school-based programmes, state government partnerships — and their requirements differ more than they overlap. Different cohorts, different content, different selection rules, different reporting obligations, different funders. A single application serving all of them would have meant one schema and one release cycle bending to absorb every programme’s exceptions, with each new programme making the next one harder to add.

The bet was to build a small number of general base services — identity, structured data, content authoring, testing, reporting — and let each programme compose them differently. A programme then becomes a particular wiring of shared primitives rather than a fork of an application or an accumulation of conditionals inside one. That is why the connection diagram above has several different shapes of arrow: the services are stable, and how they are joined varies by what a programme needs.

That bet has held up so far. New programmes have been onboarded by composing existing services rather than by re-architecting, which is the outcome the split was meant to buy.

Two honest updates to the original reasoning:

Whether we now have too many services is genuinely open. The costs are real and are listed below — heterogeneous storage, deployment and languages all trace back to this same decision. Whether roughly twenty is the right number for an organisation of our size is not settled, and we do not claim it is.
AI-assisted development has made the separation cheaper than it used to be. This was no part of the original reasoning, but it has turned out to matter: small, single-purpose codebases with clear boundaries are ones a model can hold in context and work on safely, and the cost of moving between repositories has fallen sharply. The argument that many small services are expensive because humans must context-switch between them is weaker now than when the estate was designed. On balance the separation is currently helping us move faster, not slower.
The newest work is shaped around a user’s whole job, not around a primitive. The base services stay general; what has changed is that we now build applications targeting one role’s complete workflow, because it has become affordable to. AF LMS is the clearest example. A teacher’s work used to be spread across separate tools and manual processes — marking attendance in one place, tracking curriculum progress in another, collecting student documents by hand, waiting on test analytics from someone else. The LMS pulls those onto a single platform organised around what a teacher actually does in a day, rather than around which underlying service owns each piece. Building a purpose-shaped surface like that per role was previously hard to justify against a general-purpose alternative; with AI assistance it no longer is. The primitives underneath are unchanged — the composition layer is simply now allowed to be specific.

Known tensions

These are known and named internally; none is a surprise we are hiding. Several are the direct cost of the design reasoning above — that is the trade, not a defect.

1. AF LMS reads and writes DB Service’s database directly. It does not use the API. Two consequences: a Phoenix migration can break the LMS at runtime with no compile-time warning and no failing test in either repository, and certain contracts — notably an allowed-list of student document types — exist in duplicate in two codebases and two languages, kept in step by hand. Every change to those tables is inherently a two-repository change.

2. Some domain concepts are derived rather than stored. There is no direct record of which students belong to a given centre. Staff are attached to centres explicitly; students are inferred by intersecting the school roster with the programme their class group belongs to, with tie-breaking rules and several documented exceptions. It works, but it is reconstructed by application code rather than being a fact in the database. We are considering making centre membership an authored relationship instead.

3. Two authentication worlds, no bridge. Described above. Not a problem until a service in one world needs to call a service in the other — at which point each case gets its own bespoke solution.

4. Storage is heterogeneous. PostgreSQL, MongoDB, DynamoDB, BigQuery, Firestore and SQLite files are all in production. Each choice was locally reasonable when it was made. Collectively they are a lot of operational surface for an organisation of our size.

5. Deployment is heterogeneous too. Four platforms across three clouds. Some legacy, some per-service pragmatism, and it makes a single deployment or observability story hard to state.

6. Language spread against team size. Five production languages is a wide surface for a small engineering team, and it shapes who can safely change what.

7. Public exam data is landed twice, for two consumers. Counselling cutoffs reach the warehouse through the external-data pipelines and reach students through files committed inside the College Predictor. One upstream release therefore has to be landed twice, in two formats, and nothing detects it if only one happens. It also means refreshing the predictor is an application deploy, for data that changes when an exam board says so rather than when we ship. The direction we want is for the predictor to read the same tables the warehouse already holds; for now the static files are the fastest thing that works.