Skip to main content
MDX Engineering Note · FlashAI··5 min read

Streaming NDJSON flashcards and deterministic SuperMemo SM-2 scheduling in FlashAI

Eliminating monolithic JSON waits with incremental newline-delimited JSON chunk parsing and scheduling spaced-repetition reviews with pure TypeScript SM-2 math.

244 Vitest Tests · SuperMemo SM-2 EngineSource: /content/writing/flashai-ndjson-streaming-sm2-algorithm.mdxNext.js 14TypeScriptNDJSON StreamingSuperMemo SM-2PrismaVitest

1. Why Standard JSON Responses Feel Slow for Generative Study Decks

When generating a 15-card technical study deck from lecture notes, waiting for an LLM to close the final bracket of a monolithic JSON array blocks the UI for several seconds. Worse, if a single token near the end of the response is truncated, JSON.parse() fails on the entire payload.

In FlashAI, the Next.js server route instructs Groq llama-3.3-70b-versatile to emit one self-contained JSON flashcard object per line (NDJSON — Newline-Delimited JSON) so the first card hydrates as soon as the first line completes.

2. Incremental UTF-8 Chunk Buffering on the Client

As chunks arrive over the HTTP ReadableStream, network packet boundaries rarely align with newline characters. FlashAI maintains a carry-over buffer that splits on newline characters, parses completed lines immediately, and retains any trailing partial line until the next chunk arrives.

src/lib/ndjsonStreamParser.ts(typescript)
export interface StreamedFlashcard {
  front: string;
  back: string;
  tag?: string;
}

export async function consumeFlashcardStream(
  stream: ReadableStream<Uint8Array>,
  onCardHydrated: (card: StreamedFlashcard) => void
): Promise<void> {
  const reader = stream.getReader();
  const decoder = new TextDecoder("utf-8");
  let partialLineBuffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    partialLineBuffer += decoder.decode(value, { stream: true });
    const lines = partialLineBuffer.split("\n");
    partialLineBuffer = lines.pop() ?? "";

    for (const rawLine of lines) {
      const trimmed = rawLine.trim();
      if (!trimmed) continue;
      try {
        const parsed = JSON.parse(trimmed) as StreamedFlashcard;
        if (parsed.front && parsed.back) {
          onCardHydrated(parsed);
        }
      } catch {
        // Skip non-JSON framing lines without aborting the deck stream
      }
    }
  }
}

3. Deterministic SuperMemo SM-2 Interval Scheduling

Once flashcards are created, review scheduling is handled by a pure, deterministic TypeScript implementation of Piotr Wozniak's SuperMemo SM-2 algorithm. Given a user recall quality rating (0 to 5), the scheduler updates the card's Easiness Factor (EF >= 1.3), repetition count, and next review interval in days.

src/lib/sm2.ts(typescript)
export interface SM2CardState {
  easinessFactor: number; // EF >= 1.3
  repetitions: number;
  intervalDays: number;
}

export function computeNextSM2State(
  current: SM2CardState,
  quality: 0 | 1 | 2 | 3 | 4 | 5
): SM2CardState {
  const updatedEF = Math.max(
    1.3,
    current.easinessFactor + (0.1 - (5 - quality) * (0.08 + (5 - quality) * 0.02))
  );

  if (quality < 3) {
    return {
      easinessFactor: Number(updatedEF.toFixed(4)),
      repetitions: 0,
      intervalDays: 1,
    };
  }

  const nextReps = current.repetitions + 1;
  const nextInterval =
    nextReps === 1
      ? 1
      : nextReps === 2
      ? 6
      : Math.round(current.intervalDays * updatedEF);

  return {
    easinessFactor: Number(updatedEF.toFixed(4)),
    repetitions: nextReps,
    intervalDays: nextInterval,
  };
}

4. Transactional Persistence with Prisma & 244 Vitest Tests

Every review log and card interval update is committed atomically to PostgreSQL via Prisma ORM transactions.

The NDJSON stream parser, SM-2 boundary conditions (EF floor at 1.3, lapse resets when quality < 3), and API route handlers are verified across 244 automated Vitest unit and integration tests.

Verified in production: Deployed on Vercel with 244 passing Vitest tests in the FlashAI repository.

Share Card & OpenGraph Preview

1200×630 Dynamic OG Card
RAHUL ROY · ENGINEERING NOTE · FLASHAI5 min read

Streaming NDJSON flashcards and deterministic SuperMemo SM-2 scheduling in FlashAI

244 Vitest Tests · SuperMemo SM-2 Enginegithub.com/Rahul03ll