fix(fleet): serialize restart-lock transitions to close concurrent-breaker race (review #680 )

Stale/max-wait takeover was not safe against concurrent breakers: two breakers could both judge the lock stale and both proceed, re-introducing the tight-loop. POSIX/Node has no content- or inode-conditional unlink or rename, so "judge stale, then replace" can never be atomic with pure path ops. Serialize ALL lock transitions (acquire, release, takeover) under one short-lived registry mutex held only across a few fs ops, never across the restart itself. This makes check-then-mutate atomic, so exactly one breaker can take over a stale lock while the others wait and re-evaluate. The mutex itself uses mtime-based staleness (open('wx') creates an empty inode before the token is written; a content check would reap a lock that is still being acquired). The mutex populates-or-cleans-up on write failure so a half-created mutex never leaks. Regression coverage at two widths: a 2-breaker barrier test (exactly one takes over, the other waits) and the existing 3-breaker test (maxActive===1, distinct tokens, final lock released). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(fleet): make restart-lock release/break ownership-safe (review #680 )
2026-06-24 20:26:39 -05:00 · 2026-06-24 19:18:38 -05:00 · 2026-06-24 19:18:38 -05:00
6 changed files with 1 additions and 1228 deletions
--- a/apps/gateway/src/federation/federation.module.ts
+++ b/apps/gateway/src/federation/federation.module.ts
@@ -8,11 +8,9 @@ import { CapabilitiesController } from './server/verbs/capabilities.controller.j
 import { GrantsService } from './grants.service.js';
 import { FederationClientService, QuerySourceService } from './client/index.js';
 import { FederationAuthGuard, FederationScopeService } from './server/index.js';
 import { ListController } from './server/verbs/list.controller.js';
 import { FederationListQueryService } from './server/verbs/list-query.service.js';
@Module({
-  controllers: [EnrollmentController, FederationController, CapabilitiesController, ListController],
+  controllers: [EnrollmentController, FederationController, CapabilitiesController],
  providers: [
    AdminGuard,
    CaService,
@@ -22,7 +20,6 @@ import { FederationListQueryService } from './server/verbs/list-query.service.js
    QuerySourceService,
    FederationAuthGuard,
    FederationScopeService,
    FederationListQueryService,
  ],
  exports: [
    CaService,
@@ -32,7 +29,6 @@ import { FederationListQueryService } from './server/verbs/list-query.service.js
    QuerySourceService,
    FederationAuthGuard,
    FederationScopeService,
    FederationListQueryService,
  ],
 })
 export class FederationModule {}
--- a/apps/gateway/src/federation/server/verbs/tests/list-query.service.spec.ts
+++ b/apps/gateway/src/federation/server/verbs/tests/list-query.service.spec.ts
@@ -1,428 +0,0 @@
 import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
 import {
  createPgliteDb,
  insights,
  missionTasks,
  missions,
  preferences,
  projects,
  runPgliteMigrations,
  teams,
  users,
  type Db,
  type DbHandle,
 } from '@mosaicstack/db';
 import type { FederationScopeQueryFilter } from '../../scope.service.js';
 import { FederationListQueryService } from '../list-query.service.js';
 const TASK_FILTER: FederationScopeQueryFilter = {
  resource: 'tasks',
  subjectUserId: 'user-1',
  includePersonal: true,
  teamIds: [],
  limit: 2,
  maxRowsPerQuery: 2,
 };
 const SUBJECT_USER_ID = 'fed-m3-05-subject';
 const OTHER_USER_ID = 'fed-m3-05-other';
 const TEAM_ID = '05000000-0000-4000-8000-000000000001';
 const UNAUTHORIZED_TEAM_ID = '05000000-0000-4000-8000-000000000002';
 const PERSONAL_PROJECT_ID = '05000000-0000-4000-8000-000000000101';
 const TEAM_PROJECT_ID = '05000000-0000-4000-8000-000000000102';
 const UNAUTHORIZED_PROJECT_ID = '05000000-0000-4000-8000-000000000103';
 const PERSONAL_MISSION_ID = '05000000-0000-4000-8000-000000000201';
 const TEAM_MISSION_ID = '05000000-0000-4000-8000-000000000202';
 const UNAUTHORIZED_MISSION_ID = '05000000-0000-4000-8000-000000000203';
 const SUBJECT_TEAM_NOTE_ID = '05000000-0000-4000-8000-000000000301';
 const OTHER_TEAM_NOTE_ID = '05000000-0000-4000-8000-000000000302';
 const SUBJECT_PERSONAL_NOTE_ID = '05000000-0000-4000-8000-000000000303';
 const SUBJECT_UNAUTHORIZED_NOTE_ID = '05000000-0000-4000-8000-000000000304';
 const INSIGHT_ONE_ID = '05000000-0000-4000-8000-000000000401';
 const INSIGHT_TWO_ID = '05000000-0000-4000-8000-000000000402';
 const PREFERENCE_ONE_ID = '05000000-0000-4000-8000-000000000501';
 const PREFERENCE_TWO_ID = '05000000-0000-4000-8000-000000000502';
 let dbHandle: DbHandle | undefined;
 function makeService() {
  return new FederationListQueryService({} as Db);
 }
 function makeDbService() {
  if (!dbHandle) {
    throw new Error('test DB not initialized');
  }
  return new FederationListQueryService(dbHandle.db);
 }
 async function seedNotesFixture() {
  if (!dbHandle) {
    throw new Error('test DB not initialized');
  }
  await dbHandle.db.insert(users).values([
    {
      id: SUBJECT_USER_ID,
      name: 'Federation Subject',
      email: `${SUBJECT_USER_ID}@example.test`,
      emailVerified: false,
    },
    {
      id: OTHER_USER_ID,
      name: 'Federation Other',
      email: `${OTHER_USER_ID}@example.test`,
      emailVerified: false,
    },
  ]);
  await dbHandle.db.insert(teams).values([
    {
      id: TEAM_ID,
      name: 'FED-M3-05 Team',
      slug: 'fed-m3-05-team',
      ownerId: SUBJECT_USER_ID,
      managerId: SUBJECT_USER_ID,
    },
    {
      id: UNAUTHORIZED_TEAM_ID,
      name: 'FED-M3-05 Unauthorized Team',
      slug: 'fed-m3-05-unauthorized-team',
      ownerId: OTHER_USER_ID,
      managerId: OTHER_USER_ID,
    },
  ]);
  await dbHandle.db.insert(projects).values([
    {
      id: PERSONAL_PROJECT_ID,
      name: 'FED-M3-05 Personal Project',
      ownerId: SUBJECT_USER_ID,
      ownerType: 'user',
    },
    {
      id: TEAM_PROJECT_ID,
      name: 'FED-M3-05 Team Project',
      teamId: TEAM_ID,
      ownerType: 'team',
    },
    {
      id: UNAUTHORIZED_PROJECT_ID,
      name: 'FED-M3-05 Unauthorized Project',
      teamId: UNAUTHORIZED_TEAM_ID,
      ownerType: 'team',
    },
  ]);
  await dbHandle.db.insert(missions).values([
    {
      id: PERSONAL_MISSION_ID,
      name: 'FED-M3-05 Personal Mission',
      projectId: PERSONAL_PROJECT_ID,
      userId: SUBJECT_USER_ID,
    },
    {
      id: TEAM_MISSION_ID,
      name: 'FED-M3-05 Team Mission',
      projectId: TEAM_PROJECT_ID,
      userId: SUBJECT_USER_ID,
    },
    {
      id: UNAUTHORIZED_MISSION_ID,
      name: 'FED-M3-05 Unauthorized Mission',
      projectId: UNAUTHORIZED_PROJECT_ID,
      userId: SUBJECT_USER_ID,
    },
  ]);
  await dbHandle.db.insert(missionTasks).values([
    {
      id: SUBJECT_TEAM_NOTE_ID,
      missionId: TEAM_MISSION_ID,
      userId: SUBJECT_USER_ID,
      notes: 'subject note on team mission',
      createdAt: new Date('2026-06-24T03:00:00.000Z'),
      updatedAt: new Date('2026-06-24T03:00:00.000Z'),
    },
    {
      id: OTHER_TEAM_NOTE_ID,
      missionId: TEAM_MISSION_ID,
      userId: OTHER_USER_ID,
      notes: 'other user note on team mission',
      createdAt: new Date('2026-06-24T02:00:00.000Z'),
      updatedAt: new Date('2026-06-24T02:00:00.000Z'),
    },
    {
      id: SUBJECT_PERSONAL_NOTE_ID,
      missionId: PERSONAL_MISSION_ID,
      userId: SUBJECT_USER_ID,
      notes: 'subject note on personal mission',
      createdAt: new Date('2026-06-24T01:00:00.000Z'),
      updatedAt: new Date('2026-06-24T01:00:00.000Z'),
    },
    {
      id: SUBJECT_UNAUTHORIZED_NOTE_ID,
      missionId: UNAUTHORIZED_MISSION_ID,
      userId: SUBJECT_USER_ID,
      notes: 'subject note outside grant-visible missions',
      createdAt: new Date('2026-06-24T04:00:00.000Z'),
      updatedAt: new Date('2026-06-24T04:00:00.000Z'),
    },
  ]);
  const memoryCreatedAt = new Date('2026-06-24T05:00:00.000Z');
  await dbHandle.db.insert(insights).values([
    {
      id: INSIGHT_ONE_ID,
      userId: SUBJECT_USER_ID,
      content: 'first insight',
      source: 'agent',
      createdAt: memoryCreatedAt,
      updatedAt: memoryCreatedAt,
    },
    {
      id: INSIGHT_TWO_ID,
      userId: SUBJECT_USER_ID,
      content: 'second insight',
      source: 'agent',
      createdAt: memoryCreatedAt,
      updatedAt: memoryCreatedAt,
    },
  ]);
  await dbHandle.db.insert(preferences).values([
    {
      id: PREFERENCE_ONE_ID,
      userId: SUBJECT_USER_ID,
      key: 'fed-m3-05-pref-1',
      value: { enabled: true },
      createdAt: memoryCreatedAt,
      updatedAt: memoryCreatedAt,
    },
    {
      id: PREFERENCE_TWO_ID,
      userId: SUBJECT_USER_ID,
      key: 'fed-m3-05-pref-2',
      value: { enabled: false },
      createdAt: memoryCreatedAt,
      updatedAt: memoryCreatedAt,
    },
  ]);
 }
 function stubRows(
  service: FederationListQueryService,
  ...pages: Array<Array<Record<string, unknown>>>
 ) {
  const mock = vi.fn();
  for (const page of pages) {
    mock.mockResolvedValueOnce(page);
  }
  (
    service as unknown as {
      listAllRows: (
        _filter: FederationScopeQueryFilter,
        _rowLimit: number,
        _cursor: unknown,
      ) => Promise<Array<Record<string, unknown>>>;
    }
  ).listAllRows = mock;
  return mock;
 }
 describe('FederationListQueryService', () => {
  beforeAll(async () => {
    dbHandle = createPgliteDb(`memory://fed-m3-05-list-${Date.now()}`);
    await runPgliteMigrations(dbHandle);
    await seedNotesFixture();
  });
  afterAll(async () => {
    await dbHandle?.close();
    dbHandle = undefined;
  });
  it('denies sensitive resources in native RBAC for M3 list reads', async () => {
    const service = makeService();
    await expect(
      service.evaluateReadAccess({
        grantId: 'grant-1',
        peerId: 'peer-1',
        subjectUserId: 'user-1',
        resource: 'credentials',
      }),
    ).resolves.toMatchObject({
      allowed: false,
      reason: 'credentials federation list access is not implemented in M3',
    });
  });
  it('allows personal memory reads without requiring team lookup', async () => {
    const service = makeService();
    await expect(
      service.evaluateReadAccess({
        grantId: 'grant-1',
        peerId: 'peer-1',
        subjectUserId: 'user-1',
        resource: 'memory',
      }),
    ).resolves.toEqual({
      allowed: true,
      access: { includePersonal: true, teamIds: [] },
    });
  });
  it('applies the scope row cap and returns an opaque next cursor when truncated', async () => {
    const service = makeService();
    const listAllRows = stubRows(
      service,
      [
        { id: '3', createdAt: new Date('2026-06-24T03:00:00.000Z') },
        { id: '2', createdAt: new Date('2026-06-24T02:00:00.000Z') },
        { id: '1', createdAt: new Date('2026-06-24T01:00:00.000Z') },
      ],
      [{ id: '1', createdAt: new Date('2026-06-24T01:00:00.000Z') }],
    );
    const firstPage = await service.list({ filter: TASK_FILTER });
    expect(firstPage).toEqual({
      items: [
        { id: '3', createdAt: new Date('2026-06-24T03:00:00.000Z') },
        { id: '2', createdAt: new Date('2026-06-24T02:00:00.000Z') },
      ],
      truncated: true,
      nextCursor: expect.any(String),
    });
    expect(listAllRows).toHaveBeenNthCalledWith(1, TASK_FILTER, 3, undefined);
    const secondPage = await service.list({ filter: TASK_FILTER, cursor: firstPage.nextCursor });
    expect(secondPage).toEqual({
      items: [{ id: '1', createdAt: new Date('2026-06-24T01:00:00.000Z') }],
      truncated: false,
    });
    expect(listAllRows).toHaveBeenNthCalledWith(
      2,
      TASK_FILTER,
      3,
      expect.objectContaining({ id: '2' }),
    );
  });
  it('rejects invalid cursors instead of falling back to the first page', async () => {
    const service = makeService();
    stubRows(service, [{ id: '1' }]);
    await expect(service.list({ filter: TASK_FILTER, cursor: 'not-base64-json' })).rejects.toThrow(
      'Invalid federation list cursor',
    );
  });
  it('throws when a truncated page cannot encode a resumable cursor', async () => {
    const service = makeService();
    stubRows(service, [
      { id: '2', createdAt: 'not-a-date' },
      { id: '1', createdAt: 'not-a-date' },
    ]);
    await expect(service.list({ filter: { ...TASK_FILTER, limit: 1 } })).rejects.toThrow(
      'Federation list cursor cannot be encoded',
    );
  });
  it('throws on unsupported resources instead of crashing pagination', async () => {
    const service = makeService();
    await expect(
      service.list({
        filter: {
          ...TASK_FILTER,
          resource: 'unknown-resource' as FederationScopeQueryFilter['resource'],
        },
      }),
    ).rejects.toThrow('Unsupported federation list resource');
  });
  it('does not leak another user mission task notes through team-scoped note reads', async () => {
    const service = makeDbService();
    const result = await service.list({
      filter: {
        resource: 'notes',
        subjectUserId: SUBJECT_USER_ID,
        includePersonal: false,
        teamIds: [TEAM_ID],
        limit: 10,
        maxRowsPerQuery: 10,
      },
    });
    const ids = result.items.map((item) => item['id']);
    expect(ids).toEqual([SUBJECT_TEAM_NOTE_ID]);
    expect(ids).not.toContain(OTHER_TEAM_NOTE_ID);
  });
  it('does not return subject personal mission task notes when includePersonal is false', async () => {
    const service = makeDbService();
    const result = await service.list({
      filter: {
        resource: 'notes',
        subjectUserId: SUBJECT_USER_ID,
        includePersonal: false,
        teamIds: [TEAM_ID],
        limit: 10,
        maxRowsPerQuery: 10,
      },
    });
    expect(result.items.map((item) => item['id'])).not.toContain(SUBJECT_PERSONAL_NOTE_ID);
  });
  it('does not return subject notes from missions outside the grant-visible project set', async () => {
    const service = makeDbService();
    const result = await service.list({
      filter: {
        resource: 'notes',
        subjectUserId: SUBJECT_USER_ID,
        includePersonal: true,
        teamIds: [TEAM_ID],
        limit: 10,
        maxRowsPerQuery: 10,
      },
    });
    const ids = result.items.map((item) => item['id']);
    expect(ids).toContain(SUBJECT_PERSONAL_NOTE_ID);
    expect(ids).toContain(SUBJECT_TEAM_NOTE_ID);
    expect(ids).not.toContain(SUBJECT_UNAUTHORIZED_NOTE_ID);
    expect(ids).not.toContain(OTHER_TEAM_NOTE_ID);
  });
  it('paginates memory deterministically across insights and preferences', async () => {
    const service = makeDbService();
    const filter: FederationScopeQueryFilter = {
      resource: 'memory',
      subjectUserId: SUBJECT_USER_ID,
      includePersonal: true,
      teamIds: [],
      limit: 2,
      maxRowsPerQuery: 2,
    };
    const firstPage = await service.list({ filter });
    const secondPage = await service.list({ filter, cursor: firstPage.nextCursor });
    const firstPageIds = firstPage.items.map((item) => item['id']);
    const secondPageIds = secondPage.items.map((item) => item['id']);
    const allIds = [...firstPageIds, ...secondPageIds];
    expect(firstPage).toMatchObject({ truncated: true, nextCursor: expect.any(String) });
    expect(firstPageIds).toEqual([INSIGHT_TWO_ID, INSIGHT_ONE_ID]);
    expect(secondPageIds).toEqual([PREFERENCE_TWO_ID, PREFERENCE_ONE_ID]);
    expect(new Set(allIds).size).toBe(allIds.length);
  });
 });
--- a/apps/gateway/src/federation/server/verbs/tests/list.controller.spec.ts
+++ b/apps/gateway/src/federation/server/verbs/tests/list.controller.spec.ts
@@ -1,188 +0,0 @@
 import 'reflect-metadata';
 import { RequestMethod } from '@nestjs/common';
 import type { FastifyRequest } from 'fastify';
 import { beforeEach, describe, expect, it, vi } from 'vitest';
 import { FederationAuthGuard } from '../../federation-auth.guard.js';
 import type {
  FederationScopeEvaluationResult,
  FederationScopeQueryFilter,
 } from '../../scope.service.js';
 import { ListController } from '../list.controller.js';
 import type { FederationListQueryResult } from '../list-query.service.js';
 const FEDERATION_CONTEXT = {
  grantId: 'grant-1',
  peerId: 'peer-1',
  subjectUserId: 'user-1',
  scope: { resources: ['tasks'], max_rows_per_query: 25 },
 };
 const TASK_FILTER: FederationScopeQueryFilter = {
  resource: 'tasks',
  subjectUserId: 'user-1',
  includePersonal: true,
  teamIds: ['team-1'],
  limit: 10,
  maxRowsPerQuery: 25,
 };
 function makeRequest(): FastifyRequest {
  return { federationContext: FEDERATION_CONTEXT } as unknown as FastifyRequest;
 }
 function allowedScope(
  filter: FederationScopeQueryFilter = TASK_FILTER,
 ): FederationScopeEvaluationResult {
  return { allowed: true, filter };
 }
 function makeController(opts?: {
  scopeResult?: FederationScopeEvaluationResult;
  queryResult?: FederationListQueryResult;
 }) {
  const scope = {
    evaluateAccess: vi.fn().mockResolvedValue(opts?.scopeResult ?? allowedScope()),
  };
  const query = {
    evaluateReadAccess: vi.fn(),
    list: vi.fn().mockResolvedValue(
      opts?.queryResult ?? {
        items: [
          {
            id: 'task-1',
            title: 'Federated task',
            createdAt: new Date('2026-06-24T00:00:00.000Z'),
          },
        ],
        truncated: false,
      },
    ),
  };
  return {
    controller: new ListController(scope as never, query as never),
    scope,
    query,
  };
 }
 describe('ListController', () => {
  beforeEach(() => {
    vi.clearAllMocks();
  });
  it('declares POST /api/federation/v1/list/:resource protected only by FederationAuthGuard', () => {
    expect(Reflect.getMetadata('path', ListController)).toBe('api/federation/v1/list');
    expect(Reflect.getMetadata('path', ListController.prototype.list)).toBe(':resource');
    expect(Reflect.getMetadata('method', ListController.prototype.list)).toBe(RequestMethod.POST);
    expect(Reflect.getMetadata('__guards__', ListController)).toEqual([FederationAuthGuard]);
  });
  it('runs AuthGuard context through ScopeService and returns local-source tagged rows', async () => {
    const { controller, scope, query } = makeController();
    const response = await controller.list('tasks', makeRequest(), { limit: 10 });
    expect(scope.evaluateAccess).toHaveBeenCalledWith({
      context: FEDERATION_CONTEXT,
      resource: 'tasks',
      requestedLimit: 10,
      nativeRbac: query,
    });
    expect(query.list).toHaveBeenCalledWith({ filter: TASK_FILTER, cursor: undefined });
    expect(response).toEqual({
      items: [
        {
          id: 'task-1',
          title: 'Federated task',
          createdAt: new Date('2026-06-24T00:00:00.000Z'),
          _source: 'local',
        },
      ],
    });
  });
  it('preserves pagination metadata when row cap truncates the query layer result', async () => {
    const { controller } = makeController({
      queryResult: {
        items: [{ id: 'task-1' }],
        nextCursor: 'cursor-2',
        truncated: true,
      },
    });
    const response = await controller.list('tasks', makeRequest(), { cursor: 'cursor-1' });
    expect(response).toEqual({
      items: [{ id: 'task-1', _source: 'local' }],
      nextCursor: 'cursor-2',
      _truncated: true,
    });
  });
  it('returns a federation error envelope when auth guard context is missing', async () => {
    const { controller, scope, query } = makeController();
    await expect(
      controller.list('tasks', {} as unknown as FastifyRequest, {}),
    ).rejects.toMatchObject({
      response: {
        error: {
          code: 'unauthorized',
          message: 'Federation context missing',
        },
      },
      status: 401,
    });
    expect(scope.evaluateAccess).not.toHaveBeenCalled();
    expect(query.list).not.toHaveBeenCalled();
  });
  it('returns a federation error envelope when scope evaluation denies access', async () => {
    const { controller, query } = makeController({
      scopeResult: {
        allowed: false,
        deny: {
          code: 'resource_excluded',
          stage: 'resource_exclusion',
          statusCode: 403,
          message: 'Requested federation resource is explicitly excluded by grant scope',
          grantId: 'grant-1',
          peerId: 'peer-1',
          subjectUserId: 'user-1',
          resource: 'credentials',
        },
      },
    });
    await expect(controller.list('credentials', makeRequest(), {})).rejects.toMatchObject({
      response: {
        error: {
          code: 'scope_violation',
          message: 'Requested federation resource is explicitly excluded by grant scope',
        },
      },
      status: 403,
    });
    expect(query.list).not.toHaveBeenCalled();
  });
  it('rejects malformed request body fields before querying storage', async () => {
    const { controller, scope, query } = makeController();
    await expect(controller.list('tasks', makeRequest(), { cursor: 123 })).rejects.toMatchObject({
      response: { error: { code: 'invalid_request' } },
      status: 400,
    });
    await expect(controller.list('tasks', makeRequest(), { limit: false })).rejects.toMatchObject({
      response: { error: { code: 'invalid_request' } },
      status: 400,
    });
    await expect(controller.list('tasks', makeRequest(), { limit: 'abc' })).rejects.toMatchObject({
      response: { error: { code: 'invalid_request' } },
      status: 400,
    });
    expect(scope.evaluateAccess).not.toHaveBeenCalled();
    expect(query.list).not.toHaveBeenCalled();
  });
 });
--- a/apps/gateway/src/federation/server/verbs/list-query.service.ts
+++ b/apps/gateway/src/federation/server/verbs/list-query.service.ts
@@ -1,408 +0,0 @@
 /**
 * Federation list query layer (FED-M3-05).
 *
 * Read-only DB adapter used by ListController after FederationAuthGuard and
 * FederationScopeService have established the subject user, allowed resource,
 * native-RBAC intersection, and row cap. Audit writes are intentionally
 * deferred to M4.
 */
 import { Inject, Injectable } from '@nestjs/common';
 import {
  and,
  desc,
  eq,
  inArray,
  insights,
  isNotNull,
  lt,
  missionTasks,
  missions,
  or,
  preferences,
  projects,
  tasks,
  teamMembers,
  type Db,
 } from '@mosaicstack/db';
 import type {
  FederationNativeRbacEvaluator,
  FederationNativeRbacRequest,
  FederationNativeRbacResult,
  FederationScopeQueryFilter,
 } from '../scope.service.js';
 import { DB } from '../../../database/database.module.js';
 export interface FederationListQueryRequest {
  readonly filter: FederationScopeQueryFilter;
  readonly cursor?: string;
 }
 export interface FederationListQueryResult<T extends object = Record<string, unknown>> {
  readonly items: T[];
  readonly nextCursor?: string;
  readonly truncated: boolean;
 }
 type CursorSource = 'insights' | 'preferences';
 const CURSOR_SOURCE = Symbol('federationCursorSource');
 type RowObject = Record<string, unknown> & { readonly [CURSOR_SOURCE]?: CursorSource };
 interface KeysetCursor {
  readonly createdAt: Date;
  readonly id: string;
  readonly source?: CursorSource;
 }
 function encodeCursor(row: RowObject): string {
  const createdAt = row['createdAt'];
  const id = row['id'];
  if (!(createdAt instanceof Date) || typeof id !== 'string') {
    throw new Error('Federation list cursor cannot be encoded');
  }
  const source = row[CURSOR_SOURCE];
  return Buffer.from(
    JSON.stringify({ createdAt: createdAt.toISOString(), id, ...(source ? { source } : {}) }),
    'utf8',
  ).toString('base64url');
 }
 function decodeCursor(cursor: string | undefined): KeysetCursor | undefined {
  if (cursor === undefined) {
    return undefined;
  }
  try {
    const parsed = JSON.parse(Buffer.from(cursor, 'base64url').toString('utf8')) as unknown;
    if (typeof parsed !== 'object' || parsed === null) {
      throw new Error('cursor must be an object');
    }
    const { createdAt, id, source } = parsed as {
      createdAt?: unknown;
      id?: unknown;
      source?: unknown;
    };
    if (typeof createdAt !== 'string' || typeof id !== 'string' || id.length === 0) {
      throw new Error('cursor is missing createdAt or id');
    }
    if (source !== undefined && source !== 'insights' && source !== 'preferences') {
      throw new Error('cursor source is invalid');
    }
    const date = new Date(createdAt);
    if (Number.isNaN(date.getTime())) {
      throw new Error('cursor createdAt is invalid');
    }
    return { createdAt: date, id, ...(source ? { source } : {}) };
  } catch {
    throw new Error('Invalid federation list cursor');
  }
 }
 function paginate<T extends RowObject>(rows: T[], limit: number): FederationListQueryResult<T> {
  const page = rows.slice(0, limit);
  const hasMore = rows.length > limit;
  const nextCursor = hasMore ? encodeCursor(page[page.length - 1] ?? {}) : undefined;
  return {
    items: page,
    truncated: hasMore,
    ...(nextCursor !== undefined ? { nextCursor } : {}),
  };
 }
 function markCursorSource<T extends RowObject>(row: T, source: CursorSource): T {
  Object.defineProperty(row, CURSOR_SOURCE, {
    value: source,
    enumerable: false,
    configurable: false,
  });
  return row;
 }
 function sortRows(rows: RowObject[]): RowObject[] {
  return [...rows].sort((a, b) => {
    const aTime = a['createdAt'] instanceof Date ? a['createdAt'].getTime() : 0;
    const bTime = b['createdAt'] instanceof Date ? b['createdAt'].getTime() : 0;
    if (aTime !== bTime) {
      return bTime - aTime;
    }
    return String(b['id'] ?? '').localeCompare(String(a['id'] ?? ''));
  });
 }
@Injectable()
 export class FederationListQueryService implements FederationNativeRbacEvaluator {
  constructor(@Inject(DB) private readonly db: Db) {}
  async evaluateReadAccess(
    request: FederationNativeRbacRequest,
  ): Promise<FederationNativeRbacResult> {
    if (request.resource === 'credentials' || request.resource === 'api_keys') {
      return {
        allowed: false,
        reason: `${request.resource} federation list access is not implemented in M3`,
        details: { resource: request.resource },
      };
    }
    if (request.resource === 'memory') {
      return { allowed: true, access: { includePersonal: true, teamIds: [] } };
    }
    const teamIds = await this.listSubjectTeamIds(request.subjectUserId);
    return { allowed: true, access: { includePersonal: true, teamIds } };
  }
  async list<T extends RowObject = RowObject>(
    request: FederationListQueryRequest,
  ): Promise<FederationListQueryResult<T>> {
    const cursor = decodeCursor(request.cursor);
    const rows = await this.listAllRows(request.filter, request.filter.limit + 1, cursor);
    return paginate(rows as T[], request.filter.limit);
  }
  private async listAllRows(
    filter: FederationScopeQueryFilter,
    rowLimit: number,
    cursor: KeysetCursor | undefined,
  ): Promise<RowObject[]> {
    switch (filter.resource) {
      case 'tasks':
        return this.listTasks(filter, rowLimit, cursor);
      case 'notes':
        return this.listNotes(filter, rowLimit, cursor);
      case 'memory':
        return this.listMemory(filter, rowLimit, cursor);
      case 'credentials':
      case 'api_keys':
        return [];
      default:
        throw new Error(`Unsupported federation list resource: ${String(filter.resource)}`);
    }
  }
  private async listSubjectTeamIds(subjectUserId: string): Promise<string[]> {
    const rows = await this.db
      .select({ teamId: teamMembers.teamId })
      .from(teamMembers)
      .where(eq(teamMembers.userId, subjectUserId));
    return rows.map((row) => row.teamId);
  }
  private async listAccessibleProjectIds(filter: FederationScopeQueryFilter): Promise<string[]> {
    const clauses = [];
    if (filter.includePersonal) {
      clauses.push(and(eq(projects.ownerType, 'user'), eq(projects.ownerId, filter.subjectUserId)));
    }
    if (filter.teamIds.length > 0) {
      clauses.push(
        and(eq(projects.ownerType, 'team'), inArray(projects.teamId, [...filter.teamIds])),
      );
    }
    if (clauses.length === 0) {
      return [];
    }
    const rows = await this.db
      .select({ id: projects.id })
      .from(projects)
      .where(clauses.length === 1 ? clauses[0] : or(...clauses));
    return rows.map((row) => row.id);
  }
  private async listMissionIds(projectIds: readonly string[]): Promise<string[]> {
    if (projectIds.length === 0) {
      return [];
    }
    const rows = await this.db
      .select({ id: missions.id })
      .from(missions)
      .where(inArray(missions.projectId, [...projectIds]));
    return rows.map((row) => row.id);
  }
  private async listTasks(
    filter: FederationScopeQueryFilter,
    rowLimit: number,
    cursor: KeysetCursor | undefined,
  ): Promise<RowObject[]> {
    const projectIds = await this.listAccessibleProjectIds(filter);
    const missionIds = await this.listMissionIds(projectIds);
    const clauses = [];
    if (projectIds.length > 0) {
      clauses.push(inArray(tasks.projectId, projectIds));
    }
    if (missionIds.length > 0) {
      clauses.push(inArray(tasks.missionId, missionIds));
    }
    if (clauses.length === 0) {
      return [];
    }
    const scopeClause = clauses.length === 1 ? clauses[0] : or(...clauses);
    const cursorClause = cursor
      ? or(
          lt(tasks.createdAt, cursor.createdAt),
          and(eq(tasks.createdAt, cursor.createdAt), lt(tasks.id, cursor.id)),
        )
      : undefined;
    const rows = await this.db
      .select({
        id: tasks.id,
        title: tasks.title,
        description: tasks.description,
        status: tasks.status,
        priority: tasks.priority,
        projectId: tasks.projectId,
        missionId: tasks.missionId,
        assignee: tasks.assignee,
        tags: tasks.tags,
        dueDate: tasks.dueDate,
        metadata: tasks.metadata,
        createdAt: tasks.createdAt,
        updatedAt: tasks.updatedAt,
      })
      .from(tasks)
      .where(and(scopeClause, cursorClause))
      .orderBy(desc(tasks.createdAt), desc(tasks.id))
      .limit(rowLimit);
    return sortRows(rows as RowObject[]);
  }
  private async listNotes(
    filter: FederationScopeQueryFilter,
    rowLimit: number,
    cursor: KeysetCursor | undefined,
  ): Promise<RowObject[]> {
    const projectIds = await this.listAccessibleProjectIds(filter);
    const missionIds = await this.listMissionIds(projectIds);
    if (missionIds.length === 0) {
      return [];
    }
    // mission_tasks rows are user-scoped even when the mission belongs to a team.
    // Team visibility can narrow the mission set, but it must never widen the
    // query to other users' mission task notes.
    const scopeClause = and(
      eq(missionTasks.userId, filter.subjectUserId),
      inArray(missionTasks.missionId, missionIds),
    );
    const cursorClause = cursor
      ? or(
          lt(missionTasks.createdAt, cursor.createdAt),
          and(eq(missionTasks.createdAt, cursor.createdAt), lt(missionTasks.id, cursor.id)),
        )
      : undefined;
    const rows = await this.db
      .select({
        id: missionTasks.id,
        missionId: missionTasks.missionId,
        taskId: missionTasks.taskId,
        status: missionTasks.status,
        content: missionTasks.notes,
        createdAt: missionTasks.createdAt,
        updatedAt: missionTasks.updatedAt,
      })
      .from(missionTasks)
      .where(and(scopeClause, cursorClause, isNotNull(missionTasks.notes)))
      .orderBy(desc(missionTasks.createdAt), desc(missionTasks.id))
      .limit(rowLimit);
    return sortRows(rows.filter((row) => row.content !== '') as RowObject[]);
  }
  private async listMemory(
    filter: FederationScopeQueryFilter,
    rowLimit: number,
    cursor: KeysetCursor | undefined,
  ): Promise<RowObject[]> {
    if (!filter.includePersonal) {
      return [];
    }
    if (cursor && cursor.source === undefined) {
      throw new Error('Invalid federation list cursor');
    }
    const rows: RowObject[] = [];
    // Memory spans two physical tables. To keep pagination deterministic and
    // resumable without a SQL UNION, M3 emits a fixed block order: all insights
    // first, then preferences. The opaque cursor records which table produced
    // the boundary row, so the next page never re-applies one table's keyset to
    // the other table (which could duplicate/skip rows at equal timestamps).
    if (cursor?.source !== 'preferences') {
      const insightCursorClause = cursor
        ? or(
            lt(insights.createdAt, cursor.createdAt),
            and(eq(insights.createdAt, cursor.createdAt), lt(insights.id, cursor.id)),
          )
        : undefined;
      const insightRows = await this.db
        .select({
          id: insights.id,
          kind: insights.source,
          content: insights.content,
          category: insights.category,
          relevanceScore: insights.relevanceScore,
          metadata: insights.metadata,
          createdAt: insights.createdAt,
          updatedAt: insights.updatedAt,
        })
        .from(insights)
        .where(and(eq(insights.userId, filter.subjectUserId), insightCursorClause))
        .orderBy(desc(insights.createdAt), desc(insights.id))
        .limit(rowLimit);
      rows.push(...(insightRows as RowObject[]).map((row) => markCursorSource(row, 'insights')));
    }
    const remaining = rowLimit - rows.length;
    if (remaining <= 0) {
      return rows;
    }
    const preferenceCursorClause =
      cursor?.source === 'preferences'
        ? or(
            lt(preferences.createdAt, cursor.createdAt),
            and(eq(preferences.createdAt, cursor.createdAt), lt(preferences.id, cursor.id)),
          )
        : undefined;
    const preferenceRows = await this.db
      .select({
        id: preferences.id,
        kind: preferences.category,
        key: preferences.key,
        value: preferences.value,
        source: preferences.source,
        mutable: preferences.mutable,
        createdAt: preferences.createdAt,
        updatedAt: preferences.updatedAt,
      })
      .from(preferences)
      .where(and(eq(preferences.userId, filter.subjectUserId), preferenceCursorClause))
      .orderBy(desc(preferences.createdAt), desc(preferences.id))
      .limit(remaining);
    rows.push(
      ...(preferenceRows as RowObject[]).map((row) => markCursorSource(row, 'preferences')),
    );
    return rows;
  }
 }
--- a/apps/gateway/src/federation/server/verbs/list.controller.ts
+++ b/apps/gateway/src/federation/server/verbs/list.controller.ts
@@ -1,147 +0,0 @@
 /**
 * Federation list verb (FED-M3-05).
 *
 * POST /api/federation/v1/list/:resource
 *
 * Pipeline: FederationAuthGuard attaches the active grant context, then
 * FederationScopeService enforces grant scope + native RBAC intersection, then
 * the read-only query layer returns capped rows tagged with `_source`. Read
 * audit-log writes are deferred to M4; this controller does not persist request
 * or response bodies.
 */
 import {
  Body,
  Controller,
  HttpException,
  Inject,
  Param,
  Post,
  Req,
  UseGuards,
 } from '@nestjs/common';
 import type { FastifyRequest } from 'fastify';
 import {
  FederationInvalidRequestError,
  FederationScopeViolationError,
  FederationUnauthorizedError,
  SOURCE_LOCAL,
  tagWithSource,
  type FederationListResponse,
  type SourceTag,
 } from '@mosaicstack/types';
 import { FederationAuthGuard } from '../federation-auth.guard.js';
 import '../federation-context.js';
 import { FederationScopeService } from '../scope.service.js';
 import { FederationListQueryService } from './list-query.service.js';
 interface FederationListRequestBody {
  readonly limit?: unknown;
  readonly cursor?: unknown;
 }
 type FederatedRow = Record<string, unknown> & SourceTag;
 function parseLimit(body: FederationListRequestBody | undefined): number | undefined {
  if (body?.limit === undefined) {
    return undefined;
  }
  const parsed =
    typeof body.limit === 'number'
      ? body.limit
      : typeof body.limit === 'string' && body.limit.trim().length > 0
        ? Number(body.limit)
        : Number.NaN;
  if (!Number.isSafeInteger(parsed) || parsed < 1) {
    throw new HttpException(
      new FederationInvalidRequestError(
        'Federation list limit must be a positive integer',
      ).toEnvelope(),
      400,
    );
  }
  return parsed;
 }
 function parseCursor(body: FederationListRequestBody | undefined): string | undefined {
  if (body?.cursor === undefined) {
    return undefined;
  }
  if (typeof body.cursor === 'string') {
    return body.cursor;
  }
  throw new HttpException(
    new FederationInvalidRequestError('Federation list cursor must be a string').toEnvelope(),
    400,
  );
 }
@Controller('api/federation/v1/list')
@UseGuards(FederationAuthGuard)
 export class ListController {
  constructor(
    @Inject(FederationScopeService) private readonly scope: FederationScopeService,
    @Inject(FederationListQueryService) private readonly query: FederationListQueryService,
  ) {}
  @Post(':resource')
  async list(
    @Param('resource') resource: string,
    @Req() request: FastifyRequest,
    @Body() body?: FederationListRequestBody,
  ): Promise<FederationListResponse<FederatedRow>> {
    if (!request.federationContext) {
      throw new HttpException(
        new FederationUnauthorizedError('Federation context missing').toEnvelope(),
        401,
      );
    }
    const requestedLimit = parseLimit(body);
    const cursor = parseCursor(body);
    const scopeResult = await this.scope.evaluateAccess({
      context: request.federationContext,
      resource,
      requestedLimit,
      nativeRbac: this.query,
    });
    if (!scopeResult.allowed) {
      const ErrorClass =
        scopeResult.deny.statusCode === 400
          ? FederationInvalidRequestError
          : FederationScopeViolationError;
      throw new HttpException(
        new ErrorClass(scopeResult.deny.message, scopeResult.deny).toEnvelope(),
        scopeResult.deny.statusCode,
      );
    }
    let result: Awaited<ReturnType<FederationListQueryService['list']>>;
    try {
      result = await this.query.list({ filter: scopeResult.filter, cursor });
    } catch (error: unknown) {
      if (error instanceof Error && error.message === 'Invalid federation list cursor') {
        throw new HttpException(
          new FederationInvalidRequestError('Federation list cursor is invalid').toEnvelope(),
          400,
        );
      }
      throw error;
    }
    const response: FederationListResponse<FederatedRow> = {
      items: tagWithSource(result.items, SOURCE_LOCAL),
    };
    if (result.nextCursor !== undefined) {
      response.nextCursor = result.nextCursor;
    }
    if (result.truncated) {
      response._truncated = true;
    }
    return response;
  }
 }
--- a/docs/scratchpads/FED-M3-05-list-verb.md
+++ b/docs/scratchpads/FED-M3-05-list-verb.md
@@ -1,52 +0,0 @@
 # FED-M3-05 — Federation List Verb Scratchpad
 ## Objective
 Implement `POST /api/federation/v1/list/:resource`.
 ## Scope
 - Wire `FederationAuthGuard` → `FederationScopeService` → read-only list query layer.
 - Apply `max_rows_per_query` row cap and return pagination metadata when truncated.
 - Tag returned rows with `_source: "local"`.
 - Keep audit writes deferred to M4.
 - No request/response body persistence.
 ## Base / branch
 - Branch: `feat/federation-m3-verb-list`
 - Base: `main` after M3-04 scope service merged via PR #672 (`c739256a`).
 ## Implementation notes
 - Added `ListController` under `apps/gateway/src/federation/server/verbs/`.
 - Added `FederationListQueryService` as the read-only query layer and native RBAC evaluator.
 - Query resources supported in M3 list path:
  - `tasks`: project/mission scoped tasks visible through personal/team project access.
  - `notes`: non-empty `mission_tasks.notes` rows visible through personal/team mission access.
  - `memory`: user-owned `insights` and `preferences` rows.
  - `credentials` / `api_keys`: denied by native RBAC in M3 even if present in scope; sensitive-resource implementation is not part of FED-M3-05.
 - Cursor pagination uses an opaque base64url keyset cursor over `(createdAt, id)`; DB reads fetch at most `limit + 1` rows per resource query.
 - Reviewer isolation fix: `mission_tasks.notes` rows are always constrained by `missionTasks.userId = subjectUserId` and accessible mission IDs; team scope narrows missions but never widens to other users' mission task notes.
 - Follow-up review fix: memory listing now uses deterministic table-block pagination (`insights` first, then `preferences`) with cursor source metadata, so one table's cursor is never applied to the other.
 - Follow-up hardening: missing auth-guard context returns a structured federation `unauthorized` envelope; unsupported resources and non-encodable truncated cursors throw instead of silently crashing/truncating.
 ## Tests
 - `pnpm --filter @mosaicstack/gateway test -- list.controller.spec.ts list-query.service.spec.ts` — PASS (16 tests, including PGlite regression coverage for team-scoped notes isolation, unauthorized mission notes exclusion, `includePersonal: false`, deterministic memory pagination, missing context envelope, unsupported resource, and cursor encode failure).
 - `pnpm --filter @mosaicstack/gateway typecheck` — PASS.
 - `pnpm --filter @mosaicstack/gateway lint` — PASS.
 - `pnpm format:check` — PASS.
 - `pnpm typecheck` — PASS (41/41 turbo tasks).
 - `pnpm lint` — PASS (23/23 turbo tasks).
 - `pnpm --filter @mosaicstack/gateway test` — FAIL in pre-existing/live-DB integration suite: `apps/gateway/src/__tests__/cross-user-isolation.test.ts` cleanup cannot connect to local PostgreSQL on `localhost:5433`. New list tests pass; failure is outside FED-M3-05.
 ## Review evidence
 - `~/.config/mosaic/tools/codex/codex-code-review.sh --uncommitted` — PASS after follow-up remediation; approve, no findings.
 - `~/.config/mosaic/tools/codex/codex-security-review.sh --uncommitted` — PASS after follow-up remediation; risk level none, no findings.
 - Security-review note: read-path audit logging remains intentionally deferred to M4 per orchestrator clarification and FED-M3-05 scope.
 ## Risks / follow-up
 - Read-path audit logging remains intentionally deferred to M4.