docs: add relay and jetstream guide

lib/navigation.js

@@ -63,6 +63,10 @@ export const navigation = [
       { title: "Scaffold Starter App", path: "/tools/scaffold" },
       { title: "Agent Skills", path: "/tools/hypercerts-agent-skills" },
       { title: "Hyperindex", path: "/tools/hyperindex" },
+      {
+        title: "Hypercerts Relay and Jetstream",
+        path: "/tools/hypercerts-relay",
+      },
       {
         title: "Hypercerts Feed Service",
         path: "/tools/hypercerts-feed-service",

pages/reference/certified-services.md

@@ -53,6 +53,19 @@ The Hypercerts Feed Service turns current Hyperindex data into ordered, viewer-s
 
 For usage examples and the source repository, see [Hypercerts Feed Service](/tools/hypercerts-feed-service).
 
+## Relay and Jetstream
+
+Relay provides the raw repository-event firehose. Jetstream provides the selected Hypercerts and Certified record stream and archive.
+
+| Service | Environment | Public URL |
+|---|---|---|
+| Relay | Production | [`relay.hypercerts.dev`](https://relay.hypercerts.dev) |
+| Relay | Staging | [`relay.staging.hypercerts.dev`](https://relay.hypercerts.dev) |
+| Jetstream | Production | [`jetstream.hypercerts.dev`](https://jetstream.hypercerts.dev) |
+| Jetstream | Staging | [`jetstream.staging.hypercerts.dev`](https://jetstream.hypercerts.dev) |
+
+See [Hypercerts Relay and Jetstream](/tools/hypercerts-relay) for full details.
+
 ## Labelers
 
 | Service | Environment | Public URL | Purpose |

pages/tools/hypercerts-relay.md

@@ -0,0 +1,113 @@
+---
+title: Hypercerts Relay and Jetstream
+description: Connect to the Hypercerts Relay firehose or the Jetstream record stream and archive.
+---
+
+# Hypercerts Relay and Jetstream
+
+The Hypercerts Relay receives repository events from Personal Data Servers (PDSs) which can be accessing via the standard `subscribeRepos` firehose.
+
+A Jetstream v2 service is availability which provides a full backfill of crawled lexicon collections and a live stream of these lexicon record events. You will typically be consuming the jetstream rather than firehose unless there's a need for the full CBOR verification details.
+
+## URLs
+
+| Service | URL | Use |
+| --- | --- | --- |
+| Relay firehose | `wss://relay.hypercerts.dev/xrpc/com.atproto.sync.subscribeRepos` | Receive raw AT Protocol repository events. Add `?cursor=<sequence>` to resume from a saved Relay cursor. |
+| Relay crawl request | `https://relay.hypercerts.dev/xrpc/com.atproto.sync.requestCrawl` | Ask Relay to reconnect to an approved PDS. |
+| Jetstream live stream | `wss://jetstream.hypercerts.dev/xrpc/network.bsky.jetstream.subscribeEvents` | Receive selected archived and live events in sequence order. |
+| Jetstream archive plan | `https://jetstream.hypercerts.dev/xrpc/network.bsky.jetstream.planSnapshot` | Plan an archive download before connecting to the live stream. |
+| Jetstream archive segments | `https://jetstream.hypercerts.dev/xrpc/network.bsky.jetstream.getSegment` | Download a segment named by an archive plan. |
+
+The Relay firehose uses the normal `com.atproto.sync.subscribeRepos` event format. Jetstream uses the `network.bsky.jetstream.subscribeEvents` format.
+
+## Subscribing to Relay
+
+When subscribed to the relay, persist the sequence number from each event. On reconnect, send it as the `cursor` query parameter.
+
+Relay keeps raw events for 72 hours. A cursor older than that window may not be available, so consumers should be able to recover from a newer position.
+
+## Working with Jetstream
+
+When subscribed with no parameters, the connection starts at the live tip. Add `cursor=<sequence>` to replay from a saved Jetstream sequence number, because Jetstream replays the event 
+reconnects can deliver duplicated, therefore deduplicate or handle events idempotently on reconnect. 
+
+Jetstream can filter the stream with these query parameters:
+
+| Parameter | Meaning |
+| --- | --- |
+| `kinds=commit` | Receive record changes only. |
+| `collections=<nsid>` | Receive commits for an exact collection NSID or a namespace pattern such as `org.hypercerts.context.*`. Use this together with `kinds=commit`. |
+| `dids=<did>` | Receive events for one or more repository DIDs. |
+
+If a saved Jetstream sequence is older than the retained stream, Jetstream returns `CursorTooOld`. Backfill from the archive, save the resulting sequence, then reconnect to the live stream.
+
+### Backfill
+
+**Note:** The archive/backfill endpoints require a Jetstream API Key, contact the Hypercerts team for one before using.
+
+Jetstream backfill lets a client catch up from the retained archive before it starts receiving live events.
+
+1. Send a `POST` request to `network.bsky.jetstream.planSnapshot` with the same `kinds`, `collections`, and `dids` filters that the live client will use.
+2. Download every segment or block range in the returned plan.
+3. Keep requesting pages until `plannedThroughSeq` equals `sealedTipSeq`. Keep the first `sealedTipSeq` as the fixed end of this backfill.
+4. Connect to `subscribeEvents` with `cursor` set to the next sequence after the archive backfill. De-duplicate the small overlap at the handoff.
+
+The Jetstream API Key should be included in the request as `Authorization: Bearer <api-key>` on archive-plan and segment-download requests.
+
+Backfill returns events that Jetstream has retained and cannot return data older than the archive however it should be expected that Jetsream will have backfilled the PDS jetstream is subscribed to, if you find gaps contact Hypercerts.
+
+When using `planSnapshot` for archival backfill it can return whole-segment entries or block ranges. Whole segments will use `getsegment` and block ranges require `getBlock` for each block index followed by decoding and exact filtering.
+
+See [Bluesky Jetstream Docs](https://bsky.network/docs/jetstream/) for further details.
+
+## Default lexicons
+
+Jetstream stores the following collections by default. Relay does not filter its raw firehose by this list.
+
+```text
+app.certified.actor.organization
+app.certified.actor.profile
+app.certified.badge.award
+app.certified.badge.definition
+app.certified.badge.response
+app.certified.graph.entityFollow
+app.certified.graph.follow
+app.certified.link.evm
+app.certified.location
+app.certified.signature.proof
+org.hypercerts.claim.activity
+org.hypercerts.claim.contribution
+org.hypercerts.claim.contributorInformation
+org.hypercerts.claim.rights
+org.hypercerts.collection
+org.hypercerts.context.acknowledgement
+org.hypercerts.context.attachment
+org.hypercerts.context.evaluation
+org.hypercerts.context.measurement
+org.hypercerts.funding.receipt
+org.hypercerts.workscope.tag
+```
+
+Hypercerts adds additional lexicons on request. When a lexicon is added, Jetstream backfills the newly selected collections for every approved PDS.
+
+## Request a PDS crawl
+
+To ask Relay to reconnect, send its hostname as JSON:
+
+```bash
+curl --request POST \
+  --url https://relay.hypercerts.dev/xrpc/com.atproto.sync.requestCrawl \
+  --header 'content-type: application/json' \
+  --data '{"hostname":"pds.example.com"}'
+```
+
+This public request does not add a new PDS, change its status, or re-enable a disabled PDS. Contact the Hypercerts team to have a new PDS approved.
+
+A newly approved PDS has a default limit of 100 active accounts. Relay's default configuration also sets a daily maximum of 50 new PDS subscriptions. Operators can set different limits for an individual PDS.
+
+## See also
+
+- [Certified Services](/reference/certified-services)
+- [Hypercerts Lexicons](/lexicons/hypercerts-lexicons)
+- [Certified Lexicons](/lexicons/certified-lexicons)