A nostr message queue that forgets what you've consumed.
noq stores signed nostr events in an append-only log. Consumers lease the events that match a NIP-01 filter and ack them when they're done. When every event in a segment file is consumed, the file is deleted, so the disk drains with the queue.
cargo install --git https://github.com/jb55/noq
Push, lease, ack
Output from the noq CLI on the 617 kind-1 notes in nostrdb's test data. Every push checks the event's id and signature. A lease that isn't acked ends when its holder exits, and the events are queued again.
$ noq push < notes.jsonl 617 queued, 0 not queued, 0 invalid $ noq lease '{"kinds":[1]}' -n 3 --ack {"id":"b2e03951843b…","kind":1,"content":"hello, this is my new key",…} {"id":"00000e1253a8…","kind":1,"content":"I am a robot beep boop! This note took 4 sec…",…} {"id":"a4b73fc5b901…","kind":1,"content":"I made a robot friend that mines nostr e…",…} $ noq push < acked.jsonl # send the consumed ones again duplicate b2e03951843b… 0:0 duplicate 00000e1253a8… 0:480 duplicate a4b73fc5b901… 0:1112 $ noq push < forged.jsonl invalid 1 id does not match the event hash $ noq --segment-size 64k lease '{}' -n 100000 --ack | wc -l 617 # 7 segment files before, 1 after
How it works
Events are appended to fixed-size, memory-mapped segment files in arrival order. Each record starts with a state byte. A lease is one compare-and-swap on that byte, from LIVE to LEASED, so two consumers can never take the same event.
Verified on the way in
One pass over the JSON checks it strictly, computes the event id and verifies the schnorr signature. Malformed or forged events are rejected before anything is written.
The JSON is the wire format
Each record keeps the original JSON next to a small span index into it. A leased event goes back out byte for byte, with no re-serialization.
NIP-01 filters, planned
Ids, authors, kinds, tags, since and until, matched against the packed record in place. In-memory indexes let a lease skip straight to matching events.
Leases with a ttl
Ack consumes an event. Nack or an expired ttl queues it again. wait_lease blocks until a push, nack or expiry gives it something to take.
The disk drains
Fully consumed segments are deleted. Mostly consumed ones are compacted, moving their remaining events forward so the file can go too.
Crash safe
Each record carries a CRC32C. Reopening a log after a crash drops a torn record at the tail. Sealed segments keep a sidecar index, so reopening is fast.
As a library
The CLI is a thin layer over the crate. A process opens the queue once and shares it between threads, so one pushes while others lease.
use noq::{Config, Filter, Noq}; use std::time::Duration; let q = Noq::open("queue", Config::default())?; q.push(json)?; // id and sig checked; Queued, Duplicate, Seen or Expired let dms = Filter::new().kinds([4, 1059]); for lease in q.wait_lease(&dms, 100, Duration::from_secs(30), Duration::from_secs(5)) { deliver(lease.json())?; // the original event json, as it was pushed lease.ack()?; // or lease.nack() to queue it again }
Numbers
On nostrdb's 1,004,866-event test file (2.0 GiB of JSON), on a 32-core machine with zfs. Every push verifies the id and signature.
One run each, warm cache, --fsync never. The comparison with nostrdb and strfry, and what each number does and doesn't measure, is in bench/README.md. Reproduce with noq bench events.jsonl.
Status
Early, version 0.1. The storage, leasing and filtering work and are tested. The API will still change.
- One process owns a queue directory. Its writer and consumers are threads.
- One queue per directory, with no consumer groups.
- No network front yet: an app embeds the crate or drives the CLI.
- Every event is queued on its own, replaceable and addressable kinds included. Deciding which version is newest is the consumer's job.