Rogue relays could in theory attack nostrdb by replaying ids and
signatures from other notes. This fixes this weakness by calculating the
id again in ndb_note_verify.
There is no known relays exploiting this, but lets get ahead of it
before we switch to the outbox model in damus iOS/notedeck
Signed-off-by: William Casarin <jb55@jb55.com>
This adds some helpers for adding custom filtering logic
to nostr filters. These are just a callback and a closure.
There can only be one custom callback filter per filter.
Fixes: https://github.com/damus-io/nostrdb/issues/33
Signed-off-by: William Casarin <jb55@jb55.com>
Add support for relay-based filtering in nostr queries.
Filters can now include a "relays" field. Optimal performance when
you include a kind as well:
{"relays":["wss://pyramid.fiatjaf.com/"], "kinds":[1]}
This corresponds to a `ndb` query like so:
$ ndb query -r wss://pyramid.fiatjaf.com/ -k 1 -l 1
using filter '{"relays":["wss://pyramid.fiatjaf.com/"],"kinds":[1],"limit":1}'
1 results in 0.094929 ms
{"id":"277dd4ed26d0b44576..}
Signed-off-by: William Casarin <jb55@jb55.com>
Add relay indexing for existing notes
This patch introduces a relay index for new notes and notes that have
already been stored, allowing the database to track additional relay
sources for a given note.
Changes:
- Added `NDB_WRITER_NOTE_RELAY` to handle relay indexing separately from
new note ingestion.
- Implemented `ndb_write_note_relay()` and
`ndb_write_note_relay_kind_index()` to store relay URLs.
- Modified `ndb_ingester_process_event()` to check for existing notes
and append relay info if necessary.
- Introduced `ndb_note_has_relay()` to prevent duplicate relay entries.
- Updated LMDB schema with `NDB_DB_NOTE_RELAYS` (note_id -> relay) and
`NDB_DB_NOTE_RELAY_KIND` (relay + kind + created_at -> note).
- Refactored `ndb_process_event()` to use `ndb_ingest_meta` for tracking
relay sources.
- Ensured proper memory management for relay strings in writer thread.
With this change, nostrdb can better track where notes are seen across
different relays, improving query capabilities for relay-based data
retrieval.
Signed-off-by: William Casarin <jb55@jb55.com>
Add support for type KIND for bech32-encoded entities naddr and nevent
as specified in NIP-19.
Co-authored-by: kernelkind <kernelkind@gmail.com>
Signed-off-by: William Casarin <jb55@jb55.com>
This adds support for nip50 fulltext searches. This allows you to use
the nostrdb query interface for executing fulltext searches instead of
the typical `ndb_text_search` api. The benefits of this include a
standardized query interface that also further filters on other fields
in the filter.
Changelog-Added: Add nip50 search filters and queries
Signed-off-by: William Casarin <jb55@jb55.com>
Update fulltext search queries to include an optional filter. This can
be used to narrow down the fulltext search. This is another step towards
nip50 support in nostrdb.
I noticed the code was exiting dubiously in certain situations... so we
fix that as well. It's possible we were missing search results because
of this.
Signed-off-by: William Casarin <jb55@jb55.com>
This fixes an allocation issue with ndb_filter_init_with for small
page sizes. instead of allocating the buffer around pages, we allocate
based on total buffer size.
Fixes: f7aac3215575 ("filter: introduce ndb_filter_init_with")
Signed-off-by: William Casarin <jb55@jb55.com>
Make fulltext indices and note blocks optional. This will be useful for
quickly building databases when testing, since more stuff in the write
queue when writing can slow things down.
Signed-off-by: William Casarin <jb55@jb55.com>
This was the only thing that wasn't threadsafe. Add a simple mutex
instead of a queue so that polling is quick.
This also means we can't really return the internal subscriptions
anymore, so we remove that for now until we have a safer
interface.
Fixes: https://github.com/damus-io/nostrdb/issues/55
Signed-off-by: William Casarin <jb55@jb55.com>
subset testing for filters. Can be used to see if one subset is
redundant in the presence of a another in the local relay model
Changelog-Added: Add ndb_filter_is_subset_of
Signed-off-by: William Casarin <jb55@jb55.com>
filter equality testing. this works because field elements are sorted
Changelog-Added: Add ndb_filter_eq for filter equality testing
Signed-off-by: William Casarin <jb55@jb55.com>
Expose a way to get the set of filters for a subscription. On the rust
side, we should likely ndb_filter_clone each filter asap, because the
result of this function will only be valid up until the subscription
ends.
Signed-off-by: William Casarin <jb55@jb55.com>
Since Damus iOS is not an immediate-mode UI like android, we would
rather not poll for results. Instead we need a way to register a
callback function that is called when we get new subscription results.
This is also useful on the android side, allowing us to request a new
frame to draw when we have new results, instead of drawing every second.
Signed-off-by: William Casarin <jb55@jb55.com>
We didn't have a way to unsubscribe from subscriptions. Now we do!
Apps like notecrumbs may open up many local subscriptions based on
incoming requests. We may need to make the MAX_SUBSCRIPTIONS size much
larger, but this should be okish for now.
Changelog-Added: Add ndb_unsubscribe to unsubscribe from subscriptions
Signed-off-by: William Casarin <jb55@jb55.com>
Clone filters when moving them into subscriptions. This will allow us to
fix the double free issue on the rust side.
Signed-off-by: William Casarin <jb55@jb55.com>
Instead of storing exact pointers inside of our filter elements, just
store offsets. This will allow us to clone filters very easily without
having to mess around with fixing up the pointers afterwards.
Signed-off-by: William Casarin <jb55@jb55.com>
This is a pretty scary looking function that realloc our large variable
filter buffer into a compact one. This saves up a bunch of memory when
we are done building the filter.
Signed-off-by: William Casarin <jb55@jb55.com>
The polling variant of ndb_wait_for_notes. This makes more sense for
realtime apps like notedeck
Changelog-Added: Add ndb_poll_for_notes
Signed-off-by: William Casarin <jb55@jb55.com>
Instead of running queries off filters directly, we do some simple
heuristics and determine a reasonable query plan for the given filter.
To test this, also add a kind index query plan and add a test for it.
We still need tag, author, and created_at index scans. This is up next!
Signed-off-by: William Casarin <jb55@jb55.com>
Still a lot more work to do, but this is at least a proof of concept for
querying nostrdb using filters.
Signed-off-by: William Casarin <jb55@jb55.com>
