Commit Graph

4218 Commits

Author SHA1 Message Date
William Casarin
155ac27bb5 nostrdb: introduce ndb_id_u64_ts
This will be the key used by our note_profile_kind indee

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
be1d149f4b nostrdb: misc: move some functions around
because this will make the changes nicer

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
9e0dc47e98 nostrdb: rename: ndb_u64_tsid to ndb_u64_ts
technically more accurate. we are about to introduce a new type called:

	ndb_ts_u64_id

which would be confusing if we didn't do this

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
0916b14b32 nostrdb: make the subscription monitor threadsafe
This was the only thing that wasn't threadsafe. Add a simple mutex
instead of a queue so that polling is quick.

This also means we can't really return the internal subscriptions
anymore, so we remove that for now until we have a safer
interface.

Fixes: https://github.com/damus-io/nostrdb/issues/55
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
6818d001f2 nostrdb: mem: reduce default queue size
This was overkill and was using lots of memory

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
4bf9160502 nostrdb: fix heap corruption on windows
windows thinks this is heap corruption... so I
guess we have to trust it.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
02df1e209b nostrdb: windows: fix threading bugs
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
3186b0e1d3 nostrdb: fix windows build
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
de0935582c nostrdb: ndb_filter_{eq,is_subset_of}: make interfaces const
this makes rust happier

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
573de6b881 nostrdb: ndb_filter_is_subset_of
subset testing for filters. Can be used to see if one subset is
redundant in the presence of a another in the local relay model

Changelog-Added: Add ndb_filter_is_subset_of
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
44ab702792 nostrdb: add ndb_filter_eq
filter equality testing. this works because field elements are sorted

Changelog-Added: Add ndb_filter_eq for filter equality testing
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
1fdf234c46 nostrdb: rename get_elems to find_elements
This is more accurate

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
3018200e95 nostrdb: add ndb_subscription_filters
Expose a way to get the set of filters for a subscription. On the rust
side, we should likely ndb_filter_clone each filter asap, because the
result of this function will only be valid up until the subscription
ends.

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
47b79fc02e nostrdb: ingest: support kind 6 reposts
This also enables processing raw json via ndb import

Fixes: https://github.com/damus-io/nostrdb/issues/46
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
0c483bb55a nostrdb: print search keys to stdout
otherwise it's way too annoying to grep

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
ddd30054e8 nostrdb: nostrdb: fix ndb_builder_find_str.
This will find strings which match the beginning of other strings,
which seems wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
30c5225ed0 nostrdb: content_parser: fix incorrect comment.
Sure, this format would be nice, but it's not what the code does.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
8c446f804c nostrdb: filter: retain const variant of get_int_elemnet
otherwise rust gets bitchy at as

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
e92018aee5 nostrdb: filter: allow mutable int elements
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
cfb140472d nostrdb: bolt11: remove unneeded fields.
If we make unknown_field simply discard, we can remove decoders and
have them discard those fields.

Now we can cut down struct bolt11 to only the fields needed by
invoice.c, and also speed up parsing a little.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
2f5fd54297 nostrdb: bolt11: update to latest version from CLN
Copy the latest, which has parsing fixes.  We make a new explicit
"bolt11_decode_minimal" which doesn't check sigs, rather than neutering
the bolt11_decode logic.

As a bonus, this now correctly parses "LIGHTNING:BECH32..." format
(upper case, with prefix).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
02e970eb9b nostrdb: Makefile: fix missing dependencies on bolt11 headers.
I wondered by `make check` was giving strange errors, until I realized it wasn't fully rebuilding.

Also, remove leftover CCAN files I missed previously.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b4b84e6895 nostrdb: resync with repo
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
7831ede057 nostrdb: ccan: update to latest.
Only change for us: CCAN_TAL_NEVER_RETURN_NULL can be defined if
we don't override tal error handling.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
a8d7d971b1 nostrdb: ccan: sync with normal versions.
This is the version of CCAN which CLN was using at the time these
were taken.  Unfortunately lots of whitespace has been changed,
but AFAICT no source changes.

Here's the command I ran (with ../ccan checked out to 1ae4c432):

```
make update-ccan CCAN_NEW="alignof array_size build_assert check_type container_of cppmagic likely list mem short_types str structeq take tal tal/str typesafe_cb utf8 endian crypto/sha256"
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
201cdd7edc nostrdb: Makefile: build using ccan/ versions of files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
e3ca6ca5b4 nostrdb: bolt11: move utf8_check into local function.
It isn't actually in the CCAN module (though it probably should be!).
So it breaks when we update.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
494386d211 nostrdb: ccan: copy ccan files into their own subdirectory.
This lets them be updated/bugfixed together.  I just copied them for now,
didn't change anything else.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
6c53bc75f2 nostrdb: content_parser: fix blocks_size
we are crossing cursors

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
6001063754 nostrdb: nostrdb: fix ndb_builder_find_str.
This will find strings which match the beginning of other strings,
which seems wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
eb0a1ee807 nostrdb: content_parser: fix incorrect comment.
Sure, this format would be nice, but it's not what the code does.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
827731b9cb nostrdb: filter: retain const variant of get_int_elemnet
otherwise rust gets bitchy at as

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
56d44d0004 nostrdb: filter: allow mutable int elements
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
7742c8fb3c nostrdb: bolt11: remove unneeded fields.
If we make unknown_field simply discard, we can remove decoders and
have them discard those fields.

Now we can cut down struct bolt11 to only the fields needed by
invoice.c, and also speed up parsing a little.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
7f2ee78512 nostrdb: bolt11: update to latest version from CLN
Copy the latest, which has parsing fixes.  We make a new explicit
"bolt11_decode_minimal" which doesn't check sigs, rather than neutering
the bolt11_decode logic.

As a bonus, this now correctly parses "LIGHTNING:BECH32..." format
(upper case, with prefix).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
4d75894bc4 nostrdb: Makefile: fix missing dependencies on bolt11 headers.
I wondered by `make check` was giving strange errors, until I realized it wasn't fully rebuilding.

Also, remove leftover CCAN files I missed previously.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
bbed448ccb nostrdb: ndb_filter_from_json
Changelog-Added: Add method for parsing filter json
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
3fb4d81d48 nostrdb: src: delete copies outside ccan/ dirs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
fc30b68c40 nostrdb: Makefile: build using ccan/ versions of files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
Rusty Russell
0ac25b7aa3 nostrdb: bolt11: move utf8_check into local function.
It isn't actually in the CCAN module (though it probably should be!).
So it breaks when we update.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b326f007f2 nostrdb: expose filter introspection methods
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
a86d8416fc nostrdb: expose ndb_filter_get_elements
This can be used to iterate though filter elements

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
b5c57dc935 nostrdb: make more things const
rust is happier this way

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
7d6814a481 nostrdb: add ndb_filter_json method
Changelog-Added: Add ndb_filter_json method for creating json filters
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
8dd048681b nostrdb: Fix issue where id tag filters are pushed as strings
When creating filters, sometimes IDs are pushed as strings, so if there
is ever a 0 byte, the id prematurely ends, causing the filter to not
match

Fixes: https://github.com/rust-nostr/nostr/issues/454
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
2d02a17af6 nostrdb: fix bech32 parsing and add test
was off by one

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
3171959d85 nostrdb: debug: improve tag index display
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
bca3716e33 nostrdb: fix note content parsing bug with damus.io urls
Changelog-Fixed: Fixed bug where non-bech32 damus io urls would cause corruption
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
57db252783 nostrdb: ndb_note_json: return length
Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00
William Casarin
319579f912 nostrdb: ndb: dump json in filters and fulltext queries
This is much more useful

Signed-off-by: William Casarin <jb55@jb55.com>
2025-08-11 16:40:00 -07:00