Metadata-Version: 2.1
Name: esdbclient
Version: 0.4.12
Summary: Python gRPC Client for EventStoreDB
Home-page: https://github.com/pyeventsourcing/esdbclient
License: BSD 3-Clause
Author: John Bywater
Author-email: john.bywater@appropriatesoftware.net
Requires-Python: >=3.7,<4.0
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: BSD License
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Requires-Dist: grpcio (>=1.44.0,<1.45.0)
Requires-Dist: protobuf (>=3.20.0,<3.21.0)
Requires-Dist: typing_extensions
Project-URL: Repository, https://github.com/pyeventsourcing/esdbclient
Description-Content-Type: text/markdown

# Python gRPC Client for EventStoreDB

This package provides a Python gRPC client for
[EventStoreDB](https://www.eventstore.com/). It has been
developed and tested to work with EventStoreDB LTS version 21.10,
and with Python versions 3.7, 3.8, 3.9, and 3.10.

Methods have typing annotations, the static typing is checked
with mypy, and the test coverage is 100%.

Not all the features of the EventStoreDB API are presented
by this client in its current form, however many of the most
useful aspects are presented in an easy-to-use interface (see below).


## Installation

Use pip to install this package from
[the Python Package Index](https://pypi.org/project/esdbclient/).

    $ pip install esdbclient

It is recommended to install Python packages into a Python virtual environment.


## Getting started

### Start EventStoreDB

Use Docker to run EventStoreDB from the official container image on DockerHub.

    $ docker run -d --name my-eventstoredb -it -p 2113:2113 -p 1113:1113 eventstore/eventstore:21.10.2-buster-slim --insecure

Please note, this will start the server without SSL/TLS enabled, allowing
only "insecure" connections. This version of this Python client does not
support SSL/TLS connections. A future version of this library will support
"secure" connections.


### Construct client

The class `EsdbClient` can be constructed with a `uri` that indicates the
hostname and port number of the EventStoreDB server.

```python
from esdbclient import EsdbClient

client = EsdbClient(uri='localhost:2113')
```

### Append events

The method `append_events()` can be used to append events to
a stream. If the append operation is successful, this method
will return the database "commit position" as it was when the
operation was completed. Otherwise, an exception will be raised.

The commit position value can be used to wait for downstream
processing to have proceeded the appended events, so that for
example a user interface that depends on eventually consistent
materialised views can wait after making a command before making
a query.

A "commit position" is a monotonically increasing integer representing
the position of the recorded event in a "total order" of all recorded
events in the database. The sequence of commit positions is not gapless.
It represents the position of the event record on disk, and there are
usually large differences between successive commits.

Three arguments are required, `stream_name`, `expected_position`
and `events`.

The `stream_name` argument is a string that uniquely identifies
the stream in the database.

The `expected_position` argument is an optional integer that specifies
the expected position of the end of the stream in the database: either
a positive integer representing the expected current position of the stream,
or `None` if the stream is expected not to exist. If there is a mismatch
with the actual position of the end of the stream, an exception
`ExpectedPositionError` will be raised by the client. This accomplishes
optimistic concurrency control when appending new events. If you need to
get the current position of the end of a steam, use the `get_stream_position()`
method (see below). If you wish to disable optimistic concurrency, set the
`expected_position` to a negative integer.

The `events` argument is a sequence of new event objects to be appended to the
named stream. The class `NewEvent` can be used to construct new event objects.

In the example below, a stream is created by appending a new event with
`expected_position=None`.

```python
from uuid import uuid4

from esdbclient import NewEvent

# Construct new event object.
event1 = NewEvent(
    type='OrderCreated',
    data=b'{}',
    metadata=b'{}'
)

# Define stream name.
stream_name1 = str(uuid4())

# Append list of events to new stream.
commit_position1 = client.append_events(
    stream_name=stream_name1,
    expected_position=None,
    events=[event1],
)
```

In the example below, two subsequent events are appended to an existing
stream. The sequences of stream positions are zero-based, and so when a
stream only has one recorded event, the expected position of the end of
the stream is `0`.

```python
event2 = NewEvent(
    type='OrderUpdated',
    data=b'{}',
    metadata=b'{}',
)
event3 = NewEvent(
    type='OrderDeleted',
    data=b'{}',
    metadata=b'{}',
)

commit_position2 = client.append_events(
    stream_name=stream_name1,
    expected_position=0,
    events=[event2, event3],
)
```

Please note, whilst the append operation is atomic, so that either all
or none of a given list of events will be recorded, by design it is only
possible with EventStoreDB to atomically record events in one stream.


### Get current stream position

The method `get_stream_position()` can be used to get the
position of the end of a stream (the position of the last
recorded event in the stream).

```python
stream_position = client.get_stream_position(
    stream_name=stream_name1
)

assert stream_position == 2
```

The sequence of stream positions is gapless. It is zero-based, so that
the position of the end of the stream when one event has been appended
is `0`. The position is `1` after two events have been appended, `2`
after three events have been appended, and so on.

If a stream does not exist, the returned stream position is `None`,
which corresponds to the required expected position when appending
events to a stream that does not exist (see above).

```python
stream_position = client.get_stream_position(
    stream_name='stream-unknown'
)

assert stream_position == None
```

This method takes an optional argument `timeout` which is a float that sets
a deadline for the completion of the gRPC operation.


### Read stream events

The method `read_stream_events()` can be used to read the recorded
events in a stream. An iterable object of recorded events is returned.

One argument is required, `stream_name`, which is the name of the
stream to be read. By default, the recorded events in the stream
are returned in the order they were recorded.

The example below shows how to read the recorded events of a stream
forwards from the start of the stream to the end of the stream.

```python
response = client.read_stream_events(
    stream_name=stream_name1
)
```

The iterable object is actually a Python generator, and we need to
iterate over it to actually get the recorded events from gRPC. So
let's convert it into a list, so that we can call `len()`, and so
that we can index it to check we have the individual events recorded
above.

```python
events = list(response)
```

Now that we have an actual list of events, we can check we have the
three events that we recorded in the stream above.

```python
assert len(events) == 3

assert events[0].stream_name == stream_name1
assert events[0].stream_position == 0
assert events[0].type == event1.type
assert events[0].data == event1.data

assert events[1].stream_name == stream_name1
assert events[1].stream_position == 1
assert events[1].type == event2.type
assert events[1].data == event2.data

assert events[2].stream_name == stream_name1
assert events[2].stream_position == 2
assert events[2].type == event3.type
assert events[2].data == event3.data
```

The method `read_stream_events()` also supports four optional arguments,
`position`, `backwards`, `limit`, and `timeout`.

The argument `position` is an optional integer that can be used to indicate
the position in the stream from which to start reading. This argument is `None`
by default, which means the stream will be read either from the start of the
stream (the default behaviour), or from the end of the stream if `backwards` is
`True`. When reading a stream from a specific position in the stream, the
recorded event at that position WILL be included, both when reading forwards
from that position, and when reading backwards from that position.

The argument `backwards` is a boolean, by default `False`, which means the
stream will be read forwards by default, so that events are returned in the
order they were appended, If `backwards` is `True`, the stream will be read
backwards, so that events are returned in reverse order.

The argument `limit` is an integer which limits the number of events that will
be returned.

The argument `timeout` is a float which sets a deadline for the completion of
the gRPC operation.

The example below shows how to read recorded events in a stream forwards from
a specific stream position to the end of the stream.

```python
events = list(
    client.read_stream_events(
        stream_name=stream_name1,
        position=1,
    )
)

assert len(events) == 2

assert events[0].stream_name == stream_name1
assert events[0].stream_position == 1
assert events[0].type == event2.type
assert events[0].data == event2.data

assert events[1].stream_name == stream_name1
assert events[1].stream_position == 2
assert events[1].type == event3.type
assert events[1].data == event3.data
```

The example below shows how to read the recorded events in a stream backwards from
the end of the stream to the start of the stream.

```python
events = list(
    client.read_stream_events(
        stream_name=stream_name1,
        backwards=True,
    )
)

assert len(events) == 3

assert events[0].stream_name == stream_name1
assert events[0].stream_position == 2
assert events[0].type == event3.type
assert events[0].data == event3.data

assert events[1].stream_name == stream_name1
assert events[1].stream_position == 1
assert events[1].type == event2.type
assert events[1].data == event2.data
```

The example below shows how to read a limited number (two) of the recorded events
in stream forwards from the start of the stream.

```python
events = list(
    client.read_stream_events(
        stream_name=stream_name1,
        limit=2,
    )
)

assert len(events) == 2

assert events[0].stream_name == stream_name1
assert events[0].stream_position == 0
assert events[0].type == event1.type
assert events[0].data == event1.data

assert events[1].stream_name == stream_name1
assert events[1].stream_position == 1
assert events[1].type == event2.type
assert events[1].data == event2.data
```

The example below shows how to read a limited number of the recorded events
in a stream backwards from a given stream position.

```python
events = list(
    client.read_stream_events(
        stream_name=stream_name1,
        position=2,
        backwards=True,
        limit=1,
    )
)

assert len(events) == 1

assert events[0].stream_name == stream_name1
assert events[0].stream_position == 2
assert events[0].type == event3.type
assert events[0].data == event3.data
```

### Read all recorded events

The method `read_all_events()` can be used to read all recorded events
in the database in the order they were committed. An iterable object of
recorded events is returned.

The example below shows how to read all events in the database in the
order they were recorded.

```python
events = list(client.read_all_events())

assert len(events) >= 3
```

The method `read_stream_events()` supports six optional arguments,
`position`, `backwards`, `filter_exclude`, `filter_include`, `limit`,
and `timeout`.

The argument `position` is an optional integer that can be used to specify
the commit position from which to start reading. This argument is `None` by
default, meaning that all the events will be read either from the start, or
from the end if `backwards` is `True` (see below). Please note, if specified,
the specified position must be an actually existing commit position, because
any other number will result in a server error (at least in EventStoreDB v21.10).
Please also note, when reading forwards the event at the given position
WILL be included. However when reading backwards, the event at the given position
will NOT be included.

The argument `backwards` is a boolean which is by default `False` meaning all the
events will be read forwards by default, so that events are returned in the
order they were committed, If `backwards` is `True`, all the events will be read
backwards, so that events are returned in reverse order.

The argument `filter_exclude` is a sequence of regular expressions that
match the type strings of recorded events that should not be included.
This argument is ignored if `filter_include` is set.

The argument `filter_include` is a sequence of regular expressions
that match the type strings of recorded events that should be included. By
default, this argument is an empty tuple. If this argument is set to a
non-empty sequence, the `filter_include` argument is ignored.

Please note, the filtering happens on the EventStoreDB server, and the
`limit` argument is applied after filtering. See below for more information
about filter regular expressions.

The argument `limit` is an integer which limits the number of events that will
be returned.

The argument `timeout` is a float which sets a deadline for the completion of
the gRPC operation.

The example below shows how to read all recorded events from a particular commit position.

```python
events = list(
    client.read_all_events(
        position=commit_position1
    )
)

assert len(events) == 3

assert events[0].stream_name == stream_name1
assert events[0].stream_position == 0
assert events[0].type == event1.type
assert events[0].data == event1.data

assert events[1].stream_name == stream_name1
assert events[1].stream_position == 1
assert events[1].type == event2.type
assert events[1].data == event2.data

assert events[2].stream_name == stream_name1
assert events[2].stream_position == 2
assert events[2].type == event3.type
assert events[2].data == event3.data
```

The example below shows how to read all recorded events in reverse order.

```python
events = list(
    client.read_all_events(
        backwards=True
    )
)

assert len(events) >= 3

assert events[0].stream_name == stream_name1
assert events[0].stream_position == 2
assert events[0].type == event3.type
assert events[0].data == event3.data

assert events[1].stream_name == stream_name1
assert events[1].stream_position == 1
assert events[1].type == event2.type
assert events[1].data == event2.data

assert events[2].stream_name == stream_name1
assert events[2].stream_position == 0
assert events[2].type == event1.type
assert events[2].data == event1.data
```

The example below shows how to read a limited number (one) of the recorded events
in the database forwards from a specific commit position.

```python
events = list(
    client.read_all_events(
        position=commit_position1,
        limit=1,
    )
)

assert len(events) == 1

assert events[0].stream_name == stream_name1
assert events[0].stream_position == 0
assert events[0].type == event1.type
assert events[0].data == event1.data
```

The example below shows how to read a limited number (one) of the recorded events
in the database backwards from the end. This gives the last recorded event.

```python
events = list(
    client.read_all_events(
        backwards=True,
        limit=1,
    )
)

assert len(events) == 1

assert events[0].stream_name == stream_name1
assert events[0].stream_position == 2
assert events[0].type == event3.type
assert events[0].data == event3.data
```

### Get current commit position

The method `get_commit_position()` can be used to get the current
commit position of the database.

```python
commit_position = client.get_commit_position()
```

This method is provided as a convenience when testing, and otherwise isn't
very useful. In particular, when reading all events (see above) or subscribing
to all events with a catch-up subscription (see below), the commit position
would normally be read from the downstream database, so that you are reading
from the last position that was successfully processed.

This method takes an optional argument `timeout` which is a float that sets
a deadline for the completion of the gRPC operation.


### Catch-up subscriptions

The method `subscribe_all_events()` can be used to create a
"catch-up subscription" to EventStoreDB.

This method takes an optional `position` argument, which can be
used to specify a commit position from which to subscribe for
recorded events. The default value is `None`, which means
the subscription will operate from the first recorded event
in the database.

This method returns an iterable object, from which recorded events
can be obtained by iteration, including events that are recorded
after the subscription was created.

Many catch-up subscriptions can be created, concurrently or
successively, and all will receive all the events they are
subscribed to receive.

The value of the `commit_position` attribute of recorded events can be
recorded along with the results of processing recorded events,
to track progress and to allow event processing to be resumed at
the correct position.

The example below shows how to subscribe to receive all recorded
events from a specific commit position. Three already-existing
events are received, and then three new events are recorded, which
are then received via the subscription.

```python

# Get the commit position (usually from database of materialised views).
commit_position = client.get_commit_position()

# Append three events.
stream_name1 = str(uuid4())
event1 = NewEvent(
    type='OrderCreated',
    data=b'{}',
    metadata=b'{}',
)
event2 = NewEvent(
    type='OrderUpdated',
    data=b'{}',
    metadata=b'{}',
)
event3 = NewEvent(
    type='OrderDeleted',
    data=b'{}',
    metadata=b'{}',
)
client.append_events(
    stream_name=stream_name1,
    expected_position=None,
    events=[event1, event2, event3],
)

# Subscribe from the commit position.
subscription = client.subscribe_all_events(
    position=commit_position
)

# Catch up by receiving the three events from the subscription.
events = []
for event in subscription:
    # Check the stream name is 'stream_name1'.
    assert event.stream_name == stream_name1
    events.append(event)
    if len(events) == 3:
        break

# Append three more events.
stream_name = str(uuid4())
event4 = NewEvent(
    type='OrderCreated',
    data=b'{}',
    metadata=b'{}',
)
event5 = NewEvent(
    type='OrderUpdated',
    data=b'{}',
    metadata=b'{}',
)
event6 = NewEvent(
    type='OrderDeleted',
    data=b'{}',
    metadata=b'{}',
)
client.append_events(
    stream_name=stream_name,
    expected_position=None,
    events=[event4, event5, event6],
)

# Receive the three new events from the same subscription.
events = []
for event in subscription:
    # Check the stream name is 'stream_name2'.
    assert event.stream_name == stream_name
    events.append(event)
    if len(events) == 3:
        break
```

This method also support three other optional arguments, `filter_exclude`,
`filter_include`, and `timeout`.

The argument `filter_exclude` is a sequence of regular expressions that
match the type strings of recorded events that should not be included.
This argument is ignored if `filter_include` is set.

The argument `filter_include` is a sequence of regular expressions
that match the type strings of recorded events that should be included. By
default, this argument is an empty tuple. If this argument is set to a
non-empty sequence, the `filter_include` argument is ignored.

Please note, in this version of this Python client, the filtering happens
within the client (rather than on the server as when reading all events) because
passing these filter options in the read request for subscriptions seems to cause
an error in EventStoreDB v21.10. See below for more information about filter
regular expressions.

The argument `timeout` is a float which sets a deadline for the completion of
the gRPC operation. This probably isn't very useful, but is included for
completeness and consistency with the other methods.

Catch-up subscriptions are not registered in EventStoreDB (they are not
"persistent subscriptions). It is simply a streaming gRPC call which is
kept open by the server, with newly recorded events sent to the client
as the client iterates over the subscription. This kind of subscription
is closed as soon as the subscription object goes out of memory.

```python
# End the subscription.
del subscription
```

Received events do not need to be (and indeed cannot be) acknowledged back
to the EventStoreDB server. Acknowledging events is an aspect of "persistent
subscriptions", which is a feature of EventStoreDB that is not (currently)
supported by this client. Whilst there are some advantages of persistent
subscribers, by recording in the upstream server the position in the commit
sequence of events that have been processed, there is a danger of "dual writing"
in the consumption of events. The danger is that if the event is successfully
processed but then the acknowledgment fails, the event may be processed more
than once. On the other hand, if the acknowledgment is successful but then the
processing fails, the event may not be been processed. The only protection against
this danger is to avoid "dual writing" by atomically recording the commit position
of an event that has been processed along with the results of process the event,
that is with both things being recorded in the same transaction.

To accomplish "exactly once" processing of the events, the commit position
of a recorded event should be recorded atomically and uniquely along with
the result of processing recorded events, for example in the same database
as materialised views when implementing eventually-consistent CQRS, or in
the same database as a downstream analytics or reporting or archiving
application. This avoids "dual writing" in the processing of events.

The subscription object might be used directly when processing events. It might
also be used within a thread dedicated to receiving events, with recorded events
put on a queue for processing in a different thread. This package doesn't provide
such thread or queue objects, you would need to do that yourself. Just make sure
to reconstruct the subscription (and the queue) using your last recorded commit
position when resuming the subscription after an error, to be sure all events
are processed once.

### The NewEvent class

The `NewEvent` class can be used to define new events.

The attribute `type` is a unicode string, used to specify the type of the event
to be recorded.

The attribute `data` is a byte string, used to specify the data of the event
to be recorded. Please note, in this version of this Python client,
writing JSON event data to EventStoreDB isn't supported, but it might be in
a future version.

The attribute `metadata` is a byte string, used to specify metadata for the event
to be recorded.

```python
new_event = NewEvent(
    type='OrderCreated',
    data=b'{}',
    metadata=b'{}',
)
```

### The RecordedEvent class

The `RecordedEvent` class is used when reading recorded events.

The attribute `type` is a unicode string, used to indicate the type of the event
that was recorded.

The attribute `data` is a byte string, used to indicate the data of the event
that was recorded.

The attribute `metadata` is a byte string, used to indicate metadata for the event
that was recorded.

The attribute `stream_name` is a unicode string, used to indicate the type of
the name of the stream in which the event was recorded.

The attribute `stream_position` is an integer, used to indicate
the position in the stream at which the event was recorded.

The attribute `commit_position` is an integer, used to indicate
the position in total order of all recorded events at which the
event was recorded.

```python
from esdbclient.events import RecordedEvent

recorded_event = RecordedEvent(
    type='OrderCreated',
    data=b'{}',
    metadata=b'{}',
    stream_name='stream1',
    stream_position=0,
    commit_position=512,
)
```

### Filter regular expressions

The `filter_exclude` and `filter_include` arguments in `read_all_events()` and
`subscribe_all_events()` are applied to the `type` attribute of recorded events.

The default value of the `filter_exclude` arguments is designed to exclude
EventStoreDB "system events", which otherwise would be included. System
events, by convention in EventStoreDB, all have `type` strings that
start with the `$` sign.

Please note, characters that have a special meaning in regular expressions
will need to be escaped (with double-backslash) when matching these characters
in type strings.

For example, to match EventStoreDB system events, use the sequence `['\\$.*']`.
Please note, the constant `ESDB_EVENTS_REGEX` is set to `'\\$.*'`. You
can import this value (`from esdbclient import ESDB_EVENTS_REGEX`) and use
it when building longer sequences of regular expressions. For example,
to exclude system events and snapshots, you might use the sequence
`[ESDB_EVENTS_REGEX, '.*Snapshot']` as the value of the `filter_exclude`
argument.


### Stop EventStoreDB

Use Docker to stop and remove the EventStoreDB container.

    $ docker stop my-eventstoredb
	$ docker rm my-eventstoredb


## Developers

Clone the project repository, set up a virtual environment, and install
dependencies.

Use your IDE (e.g. PyCharm) to open the project repository. Create a
Poetry virtual environment, and then update packages.

    $ make update-packages

Alternatively, use the ``make install`` command to create a dedicated
Python virtual environment for this project.

    $ make install

The ``make install`` command uses the build tool Poetry to create a dedicated
Python virtual environment for this project, and installs popular development
dependencies such as Black, isort and pytest.

Add tests in `./tests`. Add code in `./esdbclient`.

Start EventStoreDB.

    $ make start-eventstoredb

Run tests.

    $ make test

Stop EventStoreDB.

    $ make stop-eventstoredb

Check the formatting of the code.

    $ make lint

Reformat the code.

    $ make fmt

Add dependencies in `pyproject.toml` and then update installed packages.

    $ make update-packages

