Setup CI for adapter with service dependencies - #8
Open
timkpaine wants to merge 2 commits into
Open
Conversation
timkpaine
force-pushed
the
tkp/adapters
branch
3 times, most recently
from
February 4, 2024 13:29
7b48be6 to
0f91511
Compare
timkpaine
added
the
tag: internal
timkpaine
force-pushed
the
tkp/adapters
branch
7 times, most recently
from
February 19, 2024 17:01
93e5d1b to
8693a5a
Compare
timkpaine
force-pushed
the
tkp/adapters
branch
2 times, most recently
from
February 28, 2024 14:57
87d1023 to
95b9a53
Compare
timkpaine
force-pushed
the
tkp/adapters
branch
2 times, most recently
from
May 24, 2024 08:30
08995d1 to
7a07db5
Compare
timkpaine
marked this pull request as ready for review
June 30, 2024 03:29
timkpaine
requested review from
AdamGlustein,
alexddobkin,
czgdp1807,
ptomecek,
robambalu and
svatasoiu
as code owners
June 30, 2024 03:29
timkpaine
added
the
adapter: kafka
Member
Author
Collaborator
Based on the test and the error message it looks like whats happening is that the sub is missing the first pub. Its possible the daemon is late to register the subscriber before the first publish. One way to try and fix it is to delay the first pub a bit. Another possibility is to compare the last N msgs rather than the first N |
timkpaine
force-pushed
the
tkp/adapters
branch
2 times, most recently
from
September 2, 2024 00:04
fc66f1e to
8bca913
Compare
timkpaine
force-pushed
the
tkp/adapters
branch
2 times, most recently
from
September 10, 2024 14:11
c64cb93 to
21229ed
Compare
timkpaine
force-pushed
the
tkp/adapters
branch
from
November 5, 2024 21:19
e978101 to
ad2cdf0
Compare
timkpaine
force-pushed
the
tkp/adapters
branch
from
November 22, 2024 19:02
ad2cdf0 to
6f430de
Compare
timkpaine
marked this pull request as draft
March 1, 2025 00:33
timkpaine
force-pushed
the
tkp/adapters
branch
6 times, most recently
from
March 2, 2025 22:54
a7aadad to
520c30a
Compare
timkpaine
commented
Mar 4, 2025
Merged
Collaborator
|
@timkpaine Resolving conflicts looks straightforward, and so assuming the tests still pass, can we take this PR out of draft mode? |
timkpaine
force-pushed
the
tkp/adapters
branch
2 times, most recently
from
February 14, 2026 23:35
83a3f7a to
85c9eb5
Compare
timkpaine
marked this pull request as ready for review
February 18, 2026 03:29
timkpaine
force-pushed
the
tkp/adapters
branch
2 times, most recently
from
August 13, 2026 23:12
5f6236a to
a74fea0
Compare
Run the Kafka adapter integration tests in CI against a real broker. Adds a test_adapters job that stands up ci/kafka/docker-compose.yml, sets CSP_TEST_<ADAPTER>, and runs the matching tests, plus dockerup/dockerps/dockerdown targets for doing the same locally. The compose stack is trimmed to zookeeper and a single broker, with healthchecks so `docker compose up --wait` blocks until the broker accepts connections rather than relying on a fixed sleep. Only 9092 is published, bound to loopback. Broker-side topic auto-creation is disabled so test_invalid_topic can exercise the failure path, and tests create their topics explicitly through the Kafka AdminClient. Test changes target the startup race where a subscriber misses the first few records while its consumer group is being assigned. Rather than loosening the assertions, the affected tests align on the first record the subscriber saw and then require an exact contiguous run, so loss, duplication and reordering are still caught. Also drops curl from the Windows chocolatey install: the package fails whenever a new version is approved on the community feed before it is downloadable, and curl.exe ships with Windows. Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
timkpaine
force-pushed
the
tkp/adapters
branch
from
August 14, 2026 17:41
a74fea0 to
fcb6fa5
Compare
The tests bootstrapped from "localhost", which also resolves to ::1 where the container publishes nothing, so librdkafka failed whichever address it picked next. Point them at 127.0.0.1, matching the published port and the advertised listener. A single all-brokers-down report then killed the engine, though librdkafka reports it on every failed connection round and reconnects on its own. Escalation now waits for the brokers to stay down for broker_down_tolerance, checked on the poll threads rather than on the report, since librdkafka may not report again for seconds. Shutdown no longer waits on a flush that cannot complete. A topic that the broker will not auto-create left a subscriber looking idle forever. The consumer reports that through poll rather than the event callback, so escalate it there, and on the publisher through the delivery report. Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.