Skip to content

Add an endpoint to list plugin states and a partial update - #88

Open
juliens wants to merge 3 commits into
masterfrom
feat/plugin-states
Open

juliens wants to merge 3 commits into
masterfrom
feat/plugin-states

Conversation

@juliens

@juliens juliens commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Piceus calls GitHub for every plugin on each run and hits the rate limit. With these endpoints, it loads the state of all plugins in one call and sends the ETag of the last GitHub tags response. When GitHub answers 304, piceus skips the plugin, and a 304 is not counted in the rate limit.

GitHub calls per plugin that did not change (measured on 30 repositories, private mode as in production):

Calls per plugin
Today 5.8
Read the plugin files once 3.9
Skip with the tags ETag 0 (one 304, not counted)

With more than 870 repositories in production, this goes from about 5,000 calls per run to only the calls for new or changed plugins.

Changes:

  • tagsETag field on the plugin.
  • GET /internal/states: id, name, latest version, stars, hidden, disabled and tags ETag of all plugins, hidden and disabled ones included.
  • PATCH /internal/:uuid: updates only stars, hidden and tagsETag.

The piceus change that uses these endpoints: traefik/piceus#168

Piceus can load the state of all plugins in one call and update only
stars, hidden or tagsETag without sending the whole plugin.
@mloiseleur

Copy link
Copy Markdown
Member

I've not read yet the details of this PR, but I already love it 👍!
Thanks @juliens

Comment thread cmd/serve/serve.go
r.Handler(http.MethodGet, "/", otelhttp.NewHandler(http.HandlerFunc(handler.List), "internal_list"))
r.Handler(http.MethodPost, "/", otelhttp.NewHandler(http.HandlerFunc(handler.Create), "internal_create"))
r.Handler(http.MethodPut, "/:uuid", otelhttp.NewHandler(http.HandlerFunc(handler.Update), "internal_update"))
r.Handler(http.MethodPatch, "/:uuid", otelhttp.NewHandler(http.HandlerFunc(handler.Patch), "internal_patch"))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The blacklist workaround is not needed here: httprouter has one tree per method, and the GET tree only has / (no /:uuid). Keeping the route in the internal router also keeps its PanicHandler and JSON NotFound.

Suggested change
r.Handler(http.MethodPatch, "/:uuid", otelhttp.NewHandler(http.HandlerFunc(handler.Patch), "internal_patch"))
r.Handler(http.MethodPatch, "/:uuid", otelhttp.NewHandler(http.HandlerFunc(handler.Patch), "internal_patch"))
r.Handler(http.MethodGet, "/states", otelhttp.NewHandler(http.HandlerFunc(handler.ListStates), "internal_list_states"))

Then registerInternalStatesRoute and its call can be removed.

Comment thread cmd/serve/serve.go Outdated
Comment on lines +104 to +109
// registerInternalStatesRoute registers the plugin states endpoint directly on
// the ServeMux, like the blacklist endpoints.
func registerInternalStatesRoute(r *http.ServeMux, handler handlers.Handlers) {
r.Handle("GET /internal/states", otelhttp.NewHandler(http.HandlerFunc(handler.ListStates), "internal_list_states"))
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed if the route moves to buildInternalRouter (see above). Also remove the call on line 60.

Suggested change
// registerInternalStatesRoute registers the plugin states endpoint directly on
// the ServeMux, like the blacklist endpoints.
func registerInternalStatesRoute(r *http.ServeMux, handler handlers.Handlers) {
r.Handle("GET /internal/states", otelhttp.NewHandler(http.HandlerFunc(handler.ListStates), "internal_list_states"))
}

Comment thread pkg/handlers/state.go Outdated
Comment on lines +63 to +91
body, err := io.ReadAll(req.Body)
if err != nil {
span.RecordError(err)
logger.Error().Err(err).Msg("Unable to read body for patch")
JSONError(rw, http.StatusBadRequest, err.Error())

return
}

var patch db.PluginPatch

if len(body) > 0 {
dec := json.NewDecoder(bytes.NewReader(body))
dec.DisallowUnknownFields()

if err = dec.Decode(&patch); err != nil {
span.RecordError(err)
logger.Error().Err(err).Msg("Unable to decode plugin patch")
JSONError(rw, http.StatusBadRequest, err.Error())

return
}
}

if patch.IsEmpty() {
JSONError(rw, http.StatusBadRequest, "nothing to update")

return
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few things here:

  • io.ReadAll has no size limit.
  • Data after the first JSON value is ignored: {"stars":1}{"hidden":true} returns 200 and only sets stars.
  • Negative stars are accepted.

Simpler version, same behavior for an empty body:

Suggested change
body, err := io.ReadAll(req.Body)
if err != nil {
span.RecordError(err)
logger.Error().Err(err).Msg("Unable to read body for patch")
JSONError(rw, http.StatusBadRequest, err.Error())
return
}
var patch db.PluginPatch
if len(body) > 0 {
dec := json.NewDecoder(bytes.NewReader(body))
dec.DisallowUnknownFields()
if err = dec.Decode(&patch); err != nil {
span.RecordError(err)
logger.Error().Err(err).Msg("Unable to decode plugin patch")
JSONError(rw, http.StatusBadRequest, err.Error())
return
}
}
if patch.IsEmpty() {
JSONError(rw, http.StatusBadRequest, "nothing to update")
return
}
var patch db.PluginPatch
dec := json.NewDecoder(http.MaxBytesReader(rw, req.Body, 1<<20))
dec.DisallowUnknownFields()
if err = dec.Decode(&patch); err != nil && !errors.Is(err, io.EOF) {
span.RecordError(err)
logger.Error().Err(err).Msg("Unable to decode plugin patch")
JSONError(rw, http.StatusBadRequest, err.Error())
return
}
if dec.More() {
JSONError(rw, http.StatusBadRequest, "body must contain a single JSON object")
return
}
if patch.IsEmpty() {
JSONError(rw, http.StatusBadRequest, "nothing to update")
return
}
if patch.Stars != nil && *patch.Stars < 0 {
JSONError(rw, http.StatusBadRequest, "stars must not be negative")
return
}

The bytes import can then be removed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants