Repository navigation
SOLR-18320 follow-up: streaming zipDirectory, drop dead passthrough wrapper - #5260
Merged
Merged
Conversation
- Utils.zipDirectory(Path) was the only shape used by two of its three callers (CreateTool, DownloadConfigSet), which both feed the bytes into an HTTP request body - fine for them, but it forced RemoteSolrBackend to buffer the whole configset in memory just to write it straight back out to a temp file it already had open. Added a streaming zipDirectory(Path, OutputStream) overload with the walk/skip-hidden/ normalize-separator logic in one place; zipDirectory(Path) now delegates to it via a ByteArrayOutputStream, and RemoteSolrBackend calls the streaming form directly against Files.newOutputStream(tempZip), restoring the original single-pass I/O. - Removed CreateTool.zipConfigSet(Path), a pure one-line passthrough to Utils.zipDirectory with no behavior of its own; both call sites (the production upload call and CreateToolTest) now call Utils.zipDirectory directly. - Added a test confirming the two zipDirectory overloads produce equivalent archives (compared by decoded entry name/content, not raw bytes - ZipEntry defaults to the current time when unset, so two back-to-back calls aren't guaranteed byte-identical). Verified: CreateToolTest, CreateToolPicocliTest, and TestSchemaDesignerConfigSetHelper (14 tests) all pass; full compileJava compileTestJava across the repo is clean; ./gradlew tidy clean.
Contributor
Author
|
/no-changelog |
Contributor
|
I wish I had a chance to review. The title says "streaming zipDirectory" but I don't see it. The following method replacement actually streams the zip (meaning, doesn't use an intermediate temp file). I will get this in at some point in the future when I return to SolrBackend stuff: public void uploadConfigSet(Path configDir, String name) throws SolrServerException, IOException {
new ConfigSetAdminRequest.Upload()
.setConfigSetName(name)
.setUploadContent(new RequestWriter.ContentWriter() {
@Override
public void write(OutputStream os) throws IOException {
// stream it directly out; no intermediate temp file :-)
Utils.zipDirectory(configDir, os);
}
@Override
public String getContentType() {
return "application/zip";
}
})
.process(adminClient);
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I was too quick on the merge and realized I had some self review comments I had not dealt with.
Summary
Small cleanup follow-up to #4672 (already merged), from a
/simplifypass that landed one commit after that PR was squash-merged, so it never made it in.Utils.zipDirectory(Path)was shaped around "return bytes for an HTTP body" - fine for its two other callers (CreateTool,DownloadConfigSet), but it forcedRemoteSolrBackend.uploadConfigSetto buffer the entire configset zip in memory just to write it straight back out to a temp file it already had open (Files.write(tempZip, Utils.zipDirectory(configDir))). Added a streamingzipDirectory(Path, OutputStream)overload with the shared walk/skip-hidden/normalize-separator logic in one place; the byte[]-returning version now delegates to it, andRemoteSolrBackendcalls the streaming form directly againstFiles.newOutputStream(tempZip), restoring the original single-pass I/O.CreateTool.zipConfigSet(Path), a pure one-line passthrough toUtils.zipDirectorywith no behavior of its own. Both call sites (the production upload call andCreateToolTest) now callUtils.zipDirectorydirectly.zipDirectoryoverloads produce equivalent archives - compared by decoded entry name/content rather than raw bytes, sinceZipEntrydefaults to the current time when none is set explicitly, so two back-to-back calls aren't guaranteed byte-identical.Test plan
CreateToolTest,CreateToolPicocliTest,TestSchemaDesignerConfigSetHelper(14 tests) all passcompileJava/compileTestJavaacross the repo is clean./gradlew tidycleanmain(post-SOLR-18320: Use Configsets V2 API instead of direct ZooKeeper access in bin/solr create #4672 squash-merge) and re-ran the test suite there🤖 Generated with Claude Code