Skip to content

add support for automated splunk update when new version is available - #97

Open
asifulhaque786 wants to merge 4 commits into
masterfrom
automate-splunk-update
Open

add support for automated splunk update when new version is available#97
asifulhaque786 wants to merge 4 commits into
masterfrom
automate-splunk-update

Conversation

@asifulhaque786

Copy link
Copy Markdown
Member

@asifulhaque786
asifulhaque786 marked this pull request as draft August 31, 2026 05:45
@asifulhaque786 asifulhaque786 changed the title add support for automated splunk update when new version is available… add support for automated splunk update when new version is available Aug 31, 2026
@asifulhaque786
asifulhaque786 force-pushed the automate-splunk-update branch from bf4b4f3 to e4391f5 Compare August 31, 2026 16:02
@asifulhaque786
asifulhaque786 marked this pull request as ready for review September 1, 2026 02:55
@asifulhaque786
asifulhaque786 force-pushed the automate-splunk-update branch 5 times, most recently from b4d3c34 to cf3bc00 Compare September 1, 2026 04:41
Adds an opt-in auto-upgrade path for the Splunk Universal Forwarder:
leaving version/build unset tracks the latest release via a new
splunk_forwarder_latest_version() function (scraped from Splunk's
download page, cached 6h, with stale-cache/hardcoded fallbacks so a
lookup failure never breaks catalog compilation). Pinning version
requires build to match, enforced by a fail-fast check. The pin-vs-
fetch decision lives in the function itself - it takes version/build
as required (but nullable) params and returns them as-is when both are
given, only falling through to the cache/fetch path when neither is.

Upgrades/downgrades are applied via package_ensure => 'latest': both
the dpkg and rpm providers read the target version directly from the
already-staged source file and, if it differs from what's installed,
apply it in place via `dpkg -i`/`rpm -U --oldpackage` - which handles
both directions natively and leaves etc/system/local/* (including
seeded admin credentials) untouched, since it isn't a packaged
conffile.

Also extends the accept-tos exec's timeout from Puppet's default 300s
to 900s via a resource collector override (a fresh install's first
start can take longer than that), and makes Service resources respect
noop_value like the rest of this profile's resources.
Aman1994

This comment was marked as spam.

return { "version" => version, "build" => build } if version && build

cached = read_cache
return cached if cached && !stale?(cached)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 — this returns a value that violates the return_type declared on line 33, breaking every catalog on the second compile.

write_cache persists data.merge("fetched_at" => ...), and read_cache is a plain JSON.parse, so cached is {"version"=>..., "build"=>..., "fetched_at"=>...}. Puppet's Struct is strict — unlisted keys are rejected.

So: first compile fetches, returns a clean 2-key hash, passes. Every compile in the next 6h hits this line and dies with a return-type mismatch. cached || FALLBACK on line 48 has the same defect.

Stripping on read rather than on write keeps the timestamp available for stale?:

return cached.slice("version", "build") if cached && !stale?(cached)
...
rescue StandardError => e
  Puppet.warning(...)
  cached ? cached.slice("version", "build") : FALLBACK
end

# compilation for every node using this class.
#
Puppet::Functions.create_function(:splunk_forwarder_latest_version) do
DOWNLOAD_PAGE = "https://www.splunk.com/en_us/download/universal-forwarder.html"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These constants leak onto top-level Object.

Ruby resolves constant assignment lexically, and a block is not a constant scope — so inside create_function do ... end these define ::DOWNLOAD_PAGE, ::CACHE_PATH, ::CACHE_TTL_SECONDS and ::FALLBACK on Object in the puppetserver JRuby.

Two consequences: a collision with any other function that picks the same identifier, and warning: already initialized constant on every environment reload.

Private methods returning the literals (def cache_path = "/opt/..."), or nesting them under a real module, both avoid it.


def write_cache(data)
FileUtils.mkdir_p(File.dirname(CACHE_PATH))
tmp_path = "#{CACHE_PATH}.tmp"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed temp filename races under concurrent compiles.

A puppetserver compiles many catalogs in parallel. When the TTL expires, every in-flight thread fetches and writes this same path, then renames it into place — so the cache can land truncated or spliced.

read_cache rescues the parse error and returns nil, so it degrades to "refetch on every compile" rather than a hard failure. But that's a live HTTP call to splunk.com per catalog, which is exactly what the cache exists to prevent.

tmp_path = "#{cache_path}.#{Process.pid}.#{Thread.current.object_id}.tmp"

plus an ensure block to unlink it on failure.

Eit_types::Noop_Value $noop_value = undef,
Hash[String[1], Hash] $addons = {},
String[1] $password_hash,
Optional[Eit_types::Version] $version = undef,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping the 7.2.4 / 8a94541dcfac defaults is the highest-blast-radius part of this PR.

grep over data/ and hieradata/ finds no splunk version/build pin anywhere, so every node with manage => true is currently on 7.2.4 by way of these defaults. After this merge the same node resolves whatever splunk.com advertises (10.4.2 today) and package_ensure => 'latest' applies it on the next run — a 3-major-version jump on the agent that ships customer logs, unattended, keyed off a page scrape.

The machinery is right; it's the default I'd flip. Keeping the pin as the default and making tracking explicit (version => 'latest', or a track_latest boolean) gets the same capability without the fleet moving on its own.

#
Puppet::Functions.create_function(:splunk_forwarder_latest_version) do
DOWNLOAD_PAGE = "https://www.splunk.com/en_us/download/universal-forwarder.html"
CACHE_PATH = "/opt/obmondo/cache/splunk_forwarder_latest_version.json"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/opt/obmondo/cache is referenced nowhere else in the repo — nothing creates or manages it.

FileUtils.mkdir_p runs as the puppetserver user, so if /opt/obmondo isn't writable by it, write_cache warns and every compile refetches. Same end state as the tmp-path race: a live HTTP call per catalog, with open_timeout: 5, read_timeout: 10 bounding one request but not the aggregate.

Puppet[:vardir] is writable by the compiling process by construction and would remove the failure mode. A short negative-result cache would also bound the worst case when Splunk's page is slow rather than down.

…rite race

- Constants assigned directly inside create_function's do...end block
  aren't lexically scoped to the function class (Class.new-style blocks
  don't nest in Module.nesting), so they leaked onto top-level Object
  in the puppetserver JRuby - collision risk with any other function
  using the same name, plus an "already initialized constant" warning
  on every environment reload. Nested under a real module instead.

- /opt/obmondo/cache is only managed on agent nodes (common::init's
  $__opt_dir); nothing creates it on the puppetserver, where this
  function actually runs at compile time. write_cache would silently
  fail there and force a live HTTP fetch on every catalog compile.
  Switched to Puppet[:vardir], writable by the compiling process by
  construction.

- P1: write_cache persisted data merged with fetched_at, and
  read_cache/latest returned that merged hash as-is on a cache hit -
  violating the declared Struct[{version, build}] return_type on every
  compile after the first, since Struct rejects unlisted keys. Fixed
  by stripping fetched_at at both return points (cached.slice(...))
  while keeping it in the persisted/read hash for stale? to use.

- Puppetserver compiles many catalogs concurrently in one JVM, so a
  fixed temp filename let concurrent writers splice/truncate each
  other's write before either renamed it into place. Made the temp
  filename unique per writer (pid + thread object_id, since pid alone
  doesn't differentiate threads in the same process) and added an
  ensure block to clean up the orphaned temp file if the write or
  rename fails partway.

Verified all of the above against real behavior (not just review) via
puppet apply: reproduced the return_type violation before the fix,
confirmed it's gone after; confirmed no leftover .tmp files on a
successful write; confirmed the ensure block cleans up an orphaned tmp
file when rename fails after write succeeds; confirmed the constant no
longer appears on Object.
Temporary: points datadir at the test hiera-data branch directly for
continued testing on gbsherepo01.abbnoa6nlk. Revert before merging to
master - this affects Hiera lookups for every node compiling against
this environment, not just the one under test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants