Add opt-in handling for cache provider errors (1.4.0)

README.md

@@ -100,6 +100,42 @@ ActiveRemote::Cached.default_options_overwrite({})
 
 Without `:expires_in`, a cached finder writes an entry that never expires.
 
+#### Cache errors
+
+By default, an error from the cache provider goes to the caller. To make a
+cache error act as a cache miss, set `:handle_cache_error`. A cache outage then
+does not stop the finders: each call goes to the remote service. This
+increases the load on that service until the cache comes back.
+
+```ruby
+# config/initializers/active_remote_cached.rb
+ActiveRemote::Cached.default_options(
+  :handle_cache_error => true,
+  :cache_error_proc => lambda { |error| Rails.logger.error(error) }
+)
+```
+
+| Option | Default | Description |
+|---|---|---|
+| `:handle_cache_error` | not set (off) | When true, a cache error does not go to the caller. `read` and `write` return nil, `exist?` returns false, `delete` returns nil, and `fetch` calls the block without the cache. |
+| `:cache_error_proc` | not set | A callable that receives the cache error. It runs only when `:handle_cache_error` is true. If the proc raises, the library writes a warning to stderr, and the call continues. |
+
+These two options apply only in `ActiveRemote::Cached.default_options`. A value
+in a finder declaration or in a finder call has no effect, and the library does
+not pass it to the cache provider.
+
+With nested caching, the nested cache and the cache provider each handle their
+own errors. An error in the nested cache does not skip the cache provider.
+
+An error from the `fetch` block (for example,
+`ActiveRemote::RemoteRecordNotFound` from a bang finder, or an RPC error) always
+goes to the caller, and the library never caches it. The block runs at most
+once for each `fetch`.
+
+`ActiveSupport::Cache::RedisCacheStore` already catches Redis connection errors
+and sends them to its own `:error_handler`. These options also catch the errors
+that the store does not catch, for example an entry that fails to deserialize.
+
 #### Local overrides
 
 Each finder as takes an optional options hash that will override the options passed to the caching provider (override from the global defaults setup for ActiveRemote::Cached)
@@ -167,6 +203,32 @@ CI runs this matrix on Ruby 3.1, Ruby 3.4, JRuby 9.4, and JRuby 10.0.
 `active_remote` 8.0 requires Ruby 3.2 or later. CI does not run that
 version on Ruby 3.1 or JRuby 9.4.
 
+## Upgrading to 1.4.0
+
+### Cache error handling
+
+1.4.0 adds `:handle_cache_error` and `:cache_error_proc` (see "Cache errors").
+Both are off by default.
+
+### The cleanup delete in fetch no longer raises
+
+When `fetch` gets a nil or empty value (without `:allow_nil` or
+`:allow_empty`), it deletes the entry. Before 1.4.0, an error from that delete
+went to the caller, and the caller lost the value from the remote service.
+In 1.4.0, `fetch` ignores that error and returns the value. The nil or empty
+entry stays in the cache until its TTL ends. This applies with or without
+`:handle_cache_error`.
+
+An app on the internal `0.3.0.rc2` release can move to 1.4.0 and keep its
+initializer. 1.4.0 does not add the rest of that release:
+
+- `0.3.0.rc2` `fetch` called `read`, then `write`. 1.4.0 keeps the provider
+  `fetch`, so `:race_condition_ttl` now works. Redis keeps each entry for
+  5 more minutes.
+- `0.3.0.rc2` passed only known options to the cache provider. 1.4.0 passes
+  every option except the two error options, as 1.3.0 does.
+- Every cache key changes (see "Upgrading to 1.2.0").
+
 ## Upgrading to 1.3.0
 
 ### default_options merges

lib/active_remote/cached/cache.rb

@@ -9,6 +9,10 @@ class Cache < ::SimpleDelegator
       # calls on it.
       class InvalidCacheProvider < ::StandardError; end
 
+      # Options that control error handling here. They are not passed to the
+      # cache provider.
+      ERROR_HANDLING_OPTIONS = %i[handle_cache_error cache_error_proc].freeze
+
       attr_reader :cache_provider
 
       def initialize(new_cache_provider)
@@ -24,9 +28,11 @@ def initialize(new_cache_provider)
         super(@cache_provider)
       end
 
+      # The nested cache and the cache provider each get their own failsafe, so
+      # a handled error in one does not skip the other.
       def delete(*args)
-        nested_cache_provider.delete(*args)
-        super
+        failsafe { nested_cache_provider.delete(*args) }
+        failsafe { super }
       end
 
       def enable_nested_caching!
@@ -38,35 +44,131 @@ def nested_caching?
       end
 
       def exist?(*args)
-        nested_cache_provider.exist?(*args) || super
+        failsafe(:returning => false) { nested_cache_provider.exist?(*args) } ||
+          failsafe(:returning => false) { super }
       end
 
-      def fetch(name, options = {})
+      # An error from the block (the RPC call) always goes to the caller. Only
+      # an error from a cache provider goes to handle_or_reraise_cache_error.
+      # When that error is handled, the block value is returned without the
+      # cache, and the block runs at most once.
+      def fetch(name, options = {}, &block)
+        block_result = FetchBlockResult.new(block)
         provider_options = provider_fetch_options(options)
-        fetch_value = nested_cache_provider.fetch(name, provider_options) { super(name, provider_options) }
+        fetch_value = provider_fetch(name, provider_options, &block_result.to_block)
 
-        delete(name) if delete_after_fetch?(fetch_value, options, provider_options)
+        delete_quietly(name) if delete_after_fetch?(fetch_value, options, provider_options)
 
         fetch_value
+      rescue StandardError => e
+        # #value raises the block error again, so a block error goes to the
+        # caller as it was raised.
+        handle_or_reraise_cache_error(e) unless block_result.raised?(e)
+        block_result.value
       end
 
       def read(*args)
-        nested_cache_provider.read(*args) || super
+        failsafe { nested_cache_provider.read(*args) } || failsafe { super }
       end
 
       def write(*args)
-        nested_cache_provider.write(*args)
-        super
+        failsafe { nested_cache_provider.write(*args) }
+        failsafe { super }
       end
 
       private
 
       attr_reader :nested_cache_provider
 
+      # Runs the fetch block at most once, on the first call to #value, and
+      # keeps its value or its error.
+      class FetchBlockResult
+        def initialize(block)
+          @block = block
+        end
+
+        def value
+          run unless @ran
+          raise @error if @error
+
+          @value
+        end
+
+        # The block to give the provider: nil when fetch got no block, so the
+        # provider gets no block either. A proc, because the provider yields
+        # the key and #value takes no argument.
+        def to_block
+          proc { value } if @block
+        end
+
+        # True for the block error itself, and for an error that a provider
+        # raised while it rescued the block error (Ruby sets it as the cause).
+        def raised?(error)
+          return false if @error.nil? || error.nil?
+
+          error.equal?(@error) || raised?(error.cause)
+        end
+
+        private
+
+        def run
+          @ran = true
+          @value = @block&.call
+        rescue StandardError => e
+          @error = e
+        end
+      end
+      private_constant :FetchBlockResult
+
+      # An error from the nested cache is handled here, and the cache provider
+      # is still used. An error from the cache provider or the block goes to
+      # #fetch.
+      def provider_fetch(name, options, &block)
+        provider_result = FetchBlockResult.new(proc { cache_provider.fetch(name, options, &block) })
+
+        begin
+          nested_cache_provider.fetch(name, options, &provider_result.to_block)
+        rescue StandardError => e
+          handle_or_reraise_cache_error(e) unless provider_result.raised?(e)
+          provider_result.value
+        end
+      end
+
+      # Removes a nil or empty value after #fetch. If the delete fails, the
+      # value stays until its TTL ends, so the error never fails the #fetch.
+      def delete_quietly(name)
+        delete(name)
+      rescue StandardError
+        nil
+      end
+
+      def failsafe(returning: nil)
+        yield
+      rescue StandardError => e
+        handle_or_reraise_cache_error(e)
+        returning
+      end
+
+      def handle_or_reraise_cache_error(error)
+        raise error unless ::ActiveRemote::Cached.default_options[:handle_cache_error]
+
+        call_cache_error_proc(error)
+      end
+
+      # A handled cache error must not fail the call, so an error from the proc
+      # (for example, a notifier that is down) is only reported.
+      def call_cache_error_proc(error)
+        error_proc = ::ActiveRemote::Cached.default_options[:cache_error_proc]
+        error_proc.call(error) if error_proc.respond_to?(:call)
+      rescue StandardError => e
+        warn("ActiveRemote::Cached ignored an error from :cache_error_proc: #{e.class}: #{e.message}")
+      end
+
       # :skip_nil tells the provider not to write a nil at all, which saves a
       # write and the delete that follows it. Only an ActiveSupport store is
       # known to honor the option.
       def provider_fetch_options(options)
+        options = options.except(*ERROR_HANDLING_OPTIONS)
         return options if options.fetch(:allow_nil, false)
         return options unless cache_provider.is_a?(::ActiveSupport::Cache::Store)
 

lib/active_remote/cached/version.rb

@@ -2,6 +2,6 @@
 
 module ActiveRemote
   module Cached
-    VERSION = '1.3.0'
+    VERSION = '1.4.0'
   end
 end