Fix broker stuck in SYNCHRONIZING on DB error during rollback

When a service broker update job fails and attempts to revert the
  broker state, a database connection failure could cause the job to
  crash without properly handling the original error. This left the
  broker stuck in SYNCHRONIZING state with a FAILED job.

  This change wraps the state rollback operation in error handling to
  catch database errors and allow the original exception to be raised
  and the job to be retried properly.

  Changes:
  - app/jobs/v3/services/update_broker_job.rb: Add error handling around
    ServiceBroker.where().update() call in rescue block to gracefully
    handle database disconnections during state rollback
  - spec/unit/jobs/v3/services/update_broker_job_spec.rb: Add test case
    for database disconnect during state rollback

Author: kathap

app/jobs/v3/services/update_broker_job.rb

@@ -66,13 +66,18 @@ def perform
 
           @warnings
         rescue StandardError => e
-          ServiceBroker.where(id: update_request.service_broker_id).update(state: previous_broker_state)
+          begin
+            ServiceBroker.where(id: update_request.service_broker_id).update(state: previous_broker_state) if update_request
+          rescue StandardError
+            # Swallow errors during state rollback (e.g., DB connection issues)
+            # so the original exception can be raised and the job retried
+          end
 
           raise V3::ServiceBrokerUpdate::InvalidServiceBroker.new(e.message) if e.is_a?(Sequel::ValidationFailed)
 
-          raise e
+          raise
         ensure
-          update_request.destroy
+          update_request.destroy if update_request
         end
 
         private

spec/unit/jobs/v3/services/update_broker_job_spec.rb

@@ -446,6 +446,32 @@ module V3
             end
           end
 
+          context 'when database disconnects during state rollback' do
+            before do
+              setup_broker_with_invalid_catalog
+
+              # Mock the where clause to raise database error only when called from rescue block
+              # We detect this by checking if it's being called with just an id condition
+              call_count = 0
+              allow(ServiceBroker).to receive(:where).and_wrap_original do |original_method, *args|
+                call_count += 1
+                # The rescue block calls ServiceBroker.where(id: ...).update(state: ...)
+                # This is the only place that calls .where with just {:id => ...}
+                if args.first.is_a?(Hash) && args.first.keys == [:id] && call_count > 1
+                  raise Sequel::DatabaseDisconnectError.new('connection lost')
+                else
+                  original_method.call(*args)
+                end
+              end
+            end
+
+            it 'handles the database error gracefully and re-raises the original error' do
+              expect { job.perform }.to raise_error(::CloudController::Errors::ApiError) do |error|
+                expect(error.message).to include('Service broker catalog is invalid')
+              end
+            end
+          end
+
           context 'when the broker ceases to exist during the job' do
             it 'raises a ServiceBrokerGone error' do
               broker.destroy