Hi, we're Arkency

Getting nondeterministic agent into deterministic guardrails

2026-02-20T11:20:24Z

Getting nondeterministic agent into deterministic guardrails

AI agents don’t reliably follow your instructions. Here’s how I made it hurt less.

My context:

I currently work on a 12-year-old Rails legacy code base
The code base is undergoing modernization. Some of the large Active Record classes have been split into smaller ones, each into its own bounded context. Events are becoming a first-class citizens in the code. We also pay close attention to keep direction of dependencies as designed by context maps.
The client has a GitHub Copilot subscription. I mostly use Sonnet and Opus models.

The basic setup

Initially I started with the basics. I was curious where it would get us. There’s an AGENTS.md file with general rules to follow. Besides the AGENTS.md file I’ve added a few skills. The goal of the skills is to tell the agent about how it should write code. I am a big fan of Szymon’s way of using RSpec. So I put that into a skill. I also developed a few skills that tell the agent how I want it to deal with event sourcing, ddd technical patterns, hotwire, backfilling data (especially events) and mutation testing. The mutation skill is quite essential because without it the agent goes bananas and tries to achieve 100% of mutation coverage with hacking.

An example of hacking is calling send(:method).

I don’t want to have such tests. Trying to achieve mutation coverage in such a way indicates that perhaps the code should be removed because it’s just unnecessary noise.

So now the question is, is that enough?

It’s pretty good but can be better – tackling non-determinism

More than once (a day) I’ve experienced my agent to go off-rails and ignore my instructions. It doesn’t respect what I’ve specified in AGENTS.md and/or skills.

It often happens when I am asking it to introduce a very similar-yet-a-little-bit-different command and handler for a specific business use case.

Changes to the production code are going very well. This is especially true if the goal is to replicate well-structured code. However, once it gets to the “write the tests” part, it switches to commodity mode and most likely uses RSpec in the most popular way, which I don’t like. This is a large part of the existing codebase. If it doesn’t fail on writing tests the way I want it, it usually doesn’t run mutation testing, even though I expect the coverage not to drop below a certain point and the mutants to be eliminated. They should be killed properly. Using the send method is no bueno.

Dealing with non-determinism

So we’re not able to change whether the agent will respect AGENTS.md and skills all the time. At least not yet. Maybe never. So we have to deal with it differently.

What I am currently testing is to have guardrails aka dev workflows. The idea is to run tools that:

Will make me focus less on code structure, incorrect formatting, etc
Make sure tests for changed files are run
Make sure mutation tests are run
And, last but not least, make sure that the boundaries within bounded contexts are not violated. I noticed that the agent, just like humans, loves to take shortcuts to achieve a goal. The difference is that I never tell the agent we’re under a strict deadline. So I’m not sure where this choice is coming from.

The workflow is Ruby code that is wired to a /verify custom command. The command runs bash with ruby -r ./lib/dev_workflow.rb.

The dev_workflow.rb orchestrates the full pipeline. Looking at its requires tells you everything about what it runs:

require_relative 'dev_workflow/step_result'
require_relative 'dev_workflow/result'
require_relative 'dev_workflow/changed_files'
require_relative 'dev_workflow/steps/base'
require_relative 'dev_workflow/steps/rubocop_step'
require_relative 'dev_workflow/steps/rspec_step'
require_relative 'dev_workflow/steps/mutant_step'
require_relative 'dev_workflow/steps/eslint_step'
require_relative 'dev_workflow/steps/jest_step'
require_relative 'dev_workflow/verify_build'

Each step follows the same pattern: check if relevant files changed, run the tool, return a structured result. Here’s the mutation testing step as an example — the one that matters most given the problems I described earlier:

class MutantStep < Base
  ALLOWED_NAMESPACES = %w[CRM Ordering Billing].freeze

  def call
    unless changed_files.any_ruby?
      return StepResult.skipped(name: name, skip_reason: 'no ruby files changed')
    end

    subjects = mutation_subjects
    if subjects.empty?
      return StepResult.skipped(name: name, skip_reason: 'no mutant-eligible files changed')
    end

    result, duration = measure_duration do
      run_mutant(subjects)
    end

    output, success = result

    if success
      StepResult.success(name: name, duration_seconds: duration, files_checked: subjects.size)
    else
      errors = parse_mutant_output(output)
      StepResult.failure(name: name, duration_seconds: duration, files_checked: subjects.size, errors: errors)
    end
  end

  private

  def run_mutant(subjects)
    subject_args = subjects.map { |s| "'#{s}'" }.join(' ')
    run_command("bundle exec mutant run --since HEAD #{subject_args}")
  end

  def mutation_subjects
    changed_files.ruby_files
      .reject { |f| f.start_with?('spec/') }
      .filter_map { |f| file_to_subject(f) }
      .select { |subject| eligible_namespace?(subject) }
      .uniq
  end
end

The key detail is StepResult. Each step returns either .skipped, .success, or .failure with structured data. This is what the agent reads to understand what went wrong and what to fix.

Last but not least, to make sure that the non-deterministic agent won’t ignore my desire to run this command by itself, I attached it to a git pre-commit hook:

#!/usr/bin/env ruby

require_relative "../lib/dev_workflow"

result = DevWorkflow::VerifyBuild.call(staged_only: true)

puts result.to_json

exit(result.success? ? 0 : 1)

And at this point, at least calling the verify method is deterministic. So the agent gets feedback, fixes whatever is reported by the tool, reruns the verification and then it’s able to commit the changes.

Reviewing changes

Besides AGENTS.md, SKILLS.md and the workflow I described above, I still review the code. I focus on tests, architecture and security parts. I do take full ownership of the code that I ship. I don’t trust the AI enough to cut the leash. And my conclusion from working with it in a legacy codebase is currently that it will not change that fast (for me).

The timezone bug that hid in plain sight for months

2026-02-05T12:02:35Z

The timezone bug that hid in plain sight for months

We recently fixed a bug in a financial platform’s data sync that had been silently causing inconsistencies for months. The bug was elegant in its simplicity: checking DST status for “now” when converting historical dates.

The broken code

I found this while debugging a different sync issue — the real bug turned out to be hiding in a helper method I wasn’t even looking at.

def self.date_to_utc(value, timezone_key)
  offset = Time.now.in_time_zone(TIMEZONE_MAP.fetch(timezone_key)).formatted_offset
  Time.new(*value.to_s.split('-').map(&:to_i), 0, 0, 0, offset).utc
end

Looks reasonable, right? Get the timezone offset, create a Time object, convert to UTC.

The problem: Time.now.in_time_zone().formatted_offset gets the offset for right now, then applies it to any date being converted.

Why this breaks

Run this in December (EST, UTC-5):

date_to_utc(Date.new(2023, 6, 20), :eastern)
# Gets -05:00 offset, but June 20 should be EDT (-04:00)
# Result: off by one hour

Run the same code in June (EDT, UTC-4):

date_to_utc(Date.new(2023, 6, 20), :eastern)
# Gets -04:00 offset, correct for June
# Result: works fine

Same input, different output depending on when you run it. Your tests pass in summer, fail in winter. Data syncs would occasionally miss records or pull wrong date ranges, depending on DST periods.

The fix

def self.date_to_utc(value, timezone_key)
  tz = ActiveSupport::TimeZone[TIMEZONE_MAP.fetch(timezone_key)]
  tz.local(value.year, value.month, value.day, 0, 0, 0).utc
end

ActiveSupport::TimeZone#local handles DST correctly for the specific date being converted. June dates always get EDT, January dates always get EST, regardless of when the code runs.

The test that exposed it

Before touching the implementation, I wrote a test to confirm my suspicion — and it failed immediately.

it 'produces consistent results regardless of system timezone' do
  date = Date.new(2023, 6, 20)
  expected = Time.new(2023, 6, 20, 4, 0, 0, 'UTC')

  %w[UTC Asia/Tokyo America/Los_Angeles].each do |tz|
    Time.use_zone(tz) do
      expect(described_class.date_to_utc(date, :eastern)).to eq(expected)
    end
  end
end

This test runs the same conversion in UTC, Tokyo, and LA timezones. The old implementation would produce different results depending on system timezone and time of year.

Impact

We caught this before it caused visible production issues, but the potential impact for a financial data integration was significant: off-by-one-hour shifts during DST transitions could cause missed records in date-range queries and validation mismatches between systems.

Lessons

Never use Time.now for calculations on other dates. If you need timezone info for a specific date, use that date.
Test with explicit timezone manipulation. Don’t rely on your system’s timezone matching production.
DST transitions are sneaky. A bug that manifests only during certain months can survive code review and testing.
Know your tools: ActiveSupport::TimeZone

Stop using DateTime in 2026 (unless you work for UNESCO)

2026-01-14T14:08:58Z

Stop using DateTime in 2026 (unless you work for UNESCO)

DateTime has been considered deprecated in Ruby since 3.0. It’s 2026. Why are people still using it?

During a recent code review, we found this:

whatever.starts_at = DateTime.now

When asked why DateTime instead of Time, the response was: “DateTime handles a wider range of dates.”

That was partially true. In 2008. On 32-bit systems.

DateTime’s range advantage died in Ruby 1.9.2

Before Ruby 1.9.2 (released in 2010), Time was limited by the system’s time_t type — typically 32-bit signed integer covering 1901-2038. DateTime had a much wider range.

Ruby 1.9.2 changed this. Time started using a signed 63-bit integer representing nanoseconds since epoch, giving it a range of 1823-2116. For dates outside this range, Time uses Bignum or Rational — slower, but it works.

The practical range advantage is gone.

Remember Rails 4.2?

Pepperidge Farm Remembers. Some time ago when upgrading Rails app from 4.2 to 5.0, the test suite fortunately failed. The culprit was surprising: DateTime#utc changed its return type.

Rails 4.2:

DateTime.now.utc.class
# => DateTime

Rails 5.0:

DateTime.now.utc.class # => Time

This broke several Dry::Struct objects with strict type definitions expecting DateTime. But instead of “fixing” the types, we asked a better question: why were we using DateTime at all?

Rails 5’s breaking change to DateTime#utc wasn’t a bug — it was a nudge. It was telling you: stop using this class.

Struggling with upgrades? We have a solution for you.

The UNESCO problem

There’s actually one legitimate use case for DateTime: historical calendar reforms.

From Ruby’s own documentation:

It’s a common misconception that William Shakespeare and Miguel de Cervantes died on the same day in history - so much so that UNESCO named April 23 as World Book Day because of this fact. However, because England hadn’t yet adopted the Gregorian Calendar Reform (and wouldn’t until 1752) their deaths are actually 10 days apart.

Ruby’s Time uses a proleptic Gregorian calendar — it projects the Gregorian calendar backwards, ignoring historical reality. October 10, 1582 doesn’t exist in Italy (Pope Gregory XIII removed 10 days that October), but Ruby happily creates that timestamp.

DateTime can handle different calendar reform dates:

shakespeare = DateTime.iso8601('1616-04-23', Date::ENGLAND)
cervantes = DateTime.iso8601('1616-04-23', Date::ITALY)

(shakespeare - cervantes).to_i # => 10 days apart

For cataloging historical artifacts or dealing with pre-1752 dates across different countries, DateTime is your tool.

For literally everything else — which is 99.99% of applications — it’s the wrong choice.

Norbert Wójtowicz gave an excellent talk about calendars at wroclove.rb covering exactly these issues.

`DateTime`’s actual problems

No timezone support. DateTime doesn’t handle timezones.

DateTime.now
# => #<DateTime: 2026-01-14T13:00:00+00:00>
# Why +00:00 when my system is CET (+01:00)?

Incompatible with Rails. ActiveSupport extends Time with timezone support. DateTime? Barely.

Time.current   # ✅ Respects Rails.application.config.time_zone
DateTime.now   # ❌ Uses system timezone, ignores Rails config

Confusing arithmetic:

Time.now + 1      # => 1 second later
DateTime.now + 1  # => 1 day later

This has caused bugs. Many bugs.

Ignores DST. DateTime doesn’t track daylight saving time. If you use DateTime for anything involving timezones, you will have bugs.

Performance. Time is faster. Noticeably.

Why people still use it

“We’ve always used it”

The code was written in 2009. It’s 2026. Update it.

“I need to store dates without time”

Use Date. That’s what it’s for.

Date.current  # ✅ Rails.application.config.time_zone aware

“The library I’m using returns DateTime”

Convert immediately:

legacy_gem.fetch_date.to_time.in_time_zone

What to use instead

For timestamps: Time.current or Time.zone.now

For dates: Date.current

For parsing: Time.zone.parse('2026-01-14 13:00:00')

The only exception

If you’re cataloging historical documents, artifacts, or working with dates before calendar reforms in different countries, DateTime is your tool. You need to track which calendar reform date applies.

For everything else — modern applications, APIs, databases — use Time.

DateTime is deprecated for a reason.

References

Adding multi-tenancy to a DDD Rails app

2025-12-30T11:00:00Z

Adding multi-tenancy to a DDD Rails app

Many businesses when they set out to create some software they need, don’t know that one day, they might need multi-tenancy.

This is one of the features, that is not easy for programmers to add later easily. It might take months or even years.

Let me describe how I approached this in an ecommerce app. Essentially, the idea is to allow to create multiple stores, where previously it was one (or actually lack of any, all resources were global).

The ecommerce project

The project I am talking about is called ecommerce and it is part of the RailsEventStore (RES) organization on github. It started as a sample application for RES but over the last 10 years it grew to some kind of utopian Rails/DDD/CQRS/Events project.

This project does run on production, but it’s not really a production project. It’s more of a visionary/educational project to show a Rails codebase that can be highly modular in a DDD fashion.

In this project, there was no concept of a Store. All the main resources were global, as in Order.all etc.

Similarly, the events didn’t have any data or metadata that would point to a specific store.

class OrderConfirmed < Infra::Event
  attribute :order_id, Infra::Types::UUID
end

Another idea related to this whole project is that those events (and wider - domains/bounded contexts) are generic in their nature. As such they can be used in other apps without changes.

Just to prove this point that the domains can be reused, another rails app (pricing_catalog_rails_app) exists which requires (as gems) the existing bounded contexts.

The obvious solution - add store_id to all commands/events

When I talked to people, how they would approach adding multi-tenancy here this idea repeated. You need to extend existing events and commands with store_id.

I don’t like such invasive approaches by default.

Also, the domains reusability aspect - while still working, would be less elegant. Some event data would exist but never used in other apps (which don’t need multi-tenancy).

It’s definitely a concept that would work - so if you don’t have such abstract needs as reusability, this may be the way to go.

The different schema approach

Another obvious solution is to use some database concepts. Create a new schema per tenant or a new db.

I also excluded this from my choices - I didn’t want to solve this at the infra level. It also wasn’t clear to me, how would I operate on some cross-store reports which are often required for such Shopify-like platforms.

What exactly is multi-tenancy?

For some time, I didn’t have an alternative solution either. I was contemplating what it means to be multi-tenant, tried to split into smaller concepts.

Filtering data

We need to filter data. When a specific store is shown, we are displaying only the products of this store. In my architecture, that’s a read model job. Definitely, my read models would need to be extended by store_id concept. I was OK with that, even though it was an invasive change. In my book, read models are application specific and if such a big application requirement comes, the read models need to adjust.

Authorization

We need to authorize access to data.

This is where good old Rails controllers come handy. We need some concept of current_store and then pass the store_id to the read models.

Admin panel

We need some admin panel, where stores can be created, deleted and listed. In my case that’s a new read model but also a new “namespace/route” in the Rails app.

CQRS - write

So far, we have discussed the reads parts. In our CQRS split between reads and writes - how do we handle writes?

We have commands like this:

module ProductCatalog
  class RegisterProduct < Infra::Command
    attribute :product_id, Infra::Types::UUID
  end
end

Similarly as with events, the commands are part of the BCs and shouldn’t need to change.

Still, we need some way of saying that this Product is registered within a store. There’s no way around it.

How can we do it, without changing the existing command definitions?

What I wanted was a solution that:

doesn’t change existing BC APIs
keeps domains reusable
keeps multi-tenancy domain and app level, not infra

With these constraints in mind, the solution I arrived at looks almost obvious in hindsight.

My solution

I’m still polishing the edges here, but overall my attempt seems to work.

The main idea is to create a new Bounded Context - Stores. This is the home for a new kind of events. The events are tiny (I like them this way) and they are just registering the main resources within the Store.

So, they look like this:


  class ProductRegistered < Infra::Event
    attribute :store_id, Infra::Types::UUID
    attribute :product_id, Infra::Types::UUID
  end

  class CustomerRegistered < Infra::Event
    attribute :store_id, Infra::Types::UUID
    attribute :customer_id, Infra::Types::UUID
  end

  class OfferRegistered < Infra::Event
    attribute :store_id, Infra::Types::UUID
    attribute :offer_id, Infra::Types::UUID
  end

There’s not really much logic around it, though. I did solve the problem of making this change non-invasive, but I do admit, the concept of such repetitive events is not super convincing either.

So, yeah, that’s the drawback.

But there are more things that I like here.

First of all, None of other BCs had to change in any way. Maybe one of the existing process managers had to change to include Store registration.

Plenty of read models had to change, but that was expected. They all need to subscribe to the one new event. They persist the store_id and they know how to filter.

def call(event_store)
  event_store.subscribe(DraftOrder.new, to: [Pricing::OfferDrafted])
  event_store.subscribe(AssignStoreToOrder.new, to: [Stores::OfferRegistered])

module Orders
  class AssignStoreToOrder
    def call(event)
      Order.
        find_by!(uid: event.data.fetch(:order_id)).
        update!(store_id: event.data.fetch(:store_id))
    end
  end
end

In the controllers, we now need to filter and authorize data:

class InvoicesController < ApplicationController
  def show
    @invoice = Invoices.find_invoice_in_store(params[:id], current_store_id)
    not_found unless @invoice
  end

Also, in the controller, when we “create” new resources, we issue two commands, one for the original BC, the other one for Stores BC:

class CouponsController < ApplicationController
  def create
    coupon_id = params[:coupon_id]

    ActiveRecord::Base.transaction do
      create_coupon(coupon_id)
    end
  rescue Pricing::Coupon::AlreadyRegistered
    flash[:notice] = "Coupon is already registered"
    render "new"
  else
    redirect_to coupons_path, notice: "Coupon was successfully created"
  end

  private

  def create_coupon(coupon_id)
    command_bus.(
      Pricing::RegisterCoupon.new(
        coupon_id: coupon_id,
        name: params[:name],
        code: params[:code],
        discount: params[:discount]
      )
    )
    command_bus.(
      Stores::RegisterCoupon.new(
        coupon_id: coupon_id,
        store_id: current_store_id
      )
    )
  end

end

This coupling is intentional and happens only at the application boundary, not inside BCs.

It was also a nice opportunity to revise all the 16 existing read models and make some long-needed cleanups too.

How Claude Code helped me here

Hard to admit, but I wrote maybe 10% of the code changes in this whole implementation of multi-tenancy.

I assisted Claude in the original read model change - Orders. Then, I was shocked how well Claude worked with all other places. It knew the patterns and just repeated them.

It wouldn’t work, though, if not for the mutation test coverage.

Honestly, in all cases, when I followed the reasoning and the steps made by the AI, there were tiny hallucinations or tiny weird solutions, or tiny commenting out code.

Which all was caught by mutant, the main quality guard I have here against AI.

If not for mutant, I’d have to be more in control and the constant code reviews would drive me crazy. With mutant in place I was much more confident - and faster!

I’m gonna write more about this AI experience in other blogposts. This story is already a bit weird - starting from DDD, via events, to multi-tenancy, to read models, to Claude Code. Thanks for bearing with me here.

AI was crucial here, though. If not for AI, I’d hate myself by the 3rd of the 16 read models with the boring repetitive work.

If not for mutant, I’d hate myself for verifying AI in all 16 modules.

To be honest, I don’t know how people work with agents without mutation testing coverage.

Without mutant it would be like working with juniors with short attention span. Actually, agents are now senior level sometimes, but it’s seniors with dementia.

Here is an example mutant output, when AI worked on the Shipments read model. It ran mutant by itself, as it knows (CLAUDE.md) that it is required.

progress: 440/447 alive: 0 runtime: 34.21s killtime: 123.81s mutations/s: 12.86
Mutant environment:
Usage:           opensource
Matcher:         #<Mutant::Matcher::Config ignore: [] subjects: [Shipments*]>
Integration:     minitest
Jobs:            4
Includes:        ["test"]
Requires:        ["./config/environment"]
Operators:       light
MutationTimeout: 10
Subjects:        11
All-Tests:       359
Available-Tests: 359
Selected-Tests:  19
Tests/Subject:   1.73 avg
Mutations:       447
Results:         447
Kills:           447
Alive:           0
Timeouts:        0
Runtime:         34.73s
Killtime:        126.85s
Efficiency:      365.22%
Mutations/s:     12.87
Coverage:        100.00%

This was a successful run, but often it would catch its own hallucinations and thanks to mutant it fixed itself.

Summary

To summarize - it’s still too early to evaluate the solution, I still need to finish some reviews. But it does seem promising to me and I’m happy with the outcome, despite the drawbacks.

The key lesson for me wasn’t multi-tenancy itself, but that strong architectural foundation + mutation testing make large-scale AI-assisted refactors possible.

Discuss this post on X (Twitter)

Rewrite with Confidence: Validating Business Rules Through Isolated Testing

2025-12-22T20:04:19Z

Rewrite with Confidence: Validating Business Rules Through Isolated Testing

A few months back, our team at Arkency faced a challenge that many Rails developers might recognize. We needed to implement a new flow at Lemonade that would eventually replace a legacy process — but with three major constraints that couldn’t be compromised: user experience, cost efficiency, and avoiding technical debt.

The stakes were high. Any discrepancies between systems would impact customers and potentially create legal issues in the insurance domain. We had just three months to understand, replicate, and improve a complex flow that had evolved organically over years. And we needed to break free from obsolete data structures while preserving essential business rules embedded in a codebase with over 1 million lines of code.

Traditional approaches wouldn’t work. Full test coverage would take months we didn’t have. What we needed was a methodology to systematically identify, isolate, and verify each business rule independently of its implementation.

We needed a way to rewrite with confidence.

The Context: Insurtech at Scale

If you had asked me three years ago if insurtech could be exciting, I would have probably laughed. But it can be.

Lemonade is an innovative insurance company that hit $1 billion in premiums just 10 years after founding. It took other well-established insurance brands 40–60 years to reach that milestone. Even companies like Microsoft, Netflix, Salesforce, and Tesla needed more time to achieve that.

What a thrill for @lemonade_inc to be in the 𝘛𝘳𝘦𝘴 𝘊𝘰𝘮𝘮𝘢𝘴 Club! I’m not particularly moved by a car “with doors that open like 𝘵𝘩𝘪𝘴 👐”, but I’m definitely exhilarated by the ride so far, and can’t wait for our #Next10X! 🙌🏻🚀🎉 https://t.co/HKpgfyFO7Y
— Daniel Schreiber (@daschreiber) March 25, 2025

When we joined Lemonade three years ago, their Director of Engineering shared a story that perfectly illustrated the stakes of our work. They once had an issue with roof coverage in one of their product lines and had to hire a legal team for a six-month sprint to fix things. The legal costs exceeded the entire IT budget.

We couldn’t break things. We had to be 100% sure that the new flow provided the same outcome.

The Architecture: A Rails Monolith Under Transformation

Lemonade used a Rails monolith as their foundation — my favorite architecture. There’s no coincidence they became successful. Over the past few years, they’ve been transforming to a microservices architecture, with new product lines released using their internal framework. But all home and renters insurance is still handled within the Rails monolith.

Our scope was clear: implement a new quote flow for HO4 (renters insurance) in the US that would produce identical underwriting results to the legacy system.

Understanding the Problem

The God Model

Like many mature Rails applications, the system had a God model — Quote — that accumulated responsibilities over time:

class Quote < ApplicationRecord
  serialize :data
  serialize :answers

  enum :status,
       {
         pending: 'pending',
         stubbed: 'stubbed',
         bindable: 'bindable',
         uw_declined: 'uw_declined',
       },
       default: 'pending'
end

The business raised a valid point: “We don’t want pending and stubbed Quotes in the system.” Pending represented abandoned quotes with no value. Stubbed meant the system couldn’t make a risk assessment, usually due to third-party issues. This data model pollution required filtering at different levels:

class Quote < ApplicationRecord
  scope :not_pending, lambda do 
    where.not(status: :pending)
  end
end

We all have such excluding scopes in our apps — don’t pretend you don’t.

This was especially problematic for the Data Science team. Without filtering these quotes, their models would be far from accurate.

The Data Complexity

The Quote model contained two serialized columns with deeply nested data. Here’s just a glimpse of what we were dealing with:

data:
  :locale:
    :region: US
    :language: en
  :client_uuid: 30a1377e-5f06-4e6f-878b-41564c2e1221
  :user_logged_in: false
  :flags:
    - :send_pdf_sample_docs
    - :tenant_pet_damage_activated
  # ... and dozens more attributes

The answers hash was even more complex, containing everything from address components to user preferences to tracking data.

Why Traditional Approaches Failed

We initially tried static analysis to figure out which Quote attributes were necessary for the underwriting process. We quickly realized this was impossible — too many branches in the code. Imagine: every US state has its own regulations affecting insurance products. Multiply this by product editions that change over time due to legal concerns or business needs. We also share the data model and flow with home insurance.

Then we tried using Module#prepend to instrument Quote accessors and track which data was involved. This gave us better overview but was still overwhelming.

And we hadn’t even touched the HTTP communication part — all the first-party and third-party calls required for underwriting, coverage selection, deductible calculation, and premium determination.

What About Other Approaches?

We considered several alternatives before settling on our solution.

Shadow traffic was an interesting option. This technique involves routing live production traffic to both the existing backend and a new shadow backend simultaneously. The shadow backend processes requests without affecting users, while comparison mechanisms validate behavior. Tools like nginx plugins or Zalando’s Skipper can handle this elegantly.

However, shadow traffic came with significant drawbacks for our use case:

Substantial infrastructure work and ongoing costs
Potential compliance issues using production data in non-production environments
Need to implement complex comparison mechanisms
Difficulty avoiding side effects when dealing with stateful operations and third-party APIs

The infrastructure overhead alone would have consumed a significant portion of our three-month deadline.

The Solution: Testing on Production

Here’s where we took an unconventional approach. Instead of trying to replicate production conditions in a test environment, we decided to test directly on production — but safely.

The Brave New Flow

The key architectural change was simple but profound: instead of creating a Quote at the beginning of the flow and updating it on every step, we’d receive all the data gathered by the frontend client and perform our task at the very end.

This meant:

No more pending quotes
No more stubbed quotes
Only bindable or uw_declined as final states
Much cleaner data model

Implementing the Sampling Mechanism

We built a sampling system using Ruby’s prepend to non-invasively inject our verification code:

class RentersUsQuoteSampler
  module AroundFilter
    def run_prepare_for_preview(quote)
      if RentersUsQuoteSampler.conditions_met_for_sampling?(quote)
        RentersUsQuoteSampler.sampled(quote) { super }
      else
        super
      end
    end
  end
end

This allowed us to intercept the underwriting process for specific quotes without affecting the normal flow.

What We Sampled

For each qualifying quote, we captured:

Quote state before underwriting - The raw quote data as it entered the process
Quote state after underwriting - The complete quote with pricing, deductible, and coverage
Address data - All location information
HTTP interactions - Every external API call made during the process

class RentersUsQuoteSampler  
  def self.sampled(quote)
    address = Address.lemonade.find_by(quote_id: quote.id)
    before_quote = to_sample(quote)

    TyphoeusRecorder.start_recording
    begin
      result = yield
      typhoeus_requests = TyphoeusRecorder.recorded_requests
    ensure
      TyphoeusRecorder.stop_recording
    end

    Record.create!(
      quote_before: before_quote,
      address: to_sample(address),
      quote_after: to_sample(quote),
      typhoeus_requests: typhoeus_requests,
    )
    result
  rescue => e
    ::Sentry.capture_exception(e, hint: { ignore_exclusions: true })
  end
end

Recording HTTP Interactions

Lemonade used Typhoeus as the HTTP client for microservices and third-party communication. Fortunately, Typhoeus provides a callback system:

class RentersUsQuoteSampler  
  module TyphoeusRecorder
    RECORD_PROC = ->(response) do
      @@typhoeus_requests.merge!(serialize_request(response))
    end

    def self.start_recording
      @@typhoeus_requests = {}
      ::Typhoeus.on_complete(&RECORD_PROC)
    end

    def self.stop_recording
      ::Typhoeus.on_complete.delete(RECORD_PROC)
      @@typhoeus_requests = nil
    end

    def self.serialize_request(response)
      {
        {
          base_url: response.request.base_url,
          params: response.request.options[:params],
          method: response.request.options[:method],
          body: response.request.options[:body],
        } => { 
          code: response.code,
          body: response.body,
        },
      }
    end
  end
end

This gave us perfect request-response pairs to use as stubs during verification.

The Verification Process

Sampling and verification were separate processes, allowing us to:

Collect samples from production continuously
Run verification asynchronously
Re-run verification after code fixes
Iterate until we achieved parity

Leaving No Trace

The critical requirement was not polluting production with duplicate quotes:

class RentersUsQuoteDtoVerifier
  def with_rollback
    ActiveRecord::Base.transaction do
      yield
      raise ActiveRecord::Rollback
    end
  end
end

But there was a gotcha: background jobs. We needed to ensure no jobs were scheduled within our rolled-back transaction.

After Commit Handling

This feature became built-in to Rails 7.2, but we weren’t there yet. Fortunately, one of the best things about working at Arkency is that if you need a solution, there’s a good chance we’ve solved it before — like in RailsEventStore or our blog posts from 9 years before Rails introduced it.

module AfterCommitRunner
  def self.call(&schedule_proc)
    transaction = ActiveRecord::Base.connection.current_transaction

    if transaction.joinable?
      transaction.add_record(async_record(schedule_proc))
    else
      schedule_proc.call
    end
  end

  def self.async_record(schedule_proc)
    AsyncRecord.new(schedule_proc)
  end

  class AsyncRecord
    def initialize(schedule_proc)
      @schedule_proc = schedule_proc
    end

    def committed!(*) = schedule_proc.call
    def rolledback!(*) = nil
    def before_committed!() = nil

    attr_reader :schedule_proc
  end
end

This allowed us to queue jobs only after successful commits, not within rolled-back transactions.

HTTP Stubbing Strategy

We needed to stub all external HTTP calls to avoid:

Mutating state in other microservices
Making expensive third-party API calls
Affecting external systems (like credit scores)
Rate limiting issues

First, we blocked all Typhoeus requests:

def with_http_stubs_mechanism
  callback = ->(req) do
    req.block_connection = true
    req
  end
  Typhoeus.before.prepend(callback)
  yield
ensure
  Typhoeus.before.delete(callback)
  Typhoeus::Expectation.clear
end

Then we used our recorded requests as stubs:

def with_common_http_stubs
  http_stubs.each do |req, res|
    Typhoeus
      .stub(req[:base_url], req[:params])
      .and_return(Typhoeus::Response.new(**res))
  end

  yield
end

Handling Edge Cases

Some libraries used net/http directly, which wasn’t easy to stub. For AWS S3 clients, we used Ruby’s metaprogramming capabilities:

def with_no_verisk_persistence
  old_const = Storage::IamS3Resource
  no_writes_iam_resource = Class.new do
    extend old_const

    def self.put(*) = 'http://example.org'
    def self.presigned_url(*) = 'http://example.org'
  end
  Storage.send(:remove_const, :IamS3Resource)
  Storage.const_set(:IamS3Resource, no_writes_iam_resource)
  yield
ensure
  Storage.send(:remove_const, :IamS3Resource)
  Storage.const_set(:IamS3Resource, old_const)
end

This allowed us to override behavior while still downloading resources from S3 (assuming GETs don’t mutate state).

The Complete Verification Flow

Putting it all together:

def sample_remake
  sample_remake = nil

  with_rollback do
    with_http_stubs_mechanism do
      with_common_http_stubs do
        with_bouncer_stubs do
          with_census_block_stubs do
            with_no_segment do
              with_no_verisk_persistence do
                with_no_promises do
                  with_no_impressions do
                    remake = mk_quote
                    Chat::Quote.run_prepare_for_preview(remake)
                    sample_remake = RentersUsQuoteSampler.to_sample(remake)
                  end
                end
              end
            end
          end
        end
      end
    end
  end

  sample_remake
end

Yes, the nesting looks deep, but each wrapper handled a specific concern. We could experiment safely as many times as needed.

Comparing Results

We used the super_diff gem to identify discrepancies:

def verify 
  tuple_to_compare.reduce(:==)
end

def diff
  SuperDiff.diff(*tuple_to_compare)
end

Example output when things didn’t match:

{
-  "status" => "bindable",
+  "status" => "pending",
   "product" => "iso",
   "form" => "ho4",
-  "edition" => "E240716",
+  "edition" => "E240618",
   # ...
}

This worked beautifully for nested structures, which was crucial for our case.

The Results

After implementing this methodology, we achieved:

Fewer questions asked - Simplified the customer flow
Cleaner data model - Eliminated obsolete quote states
Identical outcomes - 100% parity with legacy underwriting
Confidence to ship - No surprises in production

The project leader shared across the organization:

“This is part of one of the best releases I have ever experienced.”

There was even a panic moment when he reached out on a Friday evening before both our ski vacations — but it was just to thank the team for the exceptional release quality.

Key Takeaways

This approach worked because we:

Separated collection from verification — Continuous sampling with async verification
Treated production as the specification — No need to replicate complex environments
Isolated tests from side effects — Transaction rollbacks and HTTP stubbing
Iterated until perfect — Fixed issues and re-verified until parity achieved
Leveraged Ruby’s strengths — Metaprogramming made complex stubbing manageable

The methodology is applicable beyond insurance or quote systems. Anytime you need to rewrite complex business logic while ensuring behavioral parity, consider:

Can you sample real production behavior?
Can you replay it safely in isolation?
Can you compare results programmatically?
Can you iterate until perfect?

When refactoring mission-critical business logic, traditional testing might not be enough. Sometimes the best test suite is production itself — as long as you can verify without breaking things.

Prefer watching?

This post is based on author’s conference talk delivered at wroclove.rb 2025 in Wrocław, Poland and EuRuKo 2025 in Viana do Castelo, Portugal.

This methodology emerged from real-world necessity at Lemonade. We’re grateful for their trust in letting us solve this challenge and share the solution with the Ruby community.

Hi, we're Arkency

Getting nondeterministic agent into deterministic guardrails

Getting nondeterministic agent into deterministic guardrails

The basic setup

It’s pretty good but can be better – tackling non-determinism

Dealing with non-determinism

Reviewing changes

The timezone bug that hid in plain sight for months

The timezone bug that hid in plain sight for months

The broken code

Why this breaks

The fix

The test that exposed it

Impact

Lessons

Stop using DateTime in 2026 (unless you work for UNESCO)

Stop using DateTime in 2026 (unless you work for UNESCO)

DateTime’s range advantage died in Ruby 1.9.2

Remember Rails 4.2?

The UNESCO problem

DateTime’s actual problems

Why people still use it

What to use instead

The only exception

References

Adding multi-tenancy to a DDD Rails app

Adding multi-tenancy to a DDD Rails app

The ecommerce project

The obvious solution - add store_id to all commands/events

The different schema approach

What exactly is multi-tenancy?

Filtering data

Authorization

Admin panel

CQRS - write

My solution

How Claude Code helped me here

Summary

Rewrite with Confidence: Validating Business Rules Through Isolated Testing

Rewrite with Confidence: Validating Business Rules Through Isolated Testing

The Context: Insurtech at Scale

The Architecture: A Rails Monolith Under Transformation

Understanding the Problem

The God Model

The Data Complexity

Why Traditional Approaches Failed

What About Other Approaches?

The Solution: Testing on Production

The Brave New Flow

Implementing the Sampling Mechanism

What We Sampled

Recording HTTP Interactions

The Verification Process

Leaving No Trace

After Commit Handling

HTTP Stubbing Strategy

Handling Edge Cases

The Complete Verification Flow

Comparing Results

The Results

Key Takeaways

Prefer watching?

`DateTime`’s actual problems