What would you do if a customer insists on storing all historical transaction data in a custom object for reporting?

I would not simply refuse — I would quantify the cost and offer a design that meets the underlying need:

1. Establish the real requirement: Usually it is 'we need to report on trends' rather than 'we need every row in a custom object'. Those have very different solutions.
2. Quantify: Project the row count and storage over three to five years, and show the effect on query performance, report timeouts, sharing recalculation, and sandbox refresh times. Concrete numbers change the conversation.
3. Offer the tiered alternative: Recent transactional data in a custom object where it is fully reportable; older data in a Big Object queried with Async SOQL, with aggregates written into a small summary object; or the whole history in a warehouse with Salesforce holding the summary.
4. Show the reporting story: Demonstrate that the aggregates and dashboards the business actually looks at still work, because that is the fear driving the request.
5. Document the decision: If the customer still insists after seeing the projection, I record the accepted risk and the trigger point at which we revisit it. Making the trade-off explicit is a legitimate outcome; letting it happen silently is not.