Maths on my mind

I thought we had done with maths for the year but instead, free from the tyranny of sitting by the computer with a frustrated child on their second go through a mathletics activity just because, months ago, I told them they had to get all the questions right, I find that I have room to think. Coincidentally I had tea with a good friend yesterday and she, too, is exploring how maths could look home-ed style. We are both aware of the concept of living maths and Charlotte Mason's ideas and how we embrace natural learning in so many areas but are afraid to in maths. My children are good at maths, maybe even ahead compared to their schooled counterparts, but it lacks joy and interest and that seems wrong to me.
My eldest son has never been to school which makes eight years of formal education that he's not had, yet he is literate, engaging and awe-inspiringly knowledgable. Whatever we have done for those years seems to have worked, and the cornerstone has been chapter book. It is the one part of home ed I have never changed, never let go of and have always prioritised. We have learnt a huge amount through the various books we have read and it has been a nourishing and nurturing experience to cuddle up on the sofa, or to let them draw and build, while I read. So I have made the sudden, and yet long-brewed, decision to let maths become part of chapter book and not to do any kind of formal teaching.
Well, I wish I was quite that brave. My eldest would be in his second year at secondary school and I am too scared to completely ditch his curriculum so he will continue to do some kind of formal maths, but my younger two will do none. None at all. If our books lead to "lets' do this" kind of moments, like the solar system, ellipses and siphoning sprang from Mathmagicians, then we will follow those leads but otherwise I will only read to them. Just that. My safety net comes from the fact that they are both of junior school age and if they reach the end of the year having learned nothing, I have time to catch up. But somehow, down in my home ed guts, I just don't think that will be the case.