The fundamental problem is still the way math is taught and not the students.
I am a teacher so I have a lot of insight.
The problem is, modern education doesn't focus on basic math skills. Children are expected to master basic math skills at a young age and continue to move on to more advanced skills. But children who grew up in poverty are less likely to master basic math skills. Than they get to high school expected to do algebra and calculus, but can't do basic math.
I think we need to drastically reform education away from the classical renessiance type of education. Most people don't need algebra, but would benefit from mastering basic math skills. And learning more practical math concepts like taxes, economic data, data managment in the workplace, starting a business etc.
If you can't do algebra I question how you could be able to do any of those things you state after. Kids do basic algebra in Kindergarten, or at least I did, you just don't recognize it as algebra. Asking a kid to fill in the triangle for 2 + triangle = 5 is algebra, it's just the x got replaced with the triangle. There's a simple game you can do online called Nerdle which is a math version of Wordle. My son is in first grade and loves doing it. Obviously I have to do the multiplication and division for him, but "I have this answer and I have this number over here, what do I fill in the middle with?" is algebra. If you can't do that, I question how you'll ever be able to do taxes. And I do my taxes every year and I believe being able to do your own taxes (not Turbotax does it, you do it) should be within the capabilities of every high school graduate.
The problem with:
But children who grew up in poverty are less likely to master basic math skills. Than they get to high school expected to do algebra and calculus, but can't do basic math.
...is that these kids don't understand core concepts but get promoted up the grades anyway in the school system because holding a kid back is considered horrible practice by modern school administrators. My wife's a middle school counselor and I've had this argument with her a few times. I was trained in Six Sigma unfortunately and "kid doesn't understand the basics well enough but we're moving him from 3rd grade to 4th grade anyway" is so "moving metal", where you don't do the process correctly at one station and instead of stopping it and fixing the problem there, you move it on to the next station stating to yourself "someone else's problem", where the next station then has to deal with the part that is not to print.