Quick note: we currently have a pull request open for an alternative sparse storage scheme (based on Knuth) that might be much more efficient in this scenario, since it seems to avoid almost all copying in this case. Might be worth to give it a trial.
Thanks,
Christoph