Apparently, the Moon accumulates an extra 1100 tonnes per year from in falling meteors and space dust.
By the same process, the Earth gains an extra 40,000 tonnes per year. OTOH, Earth also loses mass due to the escape of hydrogen and helium. So the
net loss for Earth is about 50,000 tones per year.
Yet despite all these fluctuations, the Earth-Moon system is pretty stable. The Moon retreats from Earth about 4cm per year. And though this may have some disrupting effects many billions of years from now, the good news is that the Sun will die long before then.
So… moving a few tonnes of stuff back and forth between the Earth and Moon won’t do much.