Some scientists are worried about technology like Elon Musk’s Neuralink. Cognitive psychologist Susan Schneider wrote an op-ed (paywall) that it could be “suicide for the human mind.”
The worry with a general merger with AI, in the more radical sense that Musk envisions, is the human brain is diminished or destroyed. Furthermore, the self may depend on the brain and if the self’s survival over time requires that there be some sort of continuity in our lives — a continuity of memory and personality traits — radical changes may break the needed continuity.
I’m no neuroscientist but I subscribe to emergentism, which is the idea that consciousness is an emergent property of the brain. An easy explanation is here, but basically it means that consciousness isn’t a property of the physical brain, but rather something that happens when you get enough neurons interconnected. This isn’t something that could be replicated with code.