That spam filters would be the first computer systems to wake up, has been a part of the AI lore for some time. There is a steep evolutionary trajectory forced upon the algorithms by the relentless pressure of incoming spam adapting itself to changing conditions.
At the same time, the complex system made up by the ‘bad guys’, their objectives, and the computer systems they control, evolve as well.
Actually computer security issues are a big problem already, without an AGI recursively rewriting the attack patterns. I have been working with Eugene Kaspersky, a renowned security expert, for the last seven years, and he has become lately rather pessimistic in the overall capacity of the industry to keep up. If you search in Google his latest remarks you will notice that: there are fewer widespread attacks, flashworms, and such, and that the self-protection mechanisms of the attacks that do spread are getting better.
The reason for the first is that the recursive evolution is happening with human intelligence being involved, and still rapidly enough: organized crime is now running the development of the latest attacks, and manage the spam operations, and the botnets. These organizations are looking for ROI, and not visibility.
The second’s roots are in the better and better application of strong cryptographic elements. Once properly implemented, and in a complete circle of vector, infection, and communication processes, the white hats will be powerless. (Unless some for the moment unsubstantiated quantum breakthroughs move the battle to a new ground.)
The intentions of the entities are malevolent already enough, and not friendly at all, as the death of a leading Russian spammer illustrates. There is also a second category: the website of the Academy of the Chinese armed forces has been hacked, or at least that is what was thought at the time. After Kaspersky alerted them that they were infecting visiting computers, and nothing happened for 6 months, the thinking is that the infection was not there with the host
unawares…
Now this is not a hard takeoff probably, but for those concerned it is already hard enough!
The Washington Post has an interesting article about the supercomputer power of the botnets.