I think AmigaOS 3 could be a nice kernel as it is. And to make it more Unix-y memory protection could be introduced but only for a new userland process with more traditional syscalls.
It's a bit how DragonflyBSD is slowly converging to.
Amiga OS 9 would have looked very different from the Amiga OS that we know (I am talking from a developer's point of view, not about the GUI).
Since inter-process communication in Amiga OS was based on message passing with memory-sharing, it was impossible to add MMU-based memory protection later. As far as I know, even Amiga OS 4 (which runs on PowerPC platforms) is not able to provide full memory protection.
There was also only minimal support for resource tracking (although it was originally planned for the user interface). If a process crashed, its windows etc. would stay open. And nobody prevented a process to pass pointers to allocated system resources (e.g. a window) to other processes.
The API was incomplete and tied to the hardware, especially for everything concerning graphics. This encouraged programmers to directly access the hardware and the internal data structures of the OS. This situation was greatly improved in Amiga OS 3, of course far too late. Amiga OS 3 was basically two or three years too late. As far as I know, Apple provided much cleaner APIs, which greatly simplified later the evolution of their OS without breaking all existing programs.
Finally, the entire OS was designed for single-core CPUs. At several places in the OS, it is assumed that only one process can run at a time. This doesn't sound like a big issue (could be fixed, right?) but so far nobody has managed to port Amiga OS to multi-core CPUs (Amiga OS4 runs on multi-core CPUs, but it can only use one core).
I have been the owner of an Amiga 500 and Amiga 1200, but to be brutally honest, I see Amiga as a one-hit wonder. After the initial design in the mid-1980s, development of the OS and the hardware basically stopped.
> Since inter-process communication in Amiga OS was based on message passing with memory-sharing, it was impossible to add MMU-based memory protection later.
Why can't you do shared memory message passing with MMU protection? There is no reason an application in a modern memory protected OS can't voluntarily share pages when the use case is appropriate. This happens today. You can mmap the same pages, you can use posix shm, X has the shm extension...
It's a bit how DragonflyBSD is slowly converging to.