My database has more than 30 schemas, and I probably have too many inactive connections, and hence the long wait.
All I could see in Alert log was "Active call for process xxxx user 'SYSTEM' program 'ORACLE.EXE (SHAD)'"
After 2 hours waiting, I gave up and started a second session and issued a Shutdown Abort.
Though this is not a bug, I was not sure how long I had to wait for all the active transactions to complete.
So, If you are planning to shutdown Immediate, make sure to query v$session for any active session, and notify the users and kill all "INACTIVE" connections.