Have you set the ulimits as per the recommendations? Set the ulimit parameter for the ora<SID> and <SID>adm user to unlimited as suggested in the SAP note 323816 - AIX: User limit settings.
It looks like machine does not have sufficient memory available. First do a quick restart of the server. You can use smitty to set them to unlimited.
Restart the Oracle database. If the database starts without issues then launch the Configtool and see if it comes up and if yes then restart SAP Java stack.
If SAP fails to start then post the logs from the work directory.