Found the solution as below.
How can I get rid of the pass-phrase dialog at Apache startup time?
The reason this dialog pops up at startup and every re-start is that the RSA private key inside your server.key file is stored in encrypted format for security reasons. The pass-phrase is needed decrypt this file, so it can be read and parsed. Removing the pass-phrase removes a layer of security from your server - proceed with caution!
- Remove the encryption from the RSA private key (while keeping a backup copy of the original file):
$ cp server.key server.key.org
$ openssl rsa -in server.key.org -out server.key
- Make sure the server.key file is only readable by root:
$ chmod 400 server.key
Now server.key contains an unencrypted copy of the key. If you point your server at this file, it will not prompt you for a pass-phrase. HOWEVER, if anyone gets this key they will be able to impersonate you on the net. PLEASE make sure that the permissions on this file are such that only root or the web server user can read it (preferably get your web server to start as root but run as another user, and have the key readable only by root).
As an alternative approach you can use the ``SSLPassPhraseDialog exec:/path/to/program'' facility. Bear in mind that this is neither more nor less secure, of course.
source: http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#removepassphrase
