Using weblogic plugin 1.0

– We get plugin 1.0 shipped along with weblogic server which we can use at web server to prxoy request from front end to backend weblogic server
– Depending on your environmental needs you can add various parameters in plugin configuration. You can refer below link for the same:

http://docs.oracle.com/cd/E13222_01/wls/docs81/plugins/plugin_params.html#1143055

– To debug any Plugin issues you need to enable below debugs and get WebLogicBridgeConfig information. This information gets details on all parameters set at plugin level and it let us know whether its able to connect to backend weblogic:

DebugConfigInfo ON
Debug ALL
WLLogFile /tmp/proxy.log

– This proxy log will capture debug information. After adding them get WebLogicBridgeConfig information by navigating as below in browser:

http://<WebServer_Host&gt;:<Port>/<App_Name>?__WebLogicBridgeConfig/

https://<WebServer_Host&gt;:<SSL_Port>/<App_Name>?__WebLogicBridgeConfig/

NSAPI PLUGIN STATISTICS REPORT

I. Runtime Statistics
A. Requests
Number of new HTTP client connection requests.
B. Successful requests
Number of HTTP client requests that reached completion without throwing runtime exceptions.
C. Exception objects created
Number of runtime exception objects allocated from heap memory. This number equal the sum of all the runtime exceptions listed below.
D. Exception objects deleted
Number of runtime exception objects returned to heap memory. A difference between exception objects created and exception objects deleted indicates a memory leak has occurred.
E. URL objects created
Number of HTTP connection objects allocated from heap memory.
F. URL objects deleted
Number of HTTP connection objects returned to heap memory. A difference between URL objects created and URL objects deleted indicates a memory leak has occurred.
G. Connections recycled
Number of HTTP connection objects that were successfully reused by the proxy.
II. Runtime Exceptions
A. CONNECTION_REFUSED
The proxy was unable to connect to a backend server or a system error occurred while connecting to a backend server. The application server may be down or unable to accept new connections.
B. CONNECTION_TIMEOUT
The proxy aborted a timed connection attempt while waiting for a backend server. The application server may be down or unable to accept new connections.
C. READ_ERROR_FROM_CLIENT
The proxy was unable to process all of the data sent in an HTTP POST request from a frontend client. An NSAPI error may have occurred or the browser aborted the request.
D. READ_ERROR_FROM_SERVER
The proxy was unable to process all of data returned by the backend server due to an NSAPI error or a socket read failure. The application server may be down.
E. READ_ERROR_FROM_FILE
The plug-in FileCaching parameter is enabled and the proxy was unable to read HTTP POST data from a temporary file due to a system error.
F. WRITE_ERROR_TO_CLIENT
An NSAPI error occurred while returning an HTTP response to a frontend client. The browser may have aborted the request.
G. WRITE_ERROR_TO_SERVER
A socket write error occurred while sending HTTP headers to a backend server. The application server may be down or unable to accept requests.
H. WRITE_ERROR_TO_FILE
The plug-in FileCaching parameter is enabled and the proxy was unable to either open a temporary file or write HTTP POST data to a temporary file due to a system error. There may be insufficient space or incorrect permissions on the partition where the temporary files are being written.
I. READ_TIMEOUT
A socket timeout occurred while waiting for data from the backend server. The application server may be down or busy.
J. WRITE_TIMEOUT
A socket timeout occurred while sending data to the backend server. The application server may be down or busy.
K. UNKNOWN_HOST
A socket error occurred while attempting to connect to the backend server. The host name of the application server may be wrong or cannot be translated by DNS.
L. NO_RESOURCES
The proxy was unable to allocate sufficient memory from the heap.
M. PROTOCOL_ERROR
Covers a variety of processing errors that may occur from malformed or empty headers, improper chunking, chunked transfer encoding (CTE) errors, or other HTTP parsing errors. These errors suggest bad data is being sent by either the frontend client or the backend server.
N. CONFIG_ERROR
Covers configuration errors such as an invalid port number, failure to specify either ‘WebLogicCluster’ or ‘WebLogicHost’, or an invalid path trim parameter. Recheck the obj.conf file for incorrect spellings or values.
O. FAILOVER_REQUIRED
Indicates that a backend server has been marked as bad and that the proxy will attempt to contact the secondary application server.
P. POST_TIMEOUT
Indicates that the maximum number of retry attempts have been made while sending HTTP POST data to a backend server. The application server may be down or unable to accept requests.
Q. REQUEST_ENTITY_TOO_LARGE
The value set by HttpServletResponse.setContentLength() exceeded the plug-in MaxPostSize parameter.
R. HALF_OPEN_SOCKET_RETRY
The proxy was using a recycled socket and that socket has already been closed by the backend server. The request will be retried on the same host with a new connection.
———————————————————————————————————————————————————————————–

– Recently I had faced an interesting issue with plugin 1.0 where if we enable SSL all requests were getting hanged at Calling InitSSL. It worked well when ssl was disbaled. We could see below logging in proxy log:

================New Request: [GET /index.html HTTP/1.1] =================
Wed Feb 18 14:14:30 2012 <2403313293368701> INFO: SSL is configured
Wed Feb 18 14:14:30 2012 <2403313293368701> SSL Main Context not set. Calling InitSSL
Wed Feb 18 14:17:07 2012 <2403513293370271>
================New Request: [GET /content HTTP/1.1] =================
Wed Feb 18 14:17:07 2012 <2403513293370271> INFO: SSL is configured
Wed Feb 18 14:17:07 2012 <2403513293370271> SSL Main Context not set. Calling InitSSL
Wed Feb 18 14:17:58 2012 <2403113293370781>

– After analyzing further we came to know that plugin 1.0 when SSL enabled uses certicom SSL implementation. This certicom code calls initSSL which performs File System (FS) scan on all FS mounted on that particular machine in order to generate the random seed which creates secret key for SSL communication.

– If any of the File system is hang then every SSL request coming to the web server will get hanged.

– To validate this we had taken strace on same web server machine with SSL enabled with help below command:

Commnad: strace -o <outout file name> -tt -r -f -s4096 -p <apache process id>

– We could see below several FS scans done on all FS mounted on apache machine:
————————————————————————————————————————————————————————————-
2732       0.000089 statfs64(“/”, 84, {f_type=”EXT2_SUPER_MAGIC”, f_bsize=4096, f_blocks=258022, f_bfree=137746, f_bavail=124639, f_files=131072, f_ffree=115432, f_fsid={0, 0}, f_namelen=255, f_frsize=4096}) = 0
2732       0.000118 statfs64(“/home”, 84, {f_type=”EXT2_SUPER_MAGIC”, f_bsize=4096, f_blocks=258022, f_bfree=249120, f_bavail=236013, f_files=131072, f_ffree=130810, f_fsid={0, 0}, f_namelen=255, f_frsize=4096}) = 0
2732       0.000136 statfs64(“/tmp”, 84, {f_type=”EXT2_SUPER_MAGIC”, f_bsize=1024, f_blocks=507748, f_bfree=495253, f_bavail=469039, f_files=131072, f_ffree=130884, f_fsid={0, 0}, f_namelen=255, f_frsize=1024}) = 0

————————————————————————————————————————————————————————————
– From strace output its been observed that since SSL request was failing to generate random seed due to one of the Network File System (NFS) was hanged, all requests failed to reach backend weblogic and ultimately failed to serve request.

– As plugin 1.0 is deprecated we could not make any changes. So the only way to avoild this issue was to make sure all file systems including NFS should be accessible to generate random seed for secret key generation.

– Alternately we can upgrade plugin to 1.1 where it seems it uses other SSL implementation. You can refer below post to configure plugin 1.1 with SSL enabled:

Configure SSL with weblogic plugin 1.1 for apache

Configure SSL with weblogic plugin 1.1 for apache

Steps to configure plugin 1.1 with apache web server on Linux:

1. Download latest plugin 1.1 by contacting Oracle support

2. Unzip it and take appropriate OS and apache bit specific zip file inside it on Linux machine

3. Unzip this file and copy all *.so files in apache modules directory

libwlssl.so
libnnz11.so
libclntsh.so.11.1
mod_wl.so

Note: In case of windows copy all *.dll files in apache/bin folder and only mod_wl.so file in modules folder.

4. Configure SSL for weblogic server by creating custom identity.jks and trust.jks.

( For creating self signed certs for weblogic refer below post:

Configure self signed certificate with Weblogic server

For getting certs signed by external authority refer below post:

Configure SSL for weblogic server

)
5. Get the copy of root certificate from weblogic machine to apache machine

6. As plugin 1.1 supports wallet we will create a wallet with the help of below commands:

orapki wallet create -wallet mywallet -auto_login_only

NOTE: orapki utility will be available with plugin 1.1 zip file. You can check it in bin folder in unzipped plugin 1.1 file

7. Now once wallet is created import root certificate which we copied from weblogic sever:

orapki wallet add -wallet mywallet -trusted_cert -cert root.der -auto_login_only

8. Call this wallet in IfModule as mentioned below and also add WLProxySSL, SecureProxy as shown in httpd.conf file:

<IfModule mod_weblogic.c>
SetHandler weblogic-handler
WebLogicHost 10.177.57.202
WebLogicPort 10002
WLProxySSL ON
SecureProxy ON
WLSSLWallet /usr/local/apache/conf/bin/mywallet
Debug ALL
WLLogFile /tmp/fileupload.log
</IfModule>

NOTE: Its not manadatory to configure SSL for apache as we have added SecureProxy so even if we hit apache URL on non-ssl it will be redirected to SSL URL at backend.
9. Refer the .so plugin 1.1 file named as mod_wl.so in httpd.conf file as below:

LoadModule weblogic_module /usr/local/apache/modules/mod_wl.so

10. Also add apache modules location in LD_LIBRARY_PATH. To set this open /etc/profile file and add the following lines at the bottom. You may already have the LD_LIBRARY_PATH variable defined, therefore simply add this path to it.

LD_LIBRARY_PATH=/usr/lib/httpd/modules/
export LD_LIBRARY_PATH

11. Now go to apache_home/bin directory and restart apache server with below command:

# apachectl -k restart

12. If prompt returns well that means apache is configured properly. Now try to access apache link and check if its working well.

Configure SSL with apache web server

As these days Verisign or Thwatte has stopped providing free trial certificates we will try to configure self signed certificates with apache:

1. Search for openSSL utility on linux box as below, it will return some output as below:

# whereis openssl
openssl: /usr/bin/openssl /usr/lib/openssl /usr/include/openssl /usr/share/man/man1/openssl.1ssl.gz

Use /usr/bin/openssl  to configure certs for apache

For Windows download OpenSSL utility from: http://www.openssl.org/source/

2. Generate a Private Key

# /usr/bin/openssl genrsa -des3 -out server.key 1024

Generating RSA private key, 1024 bit long modulus
…………………………………………………++++++
……..++++++
e is 65537 (0x10001)
Enter PEM pass phrase:
Verifying password – Enter PEM pass phrase:

Please give private key pass phrase twice as asked before.

2. Generate a CSR (Certificate Signing Request) for apache even though we are creating self signed cert we create csr and then we genearte self signed cert:

# /usr/bin/openssl req -new -key server.key -out server.csr

Country Name (2 letter code) [GB]:IN
State or Province Name (full name) [Berkshire]:KA
Locality Name (eg, city) [Newbury]: Bangalore
Organization Name (eg, company) [My Company Ltd]:Techworldrocks
Organizational Unit Name (eg, section) []:Middleware
Common Name (eg, your name or your server’s hostname) []:techworldrocks.wordpress.com …(Provide machine name on which apache server is hosted)
Email Address []:aaa.bbb@ccc.com
Please enter the following ‘extra’ attributes
to be sent with your certificate request
A challenge password []:XXXX
An optional company name []:

3. Remove Passphrase from Key

# cp server.key server.key.org
# /usr/bin/openssl rsa -in server.key.org -out server.key

The newly created server.key file has no more passphrase in it.

-rw-r–r– 1 root root 790 Mar 14 01:32 server.csr
-rw-r–r– 1 root root 891 Mar 14 01:33 server.key
-rw-r–r– 1 root root 963 Mar 14 01:33 server.key.org

4. Generating a Self-Signed Certificate

# /usr/bin/openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

This will generate server.crt apache cert with information provided in CSR.

5. Copy Private Key and Certificate in apache/conf folder to refer them in httpd-ssl.conf:

# cp -p server.crt /usr/local/apache/conf/ssl.crt
# cp -p server.key /usr/local/apache/conf/ssl.key

6. Configuring SSL Enabled Virtual Hosts in httpd-ssl.conf available in apache/conf/extra directory. Make sure below lines are uncommented:

SSLEngine on
SSLCertificateFile /usr/local/apache/conf/ssl.crt/server.crt
SSLCertificateKeyFile /usr/local/apache/conf/ssl.key/server.key

SetEnvIf User-Agent “.*MSIE.*” nokeepalive ssl-unclean-shutdown
CustomLog logs/ssl_request_log “%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \”%r\” %b”

7. Also uncomment below line in httpd.conf (If not present add it) to add mod_ssl.so along with its location:

LoadModule ssl_module /usr/lib/httpd/modules/mod_ssl.so

8. Also make sure to include extra/httpd-ssl.conf file in httpd.conf file as below:

Include conf/extra/httpd-ssl.conf

9. Now go to apache/bin folder and restart apache as below:

# apachectl -k restart

10. Try accessing URL on ssl.

 

Configure self signed certificate with Weblogic server

You can configure weblogic server to use self signed certificate which will be signed by yourself and not by any external authority. You can use these certs in non-production environments:

NOTE: You need to create keystores on each physical machine where you have servers running.

I. Generating self signed certificates:

1. Go to weblogic domain directory and create a new directory call as certs

> mkdir certs

2. Go to bin directory available in domain directory and execute setDomainEnv.sh / setDomainEnv.cmd files as below:

Windows: > setDomainEnv.cmd

Unix: . ./setDomainEnv.sh (Do not forget two dots)

3. Change directory to certs and Execute below command to create identity.jks:

> keytool -genkey -alias mycert -keyalg RSA -keypass weblogic1 -keystore identity.jks -storepass weblogic1 -validity 365

Below questions will be asked. Answer them according to your environment:

What is your first and last name?
[Unknown]: techworldrocks.wordpress.com….( NOTE: You need to provide Fully Qualified domain name of your machine where weblogic server is hosted)
What is the name of your organizational unit?
[Unknown]: Middleware
What is the name of your organization?
[Unknown]: techworldrocks
What is the name of your City or Locality?
[Unknown]: Bangalore
What is the name of your State or Province?
[Unknown]: KA
What is the two-letter country code for this unit?
[Unknown]: IN
Is CN=techworldrocks.wordpress.com, OU=Middleware, O=techworldrocks, L=Bangalore, ST=KA, C=IN correct?
[no]: yes

Now we have created a private key in identity.jks file. As we are configuring self signed certificate we will not be creating CSR to order certs from external authority.

4. As its self signed cert we will export same cert which will be also called root in this case from identity.jks:

> keytool -export -alias mycert -file root.cer -keystore identity.jks -storepass weblogic1

5. Now we will import same cert into trust.jks which generally contains only root cert.

NOTE: We can use same identity.jks file in place of trus.jks as we will be having same cert in both but as a good practice we should have both separate. In prod where we have chain of certs it recommended to create two jks stores.

> keytool -import -alias mycert -trustcacerts -file root.cer -keystore trust.jks -storepass weblogic1

Owner: CN=techworldrocks.wordpress.com, OU=Middleware, O=techworldrocks, L=Bangalore, ST=KA, C=IN
Issuer: CN=techworldrocks.wordpress.com, OU=Middleware, O=techworldrocks, L=Bangalore, ST=KA, C=IN
Serial number: 4f738bae
Valid from: Thu Mar 29 03:37:42 IST 2012 until: Fri Mar 29 03:37:42 IST 2013
Certificate fingerprints:
MD5: 19:B9:AB:EC:50:09:F6:D6:46:8D:B0:AD:71:77:22:60
SHA1: 5F:3E:D2:9D:9A:84:D4:06:13:12:A2:9B:C5:F8:A2:EA:83:FB:48:47
Signature algorithm name: SHA1withRSA
Version: 3
Trust this certificate? [no]: yes
Certificate was added to keystore

Now both keystores are ready for configuration. Repeat this process on all physical machines where weblogic servers are hosted

II. Configuring jks with weblogic:

1. Login to admin console

2. Navigate to servers>[server_name]>Configuration>Keystores

3. Select Custom Identity and Custom Trust and provide below details:

-Custom Identity Keystore: <Location of identity.jks>

-Custom Identity Keystore Type: jks

-Custom Identity Keystore Passphrase:weblogic1

-Confirm Custom Identity Keystore Passphrase:weblogic1

-Custom Trust Keystore:  <Location of trust.jks>

-Custom Trust Keystore Type: jks

-Custom Trust Keystore Passphrase:weblogic1

-Confirm Custom Trust Keystore Passphrase:weblogic1

4. Then click on SSL tab next to Keystores and provide values for below parameters:

-Private Key Alias: mycert

-Private Key Passphrase: weblogic1

-Confirm Private Key Passphrase: weblogic1

NOTE: As I have given identity keystore password same as private key password I will give same password here. If you have given different make sure to give different password

5. Then enable SSL port for that particular weblogic server by navigating servers>[server_name]>Configuration>General

Also provide Fully Qualified Domain Name (FQDN) in Listen Address field.

6. Save and activate changes. Repeat this 2nd half procedure for all weblogic servers which intend to use SSL

III. CONFIGURING NODE MANAGER

Modify nodemanager.properties file available in $WL_HOME/common/nodemanager folder. Insert the following lines at the end:

KeyStores=CustomIdentityAndCustomTrust

CustomIdentityKeystoreType=jks

CustomIdentityKeyStoreFileName=<Location of identity.jks>

CustomIdentityKeyStorePassPhrase=weblogic1

CustomTrustKeystoreType=jks

CustomTrustKeyStoreFileName=<Location of trust.jks>

CustomTrustKeyStorePassPhrase=weblogic1

CustomIdentityAlias=mycert

CustomIdentityPrivateKeyPassPhrase=weblogic1

NOTE: Make sure SecureListener is set to true which is already present in this file. Also mention Fully Qualified Domain Name (FQDN) in ListenAddress field which is blank by default.

IV. DISABLING HOSTNAME VERIFICATION

NOTE: As we are using self signed certs we need to disable host name verification whereas in production environment we should keep it enabled by configuring certificates authorized by internal or external certificate authority.

i) Disable host name verification at server level

1. Navigate to servers>[server_name]>Configuration>SSL
2. Click on advanced
3. Make Hostname Verification NONE
4. Save and activate changes.
5. Repeat these steps fore all servers in domain for which self signed certs are configureii) Disable host name verification for Node managerAdd parameter -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false in startNodeManager.sh/startNodeManager.cmd file as given below:

   “${JAVA_HOME}/bin/java” ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS} -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Djava.security.policy=”${WL_HOME}/server/lib/weblogic.policy” -Dweblogic.nodemanager.javaHome=”${JAVA_HOME}” -DListenAddress=”${LISTEN_ADDRESS}” -DListenPort=”${LISTEN_PORT}” weblogic.NodeManager -v

   else
"${JAVA_HOME}/bin/java" ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS} -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy" -Dweblogic.nodemanager.javaHome="${JAVA_HOME}" -DListenPort="${LISTEN_PORT}" weblogic.NodeManager -v
fi
else
if [ "$LISTEN_ADDRESS" != "" ]
then
"${JAVA_HOME}/bin/java" ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS} -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy" -Dweblogic.nodemanager.javaHome="${JAVA_HOME}" -DListenAddress="${LISTEN_ADDRESS}" weblogic.NodeManager -v
else
"${JAVA_HOME}/bin/java" ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS} -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy" -Dweblogic.nodemanager.javaHome="${JAVA_HOME}" weblogic.NodeManager -v

Now restart Node manager, you do not have to restart servers if they are already running. This completes SSL configuration with self signed certs.

NOTE: Sometimes you might get some SSL issue at Node manager, in that case import CertGenCA.der into custom trust store trust.jks so that Admin server can trust Node manager.

> keytool -import -alias mycert-trustcacerts -file “<location of CertGenCA.der>” -keystore trust.jks