Let's Encrypt - LE4D Issues

Mindwatering Incorporated

Author: Tripp W Black

Created: 10/18/2019 at 02:56 PM

 

Category:
Domino Server Issues Troubleshooting
Other

Issues:

- Program Document Run Error on HCL Domino v10
- Failure on Response Challenge
- Sporadic Failure on Renewals with Invalid Message
- JVM failure to run LE4D renewals after upgrade to Domino 11.0.1 or Domino 10.0.1 FP5


1. Program Document Run Error
The program document doesn't run the renewal agent, and a web site is about to expire.
Looking at the log alerts, the Domino 11.0.1 or Domino 10 FP5 server is unable to find the agent '' (empty string). It's because of a JVM update version >1.80_231-b11.

Thanks to Eknori and Andy Brunner for the work around using a text file.

Program document runs:
> Tell amgr run ‘MWUtil/LE4D.nsf’ ‘letsencrypt’

The log shows:

10/16/2019 05:30:27 AM Running Program document for 'server -c "tell amgr run 'MWUtil/MWLE4D.nsf' 'letsencrypt'"'
> tell amgr run 'MWUtil/MWLE4D.nsf' 'letsencrypt'

10/16/2019 05:30:27 AM AMgr: Start executing agent ' ' in 'MWUtil/MWLE4D.nsf'

10/16/2019 05:30:27 AM AMgr: Error attempting to load agent ' ' in 'MWUtil/MWLE4D.nsf': Entry not found in index

Evidently the SPR# is GFALB5NKTV.

Workaround:

1. Create a text file, amgr_run_le4d, in the notesdata folder. The text file needs to have the text:
tell amgr run "/MWUtil/LE4D.nsf" 'letsencrypt'

Note: Update the path of the NSF to your location.

2. If running on MS Windows, update the Program document Command line field:
Program Name: nserver to cmd
-c "< /opt/ibm/domino/amgr_run_le4d.txt"

3. Update the JavaOptions.txt file in the /local/notesdata or ../Domino/Data/ folder. If it doesn't exist, create it, owned by the Domino user, as applicable. Add the following to the javaOptions.txt and notes.ini the data notesdata folder:
$ cd /local/notesdata
$ vi javaOptions.txt
...
-Djdk.lang.Process.allowAmbiguousCommands=true
<esc>:wq, to save.

$ vi notes.ini
...
JAVAOPTIONSFILE=D:\YourDominoDataDir\javaoptions.txt
<esc>:wq, to save.

4. Restart the Domino server
$ sudo systemctl stop domino.service
<wait>
$ sudo systemctl status domino.service
<confirm>
$ sudo systemctl start domino.service
<wait>
$ sudo systemctl status domino.service
<confirm>


3. Wait till the time you set. Review the logs to confirm Program document executes okay.

________



2. Failure at Response Challenge:

If you fail after this point, you don't have port 80 open on your server. Port 80 MUST be open for the challenge.
10/18/2019 03:16:26 PM Agent Manager: Agent printing: Session URL: acme://letsencrypt.org
10/18/2019 03:16:32 PM Agent Manager: Agent printing: ... challenge:/local/notesdata/domino/html/.well-known/acme-challenge/OBfusCAt3DHexCode4321FAILur32dig
10/18/2019 03:26:32 PM Agent Manager: Agent printing: [ERROR] – Order’s status (“invalid”) was not pending

Reason #1:
Port 80 is not open on the Domino server.
Fix #1:

1. Open up your Internet Site document, change the Port 80 configuration from Redirect to SSL/TLS, to No.

2. Whatever applications were needing to run with TLS, need to be updated since the server-wide redirect is not in play. Go to each one's application properties and click the option Require SSL.


3. Note:
The developers posted with 2.0 Hints, that Port 80 is needed for first time challenge validation after the 2.0 upgrade and when adding a new host to the list of hostnames for the app.
The end result is that we've left it open since we are occasionally adding new hosts.

Reason #2:
Your Domino server is not that domain name. The challenge is NOT sent to the server sending the request but to that domain name. Therefore, if you have an "admin" server doing the requests, but the domain points to another Domino server, the other server will get the request but not have the file requested and it will be invalid.

Fix #2:
Option 1: Update the public DNS for the domain and point the domain to the primary Domino server. After the agent runs, set up something to copy the files to the correct Domino server.
Option 2: Create a replica of the LE4D app on the other Domino server and change the Domino field to that Domino server. That way the certs are already on the "right" server. In our case, we have round-robin DNS, so we have to copy the certs regardless which option we choose.

This is why we created the MW Sidecar app for LE4D. Please note that Domino 12.0 includes the new Certificate Store app, and its new manager task/process. Once you upgrade to this, the LE4D app and the MW Sidecar app are no longer needed.

__________



3. Sporadic Failure on Renewals with Invalid Message

Issue:
Domain does not renew and the Java agent gives the error:
Order's status ("invalid") is not acceptable for finalization

10/18/2019 02:51:02 PM Agent Manager: Agent error: org.shredzone.acme4j.exception.AcmeServerException: Order's status ("invalid") is not acceptable for finalization
10/18/2019 02:51:02 PM Agent Manager: Agent error: at org.shredzone.acme4j.connector.DefaultConnection.throwAcmeException(DefaultConnection.java:431)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at org.shredzone.acme4j.connector.DefaultConnection.performRequest(DefaultConnection.java:359)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at org.shredzone.acme4j.connector.DefaultConnection.sendSignedRequest(DefaultConnection.java:174)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at org.shredzone.acme4j.connector.DefaultConnection.sendSignedRequest(DefaultConnection.java:154)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at org.shredzone.acme4j.Order.execute(Order.java:133)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at de.midpoints.le4d.manager.Le4dManager.downloadCertificate(Le4dManager.java:255)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at de.midpoints.le4d.manager.Le4dManager.orderCertificateUseHTTPChallenge(Le4dManager.java:207)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at de.midpoints.le4d.manager.Le4dManager.process(Le4dManager.java:132)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at de.midpoints.le4d.manager.Le4dManager.run(Le4dManager.java:97)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at de.midpoints.MPStarter.NotesMain(MPStarter.java:16)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at lotus.domino.AgentBase.runNotes(Unknown Source)
10/18/2019 02:51:02 PM Agent Manager: Agent error: at lotus.domino.NotesThread.run(Unknown Source)
10/18/2019 02:51:02 PM Agent Manager: Agent printing: [ERROR] - Order's status ("invalid") is not acceptable for finalization

Workaround?
If we immediately (within minutes) run the letsencrypt agent again, we get the same error, again.
However, if we wait 15 minutes and rerun, the process is successful - no error.

__________



Reason #4:
The domain name is misspelled and doesn't exist. Or, you have included a variant of the domain that is not in DNS.
e.g. www.mydomin.ext, mydomain.ext, blog.mydomain.ext are in the list of domains, but blog.mydomain.ext has not been set-up in DNS. This 3rd option will cause the return to be 'invalid'.

. The challenge is NOT sent to the server sending the request but to that domain name. Therefore, if you have an "admin" server doing the requests, but the domain points to another Domino server, the other server will get the request but not have the file requested and it will be invalid.

Fix #4:
Option 1: Update the public DNS for the domain and point the domain to the primary Domino server. After the agent runs, set up something to copy the files to the correct Domino server.
Option 2: Create a replica of the LE4D app on the other Domino server and change the Domino field to that Domino server. That way the certs are already on the "right" server. In our case, we have round-robin DNS, so we have to copy the certs regardless which option we choose.


__________


4. JVM failure to run LE4D renewals after upgraded to Domino 11.0.1

Issue:

Renewal agent run fails when it is unable to run the command to import the new certificates into the LE4D application:

java.io.IOException: Cannot run program "cmd.exe": Malformed argument has embedded quote: "d:\domino\kyrtool.exe" . . .


Per the instructions at Eknori.de, perform the following:
1. If not existing, create an javaOptions.txt file in the data folder.

2. Add the following line:
jdk.lang.Process.allowAmbiguousCommands=true
Save javaOptions.txt file

3. Add the following line to the server notes.ini in the data folder.
JAVAOPTIONSFILE=DominoDataDir/javaOptions.txt
Save the notes.ini file.

Note:
If on Linux, do NOT prepend a forward slash, adding the JAVAOPTIONSFILE entry to the notes.ini, if you do, you'll see an error on the Domino console at boot, with a second line saying the JVM didn't load.


4. Restart the Domino service, or issue a rest server one the Domino console.



__________





previous page