Setup key for nx and ssh

It took me quite many hours. There are many posts online but at the same time many of them lead to dead ends. As mentioned in this post, the instruction described on help.ubuntu.com allows one to setup custom key but at the same time leaves a big security hole on ssh. Because everyone can ssh to your server without the need of a key. Here is what I did to patch such hole. After the whole procedure, password authentication for ssh will be blocked and custom key will be used for nx. I tested it in Ubuntu 10.10. After more testing, everything is fine but I can’t resume session. Still need more fixing!

1. Instead of using ssh, it is better to use SU to authenticate. I am mostly following this thread (it has a script there that probably handles Steps 1 and 2 here but I’m not very comfortable running “unidentified” script on my working computer and so I didn’t try).
   1. Modify /etc/nxserver/node.conf to set ENABLE_SU_AUTHENTICATION=”1″ and the rest (ENABLE_SSH_AUTHENTICATION, ENABLE_USERMODE_AUTHENTICATION, ENABLE_PASSDB_AUTHENTICATION) to “0”.
   2. Restart nxserver by running sudo /etc/init.d/freenx-server restart
   3. If everything is going well, you can now try to shut down password authentication for ssh as follows:
      • Add or uncomment “PasswordAuthentication no” in /etc/ssh/sshd_config
      • Restart sshd by sudo service ssh restart
   4. If nx still works well, you may want to change sshd_config back to allow password authentication for the moment
2. Now, let us set custom key as described in help.ubuntu.com as follows.
   1. Run sudo dpkg-reconfigure freenx-server and select “Create new custom keys” (or “Custom keys” if you have created keys before and don’t want to change them).
   2. In the next page, you should select SU for authentication.
   3. The above steps (2.a and 2.b) don’t work to me. It probably doesn’t work for you also if you have the “unknown job: freenx-server” error. Basically, new custom keys were not made for me. However, sudo /usr/lib/nx/nxkeygen did the job.
   4. After configuration, you need to distribute the public key to clients that want to login to your server. The key is stored at /var/lib/nxserver/home/.ssh/client.id_dsa.key. You will need root privilege to access or copy it.
   5. For Windows, you can change the key in NX Client at Configure->General->Key->Import. Or you can also directly copy the content of client.id_dsa.key to DSA Key window.
   6. For Linux, qtnx is pretty screwed up with key management. See this post if you have problem trying to have custom key worked (actually please leave me a message if you figure out how to make qtnx gui worked correctly with your custom key. I definitely want to know!). I just realized that NX Client supports Linux also. So it probably easier to just use NX Client instead of qtnx.
3. If you disable “password authentication” as suggested earlier, ssh will not work anymore unless you setup a key. Setting up key is relatively easy.
   1. ssh-keygen -t rsa (or ssh-keygen -t dsa) [I’m still not sure which one should be used always; I will suggest simply try both.]
   2. Copy the content of public key files (id_rsa.pub, id_dsa.pub) to the authorized_keys file on the remote server. This step is a bit counter-intuitive at first. But it is actually very easy to understand. When a client tries to login, the server looks into all public keys (locks) in the authorizedkeysfile and see if the client can open any of the locks. If the client is successful, the server can be certain the client is authentic.
   3. Also you need to check /etc/ssh/sshd_config to make sure that the AuthorizedKeysFile is actually defined and pointed correctly to the authorized_keys file on the server. The following two lines work for me:
      • AuthorizedKeysFile /var/lib/nxserver/home/.ssh/authorized_keys2
      • AuthorizedKeysFile2 %h/.ssh/authorized_keys
   4. Instead of going through Step 3.b above, you may also use ssh-copy-id user@remote_server to transfer your keys directly if you haven’t set password authentication to “no” yet.
4. If everything is working fine, remember to add “PasswordAuthentication no” back to /etc/ssh/sshd_config and restart ssh server (sudo service ssh restart).

If you have problem to resume session, try to set ENABLE_SLAVE=”0″ inside /etc/nxserver/node.config.

Below are the final /etc/ssh/sshd_config and /etc/nxserver/node.config that seem to work for me. The version I am using is NXSERVER – Version 3.2.0-74-SVN OS (GPL, using backend: 3.3.0)