Bug #2512

Jails missing after reboot, New jails cannot be created

Added by BrianJM - almost 2 years ago. Updated about 1 month ago.

Status:ClosedStart date:
Priority:ImportantDue date:
Assignee:William Grzybowski% Done:

0%

Category:Backend
Target version:9.2.0-RELEASE
Seen in:9.1.0-RELEASE

Description

I upgraded from version 8.3.1 (x64) to FreeNAS-9.1.0-RELEASE-x64 (dff7d13). The upgrade went well, with the exception of the previously installed plugins. I decided to just start over with the plugins, so I deleted the plugins that carried over from the upgrade as well as the ZFS datasets related to the plugins.

I was able to create two 64-bit jails (plugin and port) on the morning of 08/07 in version 9.1. I rebooted the server that night after one of the jails stopped responding.

Upon reboot, the two configured Jails are missing from the "View Jails" page. Executing /usr/local/bin/warden list -v from the command line indicates there are no jails found. The jail storage mount points remain, as well as the plugin (Transmission) in the "Installed" section but the jail name is in red. The ZFS datasets for the jails also remain intact.

Attempting to create a new jail fails. The error below is displayed in the "Add Jails" dialog (after clicking "OK"):

mkdir: /usr/jails: Read-only file system mkdir: /usr/jails: Read-only file system cp: /usr/jails/.warden-files-cache/distfiles/9.1-RELEASE/amd64 is not a directory mkdir: /usr/jails: No such file or directory mkdir: /usr/jails: Read-only file system cd: /usr/jails: No such file or directory touch: /usr/jails/.fbsd-aria-stat-amd64: No such file or directory fetch: http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/amd64/9.1-RELEASE/base.txz: Not Found ERROR: Failed downloading: [[FreeBSD]] 9.1-RELEASE ERROR: Failed create default template

Attempting to access http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/amd64/9.1-RELEASE/base.txz from my laptop yields a 404. The entire 9.1-RELEASE directory is missing from the website.

The console log yields the following:

Aug  8 22:28:42 nas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list  -v
Aug 8 22:28:42 nas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden create bs --ipv4 192.168.10.13/24 --vanilla --pluginjail --syslog --logfile /mnt/RAID1_3TB/jails/warden.log
Aug 8 22:28:42 nas warden: Fetching jail environment. This may take a while...
Aug 8 22:28:42 nas warden: get_freebsd_file amd64/9.1-RELEASE/base.txz /usr/jails/.download/base.txz
Aug 8 22:28:44 nas warden: Trying ftp-archive...
Aug 8 22:28:44 nas warden: fetch -o /usr/jails/.download/base.txz http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/amd64/9.1-RELEASE/base.txz
Aug 8 22:28:44 nas warden: ERROR: Failed downloading: [[FreeBSD]] 9.1-RELEASE
Aug 8 22:28:44 nas warden: ERROR: Failed create default template

There are two issues here:

1) The missing jails in "View Jails". Can these be restored?
2) The inability to create new jails because the template appears to be missing from the download site.

History

#1 Updated by J - almost 2 years ago

Hi,

I discovered an additional problem. After getting the Jail-System up and running as described above, the created jails don't run any service specified in the rc.conf (inside of the jail). So even if "sshd_enable=YES" is in the rc.conf. SSH wont be started at Jail-Startup.

If I go inside the Jail via jexec and run "service sshd start" SSH comes up and everything is working as expected.

#2 Updated by BrianJM - almost 2 years ago

I just re-saved the configuration (Jails -> Configuration) because I noticed the IPv4 Network was set to "192.168.10.0/24".I modified this to "192.168.10.1/24" and clicked Jails -> Jails. Both jails now appear. I just attempted to create a new jail, and that works fine as well. However, there is a problem when the server is rebooted.

I can confirm consistent behavior where upon reboot, when the jail storage resides on an encrypted ZFS dataset, the jails are not visible after the dataset is decrypted. In order to view and start the jails, I have to re-save the jails configuration. After re-saving the configuration, the jails are again visible. They must be started manually even though they are set to autostart.

In version 8.3.1, there was an option to restart the Jails when decrypting the dataset - this functionality appears to have been removed and does not occur automatically.

#3 Updated by Calle R almost 2 years ago

I can confirm this bug in 9.1.0-RELEASE-x64 on a fresh install as well. Jails on encrypted ZFS drives are not listed after unlocking the volume. They are not listed when using 'jls' in the shell either.

Thanks for the workaround BrianJM.

Comment from a developer? Can a fix be scheduled for 9.1.1?

#4 Updated by Calle R almost 2 years ago

Well, I could start the plugin jail manually, but my only plugin did not start automatically. So I tried starting it with the Plugins > Service status but got "Some error occurred".

The following was logged in /var/log/messages:

Aug 27 21:46:02 freenas manage.py: [plugins.utils:80] Couldn't retrieve http://<freenas host>/plugins/transmission/1/_s/status: No JSON object could be decoded: line 1 column 0 (char 0)

#5 Updated by Calle R almost 2 years ago

Here's the traceback of the start request:

Request


Request Method: GET
Request URL: http://192.168.11.2/plugins/transmission/1/_s/start
Software Version: FreeNAS-9.1.0-RELEASE-x64 (dff7d13)
Exception Type: ValueError
Exception Value:
The view freenasUI.plugins.views.plugin_fcgi_client didn't return an HttpResponse object.
Exception Location: /usr/local/lib/python2.7/site-packages/django/core/handlers/base.py in get_response, line 133
Server time: Tue, 27 Aug 2013 22:05:13 +0200

Traceback


Software Version: FreeNAS-9.1.0-RELEASE-x64 (dff7d13)
Request Method: GET
Request URL: http://192.168.11.2/plugins/transmission/1/_s/start
Traceback:
File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
133. raise ValueError("The view %s.%s didn't return an HttpResponse object." % (callback.__module__, view_name))
Exception Type: ValueError at /plugins/transmission/1/_s/start
Exception Value: The view freenasUI.plugins.views.plugin_fcgi_client didn't return an HttpResponse object.

#6 Updated by William Grzybowski almost 2 years ago

  • Priority changed from Expected to Important
  • Seen in set to 9.1.0-RELEASE

You need to restart ix-nginx and nginx as part of the workaround as well

service ix-nginx start
service nginx restart

#7 Updated by William Grzybowski almost 2 years ago

  • Assignee set to William Grzybowski

Can you confirm it has been fixed in 9.1.1?

Thanks

#8 Updated by Aaron Oneal over 1 year ago

This still repros on 9.1.1.

Jails stored on encrypted ZFS volumes do not show up or autostart after unlocking the volume. :-(

#9 Updated by William Grzybowski over 1 year ago

  • Status changed from Unscreened to Resolved
  • Target version set to 19

#10 Updated by Aaron Oneal over 1 year ago

I tested this patch. It seems to have 2 issues:

1. It started jails that didn't have auto-start set.

2. Plugins were all in the off position after the jails were started and could not be switched on.

This error is in the log.
Oct 8 10:35:50 freenas manage.py: [plugins.utils:87] Couldn't retrieve https://192.168.x.x/plugins/owncloud/1/_s/status: No JSON object could be decoded: line 1 column 0 (char 0)

#11 Updated by William Grzybowski over 1 year ago

The fix is composed by multiple commits, not only one.

#12 Updated by Aaron Oneal over 1 year ago

I picked up the extra commit for restarting HTTP and that resolved problem #2. Problem #1 is still present. That is, all jails are started and not just the ones that are set to autostart.

The code doesn't appear to be checking the autostart property.

for jail in Jails.objects.all():
    Warden().start(jail=jail.jail_host)

#13 Updated by Jordan Hubbard over 1 year ago

  • Target version changed from 19 to 59

#14 Updated by Jordan Hubbard over 1 year ago

  • Target version changed from 59 to 60

#15 Updated by Jordan Hubbard over 1 year ago

  • Status changed from Resolved to Closed

#16 Updated by Jordan Hubbard about 1 month ago

  • Target version changed from 60 to 9.2.0-RELEASE

Also available in: Atom PDF