Bug #838

System reboots continuously after BIOS screens when all drives are in AHCI mode

Added by Troy Warr over 2 years ago. Updated about 2 years ago.

Status:ClosedStart date:
Priority:HighDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Seen in:

Description

I built a system using the following hardware:

  • Asus M4A89GTD PRO/USB3 motherboard
  • Kingston KVR1333D3E9SK2/8G DDR3-1333 8GB (2x4GB) ECC memory
  • AMD Athlon II X2 260 Regor 3.2GHz CPU
  • (10x) Hitachi Deskstar 0S03230 3TB SATA 6.0Gb/s hard drive
  • (2x) HighPoint Rocket 620 PCI-Express 2.0 SATA 6.0Gb/s controller card
  • IDE to Compact Flash CF Adapter
  • Transcend 4GB Compact Flash TS4GCF133
  • Intel EXPI9301CTBLK PCI-E network adapter
  • Antec EarthWatts Green EA-380D 380W PSU

I'm currently running FreeNAS 8.0.1-RC2 from a USB thumb drive (I set the CompactFlash card aside in case that was contributing to the problem).

The problem is that FreeNAS won't boot when I set all 6 of the onboard SATA ports to AHCI mode in the motherboard's BIOS. The system goes through the BIOS steps normally, but when the initial FreeNAS screen should appear, the system just reboots and starts again. I see a brief flash of the Asus BIOS graphical screen, then a flicker, long pause, and the BIOS process begins again. This happens continuously until I shut down the computer.

When the drives are set to IDE mode, FreeNAS boots fine. This may be good enough, but I'd probably prefer the extra feature(s) of AHCI (e.g., NCQ) unless it really doesn't matter. FWIW, I also noticed that the Rocket 620 HBAs are in "CPU AHCI" mode by default.

Here's a summary of the conditions that will/won't allow FreeNAS to boot.

  • If I set all 6 motherboard SATA ports to IDE mode, the system boots successfully. The BIOS allows you to specify AHCI/RAID/IDE separately for ports 1-4 (as a group) and ports 5-6 (as a group). You can set all 6 as AHCI, all 6 as IDE, or 1-4 as AHCI and 5-6 as IDE (the BIOS says something about using the latter to allow the system to recognize optical drives for installing the OS).
  • If I set ports 1-4 to AHCI and ports 5-6 to IDE, the system also boots successfully.
  • If I set all 6 motherboard SATA ports to AHCI mode, the system does not boot when all 10 drives are connected. To get it to boot, I can either (A) remove one or more SATA cables from the motherboard's ports, or (B) remove one of the Rocket 620 HBAs from its PCIe 2.0 slot (or just both SATA cables from it, still leaving the card in the slot).
  • One approach I kind of expected to work that does NOT work is to remove one SATA cable from each Rocket 620 HBA (8 drives total). I have to remove both SATA cables from either Rocket 620 HBA.

I have updated the motherboard's BIOS to the latest version (2301 - http://www.asus.com/Motherboards/AMD_AM3/M4A89GTD_PROUSB3/#download), but all other firmware/drivers are untouched from what shipped with the hardware.

This forum thread contains some more details and troubleshooting helps that may or may not be helpful.

http://forums.smallnetbuilder.com/showthread.php?t=5888

Please let me know if there's any more info that I can supply to help out. Thanks very much!

History

#1 Updated by Troy Warr over 2 years ago

Sorry, nothing useful yet from either HighPoint tech support (which is essentially non-existent; remind me not to buy from them again) or Asus (they were at least more eager to help, but I haven't gotten a call back from their "Level 2" support yet).

I tried to disable the BIOS for the HighPoint HBAs, but I couldn't find a way. The BIOS itself is extremely minimal - only allows you to set up RAID (odd, since this is a non-RAID card) - and the Asus BIOS doesn't seem to have the capability to disable plug-in cards (at least I've looked everywhere), which is frustrating since I believe that's a capability that many, if not most, motherboards have these days. It's certainly not a low-end board.

I also found a copy of the HighPoint BIOS update tool (supplied with the 2620 model, and possibly other cards) online and downloaded a BIOS update for the 620 that I found on HighPoint's website. However, the update tool didn't acknowledge the 620 card. HighPoint support insists that the 620 doesn't use a BIOS, but it clearly does.

I also discovered that since I can boot with AHCI mode set with both HighPoint HBAs installed (as long as one of the cards has no drives attached), I can insert the additional 2 disks after booting and they appear to work fine. That's not really useful, of course, other than as another potential clue.

At this point, I'm not sure what to do. Asus suggested that they could advise me on what HBA cards are known to work with this motherboard, so I'll probably just give up and buy a different card(s). Syba and Rosewill make similar low-price cards based on different SATA controller chips, but they may just lead to the same problem; and depending on what the issue really is, a 4-port SATA card may not alleviate it either. Maybe replace the motherboard? I would need something compatible with my Athlon II CPU and ECC memory...

In any case, I think we've determined that this is not a FreeNAS OS issue, so I can understand if you'd like to close this ticket as invalid.

One last question, though, if you wouldn't mind - do you think I'm safe just using the drives as IDE for now (since I can boot in that mode), so that I can move ahead with this project while I figure out what to do about the hardware? In other words, if I build my RAID-Z2 array and start copying files to it, will I potentially face issues or headaches if I switch out my hardware (controller cards and/or motherboard) in the future? Does IDE/AHCI affect how the data is written to the disk in any way, leading to possible incompatibilities?

Thank you very much for your willingness to help - I sincerely appreciate it!

#2 Updated by Troy Warr over 2 years ago

Attached above is my dmesg output. Would you mind taking a look to see if my drives are indeed running as AHCI, even though they're set to IDE in the BIOS? Please note that I switched back to booting from the IDE/CompactFlash adapter, so I assume that will show up as an 11th drive (Transcend 4GB).

If this looks OK, do you think we're all set, now that (I believe) we've determined it's not a FreeBSD issue (since Windows 7 doesn't boot either), and the drives are running as AHCI in spite of the BIOS?

#3 Updated by Troy Warr over 2 years ago

Thanks, guys, for all of your help. I really appreciate it.

I tried booting Windows 7 from DVD-ROM, and it didn't work (same problem) - so I assume this means it's not a FreeBSD-related issue after all?

I tried disabling everything that seemed potentially memory-consuming in the BIOS, but no luck. I think the major culprits (Express Gate and the HighPoint 620s' BIOSes) were still enabled, though, so that may not prove anything. I've been searching and searching for how to disable the BIOS on the 620 cards - it seems like that may actually solve the issue - but I can only find instructions for a different series of cards (http://www.highpoint-tech.com/China/faq_rr2300c.htm). I'll keep looking.

If I set the drives to IDE in the BIOS to allow the system to boot, how can I verify that they're AHCI once the OS is running? Sorry, I'm not familiar with how to check for that. That would be a perfectly acceptable workaround, I just want to be sure that the drives are actually running as AHCI and not IDE.

#4 Updated by Troy Warr over 2 years ago

Thanks, mav - unfortunately, I'm a complete beginner so that's still over my head. I'm using:

dmesg | less

from the root directory to read through dmesg, and it seems to correspond to what you're saying (i.e., that AHCI is in fact attaching).

I need to read up on how to get the dmesg output copied to another thumb drive - if I can attach it here as a text file and you wouldn't mind double-checking it to see if I'm all set, I'd really appreciate it.

#5 Updated by Troy Warr over 2 years ago

Thanks, mav. I'm crossing my fingers that this can be resolved rather than having to replace the motherboard already.

FWIW, I just got a response from HighPoint:

Firstly our card don't support [[FreeNAS]],and judging from the phenomenon,it seems the out of memory issue.It cannot allocate more memory to handle the devices.

#6 Updated by Troy Warr over 2 years ago

Thanks gcooper! The motherboard BIOS is already on the latest version - updating it from the original version didn't seem to have any effect. I might be able to update the BIOS of the HighPoint HBAs if you think that's necessary (it's a little more involved) - just let me know.

Here is a video of the boot process:

http://www.youtube.com/watch?v=YBPZG_pLoY0

A rundown is included in the video description.

At the point where the screen flashes and the computer restarts, I would normally see the FreeNAS ASCII logo and continue to boot
successfully. With the motherboard's native drives set to IDE mode, the system boots completely:

http://www.youtube.com/watch?v=0UlF66qUk2Q

Note that 0:45 is the point at which the system would have rebooted if AHCI mode was selected.

Thanks! Please let me know if I can provide anything else.

#7 Updated by Troy Warr over 2 years ago

Thanks for looking into that!

I guess my next step, then, is to try to disable the BIOS for the two HighPoint HBA cards. I'm hoping that doing that will allow me to enable AHCI again in the motherboard BIOS and boot successfully. This HighPoint document (for another of their products) seems to suggest as much (halfway down):

http://www.highpoint-tech.com/China/faq_rr2300c.htm

so hopefully the same applies for the Rocket 620 cards that I have. I'll have to ask HighPoint if it's possible.

I'll give that a shot and report back with my progress ASAP.

Thanks again!

#8 Updated by Alexander Motin over 2 years ago

Replying to [comment:9 Mr. Bungle]:

If I set the drives to IDE in the BIOS to allow the system to boot, how can I verify that they're AHCI once the OS is running? Sorry, I'm not familiar with how to check for that. That would be a perfectly acceptable workaround, I just want to be sure that the drives are actually running as AHCI and not IDE.

Check kernel boot messages on the console or in /var/run/dmesg.boot file. You should see there 3 ahciX devices detected (one for AMD/ATI SATA and two for HighPoints/Marvells). Also for each detected SATA disk you should see "Command Queueing enabled" message. If ahci(4) driver won't attach, you will probably see atapci driver attached to the SATA controller and no messages about "Command Queueing". Note that for PATA controller atapci driver is fine.

#9 Updated by Alexander Motin over 2 years ago

The solution is only partial: first 4 SATA ports assigned to the first controller were successfully forced to AHCI, but two SATA ports sharing second controller with PATA remained in legacy mode. Beside those two loosing NCQ, they also sharing one ATA channel, so only one of them can execute commands at one time, reducing performance much more.

I see your board also has one more SATA channel on JMicron controller. It may be not the best, but it is independent and AHCI-compatible. So if you find no better solution, it may be reasonable to move sixth disk from chipset controller to there. May be it even allow you to boot with AHCI enabled for the fifth drive.

Also I see that your HighPoint chips are flashed with some RAID-like firmware. I don't have such one to test, but some people were reporting that hot-plug may not work in that case. So if you find some other BIOS, I would try it.

#10 Updated by Garrett Cooper over 2 years ago

This is a FreeBSD driver issue, not a backend issue. That being said.. if you had exact screenshots of how things failed (or a video), that would make visualizing where the failure point is a lot easier.

Please also update your BIOS to the latest version, in case it's a BIOS bug of some kind.

#11 Updated by Alexander Motin over 2 years ago

Just as my $0.02 against this motherboard. I had another problem with it. Even with few drives connected and without extra controllers I can't net-boot FreeBSD on it if on-board SATA configured as RAID. Switching to AHCI (AFAIR) or IDE fixed the net-boot. In my case problem was somewhere between BIOS and FreeBSD boot loader, as FreeBSD kernel wasn't even started. Problem seems to be motherboard specific, as dozen of other boards I have are booting fine in same environment.

#12 Updated by Xin Li over 2 years ago

Replying to [comment:3 Mr. Bungle]:

Here is a video of the boot process:

http://www.youtube.com/watch?v=YBPZG_pLoY0

[...]

Note that 0:45 is the point at which the system would have rebooted if AHCI mode was selected.

It looks like the system didn't reached FreeBSD boot yet, which suggests something earlier than the operating system have caused the problem. If the motherboard have a on-board speaker, did you hear the "beep" indicating that POST was successful?

By the way would you please try if you could boot another operating system, either Windows installer or Ubuntu to see if the board would boot with that? I'm sort of inclined in that this is actually a motherboard issue, but if you could boot another OS, then it would confirm this as a real FreeBSD bug which we would need to fix.

#13 Updated by Alexander Motin over 2 years ago

Can you try to disable extra BIOS extensions, such as network boot and HighPoint cards BIOSes? It may free some BIOS resources (memory) and workaround the problem.

Also it may be fine to switch all 6 on-board SATAs to the IDE mode. NVIDIA and AFAIR AMD chipsets (but not Intel) allow to easily force AHCI from the driver level, so ahci(4) driver may attach even if AHCI disabled in BIOS. Just try it and see what driver has attached to the hardware. As soon as you are booting from PATA, you technically don't need SATA BIOS at all.

#14 Updated by Xin Li over 2 years ago

Hi,

Replying to [comment:4 Mr. Bungle]:

Thanks, mav. I'm crossing my fingers that this can be resolved rather than having to replace the motherboard already.

FWIW, I just got a response from HighPoint:

Firstly our card don't support FreeNAS,and judging from the phenomenon,it seems the out of memory issue.It cannot allocate more memory to handle the devices.

I'm not quite convinced that the High Point card is not supported by FreeBSD. Am I understanding correctly that "Rocket 620 PCI-Express 2.0 SATA 6.0Gb/s controller card" is the SATA HBA? I've actually checked FreeNAS's patched source tree and the following models should be supported:

HighPoint RocketRAID 620
HighPoint RocketRAID 622
HighPoint RocketRAID 640
HighPoint RocketRAID 644

Please let us know if you hit some problem in this case, thanks!

#15 Updated by Alexander Motin over 2 years ago

Replying to [comment:7 delphij]:

Replying to [comment:4 Mr. Bungle]:

FWIW, I just got a response from HighPoint:
Firstly our card don't support FreeNAS,and judging from the phenomenon,it seems the out of memory issue.It cannot allocate more memory to handle the devices.

I'm not quite convinced that the High Point card is not supported by FreeBSD. Am I understanding correctly that "Rocket 620 PCI-Express 2.0 SATA 6.0Gb/s controller card" is the SATA HBA? I've actually checked FreeNAS's patched source tree and the following models should be supported:

HighPoint RocketRAID 620
HighPoint RocketRAID 622
HighPoint RocketRAID 640
HighPoint RocketRAID 644

Please let us know if you hit some problem in this case, thanks!

I think it is some kind of misunderstanding. On HighPoint site Rocket (non-RAID) series marked as: "Native Support in FreeBSD", while for RocketRAID series they also provide custom RAID-aware driver. Up to my knowledge all of them should work, RAID ones even in two ways. I've personally successfully tested Rocket 620 and RocketRAID 640 cards. RocketRAID 620 were also reported as working AFAIR. xx2/xx4 cards are just have eSATA ports instead of SATA.

#16 Updated by Alexander Motin over 2 years ago

Ah, sorry. JMicron port is wired to the rear eSATA connector.

#17 Updated by Alexander Motin over 2 years ago

Replying to [comment:16 Mr. Bungle]:

One last question, though, if you wouldn't mind - do you think I'm safe just using the drives as IDE for now (since I can boot in that mode), so that I can move ahead with this project while I figure out what to do about the hardware? In other words, if I build my RAID-Z2 array and start copying files to it, will I potentially face issues or headaches if I switch out my hardware (controller cards and/or motherboard) in the future? Does IDE/AHCI affect how the data is written to the disk in any way, leading to possible incompatibilities?

You should be fine running them in IDE mode:
- they won't support automatic hot-plug - may or may not be important, depending on your case;
- they won't have NCQ - that is about 20-30% of performance depending on load;
- they will share ATA channel (handle requests one at a time) - if both accessed same time it may halve their performance, depending on load;
- there is no difference for ZFS whether devices connected via AHCI or legacy mode IDE, they are detected just using ZFS on-disk metadata; if you later reconnect them otherwise, they should be fine.

Recalling about JMicron controller on board, you may look around for eSATA-SATA cable to connect one of these drives there. eSATA is mostly the same electrically, so even if such cables could be not so widespread (but I've found them in local store), they shouldn't cost more then few dollars to try.

#18 Updated by Josh Paetzel about 2 years ago

  • Resolution set to fixed
  • Status changed from Unscreened to Closed

Also available in: Atom PDF