Author Topic: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)  (Read 5707 times)

Offline fdlsys

  • Vinylist
  • Trader
  • Trade Count: (+72)
  • *****
  • Posts: 7,127
  • Do you like life, sweetheart?
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #15 on: September 03, 2013, 11:14:38 PM »
:notworthy:
The four building blocks of the universe are fire, water, gravel and vinyl. Dave Barry
Come back when you’ve lived a little. Miles Davis

Offline David Copperfield

  • Trade Count: (0)
  • AVForums Veteran
  • *****
  • Posts: 1,200
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #16 on: September 04, 2013, 06:30:29 AM »
I'm going to make a slightly smaller one tonight as it seems that it might cause problems on some 8GB usb sticks. Will post link either tonight or tomorrow.

Thanks ara :thumbs:

Offline ara

  • Trade Count: (+2)
  • AVForums Veteran
  • ***
  • Posts: 388
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #17 on: September 04, 2013, 10:54:46 PM »
Busy uploading. Will probably take the night so will post some guides tomorrow or Friday night.

Offline ara

  • Trade Count: (+2)
  • AVForums Veteran
  • ***
  • Posts: 388
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #18 on: September 08, 2013, 09:46:29 AM »
I'll post the final link sometime in the coming weeks. Helped someone yesterday and there is a couple of things I need to sort out like a guide/script to make sure that your lms, sickbeard, etc library is not on the flash drive. Also a mount issue that needs to be sorted out although for most it won't be a problem.

Offline fdlsys

  • Vinylist
  • Trader
  • Trade Count: (+72)
  • *****
  • Posts: 7,127
  • Do you like life, sweetheart?
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #19 on: September 08, 2013, 06:43:30 PM »
I'll post the final link sometime in the coming weeks. Helped someone yesterday and there is a couple of things I need to sort out like a guide/script to make sure that your lms, sickbeard, etc library is not on the flash drive. Also a mount issue that needs to be sorted out although for most it won't be a problem.
That be me, thanks! :notworthy:

Experiences and impressions so far, hopefully it helps someone:

Running high read/write applications from the flash drive is absolute no-no.
LMS for one is so slow that it's completely useless. After Ara eventually cracked the FreeBSD "Jail" quirks and reconfigured the LMS to use the HDD mount for all logs/cache/db reads/writes, things sped up greatly. Discovery of 28000 files took 1.5 minutes. Full 1st scan of these files - less than one hour. Very good. All this is using LMS 7.8, so it's the latest "bleeding edge" release.

The road to getting the LMS to run using the HDDs was painful. FreeBSD Jails are not quite ready for prime time I would say. Many issues that are preventing easy or custom configuration. From cryptic "mount" sharing and re-sharing between the Main Nas4Free OS and the Jail OS-es, to unresolvable issues with User IDs. Fortunately, "root" is root everywhere so that sorted the security issues. Not the safest way of doing things, but at least it works.

Sickbeard ... I'm a great fan of the application but boy... I could not crack changing the logs/cache/db directories. The biggest problem is that Sickbeard will simply not start if it does not like something in the configuration file so no logs, no errors - no idea what bothers it. Weird thing - it will accept writing Logs to the HDD mount, but not the cache. I could not find where to set the database path at all.

Deluge ... huh. We encountered an issue (most likely Jail related, not Deluge as such) where if max download bandwidth allowed to Deluge is over 300Kbs it would take over the complete 1000mbs NIC (!) Pings to/from that server would take 900ms which is equal to server being on NASA satellite circling the Jupiter. Then, there are issues with Deluge itself. Configuring the global Bandwidth and Queue parameters. At one point as I was tuning down the Upload ratio, Deluge stopped downloading completely and no matter what I did it just wouldn't restart. It took me randomly pulling the various levers one at the time and hitting "Apply" until it eventually started working again. All to do with Upload ratio!? Makes no sense.

I think we will eventually have to consider running the applications inside the main OS instead of Jails, even if only for reference and a sanity check?

Regretfully, there is no free lunch. For me at least since I won't compromise on the ZFS file system, thus, other mainstream Linux distros (Ubuntu) are not really offering me a solution.
Solaris that ZFS was originally developed for is not for small small scale hardware. That leaves FreeBSD and Nas4Free (and Freenas as a worst case scenario).

Other option would be looking into fully proprietary NAS systems like UnRaid. UnRaid is fine, but as with all proprietary (closed-off) development, they usually lack community support and seriously lag behind with applications ported for them. And when you have a problem with one of those, pheew... forget about a quick fix.
The four building blocks of the universe are fire, water, gravel and vinyl. Dave Barry
Come back when you’ve lived a little. Miles Davis

Offline Rotten Johnny

  • ROONed for life.
  • Administrator
  • Trade Count: (+31)
  • *****
  • Posts: 14,477
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #20 on: September 08, 2013, 06:53:08 PM »
Why the predeliction for ZFS?
Audiophile: There is almost no other group that prides themselves more on wasting good money on utterly worthless ****, and then trying to furiously blow smoke up their own ass to justify it.

Free your mind...and your ass will follow.

Offline ara

  • Trade Count: (+2)
  • AVForums Veteran
  • ***
  • Posts: 388
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #21 on: September 08, 2013, 07:27:24 PM »
Why the predeliction for ZFS?

It is a bit like Vinyl vs Digital

Once you have dealt with ZFS (although it also have it's quirks) I guess you will never go back to anything else (at least not me). The big advantage of ZFS I would say is as follows:
1. The ability to scale very large and speed (That is why the Linux crowd have tried to implement it and why they are now trying to implement their own new file system). To give an idea I recently had a HDD failure and it is Raid-6 (Raid-Z2 in ZFS speak) and it took 11 hours to recover/rebuild/resilver 36TB of data.
2. That it is very hardware, os (to a big extent), etc indepent. So you can just normally export a zfs pool (like raid) and import into the next OS that supports ZFS
3. The ability to have very fast, small footprint snapshots for rolling back to previous versions.
4. Deduplication of files to store only links to files that are the same and only keep one (this you can't though do on a MicroServer)
5. It does not suffer from "raid rot" that causes loss of data overtime without you/your system knowing about it.
« Last Edit: September 08, 2013, 07:32:39 PM by ara »

Offline ara

  • Trade Count: (+2)
  • AVForums Veteran
  • ***
  • Posts: 388
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #22 on: September 08, 2013, 07:31:02 PM »
Then, there are issues with Deluge itself. Configuring the global Bandwidth and Queue parameters. At one point as I was tuning down the Upload ratio, Deluge stopped downloading completely and no matter what I did it just wouldn't restart. It took me randomly pulling the various levers one at the time and hitting "Apply" until it eventually started working again. All to do with Upload ratio!? Makes no sense.

Eish Mike. Just leave the things alone to run  ;D but honestly no idea why that would happen. I only control my global upload and downloads normally.

Offline Rotten Johnny

  • ROONed for life.
  • Administrator
  • Trade Count: (+31)
  • *****
  • Posts: 14,477
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #23 on: September 08, 2013, 07:46:41 PM »
Thx Gideon, that recovery time you quoted is super impressive.
Audiophile: There is almost no other group that prides themselves more on wasting good money on utterly worthless ****, and then trying to furiously blow smoke up their own ass to justify it.

Free your mind...and your ass will follow.

Offline fdlsys

  • Vinylist
  • Trader
  • Trade Count: (+72)
  • *****
  • Posts: 7,127
  • Do you like life, sweetheart?
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #24 on: September 08, 2013, 08:00:25 PM »
...I only control my global upload and downloads normally.
Grrrr :ticked: that's what I tried to do! :point:
For some reason, download speed in Deluge is directly and tightly related to anything and everything about uploads (!?!). Speed, number of slots, all Ul ratios. They might be trying to internally enforce the fair/healthy DL/UL ratios but if that is the case, they have gone a fair bit overboard with it.
I had an interesting one just now - I changed - for the hell of it - the UL bandwidth allowance higher up, and before the DL even came close to 300kbs (ULs at only about 20kbs at that moment), pings to the main OS went over 1000ms :headbanger: So I just crippled the UL bandwidth down to 20kbs max, and everything returned to normal. Even though it wasn't even using more that 20kbs at that time. Weird, weird, weird...

Why the predeliction for ZFS?
As Ara said - it's as safe as it gets. I had a good few power loses and the Freenas / Nas4Free box boots up perfectly and immediately after it every time - and I don't even have Raid on it.
UnRaid server that truly is "The Raid" in terms of reliability and performance, loses a few (redundant) bits every time on power loss and than it requires another manual restart to force-start the Samba shares followed by the whole night of rebuilding the parity.
The four building blocks of the universe are fire, water, gravel and vinyl. Dave Barry
Come back when you’ve lived a little. Miles Davis

Offline Tinuva

  • Trade Count: (0)
  • AVForums Member
  • *
  • Posts: 4
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #25 on: September 23, 2013, 12:46:59 PM »
Why not give transmission-daemon a try? Much better than deluge IMHO, and works great on my VM at home and a VPS in Amsterdam, I actually won't use another nix based torrent client right now, they all suck equally, including deluge.

As for ZFS, probably the best thing ever invented at SUN, and luckily because it is forked, Oracle wont completely destroy it.

That said, I definitely prefer OpenIndiana right now, which I use on my Fileserver at home, running both KVM VMs and my Storage.


Offline fdlsys

  • Vinylist
  • Trader
  • Trade Count: (+72)
  • *****
  • Posts: 7,127
  • Do you like life, sweetheart?
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #26 on: September 23, 2013, 01:09:06 PM »
Why not give transmission-daemon a try? Much better than deluge IMHO...
Ara is working on restructuring the HP MS image as-we-speak. He is very happy with his Deluge implementation, but at my place the same (similar) installation on HP Microserver / Nas4Free / TheBrig + FreeBSD Jails ended up completely taking over a 1GB NIC as soon as you give jailed Deluge more than 300Kbs bandwidth. So, a no-go. Not to mention that Deluge is a Java app, a known cpu/memory hog :puke: I really expected more from Deluge...

I tested the built-in Transmission client on N4F and it worked splendidly, stable as a rock. It has a reasonable WebUI but more importantly, once it's set up to either sweep the .torrent files from the "blackhole" or have the torrents passed directly to it via WebUI (by Sickbeard/Couchpotato), it just works, and it's super fast! I compared the DL over some 15'-30' with my favourite uTorrent/Win7 and Transmission won! Excellent stuff.

As for ZFS, probably the best thing ever invented at SUN, and luckily because it is forked, Oracle wont completely destroy it.
:notworthy: Problem is, it's getting actively demoted on Linux even though there are at least two well behaved different packages/forks available for Ubuntu. Ubuntu guys are busy developing their own native fault-tolerant FS so they are not very keen on supporting the foreign FS.

That said, I definitely prefer OpenIndiana right now, which I use on my Fileserver at home, running both KVM VMs and my Storage.
You and Ara could book a room! :point: He's running pretty much the same OS on some 16 CPUs and countless VMs and TBs of disks.
Last time I dared ask "... but why? ..." he said something about being only days away from taking over the world... :ROFLMAO:
The four building blocks of the universe are fire, water, gravel and vinyl. Dave Barry
Come back when you’ve lived a little. Miles Davis

Offline Tinuva

  • Trade Count: (0)
  • AVForums Member
  • *
  • Posts: 4
Re: Image format suggestions (nas4free + LMS 7.2 + deluge + sickbeard)
« Reply #27 on: September 24, 2013, 07:30:31 PM »
Ara is working on restructuring the HP MS image as-we-speak. He is very happy with his Deluge implementation, but at my place the same (similar) installation on HP Microserver / Nas4Free / TheBrig + FreeBSD Jails ended up completely taking over a 1GB NIC as soon as you give jailed Deluge more than 300Kbs bandwidth. So, a no-go. Not to mention that Deluge is a Java app, a known cpu/memory hog :puke: I really expected more from Deluge...

I tested the built-in Transmission client on N4F and it worked splendidly, stable as a rock. It has a reasonable WebUI but more importantly, once it's set up to either sweep the .torrent files from the "blackhole" or have the torrents passed directly to it via WebUI (by Sickbeard/Couchpotato), it just works, and it's super fast! I compared the DL over some 15'-30' with my favourite uTorrent/Win7 and Transmission won! Excellent stuff.
Ok that is great news then for this image :)


:notworthy: Problem is, it's getting actively demoted on Linux even though there are at least two well behaved different packages/forks available for Ubuntu. Ubuntu guys are busy developing their own native fault-tolerant FS so they are not very keen on supporting the foreign FS.
Actually ZFS developers from all over are moving towards a single goal of working together, have a look at the very recently announced: http://open-zfs.org/ initiative, and zfsonlinux is one of the main ones there.

That said, I agree zfs on linux is not as easy as it should be, but then again btrfs is also not yet what it should be, and zfs is light years ahead of it, and it will be at least another 4 years before it will come close. That said, my fileserver will not see Linux as the main OS for this filesystem, I am pretty hectic about my data :P


You and Ara could book a room! :point: He's running pretty much the same OS on some 16 CPUs and countless VMs and TBs of disks.
Last time I dared ask "... but why? ..." he said something about being only days away from taking over the world... :ROFLMAO:
Heh, sounds like he must have some insane file/vm server at home. I mostly use consumer equipment, though I wish I made a better choice for my motherboard, one which has IPMI and dual onboard gig ports.