How to make a decent video encode from DVD to .mkv with chapters, subtitles, multi channel audio etc with Sabayon Linux

We’re all recording our TV shows etc onto computer files nowadays, thanks to XBMC, Windows Media Center, Media Portal, Apple TV and even Sony Playstaion, many of us are enjoying a rich online library of content to show off to our friends.  For me, the first thing I do when I buy a DVD is rip it to an mkv and store the DVD away in the cupboard.  I just wish you could take a USB stick down to the video store and have this done already.  But no go, so I do it myself.

This is something I’ve been doing for years.  There’s just no excuse for bad quality huge file size divx files anymore.  While I might have been fortunate enough in the past to have always had a dual-core while some people haven’t had that fortune, many years have passed and this type of hardware is extremely common these days.  The number one reason why using the 264 codec wasn’t popular for most was simply encoding time, if you didn’t have a dual core you would have most likely been looking at about 3-7 days to encode a single file.  But don’t fret, this is no longer the case.

This method, (although not rocket science at all) is often overlooked in favour of other, more historical methods out of sheer laziness.  With this method it is common to produce a file between only 700MB and 1.1GB that includes a full resolution video track with NO loss in quality that I’ve ever been able to see, a full uncompressed copy of the original AC3 and/or DTS soundtrack, full chapter support, multiple subtitles and even a copy of the cover art.  When I say no loss in quality, I really do mean it.  I’ve looked at these things so hard, found the smallest little error and then ended up realising it was actually a problem on the original DVD.  It’s important to note actually, that if you’re fussy, the kinds of issues you’re likely to be looking at will actually NOT be with the encoded file, but in the original source.

Why would I do it on linux?  Well, for starters, when it comes to encoding, linux is faster and not just a little bit either, quite a lot actually.  I can’t comment on Windows 7 at this point, but certainly on prior versions, Linux has had a considerable lead on encoding time.  I suspect former versions of Windows did not support the new processors properly.  This goes double when you have a quad core machine or higher.  Plus, if you’re feeling particularly greedy, you can install the 64bit version of linux and compile your own copies of the encoder in sabayon for your specific CPU and hardware using the emerge command.  If you’re going to do a lot of encoding, the 64 bit / manually compiled encoders add quite a bit of extra speed.  If you’re lucky enough to own a core i7 you would get even more advantage out of the maths capabilities of that architecture.  A typical scenario on my Intel Quad Q9300 takes about 2 hours to complete the mkv file above without manual compiling and running 32 bit Sabayon KDE.

**Before you start, please be aware that I in no way wish to support the pirating of DVD’s for spreading across the internet.  This guide is for personal use only and is intended to be used only for the transcoding of your already owned media to your own personal system.**

Righto, now that that’s over, we can begin.

Make the ISO

This script (h264enc) will actually encode/transcode from just about any format to any other format.  In fact it has presets for everything from blackberry’s, psp’s and ps3’s to generic hardware based DVD players.  For the sake of this guide however, we will concentrate on encoding an MPEG-2 ISO file made from DVD to an MPEG-4 mkv file playable on any good media centre such as XBMC or media portal.

While you can use this method to encode from other DVD structures, ISO’s are the easiest method, so we will focus on that here.  I’m not going to go into how to make an ISO from a DVD disc, however I can point you to k9copy for linux and DVD Fab HD Decrypter for windows.  Sometimes the windows method is preferred as Linux often is not up with the latest methods of extracting this information from a DVD.  The only comment I will make, is don’t bother to ‘Shrink’ the DVD to a 4.3GB ISO.  Since we won’t be trying to put it on a single layer disk (in fact we can fit four mkv files on a single layer DVD (therefore four movies)) having the original 9GB file will actually help your final encode to be higher quality since the source will have more colour bits in it to read from.

Get the software

In Sabayon you will need to install the following software:

mplayer, ogmtools (for chapter support), gpac (if you want to convert for PSP etc),

mkvtoolnix (for mkv support), h264enc, smplayer (just because it’s the best media player for linux and windows) :)

There are other packages that would be useful depending on what you want to do with h264enc, but for the sake of this example I’m leaving them out for now.

# equo install mplayer smplayer gpac mkvtoolnix ogmtools

Then you need to download h264enc from the website since it’s not currently included in Sabayon and this way you don’t have to wait for an ebuild to be made.

Browse to http://sourceforge.net/projects/h264enc/files/ and get the latest version.

Extract the version you downloaded using tar -zxvf fileyoudownloaded

go into the directory that was just extracted, probably h264enc9.0.1 or something like that.

become root

su <yourpassword>

type ./install and follow the simple questions.

Checking your config

You should now have all the relevant files installed.  To check, as your NORMAL user enter the following:

$ h264enc -sc

You should get output similar to the following:

someone@somemachine ~ $ h264enc -sc

-> Checking for ‘MPlayer’………………… OK
-> Checking for ‘MEncoder’……………….. OK
-> H.264 video support in MEncoder………… YES
-> AAC (FAAC) audio support in MEncoder……. YES
-> MP3 (LAME) audio support in MEncoder……. YES
-> AC3 (lavc) audio support in MEncoder……. YES
-> PCM audio support in MEncoder………….. YES

-> Checking for ‘bc’…………………….. OK
-> Checking for ‘pv’…………………….. FAILED! [no support for DVD ISO dumps]
-> Checking for ‘dd’…………………….. OK
-> Checking for ‘neroAacEnc’……………… FAILED! [no support for AAC+ audio]
-> Checking for ‘aacplusenc’……………… FAILED! [no support for AAC+ audio]
-> Checking for ‘oggenc’…………………. OK
-> Checking for ‘flac’…………………… OK
-> Checking for ‘less’…………………… OK
-> Checking for ‘lsdvd’………………….. OK
-> Checking for ‘dvdxchap’ (from ogmtools)…. OK
-> Checking for ‘mkvmerge’ (from mkvtoolnix).. OK
-> Checking for ‘ogmmerge’ (from ogmtools)…. OK
-> Checking for ‘MP4Box’ (from gpac)………. OK
-> Checking for ‘tsMuxeR’………………… FAILED! [no support for the TS container]

If you have installed a required program but the script
can’t find it, run ‘h264enc -r’ to reset the config file.

Don’t worry about the missing aac etc, you don’t need that.  In fact you’re all done!  All that remains is to run the script.

Before we run the script I need to point out some more things.  I’m not going to step by step you through the script, there are just too many options.  However there are some things that I can give you pointers on.

  1. Always know if your source file is interlaced or not.  If you don’t know how to tell, just play your DVD in a computer (must be a computer) and you will see if there are jagged lines in the image or not.  If you can’t see any or aren’t sure, you will have progressive.
  2. If you do have interlaced, you need to say that you DON”T want to preserve interlacing as nothing can play it otherwise.  Use the Yadif and MCDEINT deinterlacer for best quality.
  3. Always use 2 pass, don’t bother with 1 pass or 3 pass
  4. Always always choose to crop, it’ll save encoding time and it’ll look cleaner when you play it later
  5. Always export chapters (especially with a music DVD as you will be able to skip songs)
  6. If you have a slower processor, you can use vhq instead of ehq, you probably won’t notice the difference.
  7. If you change the name of the output file, you must put .avi on the end
  8. When you get to the audio section choose COPY.  Don’t bother with the others, they’re a waste of time unless you’re encoding for a PSP or something.

To execute the script to start your encode:

$ h264enc -2p -p ehq

Have a play, and feel free to post comments / questions.  Did I mention that these can be hardware accelerated by NVIDIA GPU’s using CUDA / VDPAU?  My CPU only uses about 1% playing 24GB 1080p files on an 8600GT card.   It’s sweet! :)

Written by marshalleq

12 Comments

gravatar
Damien Bird

Hi, followed your guide for my Ubuntu 9.04 install. I have been using makemkv beta to date for my dvd rips on xbmc. Thought I would see if I could save some space as the mkv’s are dvd size via makemkv. Everything did as it should however I get an error, H.264 video support in MEncoder….. NO.

If I try to run the script I get:
Checking for libx264 support in MEncoder… Failed!
download libx264 and recompile mplayer

I have tried a few things like reinstalling Mencoder and libx264 but other than that – I ain’t sure! Please help me as I would like to achieve the space saving.

Reply
gravatar
marshalleq

I’m not that familiar with Ubuntu other than in XBMC live, however I think perhaps you need the 264-encoder library. A while back the player and encoder were packaged to together but as of about 5 months ago they seperated them. On my system (Sabayon) I have x264 and x264-encoder. If you’re not compiling mencoder yourself, then it has to simply be that the 264 encoder is not present unless the package maintainer overlooked it which I doubt or I’m on the wrong track here :)

try doing h264enc -r after re-installing the encoders too, it will reset the packages it knows about. Then to see if the libraries are available, type h264enc -sc, both as normal user.

Reply
gravatar
Grozdan

2) “nothing can play it at all” is not correct. Some monitors only support interlaced content. Also, yadif will be more than enough for most of the time and you’ll see very little difference between yadif only and yadif + mcdeint, but with the latter, you’ll considerably increase encoding time as it’s very slow, even if you choose fast mcdeint option

7) this is incorrect. You don’t have to add .avi extension if you change output name. h264enc will do it for you so specifying something like ‘/home/username/my custom title’ is valid, including with spaces as h264enc is written to escape them, both in file names and directories

Further, there’s a difference between 1-pass bitrate-based encoding and 1-pass crf encoding. If you don’t care for file size, then use crf encoding with the -crf option. This will save you time and you won’t be able to tell the difference between it and a 2-pass encode. crf is a constant rate factor which roughly maps to a quality scale. For DVDs, I recommend (and use) a value of 19. For HD, I’d use something in between 21-23

ps: I’m the author of it so I should know 😛

Reply
gravatar
marshalleq

Thanks Grozdan.

For 2 perhaps I should have said, “Nothing I’ve tried” since I’ve tried preserving interlacing in an encode before but the main players (mplayer, xine etc) complained bitterly they couldn’t play it. There was also comments in mplayerhq somewhere stating it was unsupported. However perhaps now it is? If you can clarify that I would be grateful!

7 – I didn’t know! I think maybe this used to be the case and is no longer, I hadn’t tried it in quite a while.

I tried the crf encoding last week at your advice, but I’m not sure about it yet. I encoded the same video file in both methods and could not see a quality difference between the two. I did see that the crf method pretty much doubled the file size from 1GB to 2GB though so I’m not sold on it yet. More experimentation is needed. And I’m concerned about crf not having 2 pass encodes.

After taking into account my very limited experiment and http://trac.handbrake.fr/wiki/CRFGuide it seems all in all that I’m trading speed of encodes for disk space, which really goes against why I started using 264 codec in the first place (small encodes great quality). But feel free to point out any errors you see in my thinking :).

Reply
gravatar
Grozdan

Yes, in the past mplayer/libavcodec had problems with interlaced H.264 video as they didn’t implement some things in the decoder to properly decode it so you got lots of warning messages but now it works (for most of the part). Still, interlacing is a dinosaur and should die ASAP, together with telecining.

As for crf, like I said, it’s a quality-like encoding mode which works only in 1-pass mode and it wouldn’t make much sense for it to have a 2-pass mode. It resembles the constant quality option (fixed_quant) in Xvid & friends. A lot of people use it, including most on doom9, as they want to get uniform quality across all their encodes and don’t care much for final size. Personally, for DVD encodes @ crf 19, I get somewhere in between 600-850 MB which is good for me and quality is pretty awesome (bitrate fluctuates between 700-1100 kbps). If you always stick to 2-pass encoding which a fixed bitrate for all your encodes (say, 950 kbps for all 2-pass encodes), it could very well be that for some of them that bitrate will not be enough or in rare cases will be overfill. In crf, you don’t have to decide/worry which bitrate to use for your encodes as crf does it for you. As with all constant-quality-like options, it’s very difficult to predict final file size. So yeah, you need to test on your own and see which crf gives you a good balance for you between quality and file size. As I only do DVD encodings at the moment, I standardized on a value of 19 (with Nero HE-AACv1 audio @ 55 kbps) and am very pleased with the results 😉

Yes, in the past you needed to add .avi at end of output name but this is no longer the case for quite some time. You can add any other extension if you want (like .mp4, .foobar, .mkv, .blabla, etc) and h264enc will still convert it to .avi. It’s very simple as I just strip the last extension and replace it with .avi, eg OUTPUT=”${OUTPUT%.*}.avi”

Reply
gravatar
marshalleq

Thanks Grozdan. I agree, Interlacing should die!!! Unfortunately a lot of my old content is still interlaced and I still come accross it in things like documentary content and old mini dv videos. I’ll have another go at crf, but if I can’t see a quality difference (and I’m quite picky about that) and the file size goes up I still don’t see the point. My encodes seem to be about 1GB but include an original uncompressed AC3/DTS soundtrack so the video is quite small. Since you’re using AAC I guess that’s how you’re keeping them under 1GB with an increased video size.

Looks like I’m going to have to test it over and over to get an absolute result. Might also go ask the folk over at doom9 since it seems that this is the way the ‘flock’ are going. Thought that doesn’t necessarily mean anything look at VHS :) Although, if I still like 2pass I may still discover in a year or so that I’m wrong!

Reply
gravatar
Grozdan

The point is that crf decides for you how much bitrate a specific video will need or not need and does not put you within boundaries, thus taking away the decision you’ll have to make when choosing a bitrate. Since each video content is different, it could be quite challenging to find a bitrate for each of it, and you’ll have to do it individually for each video as one may be just fine with say 1000 kbps while another one may need (much) more or (much) less. Also, as said, crf does not have a the same”restriction” like 2-pass does since 2-pass tries to squeeze the most out of the given constrains (bitrate boundaries). F.ex if you give it only 400 kbps, it’ll try to get the best out of this 400 kbps but the end-result may still look like crap, even though you did a 2-pass encode. On the other hand with crf, the rate control is much more “broad” and adapts to the “needs of the content”. crf has also its boundaries since it makes its decisions on the crf (rate factor) value you give it. If it’s too high, it’ll probably look like crap and if it’s too low, it’ll probably waste bits unnecessarily. However, with crf, if you find the one value you like, you’ll get across all your encodes roughly the same quality while with 2-pass, this cannot be guaranteed since if you have “standardized” on say 900 kbps for *all* your encodes, for some of them this may be enough while for others it may be way too low or way too high so you’ll have to decide on per content which bitrate to use, thus constantly evaluating bitrate requirements for each content individually so in the end you can’t really “standardize” on a specific bitrate since on some content it’ll be too much and a waste of bits while on another one it could be too low. With crf, it decides for you how much bitrate the content needs or doesn’t need

As for audio, yes, my audio part is quite small and honestly keeping AC3/DTS audio is a waste of bits since these codecs are so inefficient that they *need* the high bitrates you see (like 448 kbps for 6-channel on DVDs and even more for DTS audio) in order not to produce noticeable artifacts. Try encoding AC3 Stereo audio @ 128 or 112 kbps and compare that to AAC Stereo @ 128 or 112 or even 96 kbps. AAC (and others like MP3/Vorbis/etc) beat the crap out of AC3 at such bitrates. 6-channel AAC @ 224-260 kbps will be indistinguishable from AC3 @ 448 kbps or DTS at even higher bitrate (768 kbps or so) so you’re really not only keeping an inefficient audio codec that wastes bits because it was designed by monkeys, but also increasing your file size for no reason at all thinking that by keeping the original audio, you somehow get “perfect” audio quality

Reply
gravatar
marshalleq

So a few years on now and I still dislike -crf. I don’t know, I get the theory behind it but practically it just seems to be more difficult and doesn’t seem to give me ANY advantages. In fact all I can see is disadvantages, namely the quality number isn’t constant and varies depending on the quality in the movie and the file size is huge and I still maintain a 1 pass cannot give the same quality as a 2 pass. My 2 pass encodes are fantastic quality and in spite of what anyone says a copy of the original compressed audio is better than compressing it again, I mean it seriously isn’t that big anyway so why would I care?

However if you’re the type that encodes only for a specific device (let’s say a macbook pro) and you’re only going to play it on that, the speaker is never going to give you good quality anyway so you won’t be able to hear the difference and you may as well compress the audio a few times to save yourself the 200MB. But that’s likely all you’ll save.

If you have a decent stereo plugged in, (be it 2 channel or surround sound (and this does not mean some cheap japanese thing with lot’s of flashing lights because sorry if you have one of those you will likely not understand what I mean here), I mean like an Arcam or a Marantz or a Linn kind of league then you are going to notice the re-compressed audio big time. This is the situation I’m in.

It’s far better to use 264 instead of XVID with a decent encode as that will save the space that the audio copy takes up and then some.

Of course these day’s we have the new and massive audio tracks that are coming with Blu-Ray also to contend with, but thankfully they also include the traditional DTS and AC3 soundtracks as well.

H264enc is a great tool, though I have now moved away from it as using a console script took too long, especially when doing multiple files. However one thing for sure, Grozdan knows how to make a good video encoding profile, and I have based my new tools profiles around his profiles which I may just write an article about somewhere. 😉

Reply

Make your mark!