GoPro + Cinelerra: High framerate slow motion and difference key

I was recently given the GoPro Hero3 Black edition by my very kind family for Fathers’ Day. Mostly I plan to use this for its high framerate capabilities to develop my skills in aggressive inline skating (it’s very hard to know how to improve your technique when you can’t see what you’re doing).

As a Linux (Ubuntu) user my software of choice for video editing is Cinelerra (I use the cinelerra-cv package maintained by Nicola Ferralis).

Cinelerra is brilliant … but infuriating. So for anyone else out there trying to get the most of their GoPro camera while working in a Linux environment, I’ll post my experiences to try to help.

My first tinkering with Cinelerra and the GoPro revealed one nice thing: By choosing “Microsoft AVI” as the file format and “MPEG-4 Video” as my compression codec when rendering from Cinelerra I seemed to get fairly fast performance processing GoPro source files for simple things like cropping, editing and dropping frame rate. That changes dramatically when video effects and transitions are applied, though…

My first hurdle was in trying to edit some footage from the skate park in which I wanted to observe my jumping technique in slow motion. I had recorded four passes in front of the stationary camera and wanted most of the action to run full speed with just the jumps slowed to 1/4 speed.

I recorded footage at 1280x720x100FPS with a wide field of view. My plan was to output 1024x576x25FPS footage, cropping off the extra pixels to zoom in a little. For the time being I’m not worrying about audio at all.

Here’s the output footage I have produced (side note: the file I uploaded to youtube was 100FPS in the end; more on that shortly):

Challenge 1: Project framerate

My first instinct was to create a Cinelerra project with 1024×576 dimensions, 16:9 aspect ratio and a frame rate of 25FPS. I expected to be able to drop a slow-mo effect onto my timeline in the desired place to make use of all the frames in my 100FPS source to get smooth slow-mo while the rest of my footage would have frames dropped by Cinelerra to output full-speed 25FPS footage. That is how things should work.

Achieving what I pictured would have this work-flow:

1. Create a new project in Cinelerra. (Settings->Format->Frame rate: 25; Width: 1024; Height: 576; W & H rations 1.0; Color model: RGBA-Float [for difference key; YUV-A reported to crash]; Aspect ratio: 16:9; Interlace mode: Not interlaced.)
2. Open GoPro source files using “create new resources only” option
3. Open GoPro source files in Cinelerra viewer (Window -> Viewer) by right-clicking Media in the Cinelerra Resources window and choosing “View”
4. Drag slider to find start and finish points from your video (ignore error windows that pop up); click “In point ([)” and “Out point (])” buttons at start and finish points
5. Click “to clip” and give the segment a name; repeat these steps for as many clips from the original footage as will be used in the final result
6. Insert in-point at the start of your editing timeline; drag and drop your first clip (from the resources window) onto the timeline
7. Fast-forward to the end of the editing timeline, move the in-point to this spot and repeat above step with your next clip
8. Select the first section you want to make slow-motion, click the “Toggle label at current position” button – this will mark these start and end points.
9. Note the start-time duration and end-time of the selection; change the start time to the end time, click in the duration field (the middle of the three marked in the below screenshot), type the duration your selection had andpress enter:
10. click the in-point and out-point buttons, marking this region
11. Edit->Paste silence repeatedly – for half speed slow-mo paste silence once; for quarter speed paste silence three times
12. Click on the label you added at the start of your slow-mo footage and move the in-point to this spot; leave the out-point where it is
13. Drag and drop the “ReframeRT” video effect into the region inside the in- and out-points
15. Make sure the “key” symbol (“generate keyframes while tweaking”) is not highlighted; Right click the “ReframeRT” bar and select “Show”. Enter your scaling factor (0.25 for quarter speed) and choose the “stretch” option

These steps should be sufficient to achieve smooth slow-motion. You have a 100FPS source (which Cinelerra does recognise as 100FPS) and a 25FPS project. The Cinelerra documentation says this of the “Stretch” option in the ReframeRT effect:

Stretch mode multiplies the current frame number of its output by the scale factor to arrive at the frame to read from its input. If its current output frame is #55 and the scale factor is 2, frame #110 is read from its input.

So… With a scale factor of 0.25 frame 2 of the output should read frame 0.5 of an input with the same frame rate. But it seems logical to me that if the input frame rate is higher, Cinelerra should be smart enough to take that into account and multiply by output framerate divided by input frame rate. It doesn’t. Checking the “interpolate” option doesn’t change anything, either. So the abve workflow achieves a jerky slow-motion effect that doesn’t include three out of every four frames in your original video, and is thus no better than having filmed with a 25FPS camera in the first place.

Cinelerra documentation talks about changing the framerate of your resources as an alternative strategy to achieve slow-mo. This would involve right-clicking the media in the resources window and choosing “Info”, then dropping the frame rate to the project frame rate. In theory you could then work in reverse, speeding up the sections of your video you want to be normal speed using ReframeRT with scale factor of 4 and leaving the parts you want slow-mo unaltered. However, in practice if I changed the frame rate of my source footage Cinelerra would only acknowledge the existence of the first quarter of the file. No good to me.

So… my solution to this issue was to change the project framerate to 100FPS (a quick way is to right-click the media in the resource window and choose the “match frame rate” option. This doesn’t alter any of your edits so is nice and safe to switch back and forth as you work on your project (in my experience so far … Cinelerra can have a habit of surprising you, though).

Working with the project in 100FPS, the rest of the workflow described above stays the same. The downside is you waste processing time rendering a 100FPS video at the end that you will then have to transcode with something like ffmpeg to drop the extra frames if you need to get down to 25FPS footage.

Challenge 2: Layering video with difference key

My particular footage involved 4 successive passes in front of my camera attempting to jump as high as I could. I really wanted to be able to compare my jumps and technique side-by-side. So I decided to try out the difference key effect.

The workflow I used was this:

1. Have one video track (at the bottom) for the first jump and the final jump; add two videos tracks for each of the next two jumps (four additional video tracks)
2. Leave a blank track above the track for the first jump; in the track above that, stagger the second jump clip so it starts a bit later than the first
3. Copy a first frame from the clip of the second jump containing only the background; paste this frame on the blank track under the footage of the second jump.
4. Drag the “freeze frame” video effect onto the track with the sigle background frame; extend the length of the effect so it is longer than the clip of the second jump; mute this track (“Don’t send to output” to the left of the track)
5. Drag the “difference key” video effect onto both the background frame track and the track of the second jump.
6. Right-click the difference key effect on the background frame track and choose “change”; click the “Video X: difference key” option under “shared effects”. This links the effect to the difference key effect on the footage of the second jump
7. Right click the difference key effect on the foreground action track (the second jump) and select “show”; drag the “threshold” and “slope” options until you can clearly see the foreground action from both the first and second jump clips – just play to find what works
8. With the recording patch turned on for only the foreground action track of the second jump, in the compositor window click the mask tool and click and drag points to mask out areas of the footage that will have no action – particularly if there things like clouds moving, as in my footage:
9. Repeat for other layered tracks

Challenge 3: Getting back to 25FPS

Now, if you’re going to play this on a reasonably-powered PC, just leave your rendered video 100FPS. But I have media players (such as my networked tv) that can’t handle 100FPS. So I used ffmpeg (note: for Ubuntu users ffmpeg has been ditched from the official packages and using the ffmpeg command actually calls a forked software package. Google to find out how to install the “real thing” because the below command won’t work otherwise):
ffmpeg -threads 0 -i CINELERRA_RENDER.avi -vcodec libx264 -crf 15.0 -acodec libmp3lame -f avi -r 25 MY_25FPS_OUTPUT.avi

This must be quality overkill because despite the quartered frame rate the files I get are as big or bigger than the 100FPS source. But at least the quality isn’t terrible so … for now that’s my method.

I hope this might be some help to some others struggling to get Cinelerra to play nice with high FPS source material.