Novacut Announcementstag:launchpad.net,2010-07-02:/novacut/+announcementshttps://launchpadlibrarian.net/190522637/novacut-64.pnghttps://launchpadlibrarian.net/190522636/novacut-14.png2015-11-22T21:47:35.417025+00:00Dropping support for V0 protocol and schema2015-11-22T21:47:35.417025+00:002015-11-22T21:47:35.358541+00:00tag:launchpad.net,2015-11-22:/+announcement/13738Jason Gerard DeRosehttps://launchpad.net/~jderose<p>The V1 Dmedia hashing protocol, V1 Dmedia document schema, and V1 Novacut document schema were finalized over 29 months ago (on June 1st 2013):</p>
<p><a href="https://launchpad.net/filestore/+announcement/11554" rel="nofollow">https:/<wbr/>/launchpad.<wbr/>net/filestore/<wbr/>+announcement/<wbr/>11554</a></p>
<p>My feeling is that this has given folks ample time to migrate any existing V0 libraries to V1. However, if anyone out there has an existing V0 library and needs help migrating it to V1, please let me know and I'll happily help!</p>
<p>Still, there is a cost in maintaining the V0 to V1 migration path, especially considering some refactoring I'd like to do in the near future. The V0 to V1 migration was unusual in terms of what we expect going forward, and the code for it is rather crufty.</p>
<p>So this migration path has now been dropped in our daily PPA builds for Ubuntu 15.10/Wily and Ubuntu 16.04/Xenial.</p>
<p>Note that support for this migration path is still present in our daily *and* stable PPA builds for Ubuntu 14.04/Trusty and Ubuntu 15.04/Vivid, and that support is still present in our *stable* PPA for Ubuntu 15.10/Wily. The current stable release for 15.10/Wily will be archived in a separate PPA prior to releasing a new stable release for 15.10/Wily that drops support for this migration path.</p>
<p>At this time, no further stable releases are planned for Ubuntu 14.04/Trusty or Ubuntu 15.04/Vivid, so Novacut releases that support the V0 to V1 migration path will be available for as long as each respective Ubuntu release is supported (almost 3 years more for Ubuntu 14.04/Trusty, and about 2 months more for Ubuntu 15.04/Vivid).</p>
<p>This is all part of an effort to trim some fat in preparation for a (hopefully) exciting Novacut release for Ubuntu 16.04/Xenial, the next Ubuntu LTS release.</p>
<p>Finally, note the plan is to indefinitely support the V1 hashing protocol, and to indefinitely support migration from the V1 document schema to whatever document schema version Dmedia and Novacut currently use.</p>Dropping support for GStreamer 1.2/1.4 (Ubuntu Trusty/Vivid)2015-10-10T23:23:22.373955+00:002015-10-10T23:23:22.280949+00:00tag:launchpad.net,2015-10-10:/+announcement/13673Jason Gerard DeRosehttps://launchpad.net/~jderose<p>In terms of the needs of Novacut, GStreamer 1.6 is an absolutely stunning release! So a huge thanks to everyone in the GStreamer community whose hard work made it happen!</p>
<p>The new rendering backend introduced with Novacut 15.08 was written to work with GStreamer 1.2 through 1.6:</p>
<p><a href="https://launchpad.net/novacut/trunk/15.08" rel="nofollow">https:/<wbr/>/launchpad.<wbr/>net/novacut/<wbr/>trunk/15.<wbr/>08</a></p>
<p>However, as we refine our new rendering backend and prepare to start adding new features, the burden of continuing to support GStreamer 1.2 and 1.4 has become difficult to justify. GStreamer 1.2 is especially problematic as it requires many work-arounds and even then can't always produce correct results (not even with perfectly conformant input videos).</p>
<p>As these new Novacut features will likely be a bit rough around the edges at first anyway, it seems like a good time to stop making new Novacut releases on Ubuntu 14.04 LTS (Trusty) and Ubuntu 15.04 (Vivid). Instead, we'll set our sights on a great Novacut release for the next Ubuntu LTS (16.04, due out in April 2016).</p>
<p>So as it stands now, the plan is that Novacut 15.08 will be the last Novacut release that supports Ubuntu 14.04 LTS and Ubuntu 15.04. In the meantime, you'll still be able to install Novacut 15.08 on Trusty and Vivid as those builds wont be going anywhere. But during the next 6 months, new Novacut releases will only be made for Ubuntu 15.10 (Wily) and Ubuntu 16.04 LTS (X).</p>
<p>Depending on how things shake out, we might make additional releases that support Ubuntu 14.04 LTS for some of the six components beneath Dmedia in the Novacut stack (UserWebKit, UserCouch, Microfiber, FileStore, Degu, and Dbase32). But as Novacut and Dmedia both directly use GStreamer, their future releases will only target GStreamer 1.6 and newer.</p>
<p>Our long-term goal is to support new Novacut releases on the current Ubuntu LTS till the next Ubuntu LTS is released, and we came pretty close this time around (the closest we have thus far, actually). But change happens and sometimes you need to cut your losses and move forward :)</p>
<p>Apologies to anyone negatively impacted by this plan. I'm confident we can make it up to you with exciting future Novacut releases that will be waiting for you whenever you're ready to upgrade to Ubuntu 16.04 LTS.</p>
<p>Cheers,<br/>
-Jason</p>Security alert: Dmedia vulnerable to Heartbleed2014-04-19T17:22:44.709573+00:002014-04-19T17:22:44.639081+00:00tag:launchpad.net,2014-04-19:/+announcement/12612Jason Gerard DeRosehttps://launchpad.net/~jderose<p>Security alert: Dmedia vulnerable to Heartbleed</p>
<p>Dmedia (and therefor Novacut) are affected by the Heartbleed[1] bug in the<br/>
OpenSSL[2] library. This bug is very serious as it allows an attacker to<br/>
capture the private keys Dmedia uses, which then allows an attacker to steal<br/>
both your Dmedia library metadata and the files it contains.</p>
<p>Please see USN-2165-1 for details about the OpenSSL fix in Ubuntu:</p>
<p> <a href="http://www.ubuntu.com/usn/usn-2165-1/" rel="nofollow">http://<wbr/>www.ubuntu.<wbr/>com/usn/<wbr/>usn-2165-<wbr/>1/</a></p>
<p>What you need to do<br/>
===================</p>
<p>To correct this problem, first make sure your packages are up-to-date:</p>
<p> sudo apt-get update<br/>
sudo apt-get dist-upgrade</p>
<p>Then you'll need to force Dmedia to generate new user and machine certificates:</p>
<p> rm ~/.local/<wbr/>share/dmedia/<wbr/>user-1.<wbr/>json<br/>
rm ~/.local/<wbr/>share/dmedia/<wbr/>machine-<wbr/>1.json<br/>
restart dmedia</p>
<p>You should do this on all your computers running Dmedia before peering them<br/>
again.</p>
<p>The next time you open Dmedia or Novacut, you'll be presented with the Dmedia<br/>
new-account screen[3].</p>
<p>On your first computer, click "New Account". On any additional computers, click<br/>
"Connect to Devices" and then accept the peering offer on the first computer.</p>
<p>More details<br/>
============</p>
<p>It's easy for an attacker on the local network to use the Heartbleed bug to<br/>
attack Dmedia on systems running a vulnerable version of OpenSSL. This includes<br/>
when you're using, for example, a public WiFi network at a coffee shop. This is<br/>
true even when you only have a single Dmedia device on a given network.</p>
<p>In practice it's probably very difficult for a remote attacker to exploit<br/>
Heartbleed in Dmedia from across the Internet. Most home routers use NAT to<br/>
prevent direct access to your computers from across Internet. Also, each time<br/>
Dmedia starts, it runs on a different, random port. Dmedia uses Avahi[4] to<br/>
advertise this random port to other Dmedia devices on the local network. Dmedia<br/>
does *not* advertise this random port to any outside servers. That said, remote<br/>
attacks could sill be possible if, for example, your router was compromised.</p>
<p>As Dmedia is not yet widely used, it's probably not yet a common attack target.<br/>
However, to play it safe, please follow the above procedure to generate new<br/>
Dmedia SSL certificates.</p>
<p>[1] Heartbleed: <a href="http://heartbleed.com/" rel="nofollow">http://<wbr/>heartbleed.<wbr/>com/</a><br/>
[2] OpenSSL: <a href="https://www.openssl.org/" rel="nofollow">https:/<wbr/>/www.openssl.<wbr/>org/</a><br/>
[3] Peering screen: <a href="http://cdn.novacut.com/Dmedia-12.10-1.png" rel="nofollow">http://<wbr/>cdn.novacut.<wbr/>com/Dmedia-<wbr/>12.10-1.<wbr/>png</a><br/>
[4] Avahi: <a href="http://avahi.org/" rel="nofollow">http://<wbr/>avahi.org/</a></p>Introducing Degu2014-03-27T03:53:10.256201+00:002014-03-27T03:53:10.123375+00:00tag:launchpad.net,2014-03-27:/+announcement/12549Jason Gerard DeRosehttps://launchpad.net/~jderose<p>There's a new component in the Novacut stack called "Degu" (a small, social<br/>
rodent):</p>
<p> <a href="https://launchpad.net/degu" rel="nofollow">https:/<wbr/>/launchpad.<wbr/>net/degu</a></p>
<p> <a href="http://docs.novacut.com/degu/index.html" rel="nofollow">http://<wbr/>docs.novacut.<wbr/>com/degu/<wbr/>index.html</a></p>
<p>Degu includes a lightweight HTTP server that's easy to embedded in desktop<br/>
applications. Degu also includes a matching HTTP client. Degu is specifically<br/>
aimed at device-to-device, peer-to-peer communication across the local network.</p>
<p>Degu is basically a 2nd generation of the internal Dmedia HTTP server. We were<br/>
also running into a few limitations with the `http.client` module in the Python<br/>
standard library. It makes a lot of sense to combine an HTTP server and client<br/>
into the same library as they have a large common intersection when it comes to<br/>
parsing the HTTP wire format and handling IO.</p>
<p>So I felt it was time to refine these elements in our stack, plus make them<br/>
easier for other projects to reuse. As always, we're very committed to<br/>
advancing the state of the art when it comes to real-time-<wbr/>collaboration, and I<br/>
hope that Degu helps enable further experimentation, even when building upon<br/>
Dmedia might seem too high-level or too constrained.</p>
<p>The only downside is that because Degu gave us a chance to re-evaluate how we<br/>
use SSL, I decided to drop Python 3.3 compatibility (and therefor Ubuntu 13.10<br/>
compatibility) a month earlier than we would have otherwise. One of the many<br/>
interesting enhancements in Python 3.4 is that TLSv1.2 support was added to the<br/>
`ssl` module.</p>
<p>So this month's Novacut release (14.03) will only be released for Ubuntu Trusty<br/>
(what will become Ubuntu 14.04 LTS on April 17th). Of course, you'll still be<br/>
able to install the Novacut 14.02 stack on Ubuntu 13.10 for the time being.</p>
<p>Note that because Degu strictly configures its `ssl.SSLContext` for TLSv1.2<br/>
only, Novacut/Dmedia running on Ubuntu 13.10 will *not* be network-compatible<br/>
with Novacut/Dmedia running on Ubuntu 14.04 LTS. If you have multiple devices<br/>
running Novacut/Dmedia, please upgrade all of them to Ubuntu 14.04 LTS at the<br/>
same time.</p>
<p>If you have any questions about using Degu in your own applications, or how this<br/>
change might effect you as a Novacut/Dmedia user, please feel free to ask on<br/>
this mailing list, and likewise to ask in the #novacut IRC channel on freenode.</p>
<p>Happy editing!<br/>
-Jason</p>Novacut 13.12 released, but not for Ubuntu 13.04 (Raring)2013-12-26T05:49:28.684941+00:002013-12-26T05:49:28.610986+00:00tag:launchpad.net,2013-12-26:/+announcement/12294Jason Gerard DeRosehttps://launchpad.net/~jderose<p>Merry Christmas, everyone! Novacut 13.12 is now in our stable PPA:</p>
<p><a href="https://launchpad.net/~novacut/+archive/stable" rel="nofollow">https:/<wbr/>/launchpad.<wbr/>net/~novacut/<wbr/>+archive/<wbr/>stable</a></p>
<p>Note that this release is not available for Ubuntu 13.04 (Raring), but support for Raring ends next month, so it's a great time to upgrade to Ubuntu 13.10 (Saucy).</p>
<p>We'll always make monthly release for the latest Ubuntu version (currently 13.10 "Saucy"). And we'll make monthly releases for the Ubuntu development version (currently 14.04 "Trusty"), although there might be a month or two of lag time there. Finally, we'll generally make releases for the previous Ubuntu version for a month or two to give people time to transition.</p>
<p>To make this a bit clearer, this is how things went down this time around:</p>
<p>Novacut 13.10 - first release for Ubuntu 14.04 dev<br/>
Novacut 13.11 - last release for Ubuntu 13.04<br/>
Novacut 13.12 - released for Ubuntu 13.10 & 14.04 only</p>
<p>Enjoy!</p>V1 hashing protocol is go!2013-06-02T17:44:00.204486+00:002013-06-02T17:44:00.104975+00:00tag:launchpad.net,2013-06-02:/+announcement/11558Jason Gerard DeRosehttps://launchpad.net/~jderose<p>So big news! The Version 1 Dmedia Hashing Protocol has been finalized, and is<br/>
now used by default.</p>
<p>The only downside is that because of the switch to our Dbase32 encoding, it<br/>
wasn't possible to support V0 alongside V1. The first time you load a FileStore<br/>
containing a V0 files/ layout, it will move this directory to files0/,<br/>
preserving your data.</p>
<p>But to actually use these files, you must run this upgrade script from a<br/>
terminal:</p>
<p> novacut-<wbr/>v0-v1-upgrade</p>
<p>Or optionally, if you only have Dmedia installed, run this (which the above<br/>
calls before upgrading the Novacut databases):</p>
<p> dmedia-<wbr/>v0-v0-upgrade</p>
<p>For more help, please watch this screencast that walks you through the upgrade<br/>
process:</p>
<p> <a href="http://www.youtube.com/watch?v=NOyEIab0E-U" rel="nofollow">http://<wbr/>www.youtube.<wbr/>com/watch?<wbr/>v=NOyEIab0E-<wbr/>U</a></p>
<p>There will certainly be additional protocols added in the future, but from V1<br/>
forward, all protocols will be supported indefinitely. Newly added files will<br/>
be hashed according to whatever the newest protocol version is at the time, but<br/>
you will always be able to resolve and verify a file according to an ID computed<br/>
with an older protocol version.</p>
<p>This is important so that we can always preserve the link integrity between,<br/>
say, a Novacut edit and the files that edit references.</p>
<p>V1 is a big milestone not just because it finalizes the details of V1, but also<br/>
because it finalizes our protocol framework, the design aspects we expect to be<br/>
the same across all protocol versions from V1 onward.</p>
<p>Each new protocol version will use a different digest size, and we'll use the<br/>
ID length to determine what protocol version was used to compute an ID. So for<br/>
example, V1 uses a 240-bit digest (48 characters when Dbase32 encoded), and V2<br/>
might use a 280-bit digest (56 characters when Dbase32 encoded).</p>
<p>Things that will (or can) change between protocol versions:</p>
<p> * By definition, the digest size will change</p>
<p> * The underlying hash function likely will change</p>
<p> * The leaf size might change</p>
<p>And things we don't expect to change:</p>
<p> * The digest size will always be a multiple of 40 bits</p>
<p> * The root-hash and leaf-hashes will use the same digest size</p>
<p> * The leaf size will be a multiple of 2 MiB</p>
<p> * The root-hash will be cryptographically tied to the file-size</p>
<p> * The leaf-hashes will be cryptographically tied to the leaf-index</p>
<p> * Dbase32 will always be the "official" digest encoding, and will be the<br/>
encoding use by standard FileStore layouts</p>
<p>We want to start research on V2 soon, but we likewise want to be very leisurely<br/>
about finalizing V2 (which is why we should start research soon). I hope it<br/>
will be at least 5 years till we add another protocol.</p>
<p>If anyone is interested in such research, the Protocol base class makes it quite<br/>
easy to build experimental protocols:</p>
<p> <a href="http://bazaar.launchpad.net/~dmedia/filestore/trunk/view/head:/filestore/protocols.py" rel="nofollow">http://<wbr/>bazaar.<wbr/>launchpad.<wbr/>net/~dmedia/<wbr/>filestore/<wbr/>trunk/view/<wbr/>head:/filestore<wbr/>/protocols.<wbr/>py</a></p>
<p>Also, folks might be interested in reading the V1 protocol specification:</p>
<p> <a href="http://docs.novacut.com/filestore/specification.html" rel="nofollow">http://<wbr/>docs.novacut.<wbr/>com/filestore/<wbr/>specification.<wbr/>html</a></p>
<p>(Thanks again to Robert von Burg for working on an experimental Java<br/>
implementation, which provided lots of great feedback that helped make the<br/>
specification clearer and more correct.)</p>
<p>And one last note, just because people tend to ask a lot: I'll explain why we<br/>
stuck with Skein for V1, rather than using the SHA-3 winner (Keccak).</p>
<p>The biggest reason we stuck with Skein is performance, specifically performance<br/>
of 64-bit software implementations. For our current 240-bit digest size, Skein<br/>
is roughly twice as fast as Keccak (on my hardware anyway). And that is<br/>
significant, because even Skein can't quite keep up with the read throughput of<br/>
todays fastest SSDs when hashing on a single core.</p>
<p>Although Keccak can be faster than Skein when implemented in hardware, we have<br/>
to make decisions based on the here and now. Today, no such hardware<br/>
implementations are readily available, and it might be many years before they<br/>
are.</p>
<p>Another issue is the Skein parameter system made our cryptographic tying very<br/>
easy, but HMAC (or equivalent) hasn't been defined yet for Keccak, and defining<br/>
our own would be very risky. Yet lots of practical experience working through<br/>
different protocol iterations, and building useful software on top, has made me<br/>
more convinced than ever that this cryptographic tying is an extremely important<br/>
and pragmatic feature for our use case.</p>
<p>And the last thing is I've embraced the idea that we will have additional<br/>
protocol versions sooner rather than later. We will support V1 forever, not use<br/>
it by default forever. I think it's important that we architect the `filestore`<br/>
package to support multiple protocol versions as soon as possible, long before<br/>
we actually have a second protocol. This is inevitable, so let's start<br/>
practicing now.</p>
<p>We really needed to commit to a stable protocol version *now*, and I think at<br/>
this moment Skein is the best choice. Plus, if we switched to a different hash<br/>
algorithm now, it would still take time to understand it as well as we<br/>
understand Skein, to be confident about the way in which we are using it. That<br/>
could easily mean another year till we have a stable protocol.</p>
<p>Anyway, thanks to the many people who have patiently reviewed the many protocol<br/>
iterations leading up to V1, especially David Jordan, Hagen Fürstenau, and<br/>
Robert von Burg.</p>
<p>-Jason</p>Seeking feedback on proposed D-Base32 encoding2013-01-22T08:17:25.638811+00:002013-01-22T08:50:26.266165+00:00tag:launchpad.net,2013-01-22:/+announcement/11099Jason Gerard DeRosehttps://launchpad.net/~jderose<p>We're seeking feedback on our proposed D-Base32 encoding.</p>
<p>The problem with standard RFC-3548 base32 encoding is the sort-order of the base32-encoded IDs is different than the sort order of the binary IDs.</p>
<p>Eventually we're going to have Dmedia-lite... a trimmed down version aimed at mobile devices, and more at content consumption than content creation. For space and performance reasons, Dmedia-lite will probably decode the base32 IDs and build its indexes according to the compact, binary IDs. In fact, even the full version of Dmedia will probably do this eventually, because it will give us better performance and allow us to handle even larger databases.</p>
<p>Still, indexing according to the base32-encoded IDs is simple and useful, something we want to support. But there are serious interoperability problems between a database using binary IDs vs one using base32 IDs, because they will have a different idea of what the first and final document IDs are in the database. So a request for a range of docs like:</p>
<p>1 <= doc_N <= 6</p>
<p>Would come out nonsense on the other end:</p>
<p>14 <= doc_N <= 3</p>
<p>Click the "Read more" link for all the gory details.</p>