As someone who usually hacks on LibreOffice, external import filters produced by the Document Liberation Project cut both ways: they are great, as they deal with obscure formats and we get them for free, OTOH hacking such code is more complex than the usual LO code. I recently contributed a few patches to libvisio and libodfgen, but before I was able to do actual code changes, I had to set up a number of repositories and configure them to talk to each other — this post describes one possible setup that suited my needs.
DLP’s central project is librevenge and everything builds on top of that, either by calling it or called by it. In case the task is to turn VSDX files into ODG ones, it looks like this:
libvisio can build a librevenge document model from Visio files (more on the various librevenge-based libraries here), libodfgen can generate ODF output from such document models (one other possibility would be e.g. libepubgen), and the writerperfect module provides kind of a controller for the remaining modules, e.g. for our purpose, a vsd2odg binary.
One possibility is to build LibreOffice, use
similar switches, then clone the repos, install them system-wide (possibly
with your modifications), and then you can test your changes just with
building the various libs, without changing your LO build (more
The drawback is that this way you pollute your system with unstable versions
of those libs.
An other possibility is to build LibreOffice as usual, and then use the external libraries patching mechanism to hack on the code. The drawback is that you have to work without git on the code, and also you can only work with a released version.
So here is what I did to avoid the above mentioned drawbacks: all DLP projects use pkg-config to find the required libraries, so you can configure them in a way that allows building as a user, avoid installing them at all, and still execute vsd2odg using the libs with your changes. Here is how to do it:
git clone git://git.code.sf.net/p/libwpd/librevenge
git clone git://gerrit.libreoffice.org/libvisio
./configure REVENGE_CFLAGS="-I/home/vmiklos/git/libreoffice/librevenge/inc" REVENGE_LIBS="-L/home/vmiklos/git/libreoffice/librevenge/src/lib/.libs/ -lrevenge-0.0" REVENGE_GENERATORS_CFLAGS="-I/home/vmiklos/git/libreoffice/librevenge/inc" REVENGE_GENERATORS_LIBS="-L/home/vmiklos/git/libreoffice/librevenge/src/lib/.libs/ -lrevenge-generators-0.0" REVENGE_STREAM_CFLAGS="-I/home/vmiklos/git/libreoffice/librevenge/inc" REVENGE_STREAM_LIBS="-L/home/vmiklos/git/libreoffice/librevenge/src/lib/.libs/ -lrevenge-stream-0.0" --enable-debug
git clone git://git.code.sf.net/p/libwpd/libodfgen
./configure REVENGE_CFLAGS="-I/home/vmiklos/git/libreoffice/librevenge/inc" REVENGE_LIBS="-L/home/vmiklos/git/libreoffice/librevenge/src/lib/.libs/ -lrevenge-0.0" REVENGE_STREAM_CFLAGS="-I/home/vmiklos/git/libreoffice/librevenge/inc" REVENGE_STREAM_LIBS="-L/home/vmiklos/git/libreoffice/librevenge/src/lib/.libs/ -lrevenge-stream-0.0" --enable-debug
git clone git://git.code.sf.net/p/libwpd/writerperfect
./configure REVENGE_CFLAGS="-I/home/vmiklos/git/libreoffice/librevenge/inc" REVENGE_LIBS="-L/home/vmiklos/git/libreoffice/librevenge/src/lib/.libs/ -lrevenge-0.0" REVENGE_STREAM_CFLAGS="-I/home/vmiklos/git/libreoffice/librevenge/inc" REVENGE_STREAM_LIBS="-L/home/vmiklos/git/libreoffice/librevenge/src/lib/.libs/ -lrevenge-stream-0.0" ODFGEN_CFLAGS="-I/home/vmiklos/git/libreoffice/libodfgen/inc" ODFGEN_LIBS="-L/home/vmiklos/git/libreoffice/libodfgen/src/.libs -lodfgen-0.1 -lrevenge-0.0 -lrevenge-stream-0.0" VISIO_CFLAGS="-I/home/vmiklos/git/libreoffice/libvisio/inc" VISIO_LIBS="-L/home/vmiklos/git/libreoffice/libvisio/src/lib/.libs -lvisio-0.1 -lrevenge-0.0" --enable-debug --with-libvisio
Of course, replace
/home/vmiklos/git/libreoffice/ with any other directory
you like, just be consistent. ;-)
Now you can hack on any of these libraries, you just need to build your changes, and then vsd2odg will produce a flat ODG that you can quickly test with any ODF processor, like LibreOffice. One remaining trick (in case you’re not an autotools expert) is that vsd2odg is a libtool shell script, not a binary. If you still want to run the underlying binary in gdb, here is how you can do that:
libtool --mode=execute gdb --args vsd2odg /home/vmiklos/git/libreoffice/test.vsdx
In case the above considered two alternatives are not sufficient for your purposes, then I hope you find this setup useful. ;-)
A vízben jó. Ez nagyon hamar tudatosult bennem, és a mai napig a víz a kedvenc közegem. Megnyugtat. Magabiztossá tesz. Otthon vagyok benne. Ha beugrom és elmerülök, megszűnik minden más. Csak a víz van, és én meg a csend. Simogató csend. De a külvilág akkor sem jut el hozzám, ha följövök. A vízben más tudatállapotba kerülök.
It turns out LibreOffice’s RTF and DOCX import filter ignored borders around Writer pictures. Given that this worked in the RTF case in the past, it’s a bit amusing that now the very same commit implements a new feature for the DOCX case and at the same time fixes a regression in the RTF filter. Code sharing FTW! :-)
Végre egy film ami nem romantikus vígjáték, nem háborús dráma és még nem is bántam meg a rászánt időt. Persze nem is véletlenül kapott 10-ből 8.2 pontot. ;-)
UPC traditionally had a setup consisting of a cable modem providing internet access to a single computer, and then it was up to the users if they use that access to really connect to a computer or to a router, providing wireless access and so on. It seems, these days they are more after actually encouraging people to use their subscription on multiple devices — possibly that way it’s easier to sell larger packages (like 60 MBit/s download rate instead of 30 MBit/s, etc). One fallout from this move is that they started to replace modems with a combination of modems and routers, in this case this is an Ubee EVW3226, with the brand removed. I wanted to try out if this new device could replace my previous router or not — so far it seems to be good enough, though there was one pitfall, hence this post.
It’s possible to define a range of IP addresses to be used for DHCP purposes, though you can’t serve fixed IP addresses based on the MAC address of the clients. Given that my home network isn’t that large, I can tolerate that: as long as there is a range that can be safely used for fixed addresses, I can configure that manually. It’s also possible to do port forwarding, e.g. redirecting the incoming ssh traffic to a given address — except you can’t do both at the same time: you can’t redirect traffic to an address that’s not known (served via DHCP) to the router. Which is a shame, the #1 use case for port forwarding is to redirect traffic to a home-server that will then also have a fixed IP internally…
So here is a hack that allowed me to still do this: set the start of the range of the DHCP served IP’s exactly to the address of the (to be used in future as) fixed address, e.g. 192.168.0.5. Connect with one client, so that the address will be known to the router. Then add the port-forwarding rule, finally set the DHCP range back to its original value (in my case I use 192.168.0.1..99 for fixed addresses and 100+ for dynamic purposes). It’s a stupid trick, but it works… ;-)
This year’s LibreOffice conference was held in Bern, Switzerland. Links to my slides:
During the sessions I also had some time to hack on the followings:
Regarding the number of attendees, draw your own conclusions from the group picture — probably around 300 attendees, counting all days.
Thanks for the organizers for this beautiful event — and also the sponsors! :-)
In June, we decided to get rid of XSLT usage in writerfilter, the module responsible for RTF and DOCX import in LibreOffice. As usual with cleaning up mess, this took time (about two months), but I’m now happy to say that I’m mostly done with this. :-)
See the doctok blog post for some background, the topic here was to clean up the OOXML tokenizer, that is that building block that turns a zipped XML document into a token stream.
The following problems are now solved:
Part of the module was generated code, the generator was implemented mostly in XSLT, but some bits were written in Perl and sed. About 4200 lines of XSLT code is now rewritten in Python, in about 1300 lines.
Given that we have much more developers who speak Python, compared to XSLT,
nontrivial changes are now much easier in the generator: Jan Holesovsky
boost::unordered_map usage at places where we depended on the order of
elements. (Yes, you read it correctly, that was the situation up till now!)
This also helps reducing the size of the resulting writerfilter shared library.
The input of the code generator was the large
model.xml file, and
generator scripts only extracted interesting information from it, so if you
mistyped something, you got no error messages, just silent failures. I’ve
removed quite some XML elements and attributes from it which were parsed by
none of the generator scripts and written a
schema for the remaining markup. Validating against this schema is part of
the default build, so no more typos without a build failure. ;-)
(The schema also contains quite some documentation, finally.)
A gperf hash of all possible OOXML elements / attribute names were duplicated in writerfilter, even if that information was already available from the oox module. This is now fixed, reducing the size of the shared library even further.
Also, both oox and writerfilter had a list of namespace URL’s, mapping them to an integer enumeration, and when the two lists didn’t match, Bad Things happened (read: usually resulted in a crash.) This is the past, I’ve refactored writerfilter to use the same namespace alias names as oox, and this allowed to get rid of the writerfilter copy of the namespace alias list. So in the future, if new namespaces have to added, only oox has to be extended.
Oh and the bonus feature: I’ve implemented a script called watch-generated-code.sh, which can record a good state of the generated code, and then compare later generated results against that, so that refactoring of the generator can now be performed in a safe way: you can change the generator in any way to make it better, and still avoid accidental output changes. :-) This is particularly useful, as it only diffs the end result of the whole generation process (cxx and hxx files), not temporarily files, which are OK to change, as long as the end result is the same.
As a conclusion, here are sizes of a stripped dbgutil version of the writerfilter shared library, from the libreoffice-4-3-branch-point and today’s master:
$ git checkout oldest HEAD is now at b3130c8... 2014-05-21 vmiklos@o9010:~/git/libreoffice/daily$ ls -lh opt/program/libwriterfilterlo.so -rwxr-xr-x 1 vmiklos users 8,3M aug 28 14:00 opt/program/libwriterfilterlo.so $ git checkout master Switched to branch 'master' vmiklos@o9010:~/git/libreoffice/daily$ ls -lh opt/program/libwriterfilterlo.so -rwxr-xr-x 1 vmiklos users 6,1M aug 28 14:01 opt/program/libwriterfilterlo.so
Again, the 8,3MB → 6,1MB size reduction is mostly thanks to Kendy’s map cleanups + the duplicated gperf hash going away. :-)
From time to time developers feel that they have little time, but they would want to take care of many bugs. Last week I was frustrated enough to actually design a T-shirt for just that. ;-)
Above is how it looks like. In case you don’t get the joke, see here for a hint. Oh, and if you would like to build your own binary… err T-shirt, you can do it: here is the ODG file that can serve as a source. Happy bugfixing! :-)
Wednesday, 16 July 2014
TextBox: complex LibreOffice Writer content inside shapes (Comments)
TL;DR: see above — it’s now possible to have complex Writer content (charts, tracked changes, tables, fields, etc.) inside drawinglayer shapes, yay! :-)
Writer in LibreOffice 4.3 can have two kind of shapes: drawinglayer ones or Writer TextFrames. (Let’s ignore OLE objects and Writer pictures for now.) Drawinglayer shapes can be triangles (non-rectangular), rectangles can have rounded corners and so on, but shape text is handled by editeng — the same engine that is used for Impress shapes or Calc cells. OTOH a Writer TextFrame can contain anything that is supported by Writer (Writer fields, styles, tables, etc.), but its drawing capabilities are quite limited: no triangle, rounded corners, etc. Together with CloudOn, we thought the best would be to be able to have both, and started to use the "shape with TextBox" term for this feature.
A user can already sort of to do this by creating a drawinglayer shape, then a Writer TextFrame, and by setting the properties of the Writer TextFrame (position, size, etc) to appear as if the TextFrame would be the shape text of the drawinglayer shape. The idea is to tie these two objects together, so the (UI and API) user sees them as a single object.
I’m providing here a few screenshots. Above, you can see an ODF document having a rectangle with rounded corners, still containing a table.
Given that OOXML has this feature since its birth, I’m also showing a few DOCX documents, which are now handled far better:
chart inside a left arrow callout:
tracked changes inside a cloud callout:
SmartArt inside a snip diagonal corner rectangle:
Table of Contents inside a pentagon:
What follows is something you can probably skip if you’re a user — however if you’re a developer and you want to understand how the above is implemented, then read on. ;-)
From the drawinglayer point of view:
SwDoc contains an
SwDoc::GetOrCreateDrawModel()), which contains a single
SdrModel::GetPage()) — Draw/Impress contain multiple sdr pages. The
SdrPage contains the shapes: e.g. a triangle is an
TextFrames, a placeholder object called
SwVirtFlyDrawObj is added to the
The writer-specific properties of an
SdrObject is stored as an
SwFrmFmt array is a member of
SwDoc ("frame format table"). The
anchor position and the node index of the frame contents counts as a property.
At UNO level, a single
DrawPage object is part of the Component (opened
document), which abstracts away the internal
For TextFrames, the UNO API works exactly the same way, except that the
implementation stores all properties of the TextFrame in the
some properties are different, compared to a drawinglayer shape).
One remaining detail is how the shape text is represented. In case of
drawinglayer shapes, this is provided by editeng: internally an
provides a container for paragraphs, at UNO API level
provides an interface that presents paragraphs and their text portions.
For TextFrames, the contents of the frames is stored in a special section in
the Writer text node array (in the 3rd toplevel section, while the 5th
toplevel section is used for body text), that’s how it can contain anything
that’s a valid Writer body text. An offset into this node array of the
"content" property of the
At a document model level, we need a way to describe that an
(provided by svx) has an associated TextFrame (provided by sw). svx can’t
depend on sw, but in the
SwFrmFmt of the SdrObject, we can use the so far
RES_CNTNT ("content") property to point to a TextFrame content.
So behind the scenes the UNO API and the UI does the following when turning on the TextBox bit for a drawinglayer shape:
creates a TextFrame
SdrObject to the TextFrame
Also, every property of the TextFrame depends on the properties of the
SdrObject, think of the followings:
position / size is the largest rectangle that fits inside the shape
borders are disabled
background is transparent
Finding the largest rectangle that fits inside the shape is probably the most
interesting here, it’s implemented in
The UNO API hides the detail that the TextFrame and the
SdrObject are in
fact two objects. To get there, the followings are done:
SwXShape is modified, so that in the TextBox case not editengine, but the
attached TextFrame is accessed when
getText() is invoked.
This was a bit tricky, as
SwXShape doesn’t have an explicit
implementation: it overrides
queryInterface() instead (see
XIndexAccess) is modified to
ignore TextFrames in the TextBox case
SwXTextPortionEnumeration is modified to ignore TextFrames in the TextBox case
modified to handle the TextBox case
This was the easiest part: the "merge TextFrame and
SdrObj into a shape with
TextBox" approach ensured that that we use existing layout features here, no
major effort was necessary here.
One interesting detail here was the positioning of as-character anchored
shapes having TextBoxes, that’s now handled in
The primary point of this feature is to improve Word (and in particular DOCX) compatibility, and of course I wanted to update ODF as necessary as well.
Regarding the new feature, I did the followings:
DOCX import now avoids setting service name from original to
css.text.TextFrame in case shape has shape text
DOCX export now handles the TextBox case: reads Writer text instead of editeng text as necessary
ODF export now adds a new optional boolean attribute to make export of the TextBox case possible
ODF import now handles the new attribute and act accordingly
Note that regarding backwards compatibility, we keep supporting editengine-based text as well. This has the best of two worlds:
existing ODF documents are unchanged, but
the TextBox feature is enabled unconditionally in DOCX import to avoid formatting loss
I took care of the followings:
the context menu of shapes now provides an item to add / remove a TextBox to/from a shape
when moving or resizing a shape, the TextBox properties are updated as well
when the shape is deleted, the associated TextBox is also deleted
editing individual TextBox properties is no longer possible, since they depend on the shape properties
Last week I reviewed those slides and realized that some of them are outdated. So here comes an updated version:
The intention is that these build nicely on top of Michael’s generic intro slides, and with that, the reader can have a good "big picture" understanding of the code base. For the gory details, you always need to read the code anyway. ;-)