Tag: scrivener

The Writing Stack: Zotero -> Scrivener -> ODT -> Docx -> Markdown -> HTML

Scrivener to Markdown and HTML

How to write in Scrivener and display in HTML, Markdown, ODT, or Doc and keep the footnotes and images.

This is the process I use for getting my chapters out of Scrivener and formatted into Markdown and HTML for putting on the web. Markdown for Github, and HTML for a static website, and Doc for turning in to advisors and the Library.

Write it

Use Scrivener to bring all of the notes and sources together in one place.

Note it

The process of writing actually begins while reading through books and looking at original source documents. For each source (whether book, document, image, or web page) I create an entry in Zotero. With an entry in Zotero, I create a child-note for that entry and take notes in that child-note. I always include the page number in the notes for easy referencing later. A typical note for an entry in Zotero looks like this:

Kaj Björn Karbo (July 4, 1920)

{ | Karbo, 1947 | | |zu:312:A6J3JADD} 

page 1,
1400 men were supposed to wash in half and hour at 20 faucets.
Longest roll call was 4 hours because a couple of men had escaped.

page 2,
Relationship to Kapos was bad, also to Russians, and somewhat so to other nationalities.
Kapos were German, Russian, Polish and Czech

page 3,
Punishments consisted of beatings with boards from a bed and truncheon. 
Stretched over a bench and held by four men and then beat

page 4,
Was part of many different work commands. In January 1945 was Schieber, 
the lowest rung of prison hierarchy. He was in charge of a 16 man work 
gang. they helped German civilian workers build a factory for synthetic fuel.

The part in curly braces { | Karbo, 1947 | | |zu:312:A6J3JADD} comes in handy later when adding citations in Scrivener.

Compose it

With all of the notes taken (for now, it can be a never ending process), copy and paste the relevant notes in the correct section of the Scrivener outline. Basically, each idea gets its own ‘page’. This boils down to each paragraph, more or less, on its own ‘page’.

Export it

First step is to export the chapter from Scrivener.

  • Export it as the OpenOffice (.odt) format. Give it a name like chapter2.odt.

Scan it

To get the footnotes into the correct format (MLA, Chicago, etc), we’ll scan the .odt file with Zotero. This creates a new file.

  • Open Zotero, click the gear, and select ‘RTF/ODF Scan’.
  • Select the file you created above (chapter2.odt).
  • Create a new name and place to save it (chapter2-citations.odt)

Cite it

The Zotero scan converts all of the coded citations from Scrivener into ‘normal’ citations.

from this: { | Blanke-Bohne, 1984 | p. 16 | |zu:312:KMQEIBU0N}

to this: Blanke-Bohne, 1984.

To get it into a different citation style, we’ll open up the file in LibreOffice and change the citation style using the Zotero ‘Set Document Preferences’ menu.

from this: Blanke-Bohne, 1984.

to this: Blanke-Bohne, Reinhold. "Die unterirdische Verlagerung von Rüstungsbetrieben und die Aßuenlager des KZ Neuengamme in Porta Westfalica bei Minden." Dissertation, University of Bremen, 1984.

After the changes finish (could take a while), then save the document as a Word, make sure to do a ‘Save As’ .docx file (chapter2-citations.docx).

Fix it

Only the .docx format is supported by pandoc for extracting images, so we’ll need to use Word as the final format before converting to Markdown and HTML. Frankly, it also has much better grammar and spell checking.

Open the .docx in Microsoft Word and fix up any formatting issues.

I also turn this version in to my advisors for review.

Convert it

In the terminal, we’ll use the pandoc command to convert the file to Markdown and HTML.

This will convert the .docx file to a markdown file, extracting the images and putting them in a ‘files/media/’ directory.

The images are named incrementally in the order they are encountered in the document. The images are given a default name, keeping the extension. If I had four images in the file (two jpegs, one png, and one gif), they would be extracted and named like so: image1.jpeg, image2.jpeg, image3.png, image4.gif, etc.

We’ll have to go in and fix the tables and check for other formatting issues.

pandoc --smart --extract-media=files -f docx -t markdown_github chapter1-citations.docx -o chapter1.md

Next we can create an HTML file using pandoc and the .docx file.

pandoc --smart --extract-media=files --ascii --html-q-tags --section-divs -f docx -t html5 chapter1-citations.docx -o chapter1.html

This creates an HTML file with the images linked to the files in the files/media/ directory and the footnotes converted to hyperlinks.

Version it

Now these files can more easily be tracked with a versioning system, like git, and the HTML files can be uploaded for a static website version of the
dissertation. Styling can easily be applied if used in a Jekyll site.

For sharing on Github, there are two repos, main and gh-pages.

main repo

The main repo is simply the chapter directories with each of the document versions and the extracted media files. Once edits and conversions are done, this is updated with a simple

git add .
git commit -m "Updates chapter X"
git push

gh-pages repo

The gh-pages repo contains the files needed to convert the html version of the doucments into a Jekyll based static website. The trick here is to get all of the updates from the main repo into this gh-pages repo. This is accomplished with doing the following command while checked out in the gh-pages branch.

```
git checkout master -- chapterX
```

Before I can push the new changes to Github, I’ll need to fix a few things in the html version of the chapter.

First is to add some YAML front matter. I add this to the beginning of the HTML version.

---
layout: page
title: Chapter X
---

Second, update the path for the images so that they will work. I open the file in Vim and do a simple search and replace:

:%s/img src="files/img src="..\/files/g

Now I can update the gh-pages branch and the site.

git add .
git commit -m "Add updates from chapterX"
git push

The Mechanics of Writing a Dissertation

Update from this post: Setting the Pace from January 3, 2013.

I’m a nerdy, geeky type of guy, fascinated with technology, so I just had to figure out a way to keep myself on track using a spreadsheet. I chose Google Docs through my Google account as it is safely backed up online and I can do cool things like provide a link for anyone to see it and thereby keep track of my progress.

https://docs.google.com/spreadsheet/ccc?key=0Aq1zQ58RVggudG5lc3FUZm54VU9DMXc5ZTJ6N3NwLUE&usp=sharing

Weekly Schedule
Weekly Schedule

The Plan

With ideas and thoughts cribbed from Eviatar Zerubavel’s The Clockwork Muse: A Practical Guide to Writing Theses, Dissertations, and Books, I came up with a deadline (December 31, 2014, so that I can edit and finalize things before graduation in May 2015), and looked at a calendar to plot the days I would be able to work. I first put down all of the events and times where I know I will NOT be able to work on the dissertation. Saturdays and Sundays are out because of family and religious reasons (my family gets me for 2 whole days of the week, where it’s my turn to take care of the kids). 40 hours a week for work. Sleeping time (although that’s the first to go). And any other times I knew I wouldn’t work on the dissertation; holidays, vacations, birthdays, anniversaries, etc.

With some basic numbers, I was ready to put together a tracking spreadsheet: 250 pages for a dissertation (on the short end for history dissertations, but I’ll have a lot of digital aspects to supplement), with roughly 40 pages a chapter; 175 days to work until December 31, 2014; 12 hours a week; 2.5 hours a day with 4.5 hours one day of the week. That works out to about 1.5 pages a day (that I plan on working) or .5 pages an hour. That seems easy enough. I can write half a page in an hour… when I have the material in hand (and mind) already. This is by far the hardest part; not waking up at 4am (that’s easy).

I have a sheet to keep track of the days I work on the dissertation, how many pages I write that day, and what I worked on.

Daily Tracking Sheet
Daily Tracking Sheet

This automatically updates my “dashboard” that tracks things on a monthly level, with a cell with giant numbers of how many pages left I have to write. The closer I get to 0, the greener the cell automatically becomes. (I just can’t help it.)

 

Dashboard
Dashboard

The downside to this, is that it doesn’t really track the progress of the actual chapters; how much of chapter one is left, how much of the introduction is written, etc. I don’t really have a set number of pages to reach in order to call a chapter “done”. It’s “done” when I get all of the information across and points made and theories explored and backed up with examples. That could take 40 pages or 100 pages. But a rough guesstimate and quick goal is for 40 pages per chapter.

The Tools

This is where Scrivener comes into play, or would “more better” if I spent more time figuring and configuring the awesome tracking abilities built into Scrivener. One of the great progress tracking features is to set a “words per section” goal. A quick calculation and another rough guesstimate puts us at 350 words per page, or 87,500 words. You can set a word goal for each section, and then see the results as you type and see all sections in a group at a glance. I have each chapter broken down into smaller sections, and each of those sections gets a word goal.

Now I just need to use all of these tools and get back to work!

 

The Tools to Do the Job – Scrivener, Zotero, LibreOffice

(This post is cross-posted at my personal blog)

Scrivener is awesome software for writing, that I’ve mentioned before, but I had yet to really test out the integration with Zotero (my citation manager of choice). So now that I have finally started on my dissertation writing in earnest (and not grant writing), I needed to make sure that footnotes are usable in my work flow. So this is a quick write up of the tools I will use in writing my dissertation, and how I will use them.

The Tools

LibreOffice: Free and Open Source document software. Who knows how long I will have access to free Microsoft Word? LibreOffice (the fork of OpenOffice) will always be free and freely available. The steps will be basically the same if you are using Microsoft Word, just substitute that program for LibreOffice when it comes to it.


Zotero: I’m certainly biased, but Zotero is the greatest citation management software evar! Also free and open source. I’m using the stand alone version, but you can use the Firefox extension as well. Should work the same.

 

 

 

 

 

Scrivener: The greatest writing software I’ve seen. So good I even paid for it. I don’t usually do that with software (as you can see, I like free and open source).

 

 

 

 

The Process

Here I will try to outline the process I found that will save footnotes from existing documents into Scrivener, and Scrivener created footnotes into exported documents. From there, it’s easy to create Zotero connected footnotes.

1. Copy existing documents with footnotes into Scrivener

Copy from LibreOffice
Copy from LibreOffice

The first issue to run across is to put your existing documents into scrivener. I wrote a paper for Hist 811 that is basically the bulk of Chapter 1 and Chapter 2 of the dissertation. It’s needs some finessing in order to fit in the dissertation. It would be a shame to lose the footnotes, which is what happens if you just use Scrivener’s import file process. This is an easy fix. Just copy the text from your document and paste it into a Scrivener text area.

Then with your Scrivener project open, create a new text area, or select an existing one, which ever, and paste it in. Nothing special there.

 

 

 

 

 

 

 

2. Create new footnotes in Scrivener

Scrivener makes a Footnote
See how Scrivener makes a footnote!

What is special, though is what Scrivener does with that footnote. See there, footnote number 20, right after the quote about the cocktail of causes and rearmament being one of the ingredients? Now in Scrivener we have the word “ingredient” highlighted and underlined, and on the right side of the Scrivener window, there is a new footnote with all of the content of the original footnote. Sweet!

 

 

 

 

 

 

Easy as Format->Footnote, or use the shorcut keys Ctrl-Cmd-8

That’s all well and good. What if we want to edit the text a little bit, add some good stuff and add another footnote in there? What do we do? Well, Scrivener has a way to add a footnote. Just highlight some text (the footnote will be inserted after the last word), and go to the Format menu and select Footnote. Or you can use the fancy shortcut keys, for faster typing and footnote inserting, Ctrl-Cmd-8 (⌃⌘8).

 

 

 

 

 

 

 

Look, Ma! A new footnote!

Now you have a new, blank, footnote area to put a footnote reverence in.

 

 

 

 

 

 

 

Select the reference in Zotero and drag it into the footnote box in Scrivener.

Zotero makes it easy to put the reference in that new empty footnote with drag and drop citations. Just pull up your Zotero (either from Firefox, or if you have the standalone version). Select the reference you want, and drag it into the empty footnote section.

 

 

 

 

 

 

 

3. Moving from Scrivener to a document, and keeping your footnotes!

So, ideally, you would be able to export your text document, and all of these lovely footnotes you have made in Scrivener, using Zotero, would just magically work in a Word or LibreOffice document. It doesn’t, yet (or ever?). So here is how to get your footnotes into a document, and then get those footnotes to be Zotero enabled.

Srivener->File->Export->Files

First, you export your Scrivener document to RTF format.

 

 

 

 

 

 

Select RTF format

Select the plain RTF format, and the first check box for only the selected files (although, you could un check this if you want to do all of your files at once. No other check boxes are needed. Then just hit the Export button.

 

 

 

 

 

Open it up with your favorite document program, LibreOffice or Word.

Next, you will want to open your new RTF document in LibreOffice (or Word if you’re using that program).

 

 

 

All my citations are in the house!

You will notice that all of your footnotes are in this file. Yeah! Sometimes the text had odd font sizes and styles. So a quick ‘Select All’ and change it to default style and Times New Roman, 12 pt should fix that right up. Now here is the labor intensive part. For each footnote, we’re going to have to recreate it so that it is handled by Zotero. Then we’ll delete the original footnote. It would be nice of Scrivener could export the footnotes in a way that Zotero could detect them, but alas it is not to be.

 

Now you add a citation through the zotero buttons to make a zotero-aware citation.

 

 

 

 

 

 

 

 

All my citations are in the house!

Insert a Zotero citation using the Zotero buttons in your document program’s menu bar.

 

 

 

 

 

 

 

 

 

 

I prefer the Zotero classic view.

 

 

 

 

 

 

 

 

 

 

The new citation find view is pretty slick, though.

 

 

 

 

 

 

 

You can add pages with a coma, space, number.

 

 

 

 

Now you have two citations.

 

 

 

 

 

 

With two citations in the document, you’ll need to delete the one that was not made by zotero.

 

 

 

 

 

Just make sure you delete the non-zotero aware citation. The Zotero citation is usually highlighted.

 

 

 

 

 

 

 

 

 

 

 

 

 

Now you can save the document as a different file format: odt, doc, docx

Now save the document as an ODT document. If it is saved as anything else, it will not be Zotero aware.

 

 

 

 

 

 

Take your pick of file types.

 

 

 

 

 

 

 

 

 

 

 

 

Save as the correct file format if you want Zotero to be able to edit them again.

 

 

 

 

 

 

 

One alternative method is to create footnotes in Scrivener using the format {Author, Year, Page#}. Then export as an RTF document as before. Then, in Zotero, use the ‘RTF Scan’ tool in the Preferences menu. Zotero will see all of the citations and replace them nicely with formatted citations (using Ibid. and short notation for repeat books, and such). Zotero will not be aware of these citations at all, so if you need them to be Zotero aware, you might as well use the steps outlined above. If you do not expect to update citations or the text once done in Scrivener, then this may be the easiest way to go.

Now I can happily transfer existing documents into Scrivener and save the footnotes!

Digging in to the dissertation

Pun intended, of course.

I found a really cool piece of software that will, I believe, be very helpful in writing the dissertation. It’s a Mac application called Scrivener. I found it while reading up on an influential digital historian’s blog, William Turkel. I like it because it organizes the writing process in the way I already think about it. I can write, or rearrange bits of text as if they were note cards, and so much more… I’ll let a few screen shots speak for themselves:

 

As you can see, I’ve been working on my outlines for the first two chapters. I was worried about integration with Zotero, but found this tip to be helpful. It’s a bit of a process, but sure beats doing all citations by hand.

Funding Update

Also, for an update, I have now applied to two big fellowships, USHMM and the GHI, with one more to go at the National Archives. I should hear back about the USHMM this month.

After that, it’s the big two, the Fulbright and the DAAD.

Sources Update

I have most of the documents scanned from USHMM. There are still a bunch of microfilms I should get digitized from the National Archives (or the originals from the German Archives). Now I just need to start going through them and translating and organizing. I’ll have a post on that later.

Detail of A4 at Hadmersleben

Above is a teaser of one of the documents. This detail shows the location of the proposed tunnels in relation to the town of Hadmersleben, in Germany. The different areas of the tunnel are labeled.