Progress Report

• 3/14/07 Just a quick update for those interested in this project...
  During the past few months I've made significant progress on this beast, however it is quite a large program, and will not be finished anytime soon. We're talking years, here, campers. I can't imagine I'll be happy with it before I've used it to write a couple novels, to make sure all the features work they way are supposed to.

  All signs point to my returning to work soon. The rough draft of Yau (the novel I am currently working on) is complete, and I am finishing the second draft. My return to work is dictated by the fact that the rough draft is now complete. I can do re-writes while working, but work uses too much creative energy to create new fiction at the same time. Once I return to work I will stop work on NovelRite until Yau is complete and the search for a publisher has started.

  So I expect to restart work on NovelRite perhaps mid summer, and then I should be able to have a version up and running by the end of the year, but not tested to the point I'm happy with it. That's what will take a couple more years.

  In other news, some of you will recall last fall I was having difficulties getting formatted text. I've overcome that using a JTextPane widget. Trouble is, due to bugs in Java (in minimalHTMLWriter), the formatted text does not always save correctly. Nothing is ever as easy as it ought to be. At least this is a problem I can work around, but it will require adding additional code, additional testing, etc.

  [ Additional work needing to be done includes creating the various db editors (the data bases are described below), which I expect to be relatively straight forward, then filling out the text editor (adding spell checking, undo, etc.) which will be less easy. ]

Introduction

NovelRite is a word processor designed especially for creating novels. It is based primarily on the ideas of Evan Marshall, as described in his book "The Marshall Plan for Writers". However, Marshall is missing elements important to literary writers (as opposed to writers of pop) and to science fiction and fantasy writers. These elements will be added.

I am posting this information primarily because of the reaction I've gotten to this program at work and at my writer's club (WriteNow! in Candor, NY). I don't think I've ever heard so much buzz about a program I've written. Since so many people are interested, perhaps they could offer feedback – Do I have all the features I need? Do the features work in the correct ways?

Many of the questions I'm getting orbit around what I plan to do with the program. The answer is simple – I plan to write novels with it. Considering how heavily this program uses material from Evan Marshall's book, I don't think it would be right to sell copies or open source is before contacting him. I have no intention of doing that before until Yau is complete (my current novel project), and I have used the program enough to know it is relatively useful and bug free. After that my next move will be to let friends who are working on novels use it to, as sort of a beta test. After that I am most inclined to open source it.

Creating a novel

What is a novel? A novel is a story. But what is a story? A story is a narrative work (often fiction) in which a main character attempts to accomplish a goal.

If the story is to carry any weight, the goal must be both worth achieving and difficult to achieve. We must understand the main character's motives for accomplishing the goal, and the main character must face almost insurmountable odds. These odds can be because others want to beat the main character to the goal, or prevent the main character from reaching the goal for some other reason. Or it can be that the nature of the goal makes it difficult to achieve. In the first circumstance we have antagonists, in the second we do not necessarily have them.

I've made these comments before to a friend interested in writing, and her reaction was that I am talking about action/adventure. But to be sure, this applies to all forms of writing. We can also come to believe that antagonists are always bad, but this stunts a person's perception of writing. I can imagine a novel about a person becoming addicted to meth, and the antagonists are trying to prevent the hero from getting more meth and getting buzzed on it.

To sum up, these "rules" might seem formulaic, and they can be if you read them incorrectly. But they do underscore most successful novels. As to those they do not underscore, when you become a good enough writer to "break the rules," then you will be good enough to not need this program.

For more on this, see The Marshal Plan.

To begin a novel you must create a main character. Part of creating the main character is describing his/her goal.

Part of creating the main character is creating background, motivations, etc. NovelRite will have a character database, and will have dialogs for creating characters. Also part of creating the main character is placing that character in a "world."

What is science-fiction? At first glance one might think sci-fi is writing about the future. But there are many sci-fi stories that take place at the current time. These might involve alien invaders (Heinlein's "The Puppet Masters", or an alternate take on the world, like the Germans and Japanese won the second world war (Philip K. Dick's "Man in the High Castle"). Some sci- fi novels actually take place in the past (Jack Dann's brilliant "The Memory Cathedral"). What is it about all these novels, all these forms, that ties them together and allows them all to be called sci-fi? They all take place in their own alternate reality, or their own "world".

There seem to be two parts to creating a "world" – creating places and creating creatures.

Creatures could be alien beings, and places could be worlds. Each world might have specific aliens with specific "properties." Or places could be dungeons, lairs, etc, and creatures could be wizards, elves, leprechauns, etc. Or places could be different cities in different parts of the world, and perhaps places the hero commonly goes in different cities. When creating a world for a novel that is not sci-fi or fantasy, one probably would not use the creature database. However, if one is writing a novel about hunting in Africa, it might be useful to keep track of the properties of dik- dik and kudu.

Sci-fi and fantasy have an interesting property in that multiple novels take place in the same imaginary world. A particular writer will develop a world and place multiple novels there, then they create another world and place other novels there, and then perhaps return to the first world.

So part of beginning a novel is creating the world the novel takes place in. It may be the real world, but not all the real world. Certain places need to be described in great detail. Depending on the subject of the novel, creatures might need to be described.

NovelRite will have a world database. When a new project is started, you will be able to select from existing worlds, or create a new world. Part of the world database will be a place database, where you can described different places that are important to the story. Another part will be a creature database, where you can describe aliens, the wee people, or cuttlefish, if they are important to your story. (I just had the thought that perhaps a technology database could be useful. For sci-fi writers, this could be used to describe phasers and transporters; for a fantasy writer it could be used to describe spells by wizards from one place, as opposed to those used by wizards from another.)

The main character will have problems caused by antagonists and will rely on confidants and will interact with other characters. Each of these characters will need to be developed, if not in the novel itself, then at least in the mind of the writer.

So the next part of creating a novel is describing the remaining characters. NovelRite will have character sheets for other characters, as well as the main character. Marshall gives an appropriate number of characters depending on the desired length of the novel. NovelRite will suggest an appropriate number of characters, and also suggest relationships with the main character (antagonist, lover, etc.) But anything NovelRite suggests can be overridden by the user.

After this, Marshall talks about "surprises." In order to pace the novel and keep it moving forward, there needs to be three surprises (more or less depending on the length) – one a quarter of the way through, one half way through, and the third just before the climax, setting up the climax. These surprises are set backs for the hero and make his/her life more difficult. This builds suspense and keeps the novel interesting.

At this time the writer needs to come up with the surprises. NovelRite will have dialogs to describe the surprises. NovelRite might also suggest the number of surprises appropriate depending on the length of the novel. The dialog will be essentially a text box for entering text. If anyone can think of anything more detailed to add, let me know.

Once this is done, Marshall advocates creating scene outlines. I want to jump over this and come back to it, because I feel there is something Marshall missed here:

Theme

What is a theme? It is what a novel is truly about. Usually the theme is subtle, but underlies all that goes on in the novel. Personally I have difficulties writing 300+ pages without something to tie them all together.

Do all novels have a theme? No. But all great novels do. By great novels here, I mean those considered great by the literary community. Dickens, Wells, Goethe, etc.

Is a theme a moral? A moral can be a theme, but this usually just does not work very well. Morals tend to be thin and simplistic. Better themes are often complex, deep and subtle. The deeper your theme and the better your novel is at illustrating it, the deeper your novel will be.

Literary novels always have themes. Pop novels most often do not.

Someone once asked me, "Does everything I write have to have a theme?" My answer: "No, but why write if you have nothing to say?"

NovelRite will have a theme dialog, which will consist of a text entry box. At this time I do not envision any more detail than that, but if anyone can think of anything appropriate, let me know.

Scene Outlines

Ah, back to these scene outline things. What is a scene? One set of cohesive actions. Part of a scene outline is determining how to get from the start of the novel to the first surprise, how to get from the first surprise to the second, how to get from the second to the third, and from there to the climax, and what has to come after the climax.

Someone once asked me where the climax comes. This really surprised me, as I thought the answer was really basic, but then I read "Interview with the Vampire," and realized even very high caliber professionals can get it wrong. The climax is the emotional climax of the story. Most people will feel that once the climax has ended there is little point in continuing to read. Therefore the climax should come as close to the end of the novel as possible. If possible it should be the final scene. But often this leaves threads hanging. So after the climax, wrap up the hanging threads as quickly as possible and be done with it.

So part of the scene outline is a roadmap of how do we get from here to there – who's scene is it - the hero, the antagonist, the lover, etc; and is it an action, or a reaction scene, etc. See Marshall for details. Marshall suggests certain roadmaps based on the length of the novel. NovelRite will make the same suggestions, but, as always, NovelRite's suggestions can be overridden by the user.

The second part of the scene outlines is outlines for each individual scene. These include information such as whether it is an action scene or a reaction scene, and which character the scene belongs too. In addition, it includes what the scene needs to accomplish, along with links to the previous scene and the next scene.

Creating the novel

Once we have all the scene outlines in place, it is time to start writing. In NovelRite, you can easily navigate to any scene, and from there bring up a word processor widget in which the scene is written. The scene outline will be visible in it's dialog, so the user can easily refer to it while writing. Also links will be available to the character database and the place database. If the character is a particular type of creature, there will be a link from the character to the creature database.

One of the important features of this tool is navigability. The scene database will always be easy to get to, as will the world database, and the character database. From there, all information they contain will be easily accessible. If navigability were not easy, then you might as well use Word!

Creating a Novel with NovelRite

When NovelRite is started, a dialog will open asking the user to create a new project or open an existing project. When a new project is started, the user will be queried for the desired number of words or pages. (Personally I think about novel length in pages, but many writers think in terms of words. It is easy to convert pages to words - multiply by 250. But it is easier for the computer to do it than for me, so the user will be allowed to enter it either way.) Marshall uses the length to decide many aspects of the novel, such as the number of characters and the flow of scenes. NovelRite will make the same recommendations, although, of course, it will be easy to override those recommendations.

Many editors feel the perfect length for a book is 350 pages. With this in mind, the program will suggest 350 pages, although that can be easily overridden. (How many words is that?)

Next a navigation dialog comes up, allowing the user to navigate to the Theme Writer, the World Creator, the Character database, the Surprise Creator or the Scene database.

The Theme Writer

This is a formatted text entry window. Formatted text is like what Word or the World Wide Web do - you can have headers, lists, bold text, italicized text, etc. (At this point I don't know how wild I'm going to get. I'm pretty sure tables will not be implemented, for example).

Displaying such text is easy in Java, and since it will be stored as xhtml, storing it will be easy too.

Technical Stuff

Java has some drawbacks, and now is as good a place to discuss them as anywhere. The first is that it has no "word processor" widget. It has a widget called JeditorPane to display xhtml (which is a strictly formatted text) or one called JtextPane, for less formally formatted text. But the programmer is responsible for creating "undo", spell checking, etc. It is not difficult in an object oriented language, once all this is created, to wrap it all up into a single class that can be used over and over. Creating this word processor class represents about 80% of the work on this project. However it is something I've wanted for some time because it will be useful for two other projects I've wanted to undertake for some time.

The other drawback that there seem to be rather tight space limitations to what Java's widgets can display. The StringBuffer class itself has limitations. Every time it needs more space it doubles in size. This is a pretty standard tactic, the idea being that it will not have to grow too often (growing a string is slow because you might have to move a lot of data from one place to another), but it will always be on the correct order of magnitude - never less than half full. However, at least according to the net, because of memory fragmentation, it is often the case that StringBuffers cannot grow to more than a few megabytes. This is enough for thousands of pages of text, so it is probably not a problem.

The JTextPane seems to have a much more severe limitation. According to the net (again), if you attempt to display more than about 100,000 bytes worth of data it does not update correctly. No one seems to know why. Since this is a swing component, the opinion seems to be the limitation holds for all systems. This is a bit more than 18 pages, so I will probably limit each scene to 18 pages. While this is a limitation, no scene should ever be that long. A scene should be about 3 pages, and at I am considering displaying a warning dialog at 6 pages to alert the user the scene is growing too long. This limit is a far bigger problem for the other projects I have in mind for the word processor widget.

I am now leaning more towards using the JEditorPane, although I am not sure it is easy to append text one character at a time, so it might not be appropriate for this application. But if it is, I have not looked into what size limitations it might have.

For you real technoheads, it might be interesting to consider a background task to spell check as you type, similar to what Word does. I have the intention to have such a beast, but it will be added after the initial program is up and running. I think once the forground spell checker is up and working, putting it into the background will not be difficult.

Another thing to think about is how to suggest spellings. I have an algorithm in one of my college texts for doing fuzzy matches, but I never used it in real life. Any words not in the dictionary will have to be matched against the character database and the place database.

Undo/redo ought to be fun, too. Design Patterns gives ideas for the basics. But one has to be careful with a word processor – undoing one keystroke at a time can be tedious, but undoing large blocks can be pointless. This is one of my biggest complaints about Notepad.

So what I will do, at least to start is undo one character at a time until a whitespace character is found. Once one space is found, all characters up to the space before that will be condensed into a single undo object. There will be three of these "word level" undo objects, then the text previous to that will be condense into sentence fragments, split up by punctuation marks. I might go further and condense previous paragraphs into single undo objects, but I'm not sure. Once the program is up and running and I am using it, I will probably want to tweak the undo feature.

A novel can be a pretty big thing, so there is some anticipation of memory problems. So instead of an infinite level undo, I will probably limit the size of the undo stack, and dump undo items past 300 or so.

The World Creator

The world creator will have options of attaching an existing world to the novel, or creating a new one. Once a world is attached, a dialog will open with two areas, one listing places, the other listing creatures. (I don't have this completely worked out, as I want to be able to handle hundreds of places or creatures. If a world gets reused for many novels, the databases can grow that large – consider the world of Star Trek.) One possibility is using a JEditorPane with hyperlinks.

The Place Database

Scenes will have links to the place database. When you create a scene, one item on the dialog will query where it takes place. After the place is selected, you will be able to click the link in the scene dialog, and a word processor widget will pop up with the place description in it (in a form that can be edited).

The Creature Database

It will be possible to link creatures to places. This will provide a view in the place database of what creatures belong there, and links between the creature database and the place database. There might also be a view of other creatures from the same place in the creature editor.

The Character database

When the character database is opened, a dialog will display all characters created so far. The program will also suggest the number of characters appropriate for the novel, and their suggested roles, based on the size of the novel.

When a character is selected, a dialog with open with a number of fields for describing the character. Most of the specific fields are those used by Marshall on his work sheets. However, there may be a few additions or extensions. (As a sci-fi writer, I insist a character have the ability to have green hair. Have you really never met anyone with green hair? I knew a girl in high school who was on the diving team, and the chlorine in the water turned her hair green.) Where I differ with Marshall, it is almost always where I think he has not gone far enough, never that he has used too many rules or used too much or incorrect information.

One item where I differ from Marshall is that the character's name comes first on the dialog. Marshall places it pretty far down on his paper worksheet. So how do you organize the worksheets and find a particular character efficiently?

An addition to Marshall will be a link to the Creature database, in case important characters are other than human. (You don't have to use it if you do not want to. But for those of us who make aliens into important characters, it is nice to have links here.)

The Surprise Creator

When the surprise creator is opened, a dialog will display the surprises. The number of surprises will be suggested, based on the length of the story. When a surprise is selected, the word processor widget will open, allowing the user to edit the surprise free form.