Yihui Xie

Learn formatR in Two Minutes

2012-05-08T00:00:00-07:00

Anthony made a video tutorial on how to use the formatR package, which I think is pretty cool:

I wish I could speak English as fast as him...

How to Make HTML5 Slides with knitr

2012-05-01T00:00:00-07:00

One week ago I made an early announcement about the markdown support in the knitr package and RStudio, and now the version 0.5 of knitr is on CRAN, so I'm back to show you how I made the HTML5 slides. For those who are not familiar with markdown, you may read the traditional documentation, but RStudio has a quicker reference (see below). The problem with markdown is that the original invention seems to be too simple, so quite a few variants were derived later (e.g. to support tables); that is another story, and you do not need to worry much about it.

Before you get started, make sure your knitr version is at least 0.5:

# install.packages(c('knitr', 'XML', 'RCurl'))
update.packages(ask = FALSE)
packageVersion('knitr') >= 0.5

Editor: RStudio

You need to install the RStudio preview version to use its new features on markdown support. With this version, you will see an interface like this when you create an R markdown file (File --> New --> R Markdown):

The button MD in the toolbar shows a quick reference of the markdown syntax, which I believe you can learn in 3 minutes. To start with, you can use my example on Github: knitr-slides.Rmd, or quickly cook up your own by Ctrl + Shift + > to insert code chunks.

You can write headers with # and bullet points with -. It is both quick to write and easy to remember (and readable too). When you are done, just hit the button Knit HTML, and you get a nice HTML page showing you R code and the output. You do not have to learn LaTeX in order to step into the realm of reproducible research. (Did you see the Binomial pmf there?!)

Converter: Pandoc

What happens behind the scene is that RStudio calls knitr to compile the Rmd document to a markdown file (you can see it under the same directory as the Rmd file), and convert this file to HTML. This is a very nice feature, and we can actually go further.

Pandoc claims itself to be a universal document converter, and it is indeed very powerful. For the above example, we can convert the markdown output (not Rmd source) to many other formats like HTML, LaTeX, Open Office or Microsoft Office documents. HTML5 slides are also supported. This is the single command that I used to convert knitr-slides.md to DZslides:

pandoc -s -S -i -t dzslides --mathjax knitr-slides.md -o knitr-slides.html

Then you get an HTML file knitr-slides.html which you can view in a modern web browser. Enjoy.

Final words

HTML5 slides is just one tiny thing that you can play with markdown; check out the pandoc documentation to see more possibilities. That being said, I feel most excited about the RStudio integration with knitr and markdown. LaTeX is beautiful but difficult to learn and laborious to write. MS Word is most widely used but you know...

I believe this combination makes reproducible research much more accessible to the general audience, and I hope to see it being used in statistical courses so that students no longer do tedious jobs of copy & paste, and professors no longer suffer from ugly Word reports.

Now I have done pretty much what I planned in the beginning. The next step will be our GSoC project, in which we will make the toolchain smoother, and work out better ways for R users to document packages and publish web pages (e.g. blogging like a hacker). If you want to follow our latest changes, you may

watch the development repository on Github https://github.com/yihui/knitr
or join the Google mailing list https://groups.google.com/group/knitr

And final ads: I will be presenting knitr at useR! 2012 with JJ from RStudio. I'm looking forward to meeting more knitters in Nashville :)

Fancy HTML5 Slides with knitr and pandoc

2012-04-22T00:00:00-07:00

Karthik Ram gave an Introduction to R a couple of weeks ago, and I strongly recommend you to take a look at his cool HTML5 slides. I started trying HTML5 slides last year, and now it is difficult for me to go back to beamer, which I have used for a few years for my presenations. It is horrible to see beamer slides everywhere at academic conferences (especially the classic blue themes).

You probably have heard of an interesting blog post by Ben Schmidt about ocean shipping animations in the 18th and 19th centuries. I also played with the dataset a little bit, and made some slides named Voyages of Sinbad the Sailor (use Left/Right or Up/Down to navigate). The source file was written in markdown, compiled by knitr, then converted to DZSlides by pandoc.

I'm using the development version of knitr, which you can install from Github. I plan to release the version 0.5 this weekend, and this version will particularly feature the markdown support. You can always read the NEWS file to know what is going on in the development.

Another piece of news which may be a little bit early to announce is the corresponding support in RStudio. I'm not going to say any details about it right now, but I'm pretty sure the so-called reproducible research and dynamic report generation can be easier than ever very soon! No LaTeX. No worries about HTML/CSS. A simple text file and a single click will give your a reasonably beautiful HTML page. Stay tuned.

Ideas on A Really Fast Statistics Journal

2012-03-15T00:00:00-07:00

I was writing comments on the blog post A proposal for a really fast statistics journal, and I realized the comment box was too small to write down my ideas. I like the proposal a lot, and I feel really bad about the current model of submitting and reviewing papers: it is just too slow! We should not blame it on editors and reviewers; everybody is busy. The question is how to reduce unnecessary efforts in this process. My ideas are close to one possible actual implementation of Jeff's proposal.

There are a few stupid rules in some journals which I hate a lot and I believe they are wasting the time of everybody, e.g.

file format of figures; I especially despite journals which require authors to use EPS (postscript) or WMF (Windows metafile)
double-spaced typesetting: oh man, stop killing trees! It makes no sense to typeset with a large line height, unless you mail the papers to reviewers and ask them to write reviews with a pen, then mail the review back to the author; you do not really do this, so why ask authors to use the ugly double-spaced typesetting?
leave figures/tables blank in the body of the paper (write Insert Figure 1 Here) and put them in other places

I have to remind myself that this is the 21st century. These are actually not a big deal in terms of statistics papers. The more important problem is journals are operated by a very small group of people relative to the large number of authors. The even more important problem is reproducibility, which Roger Peng often mentions in that blog.

An open-access journal

This journal is electronic like JSS, and you can see everything in the website. For authors, LaTeX is too laborious, so we abandon LaTeX. For website maintainers, HTML is laborious as well, and we abandon HTML.

We turn to markdown. Why? It is simple. I hate writing a paper with \documentclass{} as my first line -- I want to start with the title directly in plain text, then write my first paragraph in a simple text editor. Who cares about \title{}? Similarly, when I write a webpage, I do not like starting with <html> as my first six characters.

My own website is built on Jekyll, a blog engine based on markdown files to build static HTML pages. The first few days of building this website might be hard, but once the design is done, the rest of the job becomes extremely easy. Wordpress is good, but it is still too heavy compared to Jekyll. See my blog source code, and this blog site is hosted on GitHub, which automatically compiles my markdown files to HTML pages using Jekyll when I submit new posts or changes to the source repository.

With a good web designer, this journal website can be set up probably in a week. In the future, neither maintainers nor paper authors need to deal with HTML any more. Paper source code looks like:

# Title

## Abstract

Here is the abstract.

## First section

Write write and write.

    plot(1) # code to generate figures

...

I will explain how we maintain this journal and talk about plot(1) later.

Faster submission and review

I do not see much difference in submitting computer code and papers, so I'd like to use a version control model for papers as well. As I mentioned in my last blog post, GIT and GitHub can make this process fast and easy.

We put the journal website under version control, and all the source code is on GitHub. Authors fork the journal repository and put their papers in the repository. Paper submission is through pull requests, which means the author wants to merge his/her changes (paper) back to the main repository.

Now the chief editor sees the request, and he/she writes a comment under the request like: @jeff and @roger, please review the new paper. Then Jeff and Roger start looking at the paper. If they want to make comments, they comment right below the lines of the texts, because GitHub supports inline comments in your code. (So why the hell should we continue writing double-spaced papers?)

After they are done, the editor asks the author to read the reviews, revise the paper, commit back again and the reviews can easily see what was changed in the revision, because GIT gives you full history of the revision. You see that the reviewers do not need to re-read the whole paper -- they just look at the revision history. If the problems of the paper are too serious, reviews can write bug reports (GitHub issues) in the author's repository and ask the author to fix these "bugs".

This process can go back and forth, until the editor decides to accept the pull request. Once the new paper is pulled in, Jekyll will recompile the website automatically.

Besides the designated reviewers, all other people can see and comment on the pull request as well, so the paper can be essentially under public review like open source software! If the two reviewers are busy, there might be other good public reviewers helping the review.

You may doubt on the "quality" of public review, but just think about the quality Linux and open source software. If we only rely on Linus Torvalds or Richard Stallman to write code, you can imagine how the open source world looks like today.

The journal can find plenty of good and active reviewers automatically after a while -- they come as a result of natural selection; this is better than requesting someone to be a reviewer because it is the "internal force" that drives the public reviewers. In the current journal model, reviewers may feel it is a job to review papers, so motivation becomes a problem.

Reproducible papers

I'm quite interested in reproducible research, and I'm all for reproducible results in papers. Well, if you are a statistician, R and Sweave have probably come into your mind. Sweave is great, and the idea of literate programming is awesome. It is just a shame that great ideas were implemented with a narrow vision. I can understand Sweave was tied to LaTeX, but that is a narrow choice. I feel painful to introduce Sweave to other people because I have to teach them LaTeX first.

I do not need to repeat my motivations to write another R package knitr to replace Sweave. The relevant point is reproducible research should be easier for the general public, and LaTeX is no way to go. Again, markdown is better. You will never be trapped in all sorts of weird TeX errors. Open a notepad (can be literally Windows Notepad or other text editors), write the text you want, then the code; compile this text file, you get your paper/report. This should be the case if we want to popularize reproducible research. If you have spent many hours on writing a TeX document and carefully embedding R code in it, then compile it...

Boom! TeX errors!
Boom! Figures too wide/narrow!
Boom! Figures floating away!
Boom! Editors ask you to use EPS instead of PDF figures!
Boom! R errors!
...

The knitr package supports markdown naturally. You can see a hello-world example and a real example.

I'm not yet very confident about working with markdown and knitr because I have not started using this combination; in the past few months I have been focusing on the LaTeX support, but I'll start eating my own markdown dog food soon.

People keep on telling you LaTeX frees your mind, because you only focus on writing and do not worry about styles, and that is a beautiful lie. You worry about styles all day long, and the tricks about styles are usually gory. I would rather say CSS frees you.

OK, this journal only accepts simple markdown files. No more argument. As I showed you, you can embed R code in markdown files to compile your papers dynamically. In the future, the server of the journal may compile the papers periodically as well to check if they are still valid. It is easy for reviewers to validate the results too. All authors are required to provide datasets associated with the paper, or at least access to the datasets (e.g. URL's).

We use web pages to present papers. No PDF. A careful design of CSS can beat LaTeX/PDF and give us beautiful web pages. If readers have strong objections to this, we use pandoc to convert markdown papers to PDF. Math expressions? No problem! Use MathJax. There are no page limits, and all figures are inserted in appropriate places (they will not float and readers do not need to turn the pages desparately to find your Figure 7).

If we do everything in web pages, there are yet more possibilities.

Use SVG graphics or PNG (knitr supports more than 20 formats, and it is easy to switch). Let EPS and WMF die.
Feel R plots are ugly? Want animations? Interactive graphics? No problem! How about using D3? How about Processing.js?
Want to talk to authors? No problem! Leave a comment under the paper just like you comment on blog posts.
Want to rate a paper? Yes, hit the "Like" button. We do not only let reviewers of the journal rate papers.
Web statistics about a paper? Yes, we have page views and other statistics. We no longer use the number of times that a paper is referenced to evaluate a paper; we see how many times it has been read. People may add references for various reasons (not necessarily related to the paper).

Summary

We have been in the age of Web 2.0 for a few years, but journals rarely make good use of the "crowd". We build the world on a small group of people. Our good papers are used to fill in the shelves of libraries and collect dust, when bad articles are occupying the web (like my blog posts) and being discussed actively. We make all sorts of rules to tie the hands of authors, and we wait, wait and wait for the anonymous reviewers, like waiting for Godot.

How to Become an Efficient and Collaborative R Programmer

2011-12-12T00:00:00-08:00

I may want to add a subtitle "Why R-Forge Must Die" (thinking of Barry Rowlingson's talk earlier this year). I have been a GitHub user for two years, and I was mainly influenced by Hadley. Now I even feel a little bit addicted to GitHub (its slogan is "social coding"), because it is really convenient for collaboration and makes me more productive.

As some readers might have known, I started a new package knitr last month, and this time I decided to try to use the power of social network like Google+ and Twitter (something I used to stay away from), and so far I'm pretty satisfied with my attempts. I call GitHub the "Facebook" of programmers, and it is also very powerful to connect programmers and users. There are a few features that I think R programmers may want to try (I use knitr as an example):

Browse R code online: I hate checking out a whole package to read its source code, and R-Forge is clumsy (following the steps of sourceforge) in this aspect; the code on GitHub is highlighted and easy to thumb through. Besides, you can browse each commit and see what was changed (the difference is highlighted).
Issues: instead of writing your own TODO list which is often forgotten, both your users and you can create issues, and you can make discussions there; when you have got a fix, you can write a commit message like fixed #46 in GIT and the issue 46 will be automatically closed. This is super cool little feature to me. What is more, there will be a reference to the commit which fixed the issue, so you can come back in the future and see how the issue was fixed. Currently knitr has got 50 issues in total, with 24 of them from users.
Inline comments: you can discuss code directly along the lines; this is a super super cool feature. Ramnath started to contribute to the code theme feature recently after he forked my repository, and the original author (i.e. me) can go to the fork and check changes; each change can be commented, e.g. we had quite a few discussions in this commit. This feels like you can sit together with another programmer, and point to the code with a pen, saying "I like this and you may need to revise that, ...". In comparison, the traditional way of collaboration is usually through email -- email patches back and forth, which is way less straightforward. When Ramnath and I feel the work is mature, he can simply send me a pull request, and all the changes can be merged back to my repository. The other example is I saw the blog post by Songpants the other day, and I suggested he move the work to GitHub so I can make suggestions closer to the R code, and now the code is happily sitting on GitHub (so are my comments).
Wiki: it makes it so easy to quickly set up a documentation page; I have not done it for knitr yet, but I did it for the formatR package. It looks better than R's documentation, right? Again, other people can collaborate with you in editing the wiki pages. The other way to make your documentation better is to write vignettes in Sweave, which usually takes a lot of efforts (wrestling with LaTeX) unless you use LyX+knitr like me; I feel vignettes are easy to make, but this is another story.
Stay tuned with a package: you can watch a repository (use the button in the top-right) so that you can read the updates of a package in the dashboard; alternatively, you can follow a GitHub user like you follow people on Twitter.
GitHub pages: this is probably the coolest feature; you can use another branch (called gh-pages) in your GIT repository to build your website based on Jekyll. I made the website for knitr in this way, because I really want to make knitr a beautiful package, so everything has to be beautiful, then R documentation was ruled out. Of course, Hadley is a pioneer in documenting R packages with websites. In the future, I may want to develop a package based on knitr which turns R documentation into a website automatically (with examples parsed and evaluated, plots inserted), so you can host it on GitHub or somewhere else. This is only an idea at the moment, and feel free to contact me if you are interested.

I cannot say I'm already an efficient R programmer, but GitHub did make me much more efficient.

knitr: Elegant, flexible and fast dynamic report generation with R

2011-12-01T00:00:00-08:00

The world has changed. You can feel it on GitHub. You can smell it on Google+. The knitr package, as an alternative tool to Sweave, has features that you have been longing for, and features that you might have never imagined. Thumb through the PDF manual to see some of them.

Currently this package is still a beta version, so I'm looking for feedback from early birds on:

is the PDF documentation confusing in any places? e.g. you have no idea on how to install the package because it was not mentioned in the manual;
does the website look ugly in your browser? (I know it does with IE under Windows) I used a font from Google Font API, and it does not seem to be consistent across different web browsers/OS'es;
what kind of difficulties did you have in switching from Sweave/pgfSweave/whatever-Sweave to knitr?
do you like the idea of putting R code/output in a shaded frame in LaTeX? is the default shading (rgb(.97, .97, .97)) too dark or too light? how about the highlighting theme?
have you ever tried to hack at Sweave? I'd love to listen to your stories;
what else do you expect from knitr?

Feel free to file a bug report in the Issues page if you find any problems or have any suggestions. I appreciate your efforts in making this knitr package even neater!

The fun Package: Use R for Fun!

2011-08-16T00:00:00-07:00

A couple of days ago we released a package named fun to CRAN, but I did not dare to send an announcement to r-packages@r-project.org as usual. This package is a collection of some classical computer games (e.g. the Mine sweeper and Five in a row) as well as other funny stuff. Some examples:

## install.packages('fun')
library(fun)
if (.Platform$OS.type == "windows") x11() else x11(type = "Xlib")
mine_sweeper()

library(fun)
gomoku()

You can take a look at the list of functions in this package by reading the HTML help page (go to help.start()), and I also need to mention the demos, e.g. see demo('TurtleGraphics') for a demo of Turtle graphics (how many people know the old Logo programming language?), and demo(package = 'fun') for a list of all demos in this package.

demo('RealTurtle', package = 'fun')

Although these topics are not new, they can still be good programming exercises.

We started writing this package more than two years ago, but it was almost forgotten later until a few days ago someone mentioned the game "Five in a row" in our web forum. This forum is almost the Chinese version of R-help, and it is not unusual for people to bring forward all kinds of funny ideas with R. If you are at useR! 2011 right now, you probably have heard from George Zhang about the Chinese R conferences these years, and this forum has been the sponsor and organizer ever since the first conference (which I initiated). However, please do not get a wrong impression that Chinese useRs are doing mine sweepers with R every day.

Feel free to share with us if you have more fun. The developers' page is at: https://github.com/yihui/fun

P. S. This package may remind some people about the sudoku package (e.g. Joshua Wiley has noticed it), and some people may even remember this:

library(fortunes)
fortune('sudoku')

Sweave and pgfSweave in LyX 2.0.x (experimental)

2011-05-25T00:00:00-07:00

Please ignore this post completely, because Sweave support has become mature in LyX since 2.0.2, and I no longer plan to add the pgfSweave module in LyX. For pgfSweave users, you may consider the new knitr module (available since 2.0.3) which uses the R package knitr.

Produce Authentic Math Formulas in R Graphics

2011-04-30T00:00:00-07:00

I remember a few weeks ago, there was a challenge in the R-help list to make the prime symbol in R graphics. In LaTeX, we simply write $X'$ or $X^\prime$ . R has a rough support for math expressions (see demo(plotmath)) and they are certainly unsatisfactory for LaTeX users. In fact we can write native LaTeX code in R plots via the tikzDevice package! Why bother to use all kinds of tricks to cheat R? :)

Here is an example per request of a reader of my blog:

library(tikzDevice)
options(tikzMetricPackages = c("\\usepackage[utf8]{inputenc}",
    "\\usepackage[T1]{fontenc}", "\\usetikzlibrary{calc}",
    "\\usepackage{amssymb}"))
## I need the amssymb package because I use \mathcal and \mathbb
tikz("formula.tex", width = 4, height = 4, standAlone = TRUE,
    packages = c("\\usepackage{tikz}",
                 "\\usepackage[active,tightpage,psfixbb]{preview}",
                 "\\PreviewEnvironment{pgfpicture}",
                 "\\setlength\\PreviewBorder{0pt}",
                 "\\usepackage{amssymb}"))
par(mar = c(4, 4, 0.1, 0.1), mgp = c(2, 0.9, 0))
plot(1, type = "n", xlab = "$x_1$", ylab = "$x_2$")
text(1, c(0.8, 1, 1.2), c("$\\underbrace{1,2,\\cdots,10}_{10}$",
    "$\\mathbb{ABCDEFG}$", "$\\mathcal{HIJKLMN}$"), cex = 2.5)
dev.off()

tools::texi2pdf("formula.tex")
system(paste(getOption("pdfviewer"), "formula.pdf"))

New versions of GGobi and rggobi for Windows users

2011-04-09T00:00:00-07:00

For those who have been struggling with the installation of GGobi and the rggobi package under Windows: a major update of GGobi 2.1.9 is that GTK+ has been bundled with GGobi, so the installation of GTK+ is no longer required (I recommend you to uninstall it if it is not used elsewhere in your system); besides, the rggobi package, which interfaces R to GGobi, is now built with the GGobi 2.1.9 on CRAN too. You might know that the Windows binary of rggobi is not available on CRAN in the past (and Prof Ripley kindly provided the binary), but now things have changed. Hopefully this can make our life with GGobi easier.

You may use install.packages('rggobi') to install the new version of rggobi from CRAN.

Also note that if you are a user of the RGtk2 package, you don't need a standalone installation of GTK+ either if you have already installed GGobi 2.1.9, because the path of GGobi will be written in the PATH variable of your system and RGtk2 can load the required dll's from GGobi's directory.