Sphinx Documentation

About Sphinx

Sphinx is a popular tool for documenting Python projects, including the ability to generate automatic documentation using docstrings in your source code.

reStructuredText (RST) vs. Markdown (MD)

Because there are never enough markup languages out there, reStructuredText was created for documenting Python, but Sphinx can also support the easier and more popular, Markdown format with a couple of plugins.

I’ve chosen to mix and match RST and MD throughout the documentation, using RST for Python docstrings, and Markdown for stuff I type.

Using Sphinx

To Use sphinx, we’ll install a bunch of packages:

# bring in requirements for my app (excepting the optional database):
-r../requirements-django.txt
# stuff needed for sphinx documentation:
Sphinx==1.8.2
sphinx-markdown-tables==0.0.9
sphinx-rtd-theme==0.4.2
sphinxcontrib-apidoc==0.3.0
sphinxcontrib-confluencebuilder==0.9
sphinxcontrib-django==0.4
sphinxcontrib-websupport==1.1.0
recommonmark==0.4.0

Then run the quickstart:

sphinx-quickstart

This creates a conf.py which is the core configuration file for Sphinx. And, since it’s Python code, you can do all kinds of cool stuff. Here are a few of my changes after the quickstart, which notably includes some django-specific stuff, autmatic API documentation and support for Markdown and Markdown Tables:

diff --git b/docs/conf.py a/docs/conf.py
index 55c2351..dc4c7a4 100644
--- b/docs/conf.py
+++ a/docs/conf.py
@@ -12,22 +12,37 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import os
+import sys
+import datetime
+import django
+from recommonmark.parser import CommonMarkParser
 
+django_version = ".".join(map(str, django.VERSION[0:2]))
+python_version = ".".join(map(str, sys.version_info[0:2]))
+
+sys.path.insert(0, os.path.abspath('..'))
+
+os.environ['DJANGO_SETTINGS_MODULE'] = 'training.settings'
+django.setup()
 
 # -- Project information -----------------------------------------------------
 
+# See https://pypi.org/project/sphinxcontrib-django/
 project = 'Django {json:api} training'
-copyright = '2018, Alan Crosswell'
+year = datetime.date.today().year
+copyright = '{}, The Trustees of Columbia University in the City of New York'.format(year)
 author = 'Alan Crosswell'
 
 # The short X.Y version
-version = ''
+from myapp import VERSION
+version = VERSION
 # The full version, including alpha/beta/rc tags
-release = ''
+release = VERSION
 
+# Auto-generate API documentation.
+#os.environ['SPHINX_APIDOC_OPTIONS'] = "members,undoc-members,show-inheritance"
+os.environ['SPHINX_APIDOC_OPTIONS'] = "members,show-inheritance"
 
 # -- General configuration ---------------------------------------------------
 
@@ -39,23 +54,30 @@ release = ''
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'sphinx.ext.autodoc',
-    'sphinx.ext.intersphinx',
-    'sphinx.ext.todo',
-    'sphinx.ext.viewcode',
+    'sphinxcontrib.apidoc',   # runs sphinx-apidoc automatically as part of sphinx-build
+    'sphinx.ext.autodoc',     # the autodoc extensions uses files generated by apidoc
+    'sphinxcontrib_django',   # does some nicer django autodoc formatting, but:
+                              # https://github.com/edoburu/sphinxcontrib-django/issues/12
+    'sphinx.ext.viewcode',    # enable viewing autodoc'd code
+    'sphinx.ext.intersphinx', # make links between different sphinx-documented packages
+    'sphinx.ext.todo',        # TODO: figure out how to use this;-)
+    'sphinx_markdown_tables', # CommonMark doesn't do tables: This extensions does!
+    'sphinxcontrib.confluencebuilder', # supposedly installs docs on Confluence
 ]
-
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
+source_parsers = {
+    '.md': CommonMarkParser,
+}
+
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
-source_suffix = '.rst'
+source_suffix = ['.rst', '.md']
 
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -67,7 +89,7 @@ language = None
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = []
+exclude_patterns = ['build']
 
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = None
@@ -78,13 +100,23 @@ pygments_style = None
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'alabaster'
+# html_theme = 'alabaster'
+# html_theme = 'default'
+html_theme = 'sphinx_rtd_theme'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
 #
-# html_theme_options = {}
+html_theme_options = {
+    # these are for sphinx_rtd_theme:
+    'prev_next_buttons_location': 'both',
+    'collapse_navigation': True,
+    # these are for alabaster:
+    # 'show_relbars': True,
+    # 'fixed_sidebar': True,
+    # 'sidebar_collapse': True,
+}
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
@@ -99,8 +131,16 @@ html_static_path = ['_static']
 # default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
 # 'searchbox.html']``.
 #
-# html_sidebars = {}
-
+# also for alabaster theme:
+# html_sidebars = {
+#     '**': [
+#         'about.html',
+#         'navigation.html',
+#         'relations.html',
+#         'searchbox.html',
+#         'donate.html',
+#     ]
+# }
 
 # -- Extension configuration -------------------------------------------------
 
+autodoc_member_order = 'bysource'
+autodoc_inherit_docstrings = False
+
+apidoc_module_dir = '../myapp'
+apidoc_output_dir = 'apidoc'
+apidoc_excluded_paths = ['../myapp/migrations']
+apidoc_separate_modules = True
+apidoc_toc_file = False
+apidoc_module_first = True
+apidoc_extra_args = ['-f']
+
+confluence_publish = True
+confluence_server_url = os.environ.get('CONFLUENCE_SERVER', "https://confluence.columbia.edu")
+confluence_space_name = os.environ.get('CONFLUENCE_SPACE', None)
+confluence_parent_page = os.environ.get('CONFLUENCE_PARENT', None)
+confluence_server_user = os.environ.get('CONFLUENCE_USER', None)
+confluence_server_pass = os.environ.get('CONFLUENCE_PASS', None)
+
 # -- Options for intersphinx extension ---------------------------------------
 
-intersphinx_mapping = {'https://docs.python.org/': None}
+intersphinx_mapping = {
+    'python': ('https://docs.python.org/{}'.format(python_version), None),
+    'django': ('https://docs.djangoproject.com/en/{}/'.format(django_version), 
+               'https://docs.djangoproject.com/en/{}/_objects/'.format(django_version)),
+    # not sure why but the default lookup of objects.inv fails with None
+    'djangorestframework-jsonapi': ('https://django-rest-framework-json-api.readthedocs.io/en/stable/',
+                                    'https://django-rest-framework-json-api.readthedocs.io/en/stable/objects.inv'),
+    # DRF doesn't use sphinx but rather mkdocs:-(
+    #'djangorestframework': ('https://django-rest-framework.readthedocs.io/en/stable/', None),
+}

Viewing Sphinx-generated content locally

You can use Sphinx to generate many output formats. A sample local invocation is:

(env) django-training$ cd docs
(env) docs$ make html
Running Sphinx v1.8.2
loading pickled environment... done
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.admin.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.models.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.serializers.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.views.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.tests.test_models.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.tests.test_views.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.tests.rst.
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 25 added, 0 changed, 0 removed
reading sources... [100%] welcome                                                                                                                                   
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] welcome                                                                                                                                    
generating indices... genindex py-modindex
highlighting module code... [100%] myapp.views                                                                                                                      
writing additional pages... search
copying images... [100%] ./media/image2.png                                                                                                                         
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded.

The HTML pages are in build/html.
(env) docs$ 

Publishing to Confluence

Publishing to Confluence is not recommended due to the limitations described below

We use Confluence for an internal documentation repository and would like to host our sphinx-generated documentation there.

Configuring Confluencebuilder

You have to get a non-CAS guest user and password in order to bypass SAML login.

I use a shell script, confluence.sh to set these environment variables:

export CONFLUENCE_SERVER=https://confluence.columbia.edu/confluence
export CONFLUENCE_USER=mysphinx
export CONFLUENCE_PASS=PASSWORD
export CONFLUENCE_SPACE="~mysphinx"
export CONFLUENCE_PARENT="API"
$*

Confluencebuilder shortcomings

The sphinxcontrib-confluencebuilder attempts to generate Confluence content but suffers from several shortcomings:

1. Several common code languages are not recognized, yielding these errors:

   WARNING: unknown code language: console
   WARNING: unknown code language: ini
   WARNING: unknown code language: json
   WARNING: unknown code language: text
   WARNING: unknown code language: tsql
   WARNING: unknown code language: yaml
   

   Some of these can be easily worked-around (e.g. substitute sql for tsql) but lack of text is pretty basic stuff.

2. Certain instances of curly braces are not properly quoted, leading to 500 macro unknown errors like this:

   An unsupported Confluence API call has been made.
   
   REQ: POST
   RSP: 500
   URL: https://confluence.columbia.edu/confluence/rest/api
   API: contentbody/convert/storage
   MSG: com.atlassian.confluence.content.render.xhtml.migration.exceptions.UnknownMacroMigrationException: The macro 'json' is unknown.
   ---
   

   Curly braces appear to be OK for normal body text but break down in:

   • browser link titles such as [See {json:api}](https://jsonapi.org)

   • autodoc-generated code blocks:

     diff --git a/myapp/serializers.py b/myapp/serializers.py
     index beb962c..e8e15ac 100644
     --- a/myapp/serializers.py
     +++ b/myapp/serializers.py
     @@ -70,8 +70,8 @@ class CourseSerializer(HyperlinkedModelSerializer):
              related_link_view_name='course-related',
          )
     
     -    #: `{json:api} compound document <https://jsonapi.org/format/#document-compound-documents>`_
     -    #: (also used for `related_serializers` for DJA 2.6.0)
     +    # `JSON:API compound document <https://jsonapi.org/format/#document-compound-documents>`_
     +    # (also used for `related_serializers` for DJA 2.6.0)
          included_serializers = {
              'course_terms': 'myapp.serializers.CourseTermSerializer',
          }
     @@ -111,8 +111,8 @@ class CourseTermSerializer(HyperlinkedModelSerializer):
              related_link_view_name='course_term-related',
          )
     

     These can be worked around by excluding undocumented members and removing docstrings or #: comments (which sphinx treats like docstrings). This was supposedly fixed but is apparently not (or this is a new way to trigger the issue).

3. There’s no way to put the ToC in the sidebar so navigation sucks.

4. No search.

5. Poor formatting of autodocs.

My conclusion: Just find a way to locally host the HTML tree generated by sphinx-build rather than trying to force this into Confluence. For example, this works:

Publishing to a static web site

(env) django-training$ rsync -av -e ssh docs/build/html/ alan@cunix:public_html/django-jsonapi-training

You can see the pages at http://www.columbia.edu/~alan/django-jsonapi-training/

Or use this if you want to secure the content:

(env) django-training$ rsync -av -e ssh docs/build/html/ alan@cunix:secure_html/django-jsonapi-training

After adding an appropriate .htaccess you can see these, if you are a CUIT staff member, at https://www1.columbia.edu/~alan/django-jsonapi-training/

Publishing to RTD

https://readthedocs.io (RTD) is where most open-source projects host their documentation.

Once we’ve got sphinx working locally, and the project hosted on github, getting it working with RTD is pretty straightforward. See the sphinx getting started guide.

On the RTD dashboard import a new project and make sure to:

1. Pick a name. I’ve chosen columbia-it-django-jsonapi-training
2. Provide the github repository URL: https://github.com/columbia-it/django-jsonapi-training
3. In advanced settings configure the PIP requirements file: docs/requirements.txt and make sure to select CPython 3.x as the Python interpreter.

Fine print: pyodbc breakage

I did have to split up the project requirements.txt into multiple pieces since I import Django and myapp into conf.py to enable autoapi and autodoc. Since I had the SQL Server packages (django-pyodbc-azure and pyodbc) in requirements.txt, pyodbc failed to install on RTD since it wants to compile some C source code using headers that are installed with an ODBC OS package. In fact, this stuff is all optional as the default database used in the project is sqlite3, so I restructured the requirements into requirements.txt:

# requirements for our app:
-rrequirements-django.txt
# optional sqlserver requirements:
-rrequirements-sqlserver.txt

with the main stuff in requirements-django.txt and the additional SQL Server stuff in requirements-sqlserver.txt.

Finally, in docs/requirements.txt we bring in the necessary django and sphinx pieces:

# bring in requirements for my app (excepting the optional database):
-r../requirements-django.txt
# stuff needed for sphinx documentation:
Sphinx==1.8.2
sphinx-markdown-tables==0.0.9
sphinx-rtd-theme==0.4.2
sphinxcontrib-apidoc==0.3.0
sphinxcontrib-confluencebuilder==0.9
sphinxcontrib-django==0.4
sphinxcontrib-websupport==1.1.0
recommonmark==0.4.0

In anticipation of adding travis support on github, I also changed tox.ini to have a separate section for local sphinx builds: tox -e sphinx.