## Sphinx Documentation

### About Sphinx

[Sphinx](http://www.sphinx-doc.org) is a popular tool for documenting Python projects,
including the ability to generate automatic documentation using docstrings in your
source code.

### reStructuredText (RST) vs. Markdown (MD)

Because
[there are never enough markup languages](https://www.google.com/search?q=restructuredtext+vs+markdown)
out there,
[reStructuredText](https://en.wikipedia.org/wiki/ReStructuredText)
was created for documenting Python, but Sphinx can also support the easier and more popular,
[Markdown](https://en.wikipedia.org/wiki/Markdown) format with a couple of plugins.

I've chosen to mix and match RST and MD throughout the documentation, using RST
for Python docstrings, and Markdown for stuff I type.

### Using Sphinx

To Use sphinx, we'll install a bunch of packages:

```text
# bring in requirements for my app (excepting the optional database):
-r../requirements-django.txt
# stuff needed for sphinx documentation:
Sphinx==1.8.2
sphinx-markdown-tables==0.0.9
sphinx-rtd-theme==0.4.2
sphinxcontrib-apidoc==0.3.0
sphinxcontrib-confluencebuilder==0.9
sphinxcontrib-django==0.4
sphinxcontrib-websupport==1.1.0
recommonmark==0.4.0
```

Then run the quickstart:
```bash
sphinx-quickstart
```

This creates a `conf.py` which is the core configuration file for Sphinx. And, since it's Python code,
you can do all kinds of cool stuff.  Here are a few of my changes after the quickstart, which notably
includes some django-specific stuff, autmatic API documentation and support for Markdown and Markdown Tables:

```diff
diff --git b/docs/conf.py a/docs/conf.py
index 55c2351..dc4c7a4 100644
--- b/docs/conf.py
+++ a/docs/conf.py
@@ -12,22 +12,37 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import os
+import sys
+import datetime
+import django
+from recommonmark.parser import CommonMarkParser
 
+django_version = ".".join(map(str, django.VERSION[0:2]))
+python_version = ".".join(map(str, sys.version_info[0:2]))
+
+sys.path.insert(0, os.path.abspath('..'))
+
+os.environ['DJANGO_SETTINGS_MODULE'] = 'training.settings'
+django.setup()
 
 # -- Project information -----------------------------------------------------
 
+# See https://pypi.org/project/sphinxcontrib-django/
 project = 'Django {json:api} training'
-copyright = '2018, Alan Crosswell'
+year = datetime.date.today().year
+copyright = '{}, The Trustees of Columbia University in the City of New York'.format(year)
 author = 'Alan Crosswell'
 
 # The short X.Y version
-version = ''
+from myapp import VERSION
+version = VERSION
 # The full version, including alpha/beta/rc tags
-release = ''
+release = VERSION
 
+# Auto-generate API documentation.
+#os.environ['SPHINX_APIDOC_OPTIONS'] = "members,undoc-members,show-inheritance"
+os.environ['SPHINX_APIDOC_OPTIONS'] = "members,show-inheritance"
 
 # -- General configuration ---------------------------------------------------
 
@@ -39,23 +54,30 @@ release = ''
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'sphinx.ext.autodoc',
-    'sphinx.ext.intersphinx',
-    'sphinx.ext.todo',
-    'sphinx.ext.viewcode',
+    'sphinxcontrib.apidoc',   # runs sphinx-apidoc automatically as part of sphinx-build
+    'sphinx.ext.autodoc',     # the autodoc extensions uses files generated by apidoc
+    'sphinxcontrib_django',   # does some nicer django autodoc formatting, but:
+                              # https://github.com/edoburu/sphinxcontrib-django/issues/12
+    'sphinx.ext.viewcode',    # enable viewing autodoc'd code
+    'sphinx.ext.intersphinx', # make links between different sphinx-documented packages
+    'sphinx.ext.todo',        # TODO: figure out how to use this;-)
+    'sphinx_markdown_tables', # CommonMark doesn't do tables: This extensions does!
+    'sphinxcontrib.confluencebuilder', # supposedly installs docs on Confluence
 ]
-
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
+source_parsers = {
+    '.md': CommonMarkParser,
+}
+
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
-source_suffix = '.rst'
+source_suffix = ['.rst', '.md']
 
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -67,7 +89,7 @@ language = None
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = []
+exclude_patterns = ['build']
 
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = None
@@ -78,13 +100,23 @@ pygments_style = None
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'alabaster'
+# html_theme = 'alabaster'
+# html_theme = 'default'
+html_theme = 'sphinx_rtd_theme'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
 #
-# html_theme_options = {}
+html_theme_options = {
+    # these are for sphinx_rtd_theme:
+    'prev_next_buttons_location': 'both',
+    'collapse_navigation': True,
+    # these are for alabaster:
+    # 'show_relbars': True,
+    # 'fixed_sidebar': True,
+    # 'sidebar_collapse': True,
+}
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
@@ -99,8 +131,16 @@ html_static_path = ['_static']
 # default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
 # 'searchbox.html']``.
 #
-# html_sidebars = {}
-
+# also for alabaster theme:
+# html_sidebars = {
+#     '**': [
+#         'about.html',
+#         'navigation.html',
+#         'relations.html',
+#         'searchbox.html',
+#         'donate.html',
+#     ]
+# }
 
 # -- Extension configuration -------------------------------------------------
 
+autodoc_member_order = 'bysource'
+autodoc_inherit_docstrings = False
+
+apidoc_module_dir = '../myapp'
+apidoc_output_dir = 'apidoc'
+apidoc_excluded_paths = ['../myapp/migrations']
+apidoc_separate_modules = True
+apidoc_toc_file = False
+apidoc_module_first = True
+apidoc_extra_args = ['-f']
+
+confluence_publish = True
+confluence_server_url = os.environ.get('CONFLUENCE_SERVER', "https://confluence.columbia.edu")
+confluence_space_name = os.environ.get('CONFLUENCE_SPACE', None)
+confluence_parent_page = os.environ.get('CONFLUENCE_PARENT', None)
+confluence_server_user = os.environ.get('CONFLUENCE_USER', None)
+confluence_server_pass = os.environ.get('CONFLUENCE_PASS', None)
+
 # -- Options for intersphinx extension ---------------------------------------
 
-intersphinx_mapping = {'https://docs.python.org/': None}
+intersphinx_mapping = {
+    'python': ('https://docs.python.org/{}'.format(python_version), None),
+    'django': ('https://docs.djangoproject.com/en/{}/'.format(django_version), 
+               'https://docs.djangoproject.com/en/{}/_objects/'.format(django_version)),
+    # not sure why but the default lookup of objects.inv fails with None
+    'djangorestframework-jsonapi': ('https://django-rest-framework-json-api.readthedocs.io/en/stable/',
+                                    'https://django-rest-framework-json-api.readthedocs.io/en/stable/objects.inv'),
+    # DRF doesn't use sphinx but rather mkdocs:-(
+    #'djangorestframework': ('https://django-rest-framework.readthedocs.io/en/stable/', None),
+}
```

### Viewing Sphinx-generated content locally

You can use Sphinx to generate many output formats. A sample local invocation is:

```console
(env) django-training$ cd docs
(env) docs$ make html
Running Sphinx v1.8.2
loading pickled environment... done
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.admin.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.models.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.serializers.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.views.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.tests.test_models.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.tests.test_views.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.tests.rst.
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 25 added, 0 changed, 0 removed
reading sources... [100%] welcome                                                                                                                                   
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] welcome                                                                                                                                    
generating indices... genindex py-modindex
highlighting module code... [100%] myapp.views                                                                                                                      
writing additional pages... search
copying images... [100%] ./media/image2.png                                                                                                                         
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded.

The HTML pages are in build/html.
(env) docs$ 
```

### Publishing to Confluence

_Publishing to Confluence is not recommended due to the limitations
[described below](#confluencebuilder-shortcomings)_

We use [Confluence](https://confluence.columbia.edu) for an internal documentation repository and
would like to host our sphinx-generated documentation there.

#### Configuring Confluencebuilder

You have to get a non-CAS guest user and password in order to bypass SAML login.

I use a shell script, `confluence.sh` to set these environment variables:
```bash
export CONFLUENCE_SERVER=https://confluence.columbia.edu/confluence
export CONFLUENCE_USER=mysphinx
export CONFLUENCE_PASS=PASSWORD
export CONFLUENCE_SPACE="~mysphinx"
export CONFLUENCE_PARENT="API"
$*
```

#### Confluencebuilder shortcomings

The `sphinxcontrib-confluencebuilder` attempts to generate Confluence content but suffers from
several shortcomings:

1. Several common code languages are not recognized, yielding these errors:
   ```text
   WARNING: unknown code language: console
   WARNING: unknown code language: ini
   WARNING: unknown code language: json
   WARNING: unknown code language: text
   WARNING: unknown code language: tsql
   WARNING: unknown code language: yaml
   ```
   Some of these can be easily worked-around (e.g. substitute `sql` for `tsql`) but lack of `text` is pretty basic stuff.

2. Certain instances of curly braces are not properly quoted, leading to `500` macro unknown errors like this:
   ```text
   An unsupported Confluence API call has been made.

   REQ: POST
   RSP: 500
   URL: https://confluence.columbia.edu/confluence/rest/api
   API: contentbody/convert/storage
   MSG: com.atlassian.confluence.content.render.xhtml.migration.exceptions.UnknownMacroMigrationException: The macro 'json' is unknown.
   ---
   ```
   Curly braces appear to be OK for normal body text but break down in:
   - browser link titles such as `[See {json:api}](https://jsonapi.org)`
   - autodoc-generated code blocks:
	 ```diff
	 diff --git a/myapp/serializers.py b/myapp/serializers.py
	 index beb962c..e8e15ac 100644
	 --- a/myapp/serializers.py
	 +++ b/myapp/serializers.py
	 @@ -70,8 +70,8 @@ class CourseSerializer(HyperlinkedModelSerializer):
			  related_link_view_name='course-related',
		  )

	 -    #: `{json:api} compound document <https://jsonapi.org/format/#document-compound-documents>`_
	 -    #: (also used for `related_serializers` for DJA 2.6.0)
	 +    # `JSON:API compound document <https://jsonapi.org/format/#document-compound-documents>`_
	 +    # (also used for `related_serializers` for DJA 2.6.0)
		  included_serializers = {
			  'course_terms': 'myapp.serializers.CourseTermSerializer',
		  }
	 @@ -111,8 +111,8 @@ class CourseTermSerializer(HyperlinkedModelSerializer):
			  related_link_view_name='course_term-related',
		  )
	 ```
     These can be worked around by excluding undocumented members
     and removing docstrings or `#:` comments (which sphinx treats like docstrings).
	 This was supposedly [fixed](https://github.com/tonybaloney/sphinxcontrib-confluencebuilder/pull/43) but is apparently
     not (or this is a new way to trigger the issue).
3. There's no way to put the [ToC in the sidebar](https://jira.atlassian.com/browse/CONFSERVER-43090) so navigation sucks.
4. No search.
4. Poor formatting of autodocs.

My conclusion: Just find a way to locally host the HTML tree generated by sphinx-build rather than trying to
force this into Confluence. For example, this works:

### Publishing to a static web site

```bash
(env) django-training$ rsync -av -e ssh docs/build/html/ alan@cunix:public_html/django-jsonapi-training
```
You can see the pages at
[http://www.columbia.edu/~alan/django-jsonapi-training/](http://www.columbia.edu/~alan/django-jsonapi-training/)

Or use this if you want to secure the content:
```bash
(env) django-training$ rsync -av -e ssh docs/build/html/ alan@cunix:secure_html/django-jsonapi-training
```
After adding an appropriate `.htaccess` you can see these, if you are a CUIT staff member, at
[https://www1.columbia.edu/~alan/django-jsonapi-training/](https://www1.columbia.edu/~alan/django-jsonapi-training/)

### Publishing to RTD

[https://readthedocs.io](https://readthedocs.io) (RTD)
is where most open-source projects host their documentation.

Once we've got sphinx working locally, and the project hosted on github, getting it working with RTD is pretty
straightforward. See the
[sphinx getting started guide](https://docs.readthedocs.io/en/latest/intro/getting-started-with-sphinx.html).

On the [RTD dashboard](https://readthedocs.org/dashboard/) import a new project
and make sure to:
1. Pick a name. I've chosen
   [columbia-it-django-jsonapi-training](https://columbia-it-django-jsonapi-training.readthedocs.io)
2. Provide the github repository URL:
   [https://github.com/columbia-it/django-jsonapi-training](https://github.com/columbia-it/django-jsonapi-training)
3. In advanced settings configure the PIP requirements file: `docs/requirements.txt` and
   make sure to select `CPython 3.x` as the Python interpreter.

#### Fine print: pyodbc breakage

I did have to split up the project `requirements.txt` into multiple pieces since I import Django and myapp into
`conf.py` to enable `autoapi` and `autodoc`. Since I had the SQL Server packages (django-pyodbc-azure and pyodbc)
in `requirements.txt`, pyodbc failed to install on RTD since it wants to compile some C source code
using headers that are installed with an ODBC OS package. In fact, this stuff is all optional as the
default database used in the project is sqlite3, so I restructured the requirements into `requirements.txt`:

```text
# requirements for our app:
-rrequirements-django.txt
# optional sqlserver requirements:
-rrequirements-sqlserver.txt
```

with the main stuff in `requirements-django.txt` and the additional SQL Server stuff in `requirements-sqlserver.txt`.

Finally, in `docs/requirements.txt` we bring in the necessary django and sphinx pieces:
```
# bring in requirements for my app (excepting the optional database):
-r../requirements-django.txt
# stuff needed for sphinx documentation:
Sphinx==1.8.2
sphinx-markdown-tables==0.0.9
sphinx-rtd-theme==0.4.2
sphinxcontrib-apidoc==0.3.0
sphinxcontrib-confluencebuilder==0.9
sphinxcontrib-django==0.4
sphinxcontrib-websupport==1.1.0
recommonmark==0.4.0
```

In anticipation of adding travis support on github,
I also changed `tox.ini` to have a separate section for local sphinx builds: `tox -e sphinx`.