Welcome to the “Confluencer” manual!

_images/logo.png

A CLI tool to automate common Confluence maintenance tasks and content publishing.

https://img.shields.io/pypi/v/confluencer.svg#:alt:LatestVersion Travis CI https://img.shields.io/coveralls/1and1/confluencer.svg#:alt:Coveralls https://img.shields.io/github/issues/1and1/confluencer.svg#:alt:GitHubIssues

Installing

Confluencer can be installed from PyPI via pip install confluencer as usual, see releases on GitHub for an overview of available versions – the project uses semantic versioning and follows PEP 440 conventions.

To get a bleeding-edge version from source, use these commands:

repo="1and1/confluencer"
pip install -r "https://raw.githubusercontent.com/$repo/master/requirements.txt"
pip install -UI -e "git+https://github.com/$repo.git#egg=${repo#*/}"

See the following section on how to create a full development environment.

To add bash completion, read the Click docs about it, or just follow these instructions:

cmdname=confluencer
mkdir -p ~/.bash_completion.d
( export _$(tr a-z- A-Z_ <<<"$cmdname")_COMPLETE=source ; \
  $cmdname >~/.bash_completion.d/$cmdname.sh )
grep /.bash_completion.d/$cmdname.sh ~/.bash_completion >/dev/null \
    || echo >>~/.bash_completion ". ~/.bash_completion.d/$cmdname.sh"
. "/etc/bash_completion"

Contributing

To create a working directory for this project, call these commands:

git clone "https://github.com/1and1/confluencer.git"
cd "confluencer"
. .env --yes --develop
invoke build --docs test check

Contributing to this project is easy, and reporting an issue or adding to the documentation also improves things for every user. You don’t need to be a developer to contribute. See Contribution Guidelines for more.

Documentation Contents

Using Confluencer

Providing Login Credentials

Before you can use Confluencer, you have to provide some minimal configuration, most importantly credentials for API access. Select one of the ways outlined below to store your API credentials.

Using the ~/.netrc File

  1. Create the file ~/.netrc with the following contents (or add that to the existing file):

    machine confluence.example.org
    login «your username»
    password «your password»

  2. Call chmod 600 ~/.netrc to protect your sensitive data.

  3. Check that everything is OK by calling python3 -m netrc | less.

This way, the sensitive authentication information is separate from the rest of the configuration. Use the cfr help command to check whether your credentials actually work – if they do, the “Confluence Stats” section in the output will show some basic info about your wiki, otherwise you’ll see an error indicator.

Using a Keyring Query Command

TODO

Tools that can be used on Linux are gnome-keyring-query and gkeyring.

Cleaning Up Pages After Content Migration

The Confluence rich text editor allows you to migrate content from rendered HTML in other systems by simple copy&paste. However, certain artifacts of the source system are carried over, or active content is only copied with its current (static) state.

The cfr tidy sub-command relieves you from manually fixing all those tiny defects, based on built-in patterns and replacement rules. These rules currently target FosWiki as a source, and for example a copied table of contents is replaced by the related Confluence macro.

Pass it the URL of the page you want to clean up – adding the --recursive option includes all descendants of that page. Normally, the output shows which and how often rules are applied to the content, the --diff option adds a detailed record of the applied changes.

If you want to just show the changes without applying them, use the --no-save option (or the shorter -n). This automatically includes diff output, to just show the applied rules repeat the option (-nn).

$ cfr tidy -nn "http://confluence.local/display/~jhe/Sandbox"
INFO:confluencer:Replaced 2 matche(s) of "FosWiki: Empty anchor in headers" (16 chars removed)
INFO:confluencer:Replaced 3 matche(s) of "FosWiki: 'tok' spans in front of headers" (94 chars removed)
INFO:confluencer:Replaced 3 matche(s) of "FosWiki: Section edit icons at the end of headers" (664 chars removed)
INFO:confluencer:Replaced 1 matche(s) of "FosWiki: Replace TOC div with macro" (127 chars removed)
INFO:confluencer:WOULD save page#2393332 "Sandbox" as v. 11

Exporting Metadata for a Page Tree

cfr stats tree generates a JSON list of a page tree given its root page (other output formats will follow). You can then select more specific information from that using jq or other JSON tools.

Consider this example creating a CSV file:

$ cfr stats tree "https://confluence.local/x/_EJN" \
  | jq '.[] | .depth, .title, .version.when, .version.by.displayName' \
  | paste -sd ';;;\n'
INFO:confluencer:Got 21 results.
0;"Root Page";"2016-10-24T17:20:04.000+02:00";"Jürgen Hermann"
1;"First Immediate Child";"2020-01-22T14:24:45.111+01:00";"Jürgen Hermann"
…

Complete CLI Reference

This is a full reference of the cfr command, with the same information as you get from using --help. It is generated from source code and thus always up to date. See Using Confluencer for a more detailed description.

Available Commands

cfr

‘confluencer’ command line tool.

cfr [OPTIONS] COMMAND [ARGS]...

Options

--version

Show the version and exit.

--license

Show the license and exit.

-q, --quiet

Be quiet (show only errors).

-v, --verbose

Create extra verbose output.

-c, --config <config_paths>

Load given configuration file(s).

help

Print some information on the system environment.

cfr help [OPTIONS]

Options

-c, --config-dump

Dump the merged configuration to stdout.

pretty

Pretty-print page content markup.

cfr pretty [OPTIONS] ‹page-url›…

Options

-R, --recursive

Handle all descendants.

-J, --json

Print raw API response (JSON).

-f, --format <markup>

Markup format.

Options:view|editor|storage|export|anon

Arguments

‹page-url›…

Optional argument(s)

rm

Remove contents.

cfr rm [OPTIONS] COMMAND [ARGS]...
tree

Remove page(s) including their descendants.

cfr rm tree [OPTIONS] ‹page-url›…

Arguments

‹page-url›…

Optional argument(s)

stats

Create status reports (or data exports).

cfr stats [OPTIONS] COMMAND [ARGS]...

Options

-f, --format <serializer>

Output format (defaults to extension of OUTFILE).

Options:dict|json|html|yaml|csv|tsv|ods|xls
-s, --space <space>
-e, --entity <entity>

Entity to handle / search for.

Options:macro
-o, --outfile <outfile>
tree

Export metadata of a page tree.

cfr stats tree [OPTIONS] ROOTPAGE

Arguments

ROOTPAGE

Required argument

usage

Create report on usage of different entities (macros, labels, …).

cfr stats usage [OPTIONS] QUERY

Options

--top <top>

Show top ‹N› ranked entities.

Arguments

QUERY

Required argument

tidy

Tidy pages after cut&paste migration from other wikis.

cfr tidy [OPTIONS] ‹page-url›…

Options

--diff

Show differences after tidying.

-n, --no-save, --dry-run

Only show differences after tidying, don’t apply them (use twice for no diff).

-R, --recursive

Handle all descendants.

Arguments

‹page-url›…

Optional argument(s)

Complete API Reference

The following is a complete API reference generated from source.

confluencer package

Confluencer – A CLI tool to automate common Confluence maintenance tasks and content publishing.

Copyright © 2015 1&1 Group <git@1and1.com>

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Subpackages
confluencer.api package

Confluence API support.

https://developer.atlassian.com/cloud/confluence/rest/

class confluencer.api.ConfluenceAPI(endpoint=None, session=None)[source]

Bases: object

Support for using the Confluence API.

Since the Confluence API has excellent support for discovery by e.g. the _links attribute in results, this just adds a thin convenience layer above plain requests HTTP calls.

CACHE_EXPIRATION = 36000
UA_NAME = 'Confluencer'
add_page(space_key, title, body, parent_id=None, labels=None)[source]

Create a new page.

The body must be in ‘storage’ representation.

delete_page(page, status=None)[source]

Delete an existing page.

To permanently purge trashed content, pass status='trashed'.

get(path, **params)[source]

GET an API path and return result.

If _cached=True is provided, the cached session is used.

getall(path, **params)[source]

Yield all results of a paginated GET.

If the limit keyword argument is set, it is used to stop the generator after the given number of result items.

Parameters:
  • path – Confluence API URI.
  • params – Request parameters.
update_page(page, body, minor_edit=True)[source]

Update an existing page.

The page MUST have been retrieved using expand='body.storage,version,ancestors'.

url(path)[source]

Build an API URL from partial paths.

Parameters:path (str) – Page URL / URI in various formats (tiny, title, id).
Returns:The fully qualified API URL for the page.
Return type:str
Raises:ValueError – A path was passed that isn’t understood, or malformed.
user(username=None, key=None)[source]

Return user details.

Passing neither user name nor key retrieves the current user.

walk(path, **params)[source]

Walk a page tree recursively, and yield the root and all its children.

confluencer.api.context(*args, **kwargs)[source]

Context manager providing an API object with standard error logging.

confluencer.api.diagnostics(cause)[source]

Display diagnostic info based on the given cause.

Extract the page ID from a so-called tiny link.

See this answer <https://answers.atlassian.com/questions/87971/what-is-the-algorithm-used-to-create-the-tiny-links> for details.

confluencer.api.tiny_id(page_id)[source]

Return tiny link ID for the given page ID.

confluencer.commands package

CLI commands.

Submodules
confluencer.commands.help module

‘help’ command.

confluencer.commands.pretty module

‘pretty’ command.

confluencer.commands.remove module

‘rm’ command.

confluencer.commands.stats module

‘stats’ command group.

confluencer.commands.stats.print_result(ctx, obj)[source]

Dump a result to the console or an output file.

confluencer.commands.tidy module

‘tidy’ command.

confluencer.tools package

Confluence supporting tools.

Classes and functions in this package help you to work with Confluence entities, and provide higher level abstractions to ease API usage.

Submodules
confluencer.tools.content module

Tools to discover and modify content.

class confluencer.tools.content.ConfluencePage(cf, url, markup='storage', expand=None)[source]

Bases: object

A page that holds enough state so it can be modified.

DIFF_COLS = {'+': 'green', '-': 'red', '@': 'yellow'}
dump_diff(changed)[source]

Dump a diff to terminal between changed and stored body.

etree()[source]

Parse the page’s body into an ElementTree.

json

The full JSON response data.

page_id

The numeric page ID.

space_key

The space this page belongs to.

tidy(log=None)[source]

Return a tidy copy of this page’s body.

title

The page’s title.

update(body=None, minor=True)[source]

Update a page’s content.

version

The page’s version number in history.

confluencer.util package

Helpers.

confluencer.util.progress(*args, **kwargs)[source]

Wrapper for progress bars providing consistent customizaion.

Submodules
confluencer.config module

Configuration utilities.

confluencer.config.cache_empty_dir(name)[source]

Return absolute path to a clean staging directory.

confluencer.config.cache_file(name)[source]

Return absolute path to any app-specific caching file.

confluencer.config.envvar(name, default=None)[source]

Return an environment variable specific for this application (using a prefix).

confluencer.config.version_info(ctx=None)[source]

Return version information just like –version does.

Contribution Guidelines

Overview

Contributing to this project is easy, and reporting an issue or adding to the documentation also improves things for every user. You don’t need to be a developer to contribute.

Reporting issues

Please use the GitHub issue tracker, and describe your problem so that it can be easily reproduced. Providing relevant version information on the project itself and your environment helps with that.

Improving documentation

The easiest way to provide examples or related documentation that helps other users is the GitHub wiki.

If you are comfortable with the Sphinx documentation tool, you can also prepare a pull request with changes to the core documentation. GitHub’s built-in text editor makes this especially easy, when you choose the “Create a new branch for this commit and start a pull request” option on saving. Small fixes for typos and the like are a matter of minutes when using that tool.

Code contributions

Here’s a quick guide to improve the code:

  1. Fork the repo, and clone the fork to your machine.
  2. Add your improvements, the technical details are further below.
  3. Run the tests and make sure they’re passing (invoke test).
  4. Check for violations of code conventions (invoke check).
  5. Make sure the documentation builds without errors (invoke build --docs).
  6. Push to your fork and submit a pull request.

Please be patient while waiting for a review. Life & work tend to interfere.

Details on contributing code

This project is written in Python, and the documentation is generated using Sphinx. setuptools and Invoke are used to build and manage the project. Tests are written and executed using pytest and tox.

Set up a working development environment

To set up a working directory from your own fork, follow these steps, but replace the repository https URLs with SSH ones that point to your fork.

For that to work on Debian type systems, you need the git, python, and python-virtualenv packages installed. Other distributions are similar.

Add your changes to a feature branch

For any cohesive set of changes, create a new branch based on the current upstream master, with a name reflecting the essence of your improvement.

git branch "name-for-my-fixes" origin/master
git checkout "name-for-my-fixes"
… make changes…
invoke ci # check output for broken tests, or PEP8 violations and the like
… commit changes…
git push origin "name-for-my-fixes"

Please don’t create large lumps of unrelated changes in a single pull request. Also take extra care to avoid spurious changes, like mass whitespace diffs. All Python sources use spaces to indent, not TABs.

Make sure your changes work

Some things that will increase the chance that your pull request is accepted:

  • Follow style conventions you see used in the source already (and read PEP8).
  • Include tests that fail without your code, and pass with it. Only minor refactoring and documentation changes require no new tests. If you are adding functionality or fixing a bug, please also add a test for it!
  • Update any documentation or examples impacted by your change.
  • Styling conventions and code quality are checked with invoke check, tests are run using invoke test, and the docs can be built locally using invoke build --docs.

Following these hints also expedites the whole procedure, since it avoids unnecessary feedback cycles.

Software License

Copyright © 2015 1&1 Group <git@1and1.com>

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Full License Text

                              Apache License
                        Version 2.0, January 2004
                     http://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

1. Definitions.

   "License" shall mean the terms and conditions for use, reproduction,
   and distribution as defined by Sections 1 through 9 of this document.

   "Licensor" shall mean the copyright owner or entity authorized by
   the copyright owner that is granting the License.

   "Legal Entity" shall mean the union of the acting entity and all
   other entities that control, are controlled by, or are under common
   control with that entity. For the purposes of this definition,
   "control" means (i) the power, direct or indirect, to cause the
   direction or management of such entity, whether by contract or
   otherwise, or (ii) ownership of fifty percent (50%) or more of the
   outstanding shares, or (iii) beneficial ownership of such entity.

   "You" (or "Your") shall mean an individual or Legal Entity
   exercising permissions granted by this License.

   "Source" form shall mean the preferred form for making modifications,
   including but not limited to software source code, documentation
   source, and configuration files.

   "Object" form shall mean any form resulting from mechanical
   transformation or translation of a Source form, including but
   not limited to compiled object code, generated documentation,
   and conversions to other media types.

   "Work" shall mean the work of authorship, whether in Source or
   Object form, made available under the License, as indicated by a
   copyright notice that is included in or attached to the work
   (an example is provided in the Appendix below).

   "Derivative Works" shall mean any work, whether in Source or Object
   form, that is based on (or derived from) the Work and for which the
   editorial revisions, annotations, elaborations, or other modifications
   represent, as a whole, an original work of authorship. For the purposes
   of this License, Derivative Works shall not include works that remain
   separable from, or merely link (or bind by name) to the interfaces of,
   the Work and Derivative Works thereof.

   "Contribution" shall mean any work of authorship, including
   the original version of the Work and any modifications or additions
   to that Work or Derivative Works thereof, that is intentionally
   submitted to Licensor for inclusion in the Work by the copyright owner
   or by an individual or Legal Entity authorized to submit on behalf of
   the copyright owner. For the purposes of this definition, "submitted"
   means any form of electronic, verbal, or written communication sent
   to the Licensor or its representatives, including but not limited to
   communication on electronic mailing lists, source code control systems,
   and issue tracking systems that are managed by, or on behalf of, the
   Licensor for the purpose of discussing and improving the Work, but
   excluding communication that is conspicuously marked or otherwise
   designated in writing by the copyright owner as "Not a Contribution."

   "Contributor" shall mean Licensor and any individual or Legal Entity
   on behalf of whom a Contribution has been received by Licensor and
   subsequently incorporated within the Work.

2. Grant of Copyright License. Subject to the terms and conditions of
   this License, each Contributor hereby grants to You a perpetual,
   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
   copyright license to reproduce, prepare Derivative Works of,
   publicly display, publicly perform, sublicense, and distribute the
   Work and such Derivative Works in Source or Object form.

3. Grant of Patent License. Subject to the terms and conditions of
   this License, each Contributor hereby grants to You a perpetual,
   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
   (except as stated in this section) patent license to make, have made,
   use, offer to sell, sell, import, and otherwise transfer the Work,
   where such license applies only to those patent claims licensable
   by such Contributor that are necessarily infringed by their
   Contribution(s) alone or by combination of their Contribution(s)
   with the Work to which such Contribution(s) was submitted. If You
   institute patent litigation against any entity (including a
   cross-claim or counterclaim in a lawsuit) alleging that the Work
   or a Contribution incorporated within the Work constitutes direct
   or contributory patent infringement, then any patent licenses
   granted to You under this License for that Work shall terminate
   as of the date such litigation is filed.

4. Redistribution. You may reproduce and distribute copies of the
   Work or Derivative Works thereof in any medium, with or without
   modifications, and in Source or Object form, provided that You
   meet the following conditions:

   (a) You must give any other recipients of the Work or
       Derivative Works a copy of this License; and

   (b) You must cause any modified files to carry prominent notices
       stating that You changed the files; and

   (c) You must retain, in the Source form of any Derivative Works
       that You distribute, all copyright, patent, trademark, and
       attribution notices from the Source form of the Work,
       excluding those notices that do not pertain to any part of
       the Derivative Works; and

   (d) If the Work includes a "NOTICE" text file as part of its
       distribution, then any Derivative Works that You distribute must
       include a readable copy of the attribution notices contained
       within such NOTICE file, excluding those notices that do not
       pertain to any part of the Derivative Works, in at least one
       of the following places: within a NOTICE text file distributed
       as part of the Derivative Works; within the Source form or
       documentation, if provided along with the Derivative Works; or,
       within a display generated by the Derivative Works, if and
       wherever such third-party notices normally appear. The contents
       of the NOTICE file are for informational purposes only and
       do not modify the License. You may add Your own attribution
       notices within Derivative Works that You distribute, alongside
       or as an addendum to the NOTICE text from the Work, provided
       that such additional attribution notices cannot be construed
       as modifying the License.

   You may add Your own copyright statement to Your modifications and
   may provide additional or different license terms and conditions
   for use, reproduction, or distribution of Your modifications, or
   for any such Derivative Works as a whole, provided Your use,
   reproduction, and distribution of the Work otherwise complies with
   the conditions stated in this License.

5. Submission of Contributions. Unless You explicitly state otherwise,
   any Contribution intentionally submitted for inclusion in the Work
   by You to the Licensor shall be under the terms and conditions of
   this License, without any additional terms or conditions.
   Notwithstanding the above, nothing herein shall supersede or modify
   the terms of any separate license agreement you may have executed
   with Licensor regarding such Contributions.

6. Trademarks. This License does not grant permission to use the trade
   names, trademarks, service marks, or product names of the Licensor,
   except as required for reasonable and customary use in describing the
   origin of the Work and reproducing the content of the NOTICE file.

7. Disclaimer of Warranty. Unless required by applicable law or
   agreed to in writing, Licensor provides the Work (and each
   Contributor provides its Contributions) on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
   implied, including, without limitation, any warranties or conditions
   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
   PARTICULAR PURPOSE. You are solely responsible for determining the
   appropriateness of using or redistributing the Work and assume any
   risks associated with Your exercise of permissions under this License.

8. Limitation of Liability. In no event and under no legal theory,
   whether in tort (including negligence), contract, or otherwise,
   unless required by applicable law (such as deliberate and grossly
   negligent acts) or agreed to in writing, shall any Contributor be
   liable to You for damages, including any direct, indirect, special,
   incidental, or consequential damages of any character arising as a
   result of this License or out of the use or inability to use the
   Work (including but not limited to damages for loss of goodwill,
   work stoppage, computer failure or malfunction, or any and all
   other commercial damages or losses), even if such Contributor
   has been advised of the possibility of such damages.

9. Accepting Warranty or Additional Liability. While redistributing
   the Work or Derivative Works thereof, You may choose to offer,
   and charge a fee for, acceptance of support, warranty, indemnity,
   or other liability obligations and/or rights consistent with this
   License. However, in accepting such obligations, You may act only
   on Your own behalf and on Your sole responsibility, not on behalf
   of any other Contributor, and only if You agree to indemnify,
   defend, and hold each Contributor harmless for any liability
   incurred by, or claims asserted against, such Contributor by reason
   of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

APPENDIX: How to apply the Apache License to your work.

   To apply the Apache License to your work, attach the following
   boilerplate notice, with the fields enclosed by brackets "{}"
   replaced with your own identifying information. (Don't include
   the brackets!)  The text should be enclosed in the appropriate
   comment syntax for the file format. We also recommend that a
   file or class name and description of purpose be included on the
   same "printed page" as the copyright notice for easier
   identification within third-party archives.

Copyright {yyyy} {name of copyright owner}

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Indices and Tables