A foliage of folly

Can we circumvent the private access rules without relying on techniques that are considered “arcane” and likely to be made illegal (at some point)?

I recently read the article Profiting from the Folly of Others by Alastair Harrison, which walks through a technique to circumvent private access rules, particularly by means of (ab)using the template instantiation mechanism together with injections of inline friend definitions. I first saw this technique used in Filip Roséen’s blog article series on stateful metaprogramming back in 2015, which made use of it to implement a constexpr-counter that were, if not fully, at the very least nearly¹ standard compliant. A. Harrison’s article mentions that the origin of the technique is likely two blog articles by Johannes Schaub published in 2010-2011².

The publication of Roséen’s blog series swiftly lead to CWG issue 2118, describing the technique to be "[…] arcane and should be made ill-formed", but the issue is, still, yet³ to be addressed.

2118. Stateful metaprogramming via friend injection

Section: 17.7.5 [temp.inject]
Status: open
Submitter: Richard Smith
Date: 2015-04-27

Defining a friend function in a template, then referencing that function later provides a means of capturing and retrieving metaprogramming state. This technique is arcane and should be made ill-formed.

Notes from the May, 2015 meeting:
CWG agreed that such techniques should be ill-formed, although the mechanism for prohibiting them is as yet undetermined.

The core working group’s main concern, afaict, is that the definition of the injected friend depends on which specialization of the template entity that is first instantiated, thus allowing one to capture a metaprogramming state⁴. For the purpose of circumventing private access rules, however, we have no actual need of capturing a metaprogramming state in this kind of sense, it just so happens that the most viable mechanism (thus far; prior to C++20) by which we may legally refer to a private entity beyond its intended access rules is by means of explicit instantiation definitions, which in turn lead us onto the path of “arcane” stateful metaprogramming.

In this blog post we’ll walk through the established approach (applied to C++17), whereafter we will proceed to look into a new minor C++20 feature to see if we can create a novel approach to circumvention of access rules, one which does not rely on mechanisms that CWG issue 2118 intends to deprecate and make ill-formed.

A disclaimer before we begin

This post is, like some others on my blog, driven from a curiousity (and sometimes also language-lawyer) perspective. However, as the end result in this post can be particularly abused, I want to highlight that thou shalt not use the techniques herein in production code. Or, by the words of Herb from GotW #76: Uses and Abuses of Access Rights:

Is this a hole in C++'s access control mechanism, and therefore a hole in C++'s encapsulation? Discuss.

[…]

The real answer to the issue is: Don’t do that! Admittedly, even Scott Meyers has said publicly that there are times when it’s tempting to have a quick way to bypass the access control mechanism temporarily, such as to produce better diagnostic output during debugging… but it’s just not a habit you want to get into for production code, and it should appear on the list of “one-warning offences” in your development shop.

The only actual use case (beyond extreme-case optimizations as used e.g. in UninitializedMemoryHacks.h in Facebook’s Folly library) I can come up with for this kind of technique, on the top of my head, is for some corner case testing scenarios, e.g. fault injection testing.

The problem we want to solve

Consider the following class

// foo.h
#pragma once
#include <iostream>

class Foo {
    int bar() const {
        std::cout << __PRETTY_FUNCTION__;
        return x;
    }

    int x{42};
};

which contains a private member function bar and a private data member x. The problem we’re trying to solve is:

Given an object of type Foo, can we somehow, without changes to Foo itself, access the private data member x and/or invoke the (non-overloaded) private member function bar whilst still using only standard conforming C++ code?

or, in more general words, can we circumvent the private access rules of C++ without relying on non-conforming code or implementation-defined mechanisms?

Recalling the pre-C++20 approach (Schaub, Folly library et al)

(We will be performing this pre-C++20 journey using C++17, and thus all standard references below refer to N4659⁵)

Before we move on to investigate whether a new minor feature of C++20 can help us circumvent access rules without inciting the anger of CWG issue 2118 or not, we’ll walk through and make sure we understand the mechanisms as well as the pitfalls of the existing approach used e.g. in the Fickle library, and whose mechanisms are covered in detail also in Profiting from the Folly of Others.

The key feature of this approach is governed by [temp.explicit]/12:

The usual access checking rules do not apply to names used to specify explicit instantiations. […]

which allows the following snippet of code to be well-formed:

class A { int x; };

template<auto>
class B {};

// Explicit instantiation definition.
template class B<&A::x>;
//               ^^^^^ access rules for private data
//                     member 'x' is waived in an
//                     explicit instantiation definition.

This mechanism in itself is neither unusual nor a direct tool for language abuse, however for the purpose of circumventing private access rules it needs to be combined with a mechanism that is; namely defining, in a class template, a friend function whose definition relies on one or more template parameters, meaning the definition depends on which specialization of the class template is first instantiated.

Defining a friend within a class template

As governed by [class.friend]/6, a function that is also a friend may be defined at its friend declaration within a class [ emphasis mine]:

A function can be defined in a friend declaration of a class if and only if the class is a non-local class ([class.local]), the function name is unqualified, and the function has namespace scope. [ Example:

class M {
  friend void f() { }  // definition of global f, a friend of M,
                       // not the definition of a member function
};

— end example ]

Such a friend function is implicitly inline, as per [class.friend]/7 [ emphasis mine]:

Such a function is implicitly an inline function. A friend function defined in a class is in the (lexical) scope of the class in which it is defined. A friend function defined outside the class is not ([basic.lookup.unqual]).

which is important as it allows the definition to be present in multiple translation units (TU) without ODR-violations, given that it is the same definition. Specifically, if the definition of say the class M above were to be located in a header file that was included by several source files, then the corresponding TU:s would all include their own definition of the global (inline) function f, which would be fine, as per [basic.def.odr]/6, as the definition in each TU would be the same.

Now, although a function defined at its friend declaration has namespace scope, a friend declaration does not introduce the declared name into the enclosing namespace, as per [namespace.memdef]/3 [ emphasis mine]:

If a friend declaration in a non-local class first declares class, function, class template or function template the friend is a member of the innermost enclosing namespace. The friend declaration does not by itself make the name visible to unqualified lookup ([basic.lookup.unqual]) or qualified lookup ([basic.lookup.qual]). [ Note: The name of the friend will be visible in its namespace if a matching declaration is provided at namespace scope (either before or after the class definition granting friendship).  — end note ] […]

meaning a matching declaration needs to be provided at namespace scope to allow the friend function to be found via non-ADL lookup⁶:

struct C {
    friend void f() { }
    friend void g() { }
};
void f();

int main() {
    f(); // OK.
    g(); // Error: 'g' was not declared in this scope.
}

Now, nothing prohibits us from supplying friend declarations in class templates, and we can likewise, as for the non-template class, define the friends at their friend declarations:

template<int N>
struct D {
    friend void f() { }
};
void f();

As per the instantiation rules of template entites, the definition of the friend will not be instantiated until at least one explicit or implicit instantiation of the class template D takes place, even though f itself is not a template function nor a member of a class template; it is free namespace scope function.

template<int N>
struct D {
    friend void f() { }
};
void f();

template<int N>
struct E {
    friend void g() { }
};
void g();

int main() {
    D<1> a;  // Instantiaces _a_ specialization of 'D'
    f(); // OK (definition "instantiated" along D<1>)
    g(); // Error: undefined reference to 'g()'.
}

Danger, Will Robinson!

However, we’ve now entered a territory that comes with a number of dangers; some diagnosable and arguably at worst confusing for the developer debugging the associated compiler errors, but some that are worse than that.

1. (Diagnosable) As per above, the definition of f will only be present if the class template D is instantiated (for any specialization); prior to the first instantiation of D, f remains undefined. On the othe hand, if more than one specialization is instantiated in the same translation unit, f will be re-defined (albeit with the same definition) and the program is ill-formed by violation of [basic.def.odr]/1

   No translation unit shall contain more than one definition of any variable, function, class type, enumeration type, or template.

2. (Diagnosable) Every translation unit in which f is odr-used needs to define f, as per [basic.def.odr]/4

   […] An inline function or variable shall be defined in every translation unit in which it is odr-used outside of a discarded statement.

   meaning that if the class template D above were to be declared in a header, say d.h, then every TU that that included d.h and odr-used f would need to instantiate the class template D, implicitly or explicitly, exactly for one specialization.

3. (Non-diagnosable; undefined behaviour) If the definition of f depends on any of the template parameters of the class template in which it has been defined, e.g.

      // d.h
      #pragma once
      template<int N, bool B, typename T>
      struct D {
          friend void f() { int x{N}; (void)x; }
          // ...
      };
      void f();
   

   then every TU which includes d.h and instantiates the class template D, implicitly or explicitly, must as per above (1) instantiate it no more than for one specialization (diagnosable), but moreover that specialization needs to be one that has exactly the same template argument for each template parameter that is used within the definition of f, as the definition of f needs to be the same over all TU:s, as per [basic.def.odr]/6. In the example above, the latter would mean that the single instantiation of D in each TU would need to use exactly the same value for the non-type template parameter N.

4. (Ill-formed NDR) If we leverage explicit instantiation definitions, we may not explicitly instantiate the same specialization twice over different translation units, as per [temp.spec]/5 [ emphasis mine]:

   For a given template and a given set of template-arguments,
   — (5.1) an explicit instantiation definition shall appear at most once in a program,
   — […]
   An implementation is not required to diagnose a violation of this rule.

   meaning that two explicit instantiation definitions of D above, in different TU:s, must instantiate different specializations but, as per (3), specializations that both use the same value as template argument for the template parameter N.

Now, this is arguably quite a mess, and any sane advice would be to stop now and never enter this minefield of ODR-violations, UB and ill-formed NDR. However, for the purpose of circumventing private access rules prior to C++20, it is precisely down this path that we need to walk.

Circumvention of private access rules by injection of pointers to private members into friends

To finally construct the circumvention mechanism in C++17, we leverage [temp.explicit]/12 to supply a pointer to a private data member or pointer to a private member function as a non-template parameter as part of the explicit instantiation definition of a class template which contains a friend declaration that captures and “access-rule-erases”, if you will, the supplied pointer to member into its definition. A first attempt, focusing on invoking the private member function bar of Foo may look like follows:

// access_private_of_foo.h
#pragma once
#include "foo.h"

template <auto mem_fn_ptr>
struct InvokePrivateFooBar {
    // (Injected) friend definition.
    friend int private_Foo_bar(Foo const& foo) {
        return (foo.*mem_fn_ptr)();
    }
};
// Friend (re-)declaration.
int private_Foo_bar(Foo const& foo);

// Explicit instantiation definition immediately after
// class template definition, to ensure this instantiation
// is the first of the class template.
template class InvokePrivateFooBar<&Foo::bar>;

which is well-formed by itself, but which may stumble over both of the serious (non-diagnosable) dangers of (3) and (4) above. By our careful understanding of (3) and (4), however, they can be mitigated.

We could implement the entirety of the private access circumvention mechanism within an unnamed namespace within single TU, which would remove the cross-TU dangers of (3) and (4), but such an approach would limit re-use, and may arguably make its way into someone’s header files sooner or later anyway.

A more common approach is to introduce an additional type template parameter to the class template (InvokePrivateFooBar here), and use a TU unique tag type as template argument for that template parameter in the single explicit instantiation definition. Leveraging the fact that unnamed namespaces are unique and have internal linkage, we may wrap a type definition within an unnamed namespace in a header file, with the result that the type will be guaranteed to be a different type in each TU that includes the header file. This would also alleviate the dangers of 3 and 4, as the single explicit instantiation definition, following immediately after the class template in the header, would use the same template arguments for all template parameters who are used (/injected into) the body of the friend, but would use the dummy TU tag type to differentiate the explicit instantiation definition between different TU:s. Any attempt by a user to instantiate another specialization than the from-header provided one would violate [basic.def.odr]/1 (2 above), which is diagnosable, so at least the compiler could help us out if we enter the territory of danger 2 above (and the same holds for 1).

With this in mind, our second attempt looks as follows, where we’ve expanded the circumvention API with a structure for accessing also the private data member x of Foo.

// access_private_of_foo.h
#pragma once
#include "foo.h"

// Unique namespace in each TU.
namespace {
// Unique type in each TU (internal linkage).
struct TranslationUnitTag {};
}  // namespace

// 'Foo::bar()' invoker.
template <typename UniqueTag,
          auto mem_fn_ptr>
struct InvokePrivateFooBar {
    // (Injected) friend definition.
    friend int invoke_private_Foo_bar(Foo const& foo) {
        return (foo.*mem_fn_ptr)();
    }
};
// Friend (re-)declaration.
int invoke_private_Foo_bar(Foo const& foo);

// Single explicit instantiation definition.
template struct InvokePrivateFooBar<TranslationUnitTag, &Foo::bar>;

// 'Foo::x' accessor.
template <typename UniqueTag,
          auto mem_ptr>
struct AccessPrivateMemFooX {
    // (Injected) friend definition.
    friend int& access_private_Foo_x(Foo& foo) {
        return foo.*mem_ptr;
    }
};
// Friend (re-)declaration.
int& access_private_Foo_x(Foo& foo);

// Single explicit instantiation definition.
template struct AccessPrivateMemFooX<TranslationUnitTag, &Foo::x>;

which we may use in a particular source file as follows:

// demo.cpp
#include <iostream>
#include "access_private_of_foo.h"
#include "foo.h"

void demo() {
    Foo f{};

    int hidden = invoke_private_Foo_bar(f);  // int Foo::bar() const
    std::cout << hidden << "\n";  // 42

    access_private_Foo_x(f) = 13;
    hidden = invoke_private_Foo_bar(f);  // int Foo::bar() const
    std::cout << hidden << "\n";  // 13
}

and we have thus reached the goal of circumvention of private access rules using a C++17 approach; particularly here applied to the Foo class, invoking the private member function bar as well as writing to the private data member x.

Demo.

We move on to look into finding an alternative approach, by making use of a new minor feature of C++20.

C++20 & waiving of access checking on specializations

(As we are now moving to C++20, all standard references from hereon refers to N4861⁷)

C++20 implemented P0692R1 (Access Checking on Specializations), summarized in P2131R0 (Changes between C++17 and C++20 DIS) as

This change fixes a long-standing, somewhat obscure situation, where it was not possible to declare a template specialization for a template argument that is a private (or protected) member type. For example, given class Foo { class Bar {}; };, the access Foo::Bar is now allowed in template<class> struct X; template<> struct X<Foo::Bar>;.

particularly adding [temp.spec]/6 to the C++20 standard:

The usual access checking rules do not apply to names in a declaration of an explicit instantiation or explicit specialization, with the exception of names appearing in a function body, default argument, base-clause, member-specification, enumerator-list, or static data member or variable template initializer. [ Note: In particular, the template arguments and names used in the function declarator (including parameter types, return types and exception specifications) may be private types or objects that would normally not be accessible. — end note ]

This opens up a new approach for circumvention of private access rules, one which does not fall under CWG issue 2118, and moreover offers more freedom and fewer gotchas than that of the explicit instantiation hack.

Common top-level private accessor class

Specializations can allow us to use a single top-level utility class template for circumvention of private access rules of any class, as compared to the C++17 approach above where a new class (template) needs to be defined for each unique member of a unique type that we want to access.

With the specialization approach we still leverage the in-class friend definition trick, but leaves such friend declarations to appear only in explicit specializations of a member class to the top-level utility class. Specifically, we let the latter be a class template with a single non-type template parameter—semantically intended to take arguments that are pointers to private members—which is used to initialize a single static data member, and declare a member class in the class template that is intended to be defined only in explicit specializations:

// accessprivate.h
#pragma once
template <auto mem_ptr>
struct AccessPrivate
{
    // kMemPtr is intended to be either a pointer to a private
    // member function or pointer to a private data member.
    static constexpr auto kMemPtr = mem_ptr;
    struct Delegate;  // Define only in explicit specializations.
};

Each explicit specialization of the templated class AccessPrivate::Delegate (a member class of class template is likewise a templated class) can declare a unique friend function which is defined at its friend declaration, and whose definition makes use of the particular instantantiation of the access-erased kMemPtr for that very specialization:

// access_private_of_foo_cpp20.h
#pragma once
#include "accessprivate.h"
#include "foo.h"

// Specialize the nested Delegate class for each private
// member function or data member of Foo that we'd like to access.

template <>
struct AccessPrivate<&Foo::bar>::Delegate {
    // (Injected) friend definition.
    friend int invoke_private_Foo_bar(Foo const& foo) {
        return (foo.*kMemPtr)();
    }
};
// Friend (re-)declaration.
int invoke_private_Foo_bar(Foo const& foo);

template <>
struct AccessPrivate<&Foo::x>::Delegate {
    // (Injected) friend definition.
    friend int& access_private_Foo_x(Foo& foo) {
        return foo.*kMemPtr;
    }
};
// Friend (re-)declaration.
int& access_private_Foo_x(Foo& foo);

where, as per [temp.type]/2.6, a template argument that is a pointer to a unique member will mean a unique specialization.

Dangers, Will Robinson?

Even if this explicit specialization is placed in a header, it is ODR-safe as an explicit specialization that defines a templated class is no different than a definition of non-templated class, and both fall under [basic.def.odr]/13 in that they may have more than one definition in more than one TU, as long as the definitions are the same. This means we needn’t apply the TU-unique dummy type template parameter tag that we had to use for the C++17 approach of explicit instantiation definitions. Moreover, we are safe against the dangers of violating [temp.expl.spec]/7 [ emphasis mine]:

If a template, a member template or a member of a class template is explicitly specialized then that specialization shall be declared before the first use of that specialization that would cause an implicit instantiation to take place, in every translation unit in which such a use occurs; no diagnostic is required. If the program does not provide a definition for an explicit specialization and either the specialization is used in a way that would cause an implicit instantiation to take place or the member is a virtual member function, the program is ill-formed, no diagnostic required. An implicit instantiation is never generated for an explicit specialization that is declared but not defined.

as any explicit specialization of AccessPrivate::Delegate is intended to ever only use a non-type template argument that may otherwise not be referred to—due to private access rules—and there is thus no risk that an implicit instantiation of such an specialization precedes the explicit specialization. Moreover, the sole purpose of each explicit specialization is to declare and define a uniquely named friend, whereas the specialization of the class itself is not intended to be used directly (say as an object or in an explicit instantiation definition).

Finally, as for any external linkage entity definition, we naturally mustn’t provide different definitions for the same explicit specialization, or for the same (inline) friend function, in different TU:s. This is arguably “general ODR-sanity”, and it does not come with the same subtleness as that of the non-diagnosable and ill-formed NDR dangers of the explicit instantiation definition approach. Furthermore, as the definition of the accessor friend function is no longer depending on instantiation order in the sense that it captures a metaprogramming state, we are using a technique that does not fall under CWG issue 2118.

Circumvention of private access rules by means of explicit specializations

Thus, we have reached an alternative mechanism to that of the explicit instantiation definition approach, and we may reproduce the behaviour of the C++17 demo using our alternative approach:

// demo.cpp
#include <iostream>
#include "access_private_of_foo_cpp20.h"
#include "foo.h"

void demo() {
    Foo f{};

    int hidden = invoke_private_Foo_bar(f);  // int Foo::bar() const
    std::cout << hidden << "\n";  // 42

    access_private_Foo_x(f) = 13;
    hidden = invoke_private_Foo_bar(f);  // int Foo::bar() const
    std::cout << hidden << "\n";  // 13
}

Demo.

Setting up a simple macro API lib

We’ll conclude this post with setting up a small header-only lib of macros, used as a utility to apply the new C++20 approach as above without bothering about the details. Possibly someone can find this lib useful for, say, injection testing. As this post has become quite long-winding already, we’ll settle for writing a macro API only for the simpler case of accessing private data members, whilst leaving the case of invokers for private members functions for a later time.

We may expand the accessprivate.h header above with the addition of a (beautifully highlighted⁸) accessor creator macro:

// accessprivate/accessprivate.h
#pragma once

namespace accessprivate {
template <auto mem_ptr>
struct AccessPrivate
{
    // kMemPtr is intended to be either a pointer to a private
    // member function or pointer to a private data member.
    static constexpr auto kMemPtr = mem_ptr;
    struct Delegate;  // Define only in explicit specializations.
};

}  // namespace accessprivate

// DEFINE_ACCESSOR(<qualified class name>, <class data member>)
//
// Example usage:
//   DEFINE_ACCESSOR(foo::Foo, x)
//
// Defines:
//   auto& accessprivate::get_x(foo::Foo&)
#define DEFINE_ACCESSOR(qualified_class_name, class_data_member)\
namespace accessprivate {\
template <>\
struct AccessPrivate<&qualified_class_name::class_data_member>::Delegate {\
    friend auto& get_##class_data_member(\
        qualified_class_name& obj) { return obj.*kMemPtr; }\
};\
auto& get_##class_data_member(qualified_class_name& obj);\
}

which we may use as follows:

#include <iostream>
#include "accessprivate/accessprivate.h"

namespace bar {

struct Bar {
    int getX() const { return x; }
    int getY() const { return y; }
private:
    int x{42};
    int y{88};
};

}  // namespace bar

DEFINE_ACCESSOR(bar::Bar, x)
// -> accessprivate::get_x(Bar&)
DEFINE_ACCESSOR(bar::Bar, y)
// -> accessprivate::get_y(Bar&)

void demo() {
    bar::Bar b{};
    accessprivate::get_x(b) = 13;
    accessprivate::get_y(b) = 33;
    std::cout << b.getX() << " " << b.getY();  // 13 33
}

Demo.

The lib has been published at dfrib / accessprivate at GitHub, and we may return to it in a future post, to have a look at whether we can expand it to provide macro “overloads” (delegating, per se) for invoking (non-overloaded) private member functions with a various number of function arguments.

We may also note that P0692R1 points out that for the particular case of explicit specializations, the mechanism introduced by the paper is already implemented (non-documented) by all compilers that the authors of the paper tried out:

It is important to note that even though the above specialization of trait is not allowed according to the standard, it builds with all compilers that were tested, including various versions of gcc, clang, icc, and msvc. Already, for the sake of standardizing existing practice, one might argue that this should be allowed.

meaning that the accessprivate lib will most likely work with most compilers also in C++17, even if it is, strictly, non-conformant in C++17.

Recalling the disclaimer

Finally, before wrapping up, we’ll re-visit the disclaimer above, adding a final reminder that the technique above should not be used in production code! Even if the technique results in well-formed code, a production code “use case” for this mechanism is likely either misunderstood, a code smell, or an XY problem.

Acknowledgements

Thanks to Christian von Schultz for proofreading the article.