|
|
gcc: Background and Rationale
Copyright 2006 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
ident "@(#)d-compilers.txt 1.7 06/03/17 SMI"
OpenSolaris
SUPPORTED COMPILER POLICY, V1 [DRAFT 2]
1. Summary
This policy defines that the OpenSolaris build process for the ON
consolidation will introduce the concept of primary and secondary
supported compilers. Compilation cleanliness across all supported
compilers is a requirement for committing a changeset to ON.
ON will support the Sun Studio compiler as the primary compiler on
current platforms and the GCC compiler as the secondary compiler.
Constraints on the implementation of this policy are identified.
Specific exceptions for changesets affecting C++ source files are
defined.
Management of coding conventions is described in cases of
different valid language interpretations within the set of
supported compilers. Conventions are established for
functionality requiring knowledge of compiler behaviour and
output, beyond that embodied in existing language and platform
standards.
2. Discussion
There are a number of arguments that might be made for supporting
multiple compilers for the ON consolidation; the principal reasons
to support the GCC compiler specifically include
- decreased service costs, as distinct compiler execution can
identify potential defects prior to release to users [0],
- decreased porting costs due to compiler unavailability, and
- potential defect elimination from GCC-derived automatic code
examination systems [1, 2]
This policy recognizes that there is an additional intangible
benefit from supporting toolsets common across numerous software
development communities. This policy can be implemented so that
the benefits are maintained with no significant increase in active
developer time expended; see the Implementation Addendum [3].
2.1. Potential support scenarios
There are three levels of support we can require for gcc on each
platform; the levels are described here with the requirements they
impose on engineers and the C-team as well as any additional
resource requirements. In all scenarios, ON source, build tools,
and processes may be modified to minimize developer intervention
times. See the Implementation Addendum for details.
1. Full support. Require testing equally with both compilers on
all applicable platforms. In this scenario, any putback that
introduces compiler dependent differences other than
differential runtime performance will be backed out.
Gatekeepers will need to perform both incremental and full
nightly builds using both compilers, and make both sets of
archives available to developers and users. A mechanism will
be required to distinguish between these sets of archives both
at runtime and in cores and crash dumps. Projects will have
to provide test results for both compilers as part of the
integration criteria. This requirement increases (by about
70% at present) build time requirements for all putbacks and
gate builds. It also increases (by 100%) testing requirements
for all putbacks and administrative overhead associated with
tracking test plans and status for all projects. This would
likely require substantial additional build and test hardware.
2. Build-only support. Require only that ON build with both
compilers. Runtime behaviour differences will not be cause
for backout; however, these differences will constitute bugs
as they are often serious violations of standards anyway.
These bugs will be prioritized like all others in accordance
with business needs. Gatekeepers will need to perform both
incremental and full nightly builds with both compilers;
gcc-built archives need not be made available, and no
expectation of their correctness can be established. Projects
will need to show that they build with gcc but need not
provide duplicate results for other tests. Costs include
build time increases as for (1) and possible additional build
machine requirements as for (1). Several possible
implementations are presented in the Implementation Addendum,
and their relative costs and benefits are described there in
detail.
Owing to the distinct C++ ABIs across the primary and
secondary compilers, projects modifying C++ code must perform
actions equivalent to those outlined in (1), including
sufficient testing to verify that both runtime and link-time
dependencies are satisfied by the objects built by both
compilers.
3. One-shot. Require nothing; gcc support will be maintained on
an ad-hoc basis. Build and runtime breakage reports will be
considered RFEs. There are no requirements for project teams
nor for the C-team, and no ongoing costs since no one will be
expected to find or fix gcc breakage. However, as soon as
code that breaks the gcc build is introduced, the gcc QDS
begins and our efforts would have little long-term impact.
In principle, SPARC and x86 could have different levels of support
on temporary or permanent bases. As an interim milestone,
building with nonfatal warnings on either platform could be
supported to encourage and ease fixing of remaining bugs, most of
which are for warnings. Additionally, current and future
supported platforms could identify different primary-secondary
compiler pairs.
2.2. Proposed scenario and implementation thereof
The proposed scenario is to require build-only support. The
expectation is that there will be sufficient interest in an
alternative compiler on one or more supported platforms to manage
the defect introduction rate not covered by the first-pass
verification. Tools and source changes will be introduced to make
possible a standard invocation that executes a "complete" build in
this sense. The Implementation Addendum describes several
possible implementations. *WE ARE RECOMMENDING OPTION 1, SHADOW
COMPILATION.*
2.2.1. Consequences
A consequence of this strategy is that separate testing of builds
made with both compilers is not required at any time; all relevant
tests must be run against the primary compiler build. This choice
reduces greatly the resource commitments that full support would
require, to a level believed sustainable. Since elevating from
build-only to full support at some future time would have a
relatively low one-time cost, our options remain open if we
determine that a Studio exit strategy is required. Such an exit
strategy is neither the effect of nor the motivation for this
proposal.
2.2.2. Constraints
Although the implementation is not expected to produce an
executable proto area, package set, or archives from the secondary
compiler's output, the tools are expected to allow swapping the
primary and secondary compiler roles such that the full set of
build products can be constructed using either compiler.
Incremental builds that produce working bits with the primary
compiler must remain possible and generate correct bits even after
any secondary compiler pass executes.
2.2.3. Timeline
Each architecture requires a different amount of work before the
baseline requirement will be met in onnv. Code built into amd64
objects already offers full support for gcc, although its
correctness can no longer be certain since the default amd64
compiler was changed from gcc to cc. Code built into i386, sparc,
and sparcv9 objects requires some changes, mainly in generic
userland code, before build-only support will be provided.
Although sparc and sparcv9 code was initially at a substantial
disadvantage with respect to gcc support, this gap has been almost
entirely eliminated. In the interest of expediency, all
architectures will be made fully build-clean prior to or
simultaneous with this policy's effectiveness and the introduction
of tools changes required to support it.
2.3. Language ambiguity management
In the process of pursuing an initial clean state, a number of
standard differences between the compiler pair have been identified.
A reasonable partition of these differences is into two sets:
(a). Unsafe implementation: One compiler or the other is
defective; that is, it fails to adhere to relevant published
standards or documented behaviour, emits output which is
semantically different from its input, or fails to accept
unambiguous valid input.
(b). Local convention: An architecture's secondary compiler
rejects input which is acceptable by the primary compiler and
is considered valid according to local conventions, even
though the code's behaviour is undefined or unspecified by
standards. These differences generally involve implementation
choices where the language standard is ambiguous.
2.3.1. Unsafe implementation
A recent example of the secondary compiler emitting unsafe code is
6362144 gcc breaks libc by turning structure assignments into
calls to memcpy()
For such cases, while the release team may choose to identify a
clarified coding convention (such as explicit member assignment for
the example), the correct outcome is to engage with the compiler
vendor or our designate to same to secure a fix for the unsafe
behavior.
Project teams may not generally be blocked from putback by unsafe
secondary compiler implementation exposed by their correct code.
While the release team may direct otherwise, it is expected that
project teams in this position will generally take the following
actions:
1. File a high-priority defect against the secondary compiler.
2. Modify makefiles, flags, or other infrastructure so as to
avoid triggering any compile-time warnings or errors generated
by the secondary compiler as a result of its unsafe
implementation.
3. Document for ON developers the differences, if known,
between each compiler's output as they affect user- or
developer-visible functionality delivered by their project.
2.3.2. Local convention
The bulk of the differences have been where the secondary compiler
has found the previous local convention, which may merely derive
from the primary compiler's defaults, to be ambiguous. For
example:
6367203 Studio 10 C compiler and gcc 3.4.3 disagree about
preprocessing #pragma redefine_extname
In such cases, the release team should examine the choices before
them: they may either propose to abandon the local convention and
require an unambiguous invocation, or engage with the secondary
compiler vendor or our designate to same to secure a fix that
introduces an option to accept the local convention as
unambiguous. The proposal should weigh the costs associated with
the change, including vendor costs, the absolute number of
affected sites within the code, and the general acceptance of the
local convention. If the release team elects to pursue a change
to the secondary compiler, an affected project team will, subject
to the release team's direction, take actions similar to those
described in 2.3.1.
2.4. Compiler Implementation Knowledge
Projects which are directly or indirectly tied to the compiler
used to build ON will be expected to function properly when built
with the secondary compiler. In particular, mdb, the CTF
infrastructure, DTrace, and other similar tools and libraries must
behave substantially the same way regardless of the compiler used
to build them. This requirement is intended to apply specifically
to tools with intimate knowledge of compiler artifacts above and
beyond that provided by standards such as an ISA's ABI or a
published debugging format standard. In some cases, this
requirement may be waived for business or technical reasons if the
following are true:
(a) There is substantial marginal cost associated with
delivering an implementation compatible with the secondary
compiler, and
(b) There is no known or reasonably forseen build-time
dependency on this functionality. For example, if the project
team delivers a utility with compiler-dependent knowledge, and
the correct and complete output of that utility is required to
complete a successful build of ON, this requirement cannot be
waived. This exception is not intended to apply to arbitrary
code solely because it it executed during the build (e.g., a
non-compiler-dependent tool built natively).
If this requirement is waived, the project team is expected to
deliver stubs for use when the project's functionality is built
with the secondary compiler. If the functionality consists of
programs which would ordinarily be run interactively, these
programs should print a message when run indicating that they are
unavailable when built with the secondary compiler. LIbrary
interfaces should fail with an appropriate error return, and other
interfaces should generally make a reasonable effort to inform
callers or users of the reason for failure. In particular,
unpredictable output or program behaviour when built with the
secondary compiler is not acceptable even if this requirement is
waived.
The release team may instead, after evaluating the relative costs
and benefits, direct a project team delivering this type of
functionality to request appropriate changes to the secondary
compiler.
3. Modified roles in proposed scenario
Once all bugs preventing successful builds with gcc are are fixed,
we will need to establish a clear set of guidelines for maintaining
gcc compatibility within ON (and possibly in other consolidations as
well). Contributors in several roles will have additional
responsibilities in support of these guidelines.
3.1. ON C-Team
The ON C-team will enforce build-only support across all supported
architectures and platforms. That is, changes are not acceptable
which prevent using the C-team-approved version of gcc to
successfully complete a full build for any supported architecture.
However, implementations which simply disable compilation of a
functional component or subdirectory when gcc is in use, without
delivering alternate functionally equivalent code suitable for use
with gcc, will not be considered to have fulfilled this
requirement. The gatekeepers will be responsible for monitoring
compliance with this requirement and for backing out non-compliant
changes or, at their option, coordinating timely corrections to
non-compliant code. Changes which fulfill the build requirement
but introduce functional differences dependent on the compiler
used may be acceptable, but such functional differences will, if
discovered, be tracked and fixed as defects; such differences
often result from violation of applicable standards or
conventions.
The C-team will also be required to evaluate differences between
the primary and secondary compilers as discussed in sections 2.3
and 2.4, to direct project teams encountering these differences,
and to ensure that timely fixes to the secondary compiler are made
as required.
3.2. ON Implementors
Implementors will be expected to understand compiler-dependent
behaviours and the standards governing their operation, and to
utilize that understanding toward the creation of portable,
standards-compliant code. They will also be expected to maintain
tools such as the compiler wrapper used to support the use of
multiple compilers, and to update those tools when their other
changes require it. Implementors will correct any alternate
compiler specific build failures induced by their changes prior to
putback. Clear instructions for performing the necessary builds
or passes will be disseminated by the Tonic team prior to these
requirements becoming effective.
The first pass proposal requires that implementors modifying C++
files perform a complete build of the affected component and its
dependents with both supported compilers.
4. Acknowledgements
Keith Wesolowski developed the initial version of this policy.
Stephen Hahn clarified this policy and contributed the description
of build-only support it proposes.
5. References
[0] Of the 475 gcc defects (this count excludes those CRs which are
actual RFEs or were closed without requiring a fix), at least 28
are indisputable: the code violates an applicable de jure standard
and/or de facto ON conventions, is semantically incorrect, works
only because of undocumented and unreliable Studio behaviours that
cannot reasonably be considered de facto ON/Studio conventions, or
will be harmful to customers who use gcc (a supported feature of
Solaris) even if we don't. These 28 bugs are tangible evidence of
code quality improvement demanded by the use of multiple
compilers; many of these defects would be expected to affect
customers. Note that these defects still existed even after most
common kernel and library code was fixed by the amd64 team, so the
actual number of bugs was likely much higher. The list of these
serious bugs follows:
6254191 uninitialized variable in elfsignlib.c
6241993 SPARC siglongjump symbol aliasing is wrong
6241314 ctfconvert should understand SPARC real types in DWARF
6241287 g++ doesn't want wchar_t under any circumstances
6241296 kernel files must include sys/note.h rather than note.h
6277636 *sh* can crash if it creates too many temporary files
6358155 When compiled with gcc, the socal driver panic's the machine
6359607 kstat perl module compiled with gcc breaks on x86
6272173 cpumem diag module writes to constant memory, crashing fmd
6268345 fmd is insufficiently careful with alignment
6264775 lpadmin expects to write into string constants
6254408 many forthdebug templates are incorrect
6281909 bootadm writes into string constants
6281951 ikeadm writes into string constants
6280646 amd64 memlist structure mismatch causes panic
6277643 ksh is insufficiently careful with alignment
6274664 libcurses has an unterminated comment
6272179 relocation processing should be bypassed when building libgenunix et al
6272155 svcs writes into string constants, can crash as a result
6268299 libumem is insufficiently careful with alignment
6267468 rpcgen generated code gcc doesn't like
6267352 libld_malloc provides inadequate alignment
6267482 lex generates code gcc doesn't like
6266620 mi2cv shouldn't #include
6264767 cfgadm expects to write into string constants
6264469 fbt is confused by jump tables in code
6264473 fbt is confused by the return instruction
6258738 fbt refuses to instrument functions starting with branches
[1] Andy Chou, Junfeng Yang, Benjamin Chelf, Seth Hallem, and Dawson
Engler. An Empirical Study of Operating System Errors. Symposium on
Operating System Principles, 2001.
[2] smatch, the Source Matcher. http://smatch.sourceforge.net/
[3] OpenSolaris Supported Compiler Policy, Implementation Addendum.
/net/tonic-gate.sfbay/gates/devproc/src/implementation/d-compilers-impl.txt
|