1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10
11. 11
12. 12
13. 13
14. 14
15. 15
16. 16
17. 17
18. 18
19. 19
20. 20
21. 21
22. 22
23. 23
24. 24
25. 25
26. 26
27. 27
28. 28
29. 29
30. 30
31. 31
32. 32
33. 33
34. 34
35. 35
36. 36
37. 37
38. 38
39. 39
40. 40
41. 41
42. 42
43. 43
44. 44
45. 45
46. 46
47. 47
48. 48
49. 49
50. 50
51. 51
52. 52
53. 53
54. 54
55. 55
56. 56
57. 57
58. 58
59. 59
60. 60
61. 61
62. 62
63. 63
64. 64
65. 65
66. 66
67. 67
68. 68
69. 69
70. 70
71. 71
72. 72
73. 73
74. 74
75. 75
76. 76
77. 77
78. 78
79. 79
80. 80
81. 81
82. 82
83. 83
84. 84
85. 85
86. 86
87. 87
88. 88
89. 89
90. 90
91. 91
92. 92
93. 93
94. 94
95. 95
96. 96
97. 97
98. 98
99. 99
100. 100
101. 101
102. 102
103. 103
104. 104
105. 105
106. 106
107. 107
108. 108
109. 109
110. 110
111. 111
112. 112
113. 113
114. 114
115. 115
116. 116
117. 117
118. 118
119. 119
120. 120
121. 121
122. 122
123. 123
124. 124
125. 125
126. 126
127. 127
128. 128
129. 129
130. 130
131. 131
132. 132
133. 133
134. 134
135. 135
136. 136
137. 137
138. 138
139. 139
140. 140
141. 141
142. 142
143. 143
144. 144

# Certificate Transparency: Go Code

[![Build Status](https://travis-ci.org/google/certificate-transparency-go.svg?branch=master)](https://travis-ci.org/google/certificate-transparency-go)
[![Go Report Card](https://goreportcard.com/badge/github.com/google/certificate-transparency-go)](https://goreportcard.com/report/github.com/google/certificate-transparency-go)
[![GoDoc](https://godoc.org/github.com/google/certificate-transparency-go?status.svg)](https://godoc.org/github.com/google/certificate-transparency-go)

This repository holds Go code related to
[Certificate Transparency](https://www.certificate-transparency.org/) (CT).  The
repository requires Go version 1.9.

 - [Repository Structure](#repository-structure)
 - [Trillian CT Personality](#trillian-ct-personality)
 - [Working on the Code](#working-on-the-code)
     - [Rebuilding Generated Code](#rebuilding-generated-code)
     - [Updating Vendor Code](#updating-vendor-code)
     - [Running Codebase Checks](#running-codebase-checks)

## Repository Structure

The main parts of the repository are:

 - Encoding libraries:
   - `asn1/` and `x509/` are forks of the upstream Go `encoding/asn1` and
     `crypto/x509` libraries.  We maintain separate forks of these packages
     because CT is intended to act as an observatory of certificates across the
     ecosystem; as such, we need to be able to process somewhat-malformed
     certificates that the stricter upstream code would (correctly) reject.
     Our `x509` fork also includes code for working with the
     [pre-certificates defined in RFC 6962](https://tools.ietf.org/html/rfc6962#section-3.1).
   - `tls` holds a library for processing TLS-encoded data as described in
     [RFC 5246](https://tools.ietf.org/html/rfc5246).
   - `x509util` provides additional utilities for dealing with
     `x509.Certificate`s.
 - CT client libraries:
   - The top-level `ct` package (in `.`) holds types and utilities for working
     with CT data structures defined in
     [RFC 6962](https://tools.ietf.org/html/rfc6962).
   - `client/` and `jsonclient/` hold libraries that allow access to CT Logs
     via entrypoints described in
     [section 4 of RFC 6962](https://tools.ietf.org/html/rfc6962#section-4).
   - `scanner/` holds a library for scanning the entire contents of an existing
     CT Log.
 - Command line tools:
   - `./client/ctclient` allows interaction with a CT Log
   - `./scanner/scanlog` allows an existing CT Log to be scanned for certificates
      of interest; please be polite when running this tool against a Log.
   - `./x509util/certcheck` allows display and verification of certificates
   - `./x509util/crlcheck` allows display and verification of certificate
     revocation lists (CRLs).
 - CT Personality for [Trillian](https://github.com/google/trillian):
    - `trillian/` holds code that allows a Certificate Transparency Log to be
      run using a Trillian Log as its back-end -- see
      [below](#trillian-ct-personality).


## Trillian CT Personality

The `trillian/` subdirectory holds code and scripts for running a CT Log based
on the [Trillian](https://github.com/google/trillian) general transparency Log.

The main code for the CT personality is held in `trillian/ctfe`; this code
responds to HTTP requests on the
[CT API paths](https://tools.ietf.org/html/rfc6962#section-4) and translates
them to the equivalent gRPC API requests to the Trillian Log.

This obviously relies on the gRPC API definitions at
`github.com/google/trillian`; the code also uses common libraries from the
Trillian project for:
 - exposing monitoring and statistics via an `interface` and corresponding
   Prometheus implementation (`github.com/google/trillian/monitoring/...`)
 - dealing with cryptographic keys (`github.com/google/trillian/crypto/...`).

The `trillian/integration/` directory holds scripts and tests for running the whole
system locally.  In particular:
 - `trillian/integration/ct_integration_test.sh` brings up local processes
   running a Trillian Log server, signer and a CT personality, and exercises the
   complete set of RFC 6962 API entrypoints.
 - `trillian/integration/ct_hammer_test.sh` brings up a complete system and runs
   a continuous randomized test of the CT entrypoints.

These scripts require a local database instance to be configured as described
in the [Trillian instructions](https://github.com/google/trillian#mysql-setup).


## Working on the Code

Developers who want to make changes to the codebase need some additional
dependencies and tools, described in the following sections.  The
[Travis configuration](.travis.yml) for the codebase is also useful reference
for the required tools and scripts, as it may be more up-to-date than this
document.

### Rebuilding Generated Code

Some of the CT Go code is autogenerated from other files:

 - [Protocol buffer](https://developers.google.com/protocol-buffers/) message
   definitions are converted to `.pb.go` implementations.
 - A mock implementation of the Trillian gRPC API (in `trillian/mockclient`) is
   created with [GoMock](https://github.com/golang/mock).

Re-generating mock or protobuffer files is only needed if you're changing
the original files; if you do, you'll need to install the prerequisites:

  - `mockgen` tool from https://github.com/golang/mock
  - `protoc`, [Go support for protoc](https://github.com/golang/protobuf) (see
     documentation linked from the
     [protobuf site](https://github.com/google/protobuf))

and run the following:

```bash
go generate -x ./...  # hunts for //go:generate comments and runs them
```

### Updating Vendor Code

The codebase includes a couple of external projects under the `vendor/`
subdirectory, to ensure that builds use a fixed version (typically because the
upstream repository does not guarantee back-compatibility between the tip
`master` branch and the current stable release).  See
[instructions in the Trillian repo](https://github.com/google/trillian#updating-vendor-code)
for how to update vendored subtrees.


### Running Codebase Checks

The [`scripts/presubmit.sh`](scripts/presubmit.sh) script runs various tools
and tests over the codebase.

```bash
# Install gometalinter and all linters
go get -u github.com/alecthomas/gometalinter
gometalinter --install

# Run code generation, build, test and linters
./scripts/presubmit.sh

# Run build, test and linters but skip code generation
./scripts/presubmit.sh  --no-generate

# Or just run the linters alone:
gometalinter --config=gometalinter.json ./...
```