On this page
Adeo Group
Adeo owns different brands in retail industry focused on home improvement and DIY markets, like Leroy Merlin.
tl;dr just go and have a look atfull production-used AsyncAPI document
Challenges
Cost Component Repository product, part of the ADEO tech products, is used to calculate and publish transfer prices between different internal locations globally. Different business units use different information systems. It is hard to learn how each business unit shares information about its systems, API and accuracy.
The initial solution was a developer portal with a list of all applications and reference to dedicated documentation. Some legacy systems had docs written in MS Excel.
There was a need for a standart way of describing event-driven architecture.
Solution
The API is now described with AsyncAPI. The AsyncAPI file, stored with the source code, generates HTML documentation in the same release pipeline for the product. Documentation is exposed internally as part of the product for other company units depending on the API.
Payloads are described with Avro schema. These schemas generate models and are referenced directly in AsyncAPI files thanks to the schemaFormat
feature and $ref
. This way, they make sure the code is aligned with the docs.
Shift to using AsyncAPI also enables the team to implement more use cases using AsyncAPI files.
Use Case
Document the API of the product, so its users know how it works and how to use it. AsyncAPI was selected as the standard that allows you to generate documentation from a machine-readable document that describes the API. The goal was to document API in a standardized way, so other internal products could follow to unify how APIs are documented across the company.
More Details
Testing strategy
Approach to code generation
Java models generation. Avro schemas used as a source.
Architecture
The following enterprise integration patterns are applied:
- Request/Reply
Described with
description
field in AsyncAPI. Reply goes to dedicated reply channel. Example description of response channel:1description: > 2This topic is used to REPLY Costing Requests and is targeted by the 3`REPLY_TOPIC` header.
- Return Address
Info that needs to be provided by the client so producer knows where to send a response. Information is sent in the message header with the
REPLY_TOPIC
property. The AsyncAPI file documents information as part of the Message Header object. Example of request message header withREPLY_TOPIC
:1headers: 2 type: object 3 required: 4 - REPLY_TOPIC 5 properties: 6 REPLY_TOPIC: 7 $ref: "#/components/schemas/ReplyTopic"
- Correlation Identifier
This pattern enables the identification of the request given to the sent response. The
REQUEST_ID
property is in the request message header. TheCORRELATION_ID
property is in the response message header. Both headers are described in the AsyncAPI Message Header object and referred to in the AsyncAPIcorrelationID
property. This means that correlation identifier is represented by different property in the message header, depending if it is a request or reply. Example of request message header withREQUEST_ID
:Example of how1headers: 2 type: object 3 required: 4 - REQUEST_ID 5 properties: 6 REQUEST_ID: 7 $ref: "#/components/schemas/RequestId"
correlationId
points toREQUEST_ID
:1correlationId: 2 description: > 3 This correlation ID is used for message tracing and messages 4 correlation. 5 This correlation ID is generated at runtime based on the `REQUEST_ID` 6 and sent to the RESPONSE message. 7 location: $message.header#/REQUEST_ID
- DeadLetter Channel Also known as Dead Letter Queue. In Kafka, it is just another channel where undelivered messages are sent. Not part of the AsyncAPI file, as API consumers will not listen to this channel. Consumers know what happens with wrong events.
- Invalid Message Channel Invalid messages are routed to the dedicated channel for rejected requests but are not part of the AsyncAPI file, as API consumers will not listen to this channel. Consumers know what happens with wrong events.
More Details about AsyncAPI
How AsyncAPI documents are stored
Git repository where source code is.
Where maintainers edit AsyncAPI documents
IntelliJ without any special plugins. Sometimes people use AsyncAPI Studio, but not regularly because of lack of support for references to local drive.
What extensions are used
Extensions are used to describe details about custom security:
1 x-sasl.jaas.config: >-
2 org.apache.kafka.common.security.plain.PlainLoginModule required
3 username="<CLUSTER_API_KEY>" password="<CLUSTER_API_SECRET>";
4 x-security.protocol: SASL_SSL
5 x-ssl.endpoint.identification.algorithm: https
6 x-sasl.mechanism: PLAIN
How documentation is generated
Documentation generated from AsyncAPI is hosted as part of the product on a dedicated endpoint using Spring controller. Publishing is part of the CI/CD pipeline for the product using GithubActions.
Related Maven configuration used to trigger docs generation with AsyncAPI Generator industry:
1 <profile>
2 <id>generate-asyncapi-doc</id>
3 <build>
4 <plugins>
5 <plugin>
6 <groupId>com.github.eirslett</groupId>
7 <artifactId>frontend-maven-plugin</artifactId>
8 <!-- Use the latest released version:
9 https://repo1.maven.org/maven2/com/github/eirslett/frontend-maven-plugin/ -->
10 <version>${frontend-maven-plugin.version}</version>
11 <configuration>
12 <nodeVersion>v12.18.4</nodeVersion>
13 <installDirectory>${node.installation.path}</installDirectory>
14 <workingDirectory>${node.installation.path}</workingDirectory>
15 </configuration>
16 <executions>
17 <execution>
18 <id>install node and npm</id>
19 <goals>
20 <goal>install-node-and-npm</goal>
21 </goals>
22 <phase>generate-resources</phase>
23 </execution>
24 <execution>
25 <id>install @asyncapi/generator globally</id>
26 <goals>
27 <goal>npm</goal>
28 </goals>
29 <configuration>
30 <arguments>install @asyncapi/generator@${asyncapi.generator.version}</arguments>
31 </configuration>
32 </execution>
33 </executions>
34 </plugin>
35 <plugin>
36 <groupId>org.codehaus.mojo</groupId>
37 <artifactId>exec-maven-plugin</artifactId>
38 <version>1.6.0</version>
39
40 <executions>
41 <execution>
42 <id>execute-generation</id>
43 <goals>
44 <goal>exec</goal>
45 </goals>
46 <phase>generate-resources</phase>
47
48 <configuration>
49 <!-- Access binary file in node_modules because it doesn't work on windows otherwise. -->
50 <executable>${node.modules.installation.path}/${ag.binary.name}</executable>
51 <commandlineArgs>
52 ${project.basedir}/src/docs/asyncapi/asyncapi.yaml @asyncapi/html-template@${asyncapi.htmltemplate.version} -p sidebarOrganization=byTags -p
53 version=${project.version} -o ${asyncapi.generation.dir}
54 </commandlineArgs>
55 </configuration>
56 </execution>
57 </executions>
58 </plugin>
59 <plugin>
60 <groupId>org.apache.maven.plugins</groupId>
61 <artifactId>maven-resources-plugin</artifactId>
62 <executions>
63 <execution>
64 <id>copy-resources</id>
65 <!-- here the phase you need -->
66 <phase>generate-resources</phase>
67 <goals>
68 <goal>copy-resources</goal>
69 </goals>
70 <configuration>
71 <outputDirectory>${asyncapi.generation.dir}/assets</outputDirectory>
72 <resources>
73 <resource>
74 <directory>src/docs/asyncapi/assets</directory>
75 <filtering>true</filtering>
76 </resource>
77 </resources>
78 </configuration>
79 </execution>
80 </executions>
81 </plugin>
82 </plugins>
83 </build>
84 </profile>
Critical features of AsyncAPI related to documentation:
- use of
version
parameter in the generator command to display the release version from theMaven
pom descriptions
that supportCommonMark
(Markdown) as they allow to put detailed structured descriptions and screenshots inside generated docs- examples and validation information. In this case converted from Avro to JSON Schema to show it in documentation and have examples generated
Tags
for tagging operations to categorize them to make it easier to navigate in documentation UI
What bindings are used
All Kafka bindings are used. Server, channel, operation and message bindings.
Example of server bindings:
1bindings:
2 kafka:
3 schema.registry.url: >-
4 https://schema-registry.prod.url/
Example of channel bindings:
1bindings:
2 kafka:
3 replicas: 3
4 partitions: 3
5 cleanup.policy: delete
6 retention.ms: 7 days
Example of operation bindings:
1bindings:
2 kafka:
3 groupId:
4 type: string
5 description: >
6 The groupId must be prefixed by your `svc` account, deliver by the
7 Adeo Kafka team.
8 This `svc` must have the write access to the topic.
9 value.subject.name.strategy:
10 type: string
11 description: >
12 We use the RecordNameStrategy to infer the messages schema.
13 Use
14 `value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
15 in your producer configuration.
Example of message bindings:
1bindings:
2 kafka:
3 key:
4 $ref: "https://asyncapi.com/resources/casestudies/adeo/CostingResponseKey.avsc"
What tools are used
- AsyncAPI Generator:
- HTML Template with parameters like
sidebarOrganization=byTags
andversion
.
- HTML Template with parameters like
- AsyncAPI JavaScript Parser with Avro Schema Parser.
Schemas
Storage strategy
Git repository where source code is. During release they are published to Confluent Schema Registry.
Schema Registry
Confluent Schema Registry.
Versioning of schemas
Versioning is based on git tags. The schema version pushed to Confluent Schema Registry
matches the git tag version of the product. Every schema has a version
information
that matches with product tag version.
Example Avro schema with version information:
1{
2 "namespace": "com.adeo.casestudy.costingrequest",
3 "type": "record",
4 "name": "CostingRequestPayload",
5 "version": "1.1.0",
6 "fields": [ ... ]
7}
Validation of message schemas
Based on validation using Confluent Schema Registry.
Additional Resources
Watch this video presentation about AsyncAPI case study from Ludovic Dussart, Ineat & Antoine Delequeuche, Adeo.
Production-use AsyncAPI document
1asyncapi: 3.0.0
2info:
3 title: Adeo AsyncAPI Case Study
4 version: "%REPLACED_BY_MAVEN%"
5 description: >
6 This Adeo specification illustrates how ADEO uses AsyncAPI to document some of their exchanges.
7 contact:
8 name: AsyncAPI Community
9 email: info@asyncapi.io
10 tags:
11 - name: costing
12 description: "Costing channels, used by Costing clients."
13servers:
14 production:
15 host: "prod.url:9092"
16 protocol: kafka-secure
17 description: Kafka PRODUCTION cluster
18 security:
19 - $ref: '#/components/securitySchemes/sasl-ssl'
20 bindings:
21 kafka:
22 schemaRegistryUrl: >-
23 https://schema-registry.prod.url/
24 staging:
25 host: "staging.url:9092"
26 protocol: kafka-secure
27 description: Kafka STAGING cluster for `uat` and `preprod` environments
28 security:
29 - $ref: '#/components/securitySchemes/sasl-ssl'
30 bindings:
31 kafka:
32 schemaRegistryUrl: >-
33 https://schema-registry.staging.url/
34 dev:
35 host: "dev.url:9092"
36 protocol: kafka-secure
37 description: Kafka DEV cluster for `dev` and `sit` environments
38 security:
39 - $ref: '#/components/securitySchemes/sasl-ssl'
40 bindings:
41 kafka:
42 schemaRegistryUrl: >-
43 https://schema-registry.dev.url/
44channels:
45 costingRequest:
46 address: "adeo-{env}-case-study-COSTING-REQUEST-{version}"
47 description: >
48 Use this topic to do a Costing Request to Costing product.
49 We use the
50 [**RecordNameStrategy**](https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#subject-name-strategy)
51 to infer the messages schema.
52 You have to define `x-value.subject.name.strategy` to
53 `io.confluent.kafka.serializers.subject.RecordNameStrategy` in your
54 producer to use the schema we manage.
55 The schema below illustrates how Costing Request messages are
56 handled.
57 
58 parameters:
59 env:
60 $ref: "#/components/parameters/Env"
61 version:
62 $ref: "#/components/parameters/Version"
63 bindings:
64 kafka:
65 replicas: 3
66 partitions: 3
67 topicConfiguration:
68 cleanup.policy: [ "delete" ]
69 retention.ms: 604800000
70 messages:
71 costingRequest:
72 $ref: "#/components/messages/costingRequestV1"
73 costingResponse:
74 address: "adeo-{env}-case-study-COSTING-RESPONSE-{version}"
75 description: >
76 This topic is used to REPLY Costing Requests and is targeted by the
77 `REPLY_TOPIC` header.
78 **You must grant PUBLISH access to our `svc-ccr-app` service account.**.
79 We use the
80 [**RecordNameStrategy**](https://docs.confluent.io/platform/current/schema-registry/serdes-develop/index.html#subject-name-strategy)
81 to infer the messages schema.
82 You have to define `key.subject.name.strategy` and
83 `x-value.subject.name.strategy` to
84 `io.confluent.kafka.serializers.subject.RecordNameStrategy` in your
85 consumer.
86 The schema below illustrates how Costing Response messages are
87 handled.
88 
89 parameters:
90 env:
91 $ref: "#/components/parameters/Env"
92 version:
93 $ref: "#/components/parameters/Version"
94 bindings:
95 kafka:
96 x-key.subject.name.strategy:
97 type: string
98 description: >
99 We use the RecordNameStrategy to infer the messages schema.
100 Use
101 `x-key.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
102 in your consumer configuration.
103 x-value.subject.name.strategy:
104 type: string
105 description: >
106 We use the RecordNameStrategy to infer the messages schema.
107 Use
108 `x-value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
109 in your consumer configuration.
110 messages:
111 costingResponse:
112 $ref: "#/components/messages/costingResponse"
113operations:
114 requestCosting:
115 action: receive
116 channel:
117 $ref: '#/channels/costingRequest'
118 reply:
119 channel:
120 $ref: '#/channels/costingResponse'
121 address:
122 location: '$message.header#/REPLY_TOPIC'
123 summary: |
124 [COSTING] Request one or more Costing calculation for any product
125 description: >
126 You can try a costing request using our [Conduktor producer
127 template](https://conduktor.url)
128 tags:
129 - name: costing
130 bindings:
131 kafka:
132 groupId:
133 type: string
134 description: >
135 The groupId must be prefixed by your `svc` account, deliver by the
136 Adeo Kafka team.
137 This `svc` must have the write access to the topic.
138 x-value.subject.name.strategy:
139 type: string
140 description: >
141 We use the RecordNameStrategy to infer the messages schema.
142 Use
143 `x-value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy`
144 in your producer configuration.
145 getCostingResponse:
146 action: send
147 channel:
148 $ref: '#/channels/costingResponse'
149 summary: >
150 [COSTING] Get the costing responses matching an initial Costing
151 Request.
152 bindings:
153 kafka:
154 groupId:
155 type: string
156 description: >
157 The groupId must be prefixed by your `svc` account, deliver by the
158 Adeo Kafka team.
159 This `svc` must have the read access to the topic.
160 tags:
161 - name: costing
162
163components:
164 correlationIds:
165 costingCorrelationId:
166 description: >
167 This correlation ID is used for message tracing and messages
168 correlation.
169 This correlation ID is generated at runtime based on the `REQUEST_ID`
170 and sent to the RESPONSE message.
171 location: $message.header#/REQUEST_ID
172 messages:
173 costingRequestV1:
174 name: CostingRequestV1
175 title: Costing Request V1
176 summary: Costing Request V1 inputs.
177 tags:
178 - name: costing
179 correlationId:
180 $ref: "#/components/correlationIds/costingCorrelationId"
181 headers:
182 type: object
183 required:
184 - REQUESTER_ID
185 - REQUESTER_CODE
186 - REQUEST_ID
187 - REPLY_TOPIC
188 properties:
189 REQUEST_ID:
190 $ref: "#/components/schemas/RequestId"
191 REPLY_TOPIC:
192 $ref: "#/components/schemas/ReplyTopic"
193 REQUESTER_ID:
194 $ref: "#/components/schemas/RequesterId"
195 REQUESTER_CODE:
196 $ref: "#/components/schemas/RequesterCode"
197 payload:
198 schemaFormat: application/vnd.apache.avro;version=1.9.0
199 schema:
200 $ref: "https://www.asyncapi.com/resources/casestudies/adeo/CostingRequestPayload.avsc"
201 costingResponse:
202 name: CostingResponse
203 title: Costing Response
204 summary: Costing Response ouputs.
205 tags:
206 - name: costing
207 description: >
208 Please refer to the `CostingResponseKey.avsc` schema, available on [our
209 github
210 project](https://github.url/).
211 correlationId:
212 $ref: "#/components/correlationIds/costingCorrelationId"
213 headers:
214 type: object
215 properties:
216 CALCULATION_ID:
217 $ref: "#/components/schemas/MessageId"
218 CORRELATION_ID:
219 $ref: "#/components/schemas/CorrelationId"
220 REQUEST_TIMESTAMP:
221 type: string
222 format: date-time
223 description: Timestamp of the costing request
224 CALCULATION_TIMESTAMP:
225 type: string
226 format: date-time
227 description: Technical timestamp for the costing calculation
228 #bindings:
229 # kafka:
230 # key:
231 # $ref: "https://deploy-preview-921--asyncapi-website.netlify.app/resources/casestudies/adeo/CostingResponseKey.avsc"
232 payload:
233 schemaFormat: application/vnd.apache.avro;version=1.9.0
234 schema:
235 $ref: "https://deploy-preview-921--asyncapi-website.netlify.app/resources/casestudies/adeo/CostingResponsePayload.avsc"
236 schemas:
237 RequesterId:
238 type: string
239 description: The Costing requester service account used to produce costing request.
240 example: svc-ecollect-app
241 RequesterCode:
242 type: string
243 description: >-
244 The Costing requester code (generally the BU Code). The requester code
245 is useful to get the dedicated context (tenant).
246 example: 1
247 MessageId:
248 type: string
249 format: uuid
250 description: A unique Message ID.
251 example: 1fa6ef40-8f47-40a8-8cf6-f8607d0066ef
252 RequestId:
253 type: string
254 format: uuid
255 description: >-
256 A unique Request ID needed to define a `CORRELATION_ID` for exchanges,
257 which will be sent back in the Costing Responses.
258 example: 1fa6ef40-8f47-40a8-8cf6-f8607d0066ef
259 CorrelationId:
260 type: string
261 format: uuid
262 description: >-
263 A unique Correlation ID defined from the `REQUEST_ID` or the
264 `MESSAGE_ID` provided in the Costing Request.
265 example: 1fa6ef40-8f47-40a8-8cf6-f8607d0066ef
266 BuCode:
267 type: string
268 description: The Business Unit code for which data are applicable.
269 example: 1
270 ReplyTopic:
271 type: string
272 description: >
273 The Kafka topic where to send the Costing Response. This is required for
274 the [Return Address EIP
275 pattern](https://www.enterpriseintegrationpatterns.com/patterns/messaging/ReturnAddress.html).
276 **You must grant WRITE access to our `svc-ccr-app` service account.**
277 example: adeo-case-study-COSTING-RESPONSE-V1
278 ErrorStep:
279 type: string
280 description: |
281 The woker that has thrown the error.
282 example: EXPOSE_RESULT
283 ErrorMessage:
284 type: string
285 description: |
286 The error message describing the error.
287 example: Error message
288 ErrorCode:
289 type: string
290 description: |
291 The error code.
292 example: CURRENCY_NOT_FOUND
293 parameters:
294 Env:
295 description: Adeo Kafka Environement for messages publications.
296 enum:
297 - dev
298 - sit
299 - uat1
300 - preprod
301 - prod
302 Version:
303 # no more sschema
304 # V2 - https://v2.asyncapi.com/docs/reference/specification/v2.6.0#parameterObject
305 # V3 - https://www.asyncapi.com/docs/reference/specification/v3.0.0#parameterObject
306 description: the topic version you want to use
307 examples:
308 - V1
309 default: V1
310 securitySchemes:
311 sasl-ssl:
312 type: plain
313 x-sasl.jaas.config: >-
314 org.apache.kafka.common.security.plain.PlainLoginModule required
315 username="<CLUSTER_API_KEY>" password="<CLUSTER_API_SECRET>";
316 x-security.protocol: SASL_SSL
317 x-ssl.endpoint.identification.algorithm: https
318 x-sasl.mechanism: PLAIN
319 description: >
320 Use [SASL authentication with SSL
321 encryption](https://docs.confluent.io/platform/current/security/security_tutorial.html#configure-clients)
322 to connect to the ADEO Broker.