APIPosted by Craig Hughes Fri, December 14, 2018 15:58:22
In one of my
previous articles, the Hollywood Principle, I introduced the concept of an
event-driven architecture. This is a popular distributed asynchronous architecture
pattern used to produce highly scalable applications. It is underpinned by asynchronous
messaging and often implemented using the Publish and Subscribe (PubSub)
pattern. The premise of this pattern is that an event triggers messages for
distribution to multiple subscribers. However, in order to implement an
effective event-driven architecture, it is important to understand what an event
is in context.
the first type of Event. Application developers, and more specifically UI
developers are all too familiar with concept of an event. A user clicking a
button or moving a mouse is considered an event. Each event has an associated
event handler to perform the specific action anticipated by the user. We’ll
call these UI events and exclude them from this discussion since they are generally
handled within the UI container.
some form of business process, an artefact may be produced (a file for example)
which needs to be handed over to another system. The system creating the
artefact needs to let the other system know that that artefact is ready for
further processing. This can be considered an event. Likewise, a UI event
handler may need to trigger a process in another system. These can also be
considered events. We’ll call these process events.
have the scenario where a business domain or entity is updated or created, via some
process or another. The update of this entity can be considered an event since this
update may be relevant or of importance to other systems. A good example of
these could include a customer update, a payment or an account update. We’ll
call these business events.
events nearly always involve only two parties; the producer of the event and
the consumer of the event message. The producer of the event needs to be aware
of the consumer of the event message. To support asynchronous behavior these
events should be transported via standard messaging. Using PubSub is overkill
in this context and should not be advocated as it clutters the PubSub topic range.
events are the ideal contenders for using PubSub since these are business
activities that others may care about.
through an example – an online payment.
UI channel, the user clicks the submit payment button. This “click” (UI event)
might be handled by a function that validates all the required fields are populated.
All these activities are encapsulated with the UI. Assuming successful
validation, the payment request might be submitted to a payments engine asynchronously
(a process event) since the payment may only be cleared later.
payments engine, the payment request is processed. This process might involve using
several external systems, including funds check, AML and fraud checks. Each of
these systems might raise an event that other systems might care about (analytics,
alerts, etc.). These systems might publish these events via PubSub using a
topic relevant to the event: “overdrawn”, suspect persons” and “suspicious behavior”.
Finally, the payment is cleared – this will trigger the final event – and publish
the transaction via PubSub for other systems to react to (account balance update,
balance SMS, analytics, etc.).
events can be a very powerful architectural pattern – it promotes real-time responses
and reduces “overnight” processing. But, it is important to know which messaging
pattern to implement.
rawpixel.com from Pexels
APIPosted by Craig Hughes Fri, December 07, 2018 16:13:30
Please note that this is a little more
technical than my previous articles – sometimes I need to let my inner-geek out.
because RESTful APIs are based on resources and use the HTTP verbs (GET, POST,
PUT, DELETE, PATCH), does not mean they should only support CRUD (Create, Read,
Update, Delete) operations. RESTful APIs can also be used for performing other actions
challenge with this recently when explaining the concepts of a layered API
architecture (see my previous
article for more information) during a Domain Driven Design
workshop. Core APIs, which expose core domains objects, support pure CRUD
operations and make full use of the HTTP verbs. These are well within the perceived
CRUD definitions of RESTful services. Process and experience APIs, which
orchestrate or compose core APIs to create a defined process, tend to move away
from this understanding; and are sometimes a cause for confusion. What HTTP
verb should be used for an operation outside of the CRUD set and how should we graft
with the basics. The naming convention for resources in RESTful APIs should be
noun based; we work with “accounts”, “customers”, “products”, etc. The basic
CRUD operations we perform on these resources are defined by the HTTP verbs.
So, for example:
- gets a list of customers
- creates a new customer based on the representation in the body of the request
- replaces the customer identified by “123” with the representation in the body
of the request
PATCH http://www.api.com/customers/123 - updates
the customer identified by “123” with the representation in the body of the
- deletes a customer identified by “123”.
The PUT, PATCH
and DELETE HTTP verbs imply very specific Update and Delete actions in the
realm of HTTP; they will not work for activity-based APIs. A GET request, only consists
of the URL to fulfill the HTTP request; you cannot provide content in the
request body (we’ll ignore the use of HTTP Headers for now – they serve a
completely different purpose). A POST request allows you to provide content
within the request body, so is best placed for an activity-based API.
Now that we’ve
identified that the HTTP POST verb is the appropriate choice, how then do we graft
the URL to identify what activity (“action”) we want to perform on the resource?
driven approach to URL design is often used: http://www.api.com/authenticateCustomer
However, while this seems to make sense,
in my view, it moves away from the resource orientation of RESTful URLS. You may
struggle to consider one resource in isolation, but rather have to be aware of
multiple related or interacting resources and actions. This can lead to a plethora
of inconsistent URL patterns and approaches – making it difficult for developers
to learn how to use your APIs.
a resource-based URL representation is surprisingly simple. I use a verb-based terms
to identify the action I want to perform on the resource in question. The URL
is grafted by appending the verb to the resource-based URL. Using the POST
method, I can provide the details required for the activity in the request
consider: POST http://www.api.com/customers/123/authenticate - “authenticate” is the action I want to
execute on the customer identified by “123”. The request body contains a
representation of the customer required for the authentication.
important principle to be aware of here is that resources MUST be noun-based
(describe the resource) and actions MUST be verb-based (describe the action).
definition and for clarity, REST (Representational State Transfer) is a
software architecture style that defines a set of constraints to be used when
creating web services. These constraints include:
1. A client-server architecture which
supports the separation of concerns principle – allowing components to evolve
2. A stateless client-server
communication – where each request from the client contains all the data
required to service the request. No client context is stored on the server.
3. Responses must, implicitly or
explicitly, define themselves as cacheable or not to prevent clients from
getting stale or inappropriate data in response to further requests.
4. A layered system where the client
cannot ordinarily tell whether it is connected directly to the end server, or
to an intermediary along the way.
5. A uniform interface is fundamental
to the design of any RESTful system. It simplifies and decouples the
architecture, which enables each part to evolve independently.
based is inherent in RESTful APIs and the standard set of HTTP verbs tend to imply
CRUD operations only; so, I do understand were the misconception comes from.
APIPosted by Craig Hughes Fri, November 30, 2018 08:56:01
A good friend of mine was out walking his dog in the country side the other day. After a while, his dog noticed some turkeys in the adjacent field, and being a dog, decided to investigate. This involved some barking and general running up and down along the fence. I suppose, in the dog’s mind, he was telling the turkeys that he was defending his master and they should not come any closer. The turkeys responded with a cacophony of gobbling and rapid gurgling. The dog got scared and now has a phobia for turkeys…
This often happens when we want to share a message to a wider audience. We have something we want to say. We’ve identified the audience. We start to speak and the response is completely different to what we expected. The dog, barking at the turkeys, expects them to run away. The turkeys do the opposite, running back towards the dog and shouting “Hey! New Guy” (in turkey talk). Leaving the dog scared and confused.
To give you an example from my daily work with APIs; I was involved in a meeting recently where I ended up in a divergent debate with another architect on what the profile of a customer meant. Turns out, the opposing opinions where brought about by lack of context; I was assuming a retail customer (person) while he was assuming a corporate customer (organization). Context in any communication is crucial.
Does this mean we should start every conversation by giving some context? Yes, probably a good idea, but I believe it may be more relevant to start by assuming your audience does not have the same context as you. If you begin with that context (pun intended), you will probably begin by providing context to your audience before diving into the details.
This is also the approach we should take with APIs. During verbal conversation, we have the opportunity to change tack or adjust our speech to provide explanation or context. With APIs, we don’t have this luxury. REST APIs are based on resources. Resource sometimes need context to have more meaning. Encapsulating this resource within some context could offer meaning, but might create a confusing data contract. Ideally, we should provide context via the URL.
Let’s look at an example. Assuming I need to consume an API that presents transactions on my account. The URL https://api.somedomain.com/transactions could return this list of transactions, but they have no context – they could be for anyone’s accounts. We could provide context to these transactions by including the account these transactions belong to. For example: https://somedomain.com/accounts/12345/transactions makes the statement that these transactions belong to the account identified by “12345”. It’s a simple and seemingly innocuous approach, it can be very powerful, but often forgotten or missed.
APIPosted by Craig Hughes Fri, November 09, 2018 15:52:45
for the first time in history, we registered over a billion passenger cars on our
roads, globally. It is estimated that by 2050, there will be 2.5 billion cars
on our roads! Clearly, cars are a big part of our lives and will continue to
stay for a while.
said, have you ever opened the bonnet of your car and had a look inside the
engine compartment? It looks incredibly complex, with many individual parts each
responsible for their own task. Look underneath your car the next time it’s
raised up on a lift and notice how complicated the drivetrain and suspension are.
Yet, despite being a massively complex machine, cars have become a fundamental
part of lives – most of us are able to operate them with relative ease.
this possible are the incredibly simple interfaces cars use to hide us from the
complex internals of the car. The steering wheel allows us it change the
direction of the car. The pedals allow us to change the speed (accelerate of decelerate)
of the car. In fact, all the human interfaces within the car allow us to
operate this incredibly complex piece of machinery with relative ease, by
following simple standards.
adopt the same principle. The interface you expose to your API consumers should
be as simple and easy as possible to understand. Complex internal integrations
and business logic should not be exposed to your consumer – this is called
abstraction; and it’s something the car does incredibly well. What it basically
means is that a consumer of your API should not have to be concerned with
understanding the internals of your API in order to use it.
That is not
to say they should be completely oblivious or ignorant of the boundaries or relevant
ancillary requirements of your API. Just like a driver of a car needs to know
that a car needs fuel to run, air in its tires, keys to start, etc., so to should
the consumer of your API be aware of any external constraints like intended
audience, security, data structure, etc.
important aspect of a car that comes to mind is modularization, and I’ve mentioned
it already. The complexity of a car is the sum of all its components working
together; each performing their own unique function. Your API could be composed
of internal components, orchestrated or composed to produce the functionality
defined. I’ve discussed in my previous articles on “Making coffee to explain
APIs” and “Most of us think we know what an API is but cannot seem to agree on
what an API should be”
rawpixel.com from Pexels
APIPosted by Craig Hughes Thu, November 01, 2018 13:15:33
has at least one set of keys; don’t they? You can recognize your keys and you make
sure you take them wherever you go. Look at each key on your keyring and you know
which door or cabinet it opens (well, we should know anyway). Some of us have
keys we no longer know the purpose of – what we do know is that the lock for
that key is probably useless without it.
the content or function they provide can also be protected by one or more keys.
Each key has a purpose and it’s important to understand what each key is for.
The only difference here, is we often refer to these keys as tokens.
These are used to protect the API endpoint from consumption by non-registered
consumers. They are usually issued by an API gateway, who’s responsibility it
is to manage the APIs published via them. API keys are issued to the person or
organization who intends to use the API for their application – this key allows
them to access the API endpoint, but not necessarily access to a user’s data or
to execute a user function behind the endpoint. That’s potentially the purpose
of another key – the JWT (JSON Web Token).
perform a number of key functions (pun intended). They can be used to provide
the information of an authenticated user as well as their permissions within
the context of their authentication. They can also include custom data and other
values relevant to the API endpoint, including relevant time bounds, issuer,
subject and intended audience of the request.
To ensure JWTs
are URL safe, they are transported as base64 encoded strings. This means that although
they may not seem readable, any decoder can reconstitute them as the characters
they represent (JSON object); well, almost. A single JWT is made up of three
- the header contains information about
the JWT including the algorithm used to sign it – this part is readable.
- the body contains the actual payload
(claims) of the JWT as a JSON object – this part is also readable.
- the signature of the JWT – this is
not readable but used to verify that the header and body of the JWT have not
been tampered with (changed).
important to understand this – simply because the JWT being sent seems illegible,
does not mean is it. JWTs protect the integrity of the information inside them,
not the visibility. So, please don’t think they can contain sensitive data – that’s
another kind of key; an encrypted JWT or JWE (JSON Web Encryption).
started to get technical; let’s bring it back up a level.
A JWT carries
the information about what an authenticated user can do, their authorization. This
information is signed using a defined algorithm and a secret key (yep, another
one…), thereby enforcing its integrity. Essentially, we use a JWT to transport information
about the user, their permissions, and any other details about the request in a
way that any changes can be recognized.
In the same
way that we can look at the keys on our keyring and recognize what each is for,
we can also see the contents of a JWT. Like the profile of a key needs to
line-up with the tumblers in a lock in order to open it, the API producer
should use the profile (user and scopes) of the JWT to decide whether access to
the resource should be allowed. If you tamper with the profile of a key, it
will stop opening its corresponding lock. Tamper with the contents of a JWT and
the API producer should not trust it and therefore not allow access.
you would not leave your house keys outside your front door, so please don’t leave
your API keys where they can be used.
Photo by PhotoMIX Ltd. from Pexels
APIPosted by Craig Hughes Wed, October 17, 2018 14:21:53
Have you’ve ever visited India? How did you cope with the traffic there? It’s clearly different to the regular, organised approach we are used to in Europe.
After spending a couple of days in Bangalore recently, I became aware that my reaction to traffic had changed; I was not fazed (read terrified) by traffic behaviour anymore and actually started to notice how well it seemed to work. This surprised me and got me thinking: “is this an example of choreography and if so, is driving in Europe orchestration?” Let me explain.
In Europe (as in most of the world), traffic is managed by lanes, signs and traffic lights. Obeying these is paramount and heavy fines are imposed on those who flout the law. Everything seems to flow in unison and order seems to be present. But is it? What happens if we don’t respond to instructions in time, or don’t follow proper lane etiquette? Traffic blockages and potentially; road rage.
Counter that with the scenario in India; everyone seems to react to changes by other drivers, lanes seem to be indicators of general direction and traffic signals are only placed on major intersections (sometimes enforced by a traffic official). This apparent chaos often scares those of us not form the region, but yet it still seems to work.
APIs, by nature and design, decouple us from the complexity of the underlying systems. Inversely, when we expose our data via APIs we cannot expose our complexity (like we may have done in the past). So, with APIs, the general practice seems to follow our driving behaviour; we want to control what and who we share our data with. We want to be the masters of orchestration to maintain control. We want to feel like we are still empowered.
Choreography is a completely different approach, here we are no longer the master of what happens to our data or who uses it. As I mentioned in my article on the Hollywood Principle; we expose our data as events when our work is done. Essentially, we’re sharing the result of that piece of work with the world – “I’ve done something you may want to know about”. How and what the consumers do with the data should be of no consequence to me, but it feels like I’ve lost control. Consumers (or subscribers) react to my data changes – each choreographing their own behaviour. It’s synonymous with driving in India.
Now, I am by no means saying we should abandon all structure and formality and start driving like we are in India – that was merely a metaphor to explain choreography. Perceived chaos seems to work in India but would not necessarily work in Europe. An event driven architecture, which induces choreography, has its place, but then so does orchestration – it depends on the agenda, use and purpose of the integration. I find the problem is that people appear to be afraid of choreography, because they cannot control it, and therefore dismiss a reactive event driven architecture and the opportunities that go with it.
In summary, Orchestration is the process where a single API consumer gathers information from various API endpoints, using the data received from initial calls to make further API calls to other endpoints. Choreography is the process where multiple subscribers react to a single event, using the data received for their own purposes.
I introduced the concept of a layered API architecture in a previous article where I mentioned the creation of core domain API. These core domain API, being data-based, are in my mind the best data components for sharing in an event driven architecture – since they describe a single entity within a single domain – allowing maximum coverage by multiple subscribers.
Finally, what about composition - another strategy used with APIs which I have not mentioned? Using the traffic metaphor; composition is similar to putting the passengers into the car – a collection of objects (people) defined as one (car). Composition is the process where a single API consumer gathers data from multiple endpoints. However, unlike Orchestration, Composition does not required one or more previous API calls to provide data for subsequent calls.
Image sourced from medium.com
APIPosted by Craig Hughes Wed, October 10, 2018 08:03:47
I was out looking
for a sandwich for lunch the other day and found myself faced by two options; I
could either go to the convenience store and pick up a pre-made sandwich or go
to the deli and have one made for me.
pre-made option sounds simple, but choice it usually limited to the imagination
of the sandwich maker. Going to a deli to have a sandwich made offers freedom
of choice, but I’d have to deal with queues of people trying to make up their
minds about which ingredients they want (including me when I get there).
stores perspective, the pre-made option is easy – display what’s on offer and
let the customer pick one; it’s quick and simple. However, reduced choice may
push customers towards the Deli. The Deli owner can offer his customers choice,
but at the cost of extra staff to make sandwiches based on each customers requirement.
The little effort required by customers and possible delays could push customers
to the convenience store for the “best fit” option.
APIs could follow a similar approach.
an API producer, using the inside-out-approach, I could decide “what” my
consumers need and make a range of specific endpoints for them to consume. If
one endpoint does not cover all their requirements, they could compose the data
they need by consuming a number of my APIs; extracting the bits needed from
each. This will give me full control over my API implementation and allow me to
optimize them for better performance. Eventually however, my consumers will
complain and ask for more “specific” APIs to meet their needs. In my
experience, this is all too often the reality.
As an API producer, using the outside-in-approach, I could offer my consumers
the ability to select which elements in my API domain should be returned. This would
require me to provide proper documentation to explain my domain and I’d need to
implement the selection logic (and consequences) in my services. Like the Deli
needs to maintain the cost of a sandwich maker, I’d need to accept the cost of
maintaining this logic and implementing it for new elements in the future. The benefit,
of course, is that my API will offer complete customization, making it as
consumable as possible by more than one consumer – a desirable trait for APIs.
There is an
alternative – I could provide the full data object in the response. In other
words, provide all the data elements of the object being modelled and let the
customer extract the values needed, ignoring the rest (pun intended). This approach
creates less work for the producer – only one simple endpoint is required per
object. The effort is passed to the consumer, similar to the Deli style approach,
only the selection of elements is made on the response instead of providing
input into the request. Taking advantage of HATEAOS will reduce the payload
size and offer discoverability but could force the consumer to perform
composition in order to get all the data elements required.
So, as API
producers we have options. We can offer our consumers flexibility through
choosing data elements before or after request, or we can offer them what we
think they need – the choice is ours. Each approach has its merits and its consequences,
but they are nonetheless choices.
on APIs that follow all three of these approaches. The Deli style approach offers
benefit to the consumer and reduces payload. The challenge comes in selecting data
elements for the request input in a hierarchical data structure. The approach
we took was simple; use dot.notation to identify the data elements just as you
would when reading the JSON response.
Image provided by Pexels.com
APIPosted by Craig Hughes Sun, September 30, 2018 14:15:56
The general consensus amongst API providers is that we
should build reusable APIs, but consumers want more - they want an API that
gives them everything they need. These tactics challenge one another.
Building for the consumer
Consumers of data or functionality want everything in one single and simple
package. Facilitating this approach can lead to a plethora of bespoke
interfaces each fulfilling only one single user’s requirement. Prospect
consumers will find it hard to find the “right” API and, due to the precedent set,
may ask the API producer to create a new interface for them.
Building for reuse
On the contrary, if an API is too generic then the API consumer will
feel that they have too much work to do. These consumers will have to use composition
and/or orchestration on these generic APIs to get the level of functionality
they need. Issues of performance and the need to understand the underlying
business may become strong arguments for the consumer.
How do we address these two divergent methodologies?
My wife and I do not work in the same industry - she does people and I
do technology. I often write as if I was explaining the topic at hand to her,
but for some reason, this topic has me flummoxed. I’m struggling to explain
this one to my wife in terms she can understand. So, please bear with me, and let me know in your feedback or comments how to simplify the following descriptions.
Domain Driven Design (DDD) allows us to break business domains into
smaller objects called domain entities or value objects – this concept is briefly explained in my previous post on Making
Coffee. In banking we have a number of business domains: customer, account,
product, etc. Customer, in turn, can be represented as a number of domain entities
including: core information, addresses, contact information, etc. The same applies for the other business domains. Domain entities can be exposed as our most reusable
API. We call these APIs core or domain APIs.
However, business practices are not only about managing domain entities, we also need to be able
to use and or manipulate them to fulfil some business process. To facilitate this, we can compose or orchestrate core APIs to perform a defined function, which we expose
as a reusable interface (payments are a good example, see my previous post on The
Butterfly Effect). We call these process or composite APIs.
Finally, we have the user interaction channels. This is where the most
specific APIs are usually required by consumers and where it is ok to have bespoke
APIs for a defined purpose. As long as these APIs are composed of core or
process APIs and do not go direct to our core systems. We call these presentation or experience APIs.
Essentially, we create a layered API Architecture. Core API, the building
blocks, form the foundation and are predominantly data driven APIs. Process APIs offer the reusable business functionality
by manipulating the core API for a defined process. Finally, experience APIs offer custom user
interaction by orchestrating or composing core and/or process APIs for a specific user interaction.
Now, I admit this is overly simplistic and does not cover the
performance argument. That can perhaps be addressed by an event driven
architecture which publishes core domain entities (see the Hollywood
Principle) and implementing the CQRS (Command Query Responsibility
Segregation) pattern. Perhaps the topic for a future discussion.
Image from Pexels.com