Personal API Experience

Personal API Experience

Daily musings

Sharing my day to day lessons, ideas and contemplations of architecting, developing and maintaining services (API) in the global marketplace we call the internet.

API: Arrays or named elements?

APIPosted by Craig Hughes Tue, January 15, 2019 16:58:24

The humble restaurant menu follows a common pattern where menu items are generally separated into a logical hierarchy. At the top level, we may have breakfast items, dinner items or drinks items. Each of these could be grouped further into categories like meat dishes, fish dishes or pasta dishes (dinner items), and again into burgers and steaks (meat dishes). Each item is then described so that we (as humans) can make a choice on what to eat.

Now, consider making that same menu available as an API (for machine consumption). Think of the challenge that becomes immediately apparent; how do we describe objects that contain similar properties? How should we model burgers for example?

The natural human tendency might be to describe each burger as an object with defined properties. We may describe the “cheeseBurger”, the “baconBurger”, the “hawaiianBurger” – this follows the menu approach (the intended business purpose). Alternatively, we could define the “burgers” as an array and identify each burger as an element with that array based on its property values.



There are a number of pros and cons to consider for each approach.

The “Named Object” approach

Pros

· The model is easy to understand (from a human perspective – developers are sometimes human)

· Consumers can navigate directly to the object needed; e.g. json.burgers.baconBurger

· Each object (burger) can evolve in isolation

· Each object can be managed via individual API end-points; e.g. http://api.sample.com/burgers/cheeseBurger

Cons

· Renaming or removing objects will lead to contract-breaking changes – forcing a new API version

· Filtering is usually done on attribute values. Doing so on objects is not intuitive.

· The object name provides identity within the interface schema. This name may need to be translated into an appropriate UI label and also makes localization challenging. The obvious solution would be to include a “label” attribute – but wouldn’t that be redundant?

The “Array” approach

Pros

· We can add, rename or remove objects (burgers) without effecting the interface model (no potential version changes).

· We can still retrieve a defined object via an API end-point; e.g. http://api.sample.com/burgers?type=Cheese%20Burger

Cons

· Consumers may need to loop through the array to find the required object by value.

· Objects have to maintain the same structure and cannot evolve in isolation.

The choice of which approach to take depends on the domain you are modeling. However, as the domain expert, the decision is yours – just understand the consequences of your choice.





  • Comments(0)//blog.craighughes.org/#post14

DDD – what isle is your attribute in?

APIPosted by Craig Hughes Wed, January 09, 2019 18:41:59

During a Domain Driven Design (DDD) workshop this week, one of my colleagues very cleverly used grocery store aisles as an analogy to explain why attributes should be grouped into entities. He explained that grocery stores group logical produce together to make it easier for customers to find what they are looking for. Thereby suggesting that grouping attributes together in entities would help API consumers find relevant data. I really liked this analogy since it highlights some of the challenges of Domain Driven Design.

Have a look next time you go to your local grocery store. The teas, coffees and other hot drinks will be in one aisle. Condiments may be in another, so too will dry goods. Each isle can be seen as a group of items that logically go together. But, and there is always the exception, there are some items that don’t clearly fit into one category; like sugar for example. Should sugar be with the hot drinks, the dry goods or the baking goods? The stores around me can’t seem to agree, but at least they’ve each made a decision and mostly keep to it. As regular shoppers, we begin to know where to find our favorite items and understand the store's layout.

DDD can be a very time-consuming and on-going exercise. Not everyone is going to agree with the models you define, but you can’t go on ad infinitum. Eventually, at some point, you have to make a decision. That decision will be based on the best available knowledge your team has at that point in time. Yes, it may not be 100% right, but it is a decision none-the-less.

Many of these decisions will be around which entities attributes logically belong to. Once all the decisions have been made, you are going to publish your model as an API. Consumers may initially be confused by the placement of one or more attributes and will either accept or challenge them.

Being challenged on a model is not a big deal. The beauty of APIs is that they can be versioned. This allows us a little flexibility in making a decision on our models. A change to the model (that breaks the current contract of your API) can be published as a new API version. That’s not to say we are free to change the model whenever the whim takes us, but, we don’t have to wait for the absolutely perfect model before releasing our APIs. We can start with a minimum viable product (MVP) and evolve it as our knowledge and experience in the domain grows.

The alternative is, of course, to not model your domain at all, and simply publish an API with a long list of root attributes. Imagine trying to shop in a store that randomly puts stuff on shelves without any thought…

Photo by Fancycrave.com from Pexels



  • Comments(0)//blog.craighughes.org/#post13

Fireworks and APIs - “meh”

APIPosted by Craig Hughes Fri, January 04, 2019 14:34:58

Every New Year’s Eve we are wowed by extravagant fireworks from around the world - we anticipate them and we expect them. This New Year’s Eve was no different, except for me – this year I was in Copenhagen city center; on the balcony of a friend’s apartment, watching.

In Denmark anyone, over a certain age, can purchase and set off fireworks during New Year’s Eve. It’s quite spectacular – bangs, fizzes, pops and smoke for a number of hours; explosions are constant and everywhere.

Some of the fireworks are spectacular and clearly cost a small fortune, others are unexceptional, but the result of all of them is the same. Firstly, there’s great expectation followed by excitement in lighting the fuse. Then there’s anticipation; waiting for them to go off, tailed by wonder as the explosion and accompanying light show is released. In the end, there’s the “meh” moment – either because it was the same as the previous one, not as good as the next one, or a little lack-luster when compared to others.

Finally, it’s all done. The show is over, the money spent and everyone goes home.

Since this is a new year, and we generally try to aspire to greater things, let’s try with our APIs. Let’s not simply build excitement and anticipation only to release an API that’s “meh” and soon forgotten. Let’s try to do them right this time.

Understand

Before you begin, make sure you understand your business. Make sure you understand how your API will be used in the context of your business. Make sure your API will be used to grow or enhance your business. And finally, make sure the purpose of your API is clearly understood.

Collaborate

Work with your colleagues, your business, your partners and your consumers. Understand their needs, work through their ideas, grasp their constraints – work with the requirements. Agree the contractual interface (payload, URLs, operations, etc.) and simulate them (using SwaggerHub or similar). Let everyone play and plan ahead with the simulations – there is no need for consumers to wait for the final product. Get feedback early.

Simple and reusable

Keep your APIs simple and try make them useable by as many parties as possible (avoid single consumer builds). Make them intuitive and keep them constrained to a single functional responsibility. If you have to document your APIs too much, they are probably too complicated. Mocking your APIs will help you understand if you’re on the right path.

Loose Coupling

Don’t place assumptions on consumers; don’t assume they understand the complexity of the system behind your APIs – it’s not their responsibility. Your API interface should abstract them from your complexity. After all, managing complexity is part of the service that an API offers.

Manage

Finally, manage you APIs through their lifecycle. Give them an explicit version (using Symantec versioning) and let your consumers know when new versions are available. Of course, if you are already collaborating with your consumers, they should be ready for your API changes.

So, fireworks offer all the excitement, anticipation and wonder; followed by the “meh” moment. Your API’s can offer the same, but try not to aspire to that – try to maintain the feel-good excitement and wonder by delivering something lasting and worthwhile to all.



  • Comments(0)//blog.craighughes.org/#post12

What event are you waiting for?

APIPosted by Craig Hughes Fri, December 14, 2018 15:58:22

In one of my previous articles, the Hollywood Principle, I introduced the concept of an event-driven architecture. This is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. It is underpinned by asynchronous messaging and often implemented using the Publish and Subscribe (PubSub) pattern. The premise of this pattern is that an event triggers messages for distribution to multiple subscribers. However, in order to implement an effective event-driven architecture, it is important to understand what an event is in context.

Let’s consider the first type of Event. Application developers, and more specifically UI developers are all too familiar with concept of an event. A user clicking a button or moving a mouse is considered an event. Each event has an associated event handler to perform the specific action anticipated by the user. We’ll call these UI events and exclude them from this discussion since they are generally handled within the UI container.

Secondly, during some form of business process, an artefact may be produced (a file for example) which needs to be handed over to another system. The system creating the artefact needs to let the other system know that that artefact is ready for further processing. This can be considered an event. Likewise, a UI event handler may need to trigger a process in another system. These can also be considered events. We’ll call these process events.

Finally, we have the scenario where a business domain or entity is updated or created, via some process or another. The update of this entity can be considered an event since this update may be relevant or of importance to other systems. A good example of these could include a customer update, a payment or an account update. We’ll call these business events.

Process events nearly always involve only two parties; the producer of the event and the consumer of the event message. The producer of the event needs to be aware of the consumer of the event message. To support asynchronous behavior these events should be transported via standard messaging. Using PubSub is overkill in this context and should not be advocated as it clutters the PubSub topic range.

Business events are the ideal contenders for using PubSub since these are business activities that others may care about.

Let’s walk through an example – an online payment.

Using the UI channel, the user clicks the submit payment button. This “click” (UI event) might be handled by a function that validates all the required fields are populated. All these activities are encapsulated with the UI. Assuming successful validation, the payment request might be submitted to a payments engine asynchronously (a process event) since the payment may only be cleared later.

Within the payments engine, the payment request is processed. This process might involve using several external systems, including funds check, AML and fraud checks. Each of these systems might raise an event that other systems might care about (analytics, alerts, etc.). These systems might publish these events via PubSub using a topic relevant to the event: “overdrawn”, suspect persons” and “suspicious behavior”. Finally, the payment is cleared – this will trigger the final event – and publish the transaction via PubSub for other systems to react to (account balance update, balance SMS, analytics, etc.).

Reacting to events can be a very powerful architectural pattern – it promotes real-time responses and reduces “overnight” processing. But, it is important to know which messaging pattern to implement.



Photo by rawpixel.com from Pexels



  • Comments(0)//blog.craighughes.org/#post11

Nouns and Verbs – in the world of APIs

APIPosted by Craig Hughes Fri, December 07, 2018 16:13:30

Please note that this is a little more technical than my previous articles – sometimes I need to let my inner-geek out.

Simply because RESTful APIs are based on resources and use the HTTP verbs (GET, POST, PUT, DELETE, PATCH), does not mean they should only support CRUD (Create, Read, Update, Delete) operations. RESTful APIs can also be used for performing other actions on resources.

I was challenge with this recently when explaining the concepts of a layered API architecture (see my previous article for more information) during a Domain Driven Design workshop. Core APIs, which expose core domains objects, support pure CRUD operations and make full use of the HTTP verbs. These are well within the perceived CRUD definitions of RESTful services. Process and experience APIs, which orchestrate or compose core APIs to create a defined process, tend to move away from this understanding; and are sometimes a cause for confusion. What HTTP verb should be used for an operation outside of the CRUD set and how should we graft the URL?

Let’s start with the basics. The naming convention for resources in RESTful APIs should be noun based; we work with “accounts”, “customers”, “products”, etc. The basic CRUD operations we perform on these resources are defined by the HTTP verbs. So, for example:

GET http://www.api.com/customers - gets a list of customers

POST http://www.api.com/customers - creates a new customer based on the representation in the body of the request

PUT http://www.api.com/customers/123 - replaces the customer identified by “123” with the representation in the body of the request

PATCH http://www.api.com/customers/123 - updates the customer identified by “123” with the representation in the body of the request

DELETE http://www.api.com/customers/123 - deletes a customer identified by “123”.

The PUT, PATCH and DELETE HTTP verbs imply very specific Update and Delete actions in the realm of HTTP; they will not work for activity-based APIs. A GET request, only consists of the URL to fulfill the HTTP request; you cannot provide content in the request body (we’ll ignore the use of HTTP Headers for now – they serve a completely different purpose). A POST request allows you to provide content within the request body, so is best placed for an activity-based API.

Now that we’ve identified that the HTTP POST verb is the appropriate choice, how then do we graft the URL to identify what activity (“action”) we want to perform on the resource?

A method driven approach to URL design is often used: http://www.api.com/authenticateCustomer or http://www.api.com/validatePaymentRequest. However, while this seems to make sense, in my view, it moves away from the resource orientation of RESTful URLS. You may struggle to consider one resource in isolation, but rather have to be aware of multiple related or interacting resources and actions. This can lead to a plethora of inconsistent URL patterns and approaches – making it difficult for developers to learn how to use your APIs.

Maintaining a resource-based URL representation is surprisingly simple. I use a verb-based terms to identify the action I want to perform on the resource in question. The URL is grafted by appending the verb to the resource-based URL. Using the POST method, I can provide the details required for the activity in the request body.

For example, consider: POST http://www.api.com/customers/123/authenticate - “authenticate” is the action I want to execute on the customer identified by “123”. The request body contains a representation of the customer required for the authentication.

The important principle to be aware of here is that resources MUST be noun-based (describe the resource) and actions MUST be verb-based (describe the action).

By definition and for clarity, REST (Representational State Transfer) is a software architecture style that defines a set of constraints to be used when creating web services. These constraints include:

1. A client-server architecture which supports the separation of concerns principle – allowing components to evolve independently.

2. A stateless client-server communication – where each request from the client contains all the data required to service the request. No client context is stored on the server.

3. Responses must, implicitly or explicitly, define themselves as cacheable or not to prevent clients from getting stale or inappropriate data in response to further requests.

4. A layered system where the client cannot ordinarily tell whether it is connected directly to the end server, or to an intermediary along the way.

5. A uniform interface is fundamental to the design of any RESTful system. It simplifies and decouples the architecture, which enables each part to evolve independently.

Being resource based is inherent in RESTful APIs and the standard set of HTTP verbs tend to imply CRUD operations only; so, I do understand were the misconception comes from.





  • Comments(0)//blog.craighughes.org/#post10

Turkey Talk and APIs

APIPosted by Craig Hughes Fri, November 30, 2018 08:56:01

A good friend of mine was out walking his dog in the country side the other day. After a while, his dog noticed some turkeys in the adjacent field, and being a dog, decided to investigate. This involved some barking and general running up and down along the fence. I suppose, in the dog’s mind, he was telling the turkeys that he was defending his master and they should not come any closer. The turkeys responded with a cacophony of gobbling and rapid gurgling. The dog got scared and now has a phobia for turkeys…

This often happens when we want to share a message to a wider audience. We have something we want to say. We’ve identified the audience. We start to speak and the response is completely different to what we expected. The dog, barking at the turkeys, expects them to run away. The turkeys do the opposite, running back towards the dog and shouting “Hey! New Guy” (in turkey talk). Leaving the dog scared and confused.

To give you an example from my daily work with APIs; I was involved in a meeting recently where I ended up in a divergent debate with another architect on what the profile of a customer meant. Turns out, the opposing opinions where brought about by lack of context; I was assuming a retail customer (person) while he was assuming a corporate customer (organization). Context in any communication is crucial.

Does this mean we should start every conversation by giving some context? Yes, probably a good idea, but I believe it may be more relevant to start by assuming your audience does not have the same context as you. If you begin with that context (pun intended), you will probably begin by providing context to your audience before diving into the details.

This is also the approach we should take with APIs. During verbal conversation, we have the opportunity to change tack or adjust our speech to provide explanation or context. With APIs, we don’t have this luxury. REST APIs are based on resources. Resource sometimes need context to have more meaning. Encapsulating this resource within some context could offer meaning, but might create a confusing data contract. Ideally, we should provide context via the URL.

Let’s look at an example. Assuming I need to consume an API that presents transactions on my account. The URL https://api.somedomain.com/transactions could return this list of transactions, but they have no context – they could be for anyone’s accounts. We could provide context to these transactions by including the account these transactions belong to. For example: https://somedomain.com/accounts/12345/transactions makes the statement that these transactions belong to the account identified by “12345”. It’s a simple and seemingly innocuous approach, it can be very powerful, but often forgotten or missed.



  • Comments(0)//blog.craighughes.org/#post9

APIs and the Abstraction of Cars

APIPosted by Craig Hughes Fri, November 09, 2018 15:52:45

In 2010, for the first time in history, we registered over a billion passenger cars on our roads, globally. It is estimated that by 2050, there will be 2.5 billion cars on our roads! Clearly, cars are a big part of our lives and will continue to stay for a while.

That being said, have you ever opened the bonnet of your car and had a look inside the engine compartment? It looks incredibly complex, with many individual parts each responsible for their own task. Look underneath your car the next time it’s raised up on a lift and notice how complicated the drivetrain and suspension are. Yet, despite being a massively complex machine, cars have become a fundamental part of lives – most of us are able to operate them with relative ease.

What makes this possible are the incredibly simple interfaces cars use to hide us from the complex internals of the car. The steering wheel allows us it change the direction of the car. The pedals allow us to change the speed (accelerate of decelerate) of the car. In fact, all the human interfaces within the car allow us to operate this incredibly complex piece of machinery with relative ease, by following simple standards.

APIs should adopt the same principle. The interface you expose to your API consumers should be as simple and easy as possible to understand. Complex internal integrations and business logic should not be exposed to your consumer – this is called abstraction; and it’s something the car does incredibly well. What it basically means is that a consumer of your API should not have to be concerned with understanding the internals of your API in order to use it.

That is not to say they should be completely oblivious or ignorant of the boundaries or relevant ancillary requirements of your API. Just like a driver of a car needs to know that a car needs fuel to run, air in its tires, keys to start, etc., so to should the consumer of your API be aware of any external constraints like intended audience, security, data structure, etc.

Another important aspect of a car that comes to mind is modularization, and I’ve mentioned it already. The complexity of a car is the sum of all its components working together; each performing their own unique function. Your API could be composed of internal components, orchestrated or composed to produce the functionality defined. I’ve discussed in my previous articles on “Making coffee to explain APIs” and “Most of us think we know what an API is but cannot seem to agree on what an API should be”

Photo by rawpixel.com from Pexels



  • Comments(0)//blog.craighughes.org/#post8

API Security – Who has the right key?

APIPosted by Craig Hughes Thu, November 01, 2018 13:15:33


Everyone has at least one set of keys; don’t they? You can recognize your keys and you make sure you take them wherever you go. Look at each key on your keyring and you know which door or cabinet it opens (well, we should know anyway). Some of us have keys we no longer know the purpose of – what we do know is that the lock for that key is probably useless without it.

API’s and the content or function they provide can also be protected by one or more keys. Each key has a purpose and it’s important to understand what each key is for. The only difference here, is we often refer to these keys as tokens.

API Keys. These are used to protect the API endpoint from consumption by non-registered consumers. They are usually issued by an API gateway, who’s responsibility it is to manage the APIs published via them. API keys are issued to the person or organization who intends to use the API for their application – this key allows them to access the API endpoint, but not necessarily access to a user’s data or to execute a user function behind the endpoint. That’s potentially the purpose of another key – the JWT (JSON Web Token).

JWT’s can perform a number of key functions (pun intended). They can be used to provide the information of an authenticated user as well as their permissions within the context of their authentication. They can also include custom data and other values relevant to the API endpoint, including relevant time bounds, issuer, subject and intended audience of the request.

To ensure JWTs are URL safe, they are transported as base64 encoded strings. This means that although they may not seem readable, any decoder can reconstitute them as the characters they represent (JSON object); well, almost. A single JWT is made up of three components

  • the header contains information about the JWT including the algorithm used to sign it – this part is readable.
  • the body contains the actual payload (claims) of the JWT as a JSON object – this part is also readable.
  • the signature of the JWT – this is not readable but used to verify that the header and body of the JWT have not been tampered with (changed).

It’s important to understand this – simply because the JWT being sent seems illegible, does not mean is it. JWTs protect the integrity of the information inside them, not the visibility. So, please don’t think they can contain sensitive data – that’s another kind of key; an encrypted JWT or JWE (JSON Web Encryption).

Anyway, we’ve started to get technical; let’s bring it back up a level.

A JWT carries the information about what an authenticated user can do, their authorization. This information is signed using a defined algorithm and a secret key (yep, another one…), thereby enforcing its integrity. Essentially, we use a JWT to transport information about the user, their permissions, and any other details about the request in a way that any changes can be recognized.

In the same way that we can look at the keys on our keyring and recognize what each is for, we can also see the contents of a JWT. Like the profile of a key needs to line-up with the tumblers in a lock in order to open it, the API producer should use the profile (user and scopes) of the JWT to decide whether access to the resource should be allowed. If you tamper with the profile of a key, it will stop opening its corresponding lock. Tamper with the contents of a JWT and the API producer should not trust it and therefore not allow access.

Finally, you would not leave your house keys outside your front door, so please don’t leave your API keys where they can be used.


Photo by PhotoMIX Ltd. from Pexels







  • Comments(0)//blog.craighughes.org/#post7
Next »