Service calls — including web server, API calls, and back-end database calls — are the bread-and-butter of observability. This gives you visibility into the different workloads that the service is performing, and allow you to either measure and categorize the various dimensions inside the request. Typical questions that you may want to ask may include:
The entry point where requests enter and are processed. This can be on a HTTP request handler in a multiplexer, a SOAP-XML function callback or even a socket based TCP call. Occasionally, it might be useful to start instrumenting from the load balancer.
The call itself: the entry point and the response. In a web server, this is the top level request that hit the server and the response it got.
If the front-end service uses other services to fulfill the request, then these can be instrumented. For example, a front-end web service will often be fulfilled by an API service and a database service. If you use a consistent request ID, than the propagating request can be fit into a tracing schema.