Working with logs in Kibana UI

1. User manual

Kibana is a tool that visualizes all application logs. Logs of any virtual environment are recorded into the console and duplicated into Elasticsearch.

To access the records, navigate to Kibana.

You can get the link to Kibana technical logs in the Networking → Routes of Openshift console by specifying service name in the search.

kibana section1 figure0

To do that, take the following steps:

  1. Open the application and navigate to the Management section.

  2. Click Create index pattern to view logs from the indexes that get into Elasticsearch.

    kibana section1 figure1

  3. In the Define Index Pattern create your index-pattern according to the template. For example, if all the logs start with app-, create app-* index pattern to view the corresponding logs.

  4. Click Next step to continue.

    kibana section1 figure2

  5. Use a filter in the Configure Settings tab to select the time period for the shown data.

    By default, the system shows logs from the last 15 minutes.

  6. Click Create Index Pattern.

    kibana section1 figure3

  7. After the app-* index-pattern has been created, navigate to the Discover tab to get the following info:

    • the number of logs created in the selected period;

    • log list.

    kibana section1 figure4

  8. Click the icon to the left of the corresponding row to view event parameters:

    • time of the event;

    • place of the event;

    • type of the event (info/warning/error);

    • event message.

    kibana section1 figure5

2. Errors

2.1. Data filtering

If there is an error during request processing, you can view all the info on this error with trace id, using the following methods:

Method 1.

To find the logs that contain information on the corresponding event, copy the trace id in the Registry on the portal, or Openshift. Then paste it to the search line in the Discover tab and specify the time period that you need the data for.

kibana section2.1 figure1

Method 2.

Navigate to the Filter window in Discover tab, select X-B3-Traceid, then is, and click Save.

kibana section2.1 figure2

The resulting logs will have the given trace id. To control the number of results, and filter out less relevant search data, configure the corresponding fields, selecting Message and Level in the list to the left.

kibana section2.1 figure3

2.2. Data visualisation

Kibana allows you to visualize information using different charts. To do that, take the following steps:

  1. Select the Visualize tab;

  2. Click Create visualization;

  3. Select visualization type;

  4. Specify the timpe period for the data.

    kibana section2.2 figure1

You can see all the visualizations on the same dashboard in the Dashboard tab.

kibana section2.2 figure2

2.3. Dashboard import

To import a dashboard, download the kibana-logs-dashboard.json file.

When done, take the following steps:

  1. Select the Management tab.

  2. Click Saved Objects.

  3. Click Import.

  4. In the prompt window, click Import again.

  5. When the download is complete, navigate to the Downloads folder and select kibana-logs-dashboard.json.

  6. Click Import and Done, to import all the objects that contain:

    • dashboard;

    • the created index-pattern;

    • all visualizations.

    kibana section2.3 figure1

After completing the import, take the following steps to get consolidated information from all the logs:

  1. Navigate to the Dashboard tab.

  2. Select Logs Dashboard from the list. kibana section2.3 figure2

  3. Select the environment using filters.

  4. Click Apply changes to get information on the environment aggregated in the last 15 minutes.

  5. Use filter to specify the required time period. kibana section2.3 figure3

2.4. Working with charts

Charts and tables contain the following information:

  1. the general number of requests in the specified time period:

    • incoming;

    • outgoing.

  2. the number of incoming requests by environment.

    kibana section2.4 figure1

    kibana section2.4 figure2

  3. the general number of:

    • errors;

    • replies.

  4. the number of replies by environment.

    Requests and replies are events recorded in dedicated logs. The number of replies must be equal to the number of requests.

    Successful replies are characterized by a numeric code, which is less than 500. Replies with a code of 500 or higher are errors.

  5. http traffic (the list of requests and replies)

    kibana section2.4 figure3

  6. all the logs that contain information required to investigate the reasons for failed requests;

    kibana section2.4 figure4

  7. next, filter the info by the specified trace id;

    kibana section2.4 figure5

  8. you can see how many requests and replies match this trace id, and all the logs and log records that contain the following info:

    • request duration;

    • headers;

    • reply status-code;

    • message;

    • all the logs that match the id.

      kibana section2.4 figure6

  9. exceptions are sorted by number from high to low, and contain the following info:

    • which exceptions occured;

    • the service the exceptions occured on.

      kibana section2.4 figure7

To get detailed info on an exception:

  1. select the time;

  2. navigate to the container field and select the name of the service, where the error has occured;

  3. click apply changes.

    kibana section2.4 figure8

If the error does not belong to a particular service:

  1. Copy error name.

    kibana section2.4 figure9

  1. Enter the name into the search field (it is recommended to use quotes "" to get a more precise result).

    kibana section2.4 figure10

    The table below shows the following data:

  • time;

  • error level;

  • message.

    kibana section2.4 figure11

The message section contains detailed information on the error and its reasons.

kibana section2.4 figure12

The Dashboard tab shows the number of errors that occured in the selected time period.

kibana section2.4 figure13

Clicking a number will filter errors by reply code: 501, 502, 503, 504, 505.

kibana section2.4 figure14

You can find the information on these codes below. Requests can be filtered by trace id in the search field of the Dashboard tab.

kibana section2.4 figure15

Here you can see the number of requests, replies, and errors with the specified trace id.

kibana section2.4 figure16

Navigate to the table containing all the information on the failed request. The information is sorted by time, so start reading from the bottom.

kibana section2.4 figure17

The table allows you to follow the entire path of the request and find where the error appeared, and its reasons.

The request with an error always has an exception that tells us why the error appeared. To see it, check the level section, which may have one of the following values:

  • info - information on the state of events;

  • warning - warning;

  • error - error.

The former two values are non-critical and don’t require any special attention.

To find the error, filter by err on the Dashboard tab.

kibana section2.4 figure18

In the resulting logs, click the arrow on the right to see the detailed info.

kibana section2.4 figure19

This will provide all the information on the error and its reason.

kibana section2.4 figure20