What should I check in my data before kibana?

Today, I wanted to use Kibana with a new dataset I found. A file of data that lists all the best Amazon products over a period of several months, which would have allowed me to make a dashboard for our next webinar.

Convinced that I would again be able to show you easily new possibilities offered by Kibana, I try to import my file into Kibana again. I had quickly forgotten the few check points I had to perform before I could play in Kibana. Here doc below the story of my tumultuous journey

My file never appeared in kibana.

I didn’t get any confirmation messages or emails, and it was only available on the Octave application.

So there must have been something preventing Kibana from processing it. So, let’s try to find out together what the problem is.

To begin with, I got out of Kibana, and opened my file
I decided to open it in three different ways. The first time, directly on my PC with Open Office and look at my file.

Ocave.io - data Kibana

We do have our file in a spreadsheet format, with our header line specifying our categories. Everything seems OK, except maybe the big space after “computer”, and in the same cell, we have an open quote, but it doesn’t seem to be closed.

We may have a clue as to the reason for our file not being processed by kibana.

To be sure, and to continue the checks, I reopened my file via a free software: Textpad.

Ocave.io - data Kibana

This time, everything seems to be displayed correctly. We don’t have a large space like when we opened the file with Open Office. Now let’s see if our quote is still there, and if it is closed at some point.

Ocave.io - data Kibana

So he is always present. And, visibly, he closes a few words further on:

Ocave.io - data Kibana

Therefore, the hypothesis that it is the never-closed inverted comma that prevents Kibana from processing the file is excluded.

Now let’s focus on the header.

We’ll just do a simple check and see if there are any special characters, spaces or anything that looks suspicious.

Ocave.io  Blog

Here there is nothing suspicious, no special characters or anything else.

On the whole, nothing looks suspicious and we don’t really know what is causing Kibana’s mishandling of the file.

But there is one thing, which we haven’t talked about yet, that changes everything:

The separators in the file.

The separators in our file are commas. So far, everything is normal.
But, if we put our characters in inverted commas, they correspond to the “categories” field, which corresponds to text :

Ocave.io - blog

Note that each of the categories within the inverted commas is separated by a comma. This causes problems when processing in Kibana.
Indeed, the categories to which the product belongs are separated by a comma, like the separator of our entire file.

So, when Kibana processes the file, it takes into account all separators, even those that are included in “categories” and therefore between inverted commas.

These are called multivalued fields, and Kibana is not designed to handle them. So our file is impossible to process in the report.

There are solutions

  • The first, the simplest and the most definitive=> Take another file
  • The second => delete the “column” with the multivalued fields. So our “categories” on all the lines of the file.
  • The third => Delete the commas inside the category fields and the inverted commas.

It’s up to you to choose the option that suits you best.

Which solution have I chosen?

For the case I was confronted with, I decided to opt for the third option. Removed the commas inside the problem fields. I re-opened my file and used the “Search and Replace” function of Open Office on my entire file. This allows me to remove the commas that interfere with the separators.
If you would like to see in more detail how I did this, we will soon offer you an article on how to best prepare a file for Kibana.

In the meantime, you can read our previous articles on Kibana. Follow us on  LinkedInTwitter et Facebook